[
https://issues.apache.org/jira/browse/ARROW-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090795#comment-17090795
]
Wes McKinney commented on ARROW-8199:
-------------------------------------
> Mainly I'd like to know if this looks like the direction you're thinking for
> arrow::DataFrame?
No, to be honest from a glance it's a different direction from what I've been
thinking. My thoughts there actually are for the data frame internally to be a
mix of yet-to-be-scanned Datasets (e.g. from CSV or Parquet files), manifest
(materialized in-memory) chunked arrays, and unevaluated expressions. Analytics
requests translate requests into physical query plans to be executed by the
to-be-developed query engine. I haven't been able to give this my full
attention since writing the design docs last year but I intend to spend a large
fraction of my time on it the rest of the year.
The reasoning for wanting to push data frame operations into a query engine is
to get around the memory use issues and performance problems associated with
"eager evaluation" data frame libraries like pandas (for example, a join in
pandas materializes the entire joined data frame in memory). There are similar
issues around sorting (particular with the knowledge of what you want to do
with the sorted data -- e.g. sort followed by a slice can be executed as a
Top-K operation for substantially less memory use)
That said, I know a number of people have expressed interest in having STL
interface layers in Arrow to the data structures. This would be a valuable
thing to contribute to the project. It's not mutually exclusive with the stuff
I wrote above but wanted to give some idea
> [C++] Guidance for creating multi-column sort on Table example?
> ---------------------------------------------------------------
>
> Key: ARROW-8199
> URL: https://issues.apache.org/jira/browse/ARROW-8199
> Project: Apache Arrow
> Issue Type: Wish
> Components: C++
> Affects Versions: 0.16.0
> Reporter: Scott Wilson
> Priority: Minor
> Labels: c++, newbie
> Attachments: ArrowCsv.cpp
>
>
> I'm just coming up to speed with Arrow and am noticing a dearth of examples
> ... maybe I can help here.
> I'd like to implement multi-column sorting for Tables and just want to ensure
> that I'm not duplicating existing work or proposing a bad design.
> My thought was to create a Table-specific version of SortToIndices() where
> you can specify the columns and sort order.
> Then I'd create Array "views" that use the Indices to remap from the original
> Array values to the values in sorted order. (Original data is not sorted, but
> could be as a second step.) I noticed some of the array list variants keep
> offsets, but didn't see anything that supports remapping per a list of
> indices, but this may just be my oversight?
> Thanks in advance, Scott
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
