[jira] [Resolved] (ARROW-24) C++: Add logical "Table" container

2016-03-04 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-24?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-24. --- Resolution: Fixed Issue resolved by pull request 16 [https://github.com/apache/arrow/pull/16] > C++: Ad

Distributed arrow?

2016-03-04 Thread Venkat Krishnamurthy
All I've been following along with great interest, and have a n00b question. What happens when any of the arrow-capable processing tools needs to work with a data set or structure that is bigger than a single node's memory capacity? Does arrow itself handle the distribution of the resulting colum

[jira] [Commented] (ARROW-24) C++: Add logical "Table" container

2016-03-04 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181428#comment-15181428 ] Wes McKinney commented on ARROW-24: --- See patch https://github.com/apache/arrow/pull/16 >

[jira] [Assigned] (ARROW-24) C++: Add logical "Table" container

2016-03-04 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-24?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reassigned ARROW-24: - Assignee: Wes McKinney > C++: Add logical "Table" container > -- >

[jira] [Created] (ARROW-41) C++: Convert table to std::vector of Struct arrays

2016-03-04 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-41: - Summary: C++: Convert table to std::vector of Struct arrays Key: ARROW-41 URL: https://issues.apache.org/jira/browse/ARROW-41 Project: Apache Arrow Issue Type: New

[jira] [Created] (ARROW-40) C++: Reinterpret Struct arrays as tables

2016-03-04 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-40: - Summary: C++: Reinterpret Struct arrays as tables Key: ARROW-40 URL: https://issues.apache.org/jira/browse/ARROW-40 Project: Apache Arrow Issue Type: New Feature

[jira] [Resolved] (ARROW-23) C++: Add logical "Column" container for chunked data

2016-03-04 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-23. --- Resolution: Fixed Issue resolved by pull request 15 [https://github.com/apache/arrow/pull/15] > C++: Ad

[jira] [Updated] (ARROW-39) C++: Logical chunked arrays / columns: conforming to fixed chunk sizes

2016-03-04 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-39: -- Summary: C++: Logical chunked arrays / columns: conforming to fixed chunk sizes (was: C++: Logical chunke

[jira] [Created] (ARROW-39) C++: Logical chunked arrays / columns: conforming to a fixed chunk sizes

2016-03-04 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-39: - Summary: C++: Logical chunked arrays / columns: conforming to a fixed chunk sizes Key: ARROW-39 URL: https://issues.apache.org/jira/browse/ARROW-39 Project: Apache Arrow

[jira] [Created] (ARROW-38) C++: Algorithms for using nested types in a hash table context

2016-03-04 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-38: - Summary: C++: Algorithms for using nested types in a hash table context Key: ARROW-38 URL: https://issues.apache.org/jira/browse/ARROW-38 Project: Apache Arrow Is

Re: Should Nullable be a nested type?

2016-03-04 Thread Daniel Robinson
That's convincing, thanks for the response. On Fri, Mar 4, 2016 at 8:14 AM -0800, "Wes McKinney" wrote: Moving this thread over from the discussion about adding null count to the physical format. I never said that what you're describing an invalid approach, only that it will yield

Re: Should Nullable be a nested type?

2016-03-04 Thread Wes McKinney
Moving this thread over from the discussion about adding null count to the physical format. I never said that what you're describing an invalid approach, only that it will yield more complexity for both library developers and users without any clear performance or net productivity benefits. This i

Re: Format: storing null count + required/non-nullable types

2016-03-04 Thread Wes McKinney
This thread was about adding null count to the data structures (and making nullability a property of the metadata, if at all). I'll reply to the other thread about implementation matters. On Fri, Mar 4, 2016 at 6:50 AM, Daniel Robinson wrote: > Wes, > > Thanks for soliciting so much input on thes

Format: storing null count + required/non-nullable types

2016-03-04 Thread Daniel Robinson
Wes, Thanks for soliciting so much input on these questions, and sharing the new prototypes. In response to point 2 and your e-mail from last week, I created some prototypes to illustrate what I think could be useful about having a Nullable template in the C++ implementation. As far as code comp