[
https://issues.apache.org/jira/browse/PARQUET-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney reassigned PARQUET-478:
------------------------------------
Assignee: Wes McKinney
> Reassembly algorithms for nested in-memory columnar memory layout
> -----------------------------------------------------------------
>
> Key: PARQUET-478
> URL: https://issues.apache.org/jira/browse/PARQUET-478
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-cpp
> Reporter: Wes McKinney
> Assignee: Wes McKinney
>
> I plan to use parquet-cpp primarily in conjunction with columnar data
> structures.
> Specifically, this requires in the interpretation of repetition / definition
> levels:
> * Computing null bits / bytes for each logical level of nested tree (group,
> array, primitive leaf)
> * Computing implied array sizes for each repeated group (according to 1, 2,
> or 3-level array encoding)
> The results of this reconstruction will be simply C arrays accompanied by the
> parquet-cpp logical schema; this way we can make it easy to adapt to
> different in-memory columnar memory schemes.
> As far as implementation, it would make sense to proceed first with
> functional unit tests of the reassembly algorithms using repetition /
> definition levels declared in the test suite as C++ vectors -- otherwise it's
> going to be too tedious trying to produce valid Parquet test data files which
> explore all of the different edge cases.
> Several other teams (Spark, Drill, Parquet-Java) are currently working on
> related efforts along these lines, so we can engage when appropriate to
> collaborate on algorithms and nuances of this approach to avoid unnecessary
> code churn / bugs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
