[
https://issues.apache.org/jira/browse/PARQUET-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145671#comment-15145671
]
Daniel Weeks commented on PARQUET-427:
--------------------------------------
[~proflin] Thanks for putting together such a comprehensive document describing
this.
Overall, I think I agree with the intent of everything in the document. I'm
not sure I agree we want to tag the primitive types with (#1 under proposed
changes). I think it would be better to use explicit set logic with the three
schemas (requested, file, predicate) which would help with both readability and
wouldn't require someone to be aware of the set intersection mapping to
understand the logic in the implementation.
You could simply add some set notation/grammar with the schema object to
facilitate this.
Another note (I'm not clear how spark handles this), but I believe row F is
actually handled by the implementation in the engines (hive, pig, presto),
though this is accurate in that it's not handled currently in the parquet
implementation. I do agree this should be handled gracefully by parquet
though.
Do you have a working implementation for the proposed changes?
> Push predicates into the whole read path
> ----------------------------------------
>
> Key: PARQUET-427
> URL: https://issues.apache.org/jira/browse/PARQUET-427
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.8.0, 1.8.1
> Reporter: Liwei Lin
> Assignee: Liwei Lin
> Fix For: 1.9.0
>
> Attachments: Parquet-427 v0.1.pdf
>
>
> Regarding primitive types, there are 3 import primitive types set:
> - the file schema primitive type set
> - the requested schema primitive type set
> - the predicates primitive type set
> Currently the file schema primitive type set and the requested schema
> primitive type set has been pushed into the whole read path, but not the the
> predicates primitive type set.
> This brings some problems like:
> - PARQUET-389, SPARK-11103, SPARK-11434
> - PARQUET-425, PARQUET-426
> - PARQUET-295
> This issue propose to push the predicates primitive type set into the whole
> read path as well. When this is resolved, hopefully the issues listed above
> would also be resolved as well.
> Attached is a change proposal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
