[
https://issues.apache.org/jira/browse/DRILL-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004429#comment-15004429
]
Jacques Nadeau commented on DRILL-4070:
---------------------------------------
I see the confusion. I was looking at what he said later: "I don't think there
are changes that should be made to Drill to solve this issue" and he didn't
assign this to himself. I'll check with him when he gets in but I don't think
he is working on a fix.
What is the fix that people think should be done? I outlined two possibilities
in the email thread:
- give people a migration tool for their previous Drill-created Parquet files
- provide people a switch to enable the old behavior. (This will possibly give
users incorrect results if they use this in the wrong context--ick...)
It sounds like given what Jason said above, it is possible there could be a
third option if every other tool did infact name their footer file with
additional metadata. Before choosing that path, it would probably be necessary
for someone to do some analysis of the actual footers written by other older
tools (e.g. impala 1.x, hive 12/13, spark 1.1-1.3, etc).
> Metadata Caching : min/max values are null for varchar columns in auto
> partitioned data
> ---------------------------------------------------------------------------------------
>
> Key: DRILL-4070
> URL: https://issues.apache.org/jira/browse/DRILL-4070
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
> Affects Versions: 1.3.0
> Reporter: Rahul Challapalli
> Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: cache.txt, fewtypes_varcharpartition.tar.tgz
>
>
> git.commit.id.abbrev=e78e286
> The metadata cache file created contains incorrect values for min/max fields
> for varchar colums. The data is also partitioned on the varchar column
> {code}
> refresh table metadata fewtypes_varcharpartition;
> {code}
> As a result partition pruning is not happening. This was working after
> DRILL-3937 has been fixed (d331330efd27dbb8922024c4a18c11e76a00016b)
> I attached the data set and the cache file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
