[
https://issues.apache.org/jira/browse/IMPALA-10115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188860#comment-17188860
]
ASF subversion and git services commented on IMPALA-10115:
----------------------------------------------------------
Commit 329bb41294a57bfd63dc0d90d57966e8562686b1 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=329bb41 ]
IMPALA-10115: Impala should check file schema as well to check full ACIDv2 files
Currently Impala checks file metadata 'hive.acid.version' to decide the
full ACID schema. There are cases when Hive forgets to set this value
for full ACID files, e.g. query-based compactions.
So it's more robust to check the schema elements instead of the metadata
field. Also, sometimes Hive write the schema with different character
cases, e.g. originalTransaction vs originaltransaction, so we should
rather compare the column names in a case insensitive way.
Testing:
* added test for full ACID compaction
* added test_full_acid_schema_without_file_metadata_tag to test full
ACID file without metadata 'hive.acid.version'
Change-Id: I52642c1755599efd28fa2c90f13396cfe0f5fa14
Reviewed-on: http://gerrit.cloudera.org:8080/16383
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Impala should check file schema as well to check full ACIDv2 files
> ------------------------------------------------------------------
>
> Key: IMPALA-10115
> URL: https://issues.apache.org/jira/browse/IMPALA-10115
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Priority: Major
>
> Currently Impala checks file metadata 'hive.acid.version' to decide the full
> ACID schema.
> There are cases when Hive forgets to set this value for full ACID files, e.g.
> major query-based compactions.
> So if 'hive.acid.version' is not present, Impala should still look at the
> schema elements to be sure about the file format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
