[
https://issues.apache.org/jira/browse/IMPALA-11032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452698#comment-17452698
]
ASF subversion and git services commented on IMPALA-11032:
----------------------------------------------------------
Commit 4077bc849ae14bb92a463aeeb6c8f5c1fca658c9 in impala's branch
refs/heads/master from Yu-Wen Lai
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4077bc8 ]
IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after
Compaction
After compaction happened in Hive(HIVE ACID table), queries made in
Impala possibly fail with a FileNotFoundException if files already
removed by the Hive cleaner.
In IMPALA-10801, catalogd checks the latest compaction id before serving
metadata. However, coordinators don't take advantage of that.
Coordinators have their own local cache, so we will have to do the
same check for coordinators as well. Besides, we also need to attach
writeIdList to requests that need to fetch file metadata. Since this
checking brings additional overhead for queries, we introduce a flag
auto_check_compaction and set it as false by default for now. We will
find some other efficient ways to do compaction checking in the future.
Tests:
Added unit tests to CatalogdMetaProviderTest
Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b
Reviewed-on: http://gerrit.cloudera.org:8080/18043
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Automatic Refresh of Metadata for Local Catalog after Compaction
> ----------------------------------------------------------------
>
> Key: IMPALA-11032
> URL: https://issues.apache.org/jira/browse/IMPALA-11032
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Yu-Wen Lai
> Assignee: Yu-Wen Lai
> Priority: Major
>
> After Hive compaction for tables created in the Hive warehouse, queries(HIVE
> ACID table) made in Impala possibly fail with a FileNotFoundException if a
> file removed by the Hive cleaner.
> In IMPALA-10801, we check latest compaction id before serving metadata from
> Catalogd. However, coordinators don't take advantage of that. Coordinators
> have local cache so we will have do the same compaction check for coordinator
> as well.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
