[
https://issues.apache.org/jira/browse/IMPALA-8600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901863#comment-16901863
]
Gabor Kaszab commented on IMPALA-8600:
--------------------------------------
Apparently, Hive doesn't update the writeIds during a major compaction. This
breaks the minor optimisation I implemented: For non-partitioned tables during
a refresh I first get the table-level writeId and compare it with the cached
writeId. If they match then I don't reload the table from HMS. However, this
way Impala won't see the new base directory after a major compaction even if
the user ran a REFRESH on the table.
So I have to remove that optimisation and simply reload the whole table if it's
transactional.
> Reload partition does not work for transactional tables
> -------------------------------------------------------
>
> Key: IMPALA-8600
> URL: https://issues.apache.org/jira/browse/IMPALA-8600
> Project: IMPALA
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: impala-acid
> Fix For: Impala 3.3.0
>
>
> If a table is transactional, a reload partition call should fetch the valid
> writeIds. Without doing this, the reload will skip adding all the newly
> created delta files of the transactional table pertaining to the new writeIds.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
