[
https://issues.apache.org/jira/browse/IMPALA-8600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898332#comment-16898332
]
ASF subversion and git services commented on IMPALA-8600:
---------------------------------------------------------
Commit 2d819655118c8c6e82649e3c3821311f3dd01174 in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2d81965 ]
IMPALA-8600: Refresh transactional tables
Refreshing a subset of partitions in a transactional table might lead
us to an inconsistent state of that transactional table. As a fix
user initiated partition refreshes are no longer allowed on ACID
tables. Additionally, a refresh partition Metastore event actually
triggers a refresh on the whole ACID table.
An optimisation is implemented to check the locally latest table
level writeId, fetch the same from HMS and do a refresh only if they
don't match.
This couldn't be done for partitioned tables as apparently Hive
doesn't update the table level writeId if the transactional table is
partitioned. Similarly, checking the writeId for each partition and
refresh only the ones where the writeId is not up to date is not
feasible either as there is no writeId update when Hive makes schema
changes like adding a column neither on table level or on partition
level. So after a adding a column in Hive to a partitioned ACID table
and refreshing that table in Impala, still Impala wouldn't see the
new column. Hence, I unconditionally refresh the whole table if it's
ACID and partitioned. Note, that for non-partitioned ACID tables Hive
updates the table level writeId even for schema changes.
Change-Id: I1851da22452074dbe253bcdd97145e06c7552cd3
Reviewed-on: http://gerrit.cloudera.org:8080/13938
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Reload partition does not work for transactional tables
> -------------------------------------------------------
>
> Key: IMPALA-8600
> URL: https://issues.apache.org/jira/browse/IMPALA-8600
> Project: IMPALA
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: impala-acid
> Fix For: Impala 3.3.0
>
>
> If a table is transactional, a reload partition call should fetch the valid
> writeIds. Without doing this, the reload will skip adding all the newly
> created delta files of the transactional table pertaining to the new writeIds.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
