[
https://issues.apache.org/jira/browse/IMPALA-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930234#comment-17930234
]
ASF subversion and git services commented on IMPALA-13769:
----------------------------------------------------------
Commit e5b785cd310116a973746e26b1181830f58fc93c in impala's branch
refs/heads/branch-4.5.0 from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e5b785cd3 ]
IMPALA-13759: Fix Hive ACID INSERT OVERWRITE base detection
Base directory created by INSERT OVERWRITE / TRUNCATE should be
treated differently than bases created by compaction because
IOW/TRUNCATE bases must be accepted even if there is an earlier
open writeId. This scenario can easily occur if there is
a pending write to a single partition, as this doesn't block
an IOW/TRUNCATE to another partition, while the global
minOpenWrite affects whether the base is accepted.
This change updates Impala logic to consider these bases
valid similarly to Hive.
Note that differentiating IOW/TRUNCATE from compaction is
different than in Hive, as metadata files are not considered
in Impala (IMPALA-13769). This can only cause problems when
interacting with earlier Hive versions that did not use
visibilityTxnId in the base path. I don't consider this
to be a significant regression that should block the current
critical fix.
Testing:
- added regression EE/FE tests
Change-Id: I838eaf4f41bae148e558f64288a1370c0908efa4
Reviewed-on: http://gerrit.cloudera.org:8080/22499
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Use Hive ACID metadata files when detecting base dirs created by compaction
> ----------------------------------------------------------------------------
>
> Key: IMPALA-13769
> URL: https://issues.apache.org/jira/browse/IMPALA-13769
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: ACID
>
> Hive uses metadata files to differentiate between base dirs created by INSERT
> OVERWRITE / TRUNCATE and compacted bases created by older Hives (that did not
> use visibiliyTxnId).
> This doesn't cause issues at the moment, but once IMPALA-13759 is merged,
> Impala will treat IO/TRUNCATE base dirs differently and accept them even if
> there is an earlier open write id. I don't think that this is a significant
> problem because if Impala sees folders written by older Hive versions then
> these transactions must be already closed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
