[
https://issues.apache.org/jira/browse/HIVE-29251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-29251:
----------------------------------
Description:
The {{HiveConf}} object is shared across multiple cleanup tasks. When the
Cleaner runs with multiple threads, this can lead to a race condition.
At the end of a cleanup task, the open transaction cap is lifted to check for
any deltas that may become eligible for cleanup once the open transaction
completes. If none are found, the cleanup request is marked complete;
otherwise, it’s returned to the queue with a “ready-for-cleaning” status.
{code:java}
conf.set(ValidTxnList.VALID_TXNS_KEY, new ValidReadTxnList().toString());
dir = AcidUtils.getAcidState(fs, path, conf,
new ValidReaderWriteIdList(info.getFullTableName(), new long[0], new
BitSet(), info.highestWriteId, Long.MAX_VALUE),
Ref.from(false), false,
dirSnapshots);
List<Path> remained = subtract(CompactorUtil.getObsoleteDirs(dir,
isDynPartAbort), deleted);
if (!remained.isEmpty()) {
LOG.warn("Remained {} obsolete directories from {}. {}",
remained.size(), location, CompactorUtil.getDebugInfo(remained));
} else {
LOG.debug("All cleared below the watermark: {} from {}", info.highestWriteId,
location);
success = true;
}{code}
that might override the `{{{}ValidTxnList.VALID_TXNS_KEY`{}}} set by a
concurrent task, leading to premature cleanup of data still in use by an active
transaction.
{code:java}
private void cleanUsingAcidDir(CompactionInfo ci, String location, long
minOpenTxn) throws Exception {
ValidTxnList validTxnList =
TxnUtils.createValidTxnListForCleaner(txnHandler.getOpenTxns(), minOpenTxn,
false);
//save it so that getAcidState() sees it
conf.set(ValidTxnList.VALID_TXNS_KEY, validTxnList.writeToString());{code}
was:
The {{HiveConf}} object is shared across multiple cleanup tasks. When the
Cleaner runs with multiple threads, this can lead to a race condition.
At the end of a cleanup task, the open transaction cap is lifted to check for
any deltas that may become eligible for cleanup once the open transaction
completes. If none are found, the cleanup request is marked complete;
otherwise, it’s returned to the queue with a “ready-for-cleaning” status.
{code:java}
conf.set(ValidTxnList.VALID_TXNS_KEY, new ValidReadTxnList().toString());
dir = AcidUtils.getAcidState(fs, path, conf,
new ValidReaderWriteIdList(info.getFullTableName(), new long[0], new
BitSet(), info.highestWriteId, Long.MAX_VALUE),
Ref.from(false), false,
dirSnapshots);
List<Path> remained = subtract(CompactorUtil.getObsoleteDirs(dir,
isDynPartAbort), deleted);
if (!remained.isEmpty()) {
LOG.warn("Remained {} obsolete directories from {}. {}",
remained.size(), location, CompactorUtil.getDebugInfo(remained));
} else {
LOG.debug("All cleared below the watermark: {} from {}", info.highestWriteId,
location);
success = true;
}{code}
that might override the override the `{{{}ValidTxnList.VALID_TXNS_KEY`{}}} set
by a concurrent task, leading to premature cleanup of data still in use by an
active transaction.
{code:java}
private void cleanUsingAcidDir(CompactionInfo ci, String location, long
minOpenTxn) throws Exception {
ValidTxnList validTxnList =
TxnUtils.createValidTxnListForCleaner(txnHandler.getOpenTxns(), minOpenTxn,
false);
//save it so that getAcidState() sees it
conf.set(ValidTxnList.VALID_TXNS_KEY, validTxnList.writeToString());{code}
> Hive ACID: HiveConf object shouldn't be shared between multiple cleanup tasks
> -----------------------------------------------------------------------------
>
> Key: HIVE-29251
> URL: https://issues.apache.org/jira/browse/HIVE-29251
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 4.1.0
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: ACID, Compaction
>
> The {{HiveConf}} object is shared across multiple cleanup tasks. When the
> Cleaner runs with multiple threads, this can lead to a race condition.
> At the end of a cleanup task, the open transaction cap is lifted to check for
> any deltas that may become eligible for cleanup once the open transaction
> completes. If none are found, the cleanup request is marked complete;
> otherwise, it’s returned to the queue with a “ready-for-cleaning” status.
>
> {code:java}
> conf.set(ValidTxnList.VALID_TXNS_KEY, new ValidReadTxnList().toString());
> dir = AcidUtils.getAcidState(fs, path, conf,
> new ValidReaderWriteIdList(info.getFullTableName(), new long[0], new
> BitSet(), info.highestWriteId, Long.MAX_VALUE),
> Ref.from(false), false,
> dirSnapshots);
> List<Path> remained = subtract(CompactorUtil.getObsoleteDirs(dir,
> isDynPartAbort), deleted);
> if (!remained.isEmpty()) {
> LOG.warn("Remained {} obsolete directories from {}. {}",
> remained.size(), location, CompactorUtil.getDebugInfo(remained));
> } else {
> LOG.debug("All cleared below the watermark: {} from {}",
> info.highestWriteId, location);
> success = true;
> }{code}
> that might override the `{{{}ValidTxnList.VALID_TXNS_KEY`{}}} set by a
> concurrent task, leading to premature cleanup of data still in use by an
> active transaction.
> {code:java}
> private void cleanUsingAcidDir(CompactionInfo ci, String location, long
> minOpenTxn) throws Exception {
> ValidTxnList validTxnList =
> TxnUtils.createValidTxnListForCleaner(txnHandler.getOpenTxns(),
> minOpenTxn, false);
> //save it so that getAcidState() sees it
> conf.set(ValidTxnList.VALID_TXNS_KEY, validTxnList.writeToString());{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
