[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=470224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-470224
]
ASF GitHub Bot logged work on HIVE-24020:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Aug/20 13:18
Start Date: 13/Aug/20 13:18
Worklog Time Spent: 10m
Work Description: vpnvishv commented on pull request #1382:
URL: https://github.com/apache/hive/pull/1382#issuecomment-673473084
@pvary @laszlopinter86 @klcopp Can you please review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 470224)
Time Spent: 20m (was: 10m)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---------------------------------------------------------------------------------------------------
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
> Affects Versions: 4.0.0, 3.1.2
> Reporter: Vipin Vishvkarma
> Assignee: Vipin Vishvkarma
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
