[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=473617&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-473617
]
ASF GitHub Bot logged work on HIVE-24020:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Aug/20 08:19
Start Date: 23/Aug/20 08:19
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1382:
URL: https://github.com/apache/hive/pull/1382#discussion_r475189153
##########
File path:
streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java
##########
@@ -581,16 +582,9 @@ protected RecordUpdater getRecordUpdater(List<String>
partitionValues, int bucke
destLocation = new Path(table.getSd().getLocation());
} else {
PartitionInfo partitionInfo =
conn.createPartitionIfNotExists(partitionValues);
- // collect the newly added partitions.
connection.commitTransaction() will report the dynamically added
- // partitions to TxnHandler
- if (!partitionInfo.isExists()) {
- addedPartitions.add(partitionInfo.getName());
- } else {
- if (LOG.isDebugEnabled()) {
- LOG.debug("Partition {} already exists for table {}",
- partitionInfo.getName(), fullyQualifiedTableName);
- }
- }
+ // collect the newly added/updated partitions.
connection.commitTransaction() will report the dynamically
+ // added partitions to TxnHandler
+ addedPartitions.add(partitionInfo.getName());
Review comment:
Can we please rename the 'addedPartitions' to something like more
resembling the actual usage, like 'updatedPartitions', or 'changedPartitions',
or whatever? Maybe a comment on the attribute/and getter should be good as well.
Otherwise LGTM.
Thanks, Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 473617)
Time Spent: 0.5h (was: 20m)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---------------------------------------------------------------------------------------------------
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
> Affects Versions: 4.0.0, 3.1.2
> Reporter: Vipin Vishvkarma
> Assignee: Vipin Vishvkarma
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
