[jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=474927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474927
]
ASF GitHub Bot logged work on HIVE-24020:
-
Author: ASF GitHub Bot
Created on: 26/Aug/20 17:51
Start Date: 26/Aug/20 17:51
Worklog Time Spent: 10m
Work Description: prasanthj merged pull request #1382:
URL: https://github.com/apache/hive/pull/1382
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 474927)
Time Spent: 1h (was: 50m)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=474904=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474904
]
ASF GitHub Bot logged work on HIVE-24020:
-
Author: ASF GitHub Bot
Created on: 26/Aug/20 17:21
Start Date: 26/Aug/20 17:21
Worklog Time Spent: 10m
Work Description: vpnvishv commented on pull request #1382:
URL: https://github.com/apache/hive/pull/1382#issuecomment-681015240
@pvary @prasanthj Can you please merge this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 474904)
Time Spent: 50m (was: 40m)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=473644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-473644
]
ASF GitHub Bot logged work on HIVE-24020:
-
Author: ASF GitHub Bot
Created on: 23/Aug/20 13:22
Start Date: 23/Aug/20 13:22
Worklog Time Spent: 10m
Work Description: vpnvishv commented on a change in pull request #1382:
URL: https://github.com/apache/hive/pull/1382#discussion_r475219031
##
File path:
streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java
##
@@ -581,16 +582,9 @@ protected RecordUpdater getRecordUpdater(List
partitionValues, int bucke
destLocation = new Path(table.getSd().getLocation());
} else {
PartitionInfo partitionInfo =
conn.createPartitionIfNotExists(partitionValues);
- // collect the newly added partitions.
connection.commitTransaction() will report the dynamically added
- // partitions to TxnHandler
- if (!partitionInfo.isExists()) {
-addedPartitions.add(partitionInfo.getName());
- } else {
-if (LOG.isDebugEnabled()) {
- LOG.debug("Partition {} already exists for table {}",
- partitionInfo.getName(), fullyQualifiedTableName);
-}
- }
+ // collect the newly added/updated partitions.
connection.commitTransaction() will report the dynamically
+ // added partitions to TxnHandler
+ addedPartitions.add(partitionInfo.getName());
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 473644)
Time Spent: 40m (was: 0.5h)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=473617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-473617
]
ASF GitHub Bot logged work on HIVE-24020:
-
Author: ASF GitHub Bot
Created on: 23/Aug/20 08:19
Start Date: 23/Aug/20 08:19
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1382:
URL: https://github.com/apache/hive/pull/1382#discussion_r475189153
##
File path:
streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java
##
@@ -581,16 +582,9 @@ protected RecordUpdater getRecordUpdater(List
partitionValues, int bucke
destLocation = new Path(table.getSd().getLocation());
} else {
PartitionInfo partitionInfo =
conn.createPartitionIfNotExists(partitionValues);
- // collect the newly added partitions.
connection.commitTransaction() will report the dynamically added
- // partitions to TxnHandler
- if (!partitionInfo.isExists()) {
-addedPartitions.add(partitionInfo.getName());
- } else {
-if (LOG.isDebugEnabled()) {
- LOG.debug("Partition {} already exists for table {}",
- partitionInfo.getName(), fullyQualifiedTableName);
-}
- }
+ // collect the newly added/updated partitions.
connection.commitTransaction() will report the dynamically
+ // added partitions to TxnHandler
+ addedPartitions.add(partitionInfo.getName());
Review comment:
Can we please rename the 'addedPartitions' to something like more
resembling the actual usage, like 'updatedPartitions', or 'changedPartitions',
or whatever? Maybe a comment on the attribute/and getter should be good as well.
Otherwise LGTM.
Thanks, Peter
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 473617)
Time Spent: 0.5h (was: 20m)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=470224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-470224
]
ASF GitHub Bot logged work on HIVE-24020:
-
Author: ASF GitHub Bot
Created on: 13/Aug/20 13:18
Start Date: 13/Aug/20 13:18
Worklog Time Spent: 10m
Work Description: vpnvishv commented on pull request #1382:
URL: https://github.com/apache/hive/pull/1382#issuecomment-673473084
@pvary @laszlopinter86 @klcopp Can you please review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 470224)
Time Spent: 20m (was: 10m)
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24020) Automatic Compaction not working in existing partitions for Streaming Ingest with Dynamic Partition
[
https://issues.apache.org/jira/browse/HIVE-24020?focusedWorklogId=468437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-468437
]
ASF GitHub Bot logged work on HIVE-24020:
-
Author: ASF GitHub Bot
Created on: 10/Aug/20 07:24
Start Date: 10/Aug/20 07:24
Worklog Time Spent: 10m
Work Description: vpnvishv opened a new pull request #1382:
URL: https://github.com/apache/hive/pull/1382
### What changes were proposed in this pull request?
We need to keep info about the updated partition as well, and pass this to
addDynamicPartitions. Here i am reusing addedParitions, as I don't see any
effect on stats either. Reviewers please suggest if you think it can affect.
### Why are the changes needed?
To fix automatic compaction in case of streaming ingest with dynamic
partition.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit-test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 468437)
Remaining Estimate: 0h
Time Spent: 10m
> Automatic Compaction not working in existing partitions for Streaming Ingest
> with Dynamic Partition
> ---
>
> Key: HIVE-24020
> URL: https://issues.apache.org/jira/browse/HIVE-24020
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This issue happens when we try to do streaming ingest with dynamic partition
> on already existing partitions. I checked in the code, we have following
> check in the AbstractRecordWriter.
>
> {code:java}
> PartitionInfo partitionInfo =
> conn.createPartitionIfNotExists(partitionValues);
> // collect the newly added partitions. connection.commitTransaction() will
> report the dynamically added
> // partitions to TxnHandler
> if (!partitionInfo.isExists()) {
> addedPartitions.add(partitionInfo.getName());
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Partition {} already exists for table {}",
> partitionInfo.getName(), fullyQualifiedTableName);
> }
> }
> {code}
> Above *addedPartitions* is passed to *addDynamicPartitions* during
> TransactionBatch commit. So in case of already existing partitions,
> *addedPartitions* will be empty and *addDynamicPartitions* **will not move
> entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in
> Initiator not able to trigger auto compaction.
> Another issue which has been observed is, we are not clearing
> *addedPartitions* on writer close, which results in information flowing
> across transactions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
