[
https://issues.apache.org/jira/browse/HIVE-23410?focusedWorklogId=521729&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-521729
]
ASF GitHub Bot logged work on HIVE-23410:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Dec/20 15:06
Start Date: 08/Dec/20 15:06
Worklog Time Spent: 10m
Work Description: kuczoram commented on a change in pull request #1660:
URL: https://github.com/apache/hive/pull/1660#discussion_r538474101
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java
##########
@@ -189,6 +191,49 @@ public WriteEntity getAcidAnalyzeTable() {
return acidSinks;
}
+ public Integer getStatementIdForAcidWriteType(long writeId, String
moveTaskId, AcidUtils.Operation acidOperation, Path path) {
+ FileSinkDesc result = null;
+ for (FileSinkDesc acidSink : acidSinks) {
+ if (acidOperation.equals(acidSink.getAcidOperation()) &&
path.equals(acidSink.getDestPath())
+ && acidSink.getTableWriteId() == writeId
+ && (moveTaskId == null || acidSink.getMoveTaskId() == null ||
moveTaskId.equals(acidSink.getMoveTaskId()))) {
+ // There is a problem with the union all optimisation. In this case,
there will be multiple FileSinkOperators
+ // with the same operation, writeId and moveTaskId. But one of these
FSOs doesn't write data and its statementId
+ // is not valid, so if this FSO is selected and its statementId is
returned, the file listing will find nothing.
+ // So check the acidSinks and if two of them have the same writeId,
path and moveTaskId, then return -1 as statementId.
+ // Like this, the file listing will find all partitions and files
correctly.
+ if (result != null) {
+ return -1;
+ }
+ result = acidSink;
+ }
+ }
+ if (result != null) {
+ return result.getStatementId();
+ } else {
+ return -1;
+ }
+ }
+
+ public Set<String> getDynamicPartitionSpecs(long writeId, String moveTaskId,
AcidUtils.Operation acidOperation, Path path) {
Review comment:
Added it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 521729)
Time Spent: 2h 10m (was: 2h)
> ACID: Improve the delete and update operations to avoid the move step
> ---------------------------------------------------------------------
>
> Key: HIVE-23410
> URL: https://issues.apache.org/jira/browse/HIVE-23410
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 4.0.0
> Reporter: Marta Kuczora
> Assignee: Marta Kuczora
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23410.1.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This is a follow-up task for
> [HIVE-21164|https://issues.apache.org/jira/browse/HIVE-21164], where the
> insert operation has been modified to write directly to the table locations
> instead of the staging directory. The same improvement should be done for the
> ACID update and delete operations as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
