[
https://issues.apache.org/jira/browse/HIVE-25071?focusedWorklogId=590914&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-590914
]
ASF GitHub Bot logged work on HIVE-25071:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Apr/21 10:05
Start Date: 29/Apr/21 10:05
Worklog Time Spent: 10m
Work Description: kuczoram commented on pull request #2231:
URL: https://github.com/apache/hive/pull/2231#issuecomment-829104540
Hi Krisztian!
Thanks for this patch, it is very interesting.
I would have one question about using multiple reducers. Do you know how it
will be guaranteed that all rows with the same bucket would go into the same
reducer and at the end to the same file?
I am asking it because during compaction we had issues that rows with the
same bucket number went into different reducers and we ended up with corrupted
files (when rows with the same bucket numbers went into different files or
files contained rows with different bucket numbers). I saw this issue when
created an unbucketed table, but inserted a bigger amount of data, so at the
end, the table contained multiple bucket files.
I know that compaction is a different story, I am just curious whether or
not something similar could happen with deletes/updates using multiple
reducers. If you know how the row distribution between the reducers would work
for deletes and updates, I would be really grateful if you could share some
details.
Thanks and regards,
Marta
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 590914)
Time Spent: 20m (was: 10m)
> Number of reducers limited to fixed 1 when updating/deleting
> ------------------------------------------------------------
>
> Key: HIVE-25071
> URL: https://issues.apache.org/jira/browse/HIVE-25071
> Project: Hive
> Issue Type: Bug
> Reporter: Krisztian Kasa
> Assignee: Krisztian Kasa
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When updating/deleting bucketed tables an extra ReduceSink operator is
> created to enforce bucketing. After HIVE-22538 number of reducers limited to
> fixed 1 in these RS operators.
> This can lead to performance degradation.
> Prior HIVE-22538 multiple reducers was available such cases. The reason for
> limiting the number of reducers is to ensure RowId ascending order in delete
> delta files produced by the update/delete statements.
> This is the plan of delete statement like:
> {code}
> DELETE FROM t1 WHERE a = 1;
> {code}
> {code}
> TS[0]-FIL[8]-SEL[2]-RS[3]-SEL[4]-RS[5]-SEL[6]-FS[7]
> {code}
> RowId order is ensured by RS[3] and bucketing is enforced by RS[5]: number of
> reducers were limited to bucket number in the table or
> hive.exec.reducers.max. However RS[5] does not provide any ordering so above
> plan may generate unsorted deleted deltas which leads to corrupted data reads.
> Prior HIVE-22538 these RS operators were merged by ReduceSinkDeduplication
> and the resulting RS kept the ordering and enabled multiple reducers. It
> could do because ReduceSinkDeduplication was prepared for ACID writes. This
> was removed by HIVE-22538 to get a more generic ReduceSinkDeduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
