[
https://issues.apache.org/jira/browse/HIVE-25722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arko Sharma updated HIVE-25722:
-------------------------------
Description:
Compaction fails if there are empty buckets.
This patch detects and deletes empty buckets before launching compaction in
order to protect it from failing.
Error stacktrace :
{code:java}
Caused by: java.lang.IllegalStateException: No 'original' files found for
bucketId=3 in
file:/Users/asharma/hive-fork/hive/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.txn.compactor.TestCompactor-1638241161113_-1801963913/warehouse/comp3/delta_0000002_0000002_0000
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPairToCompact.<init>(OrcRawRecordMerger.java:602)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:1154)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2462)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:811)
~[hive-exec-4.0.0-SNAPSHOT.jar:?]
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:787)
~[hive-exec-4.0.0-SNAPSHOT.jar:?]
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
{code}
was:
Compaction fails if there are empty buckets.
This patch detects and deletes empty buckets before launching compaction in
order to protect it from failing.
> Compaction fails if there are empty buckets.
> --------------------------------------------
>
> Key: HIVE-25722
> URL: https://issues.apache.org/jira/browse/HIVE-25722
> Project: Hive
> Issue Type: Bug
> Reporter: Arko Sharma
> Assignee: Arko Sharma
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Compaction fails if there are empty buckets.
> This patch detects and deletes empty buckets before launching compaction in
> order to protect it from failing.
> Error stacktrace :
> {code:java}
> Caused by: java.lang.IllegalStateException: No 'original' files found for
> bucketId=3 in
> file:/Users/asharma/hive-fork/hive/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.txn.compactor.TestCompactor-1638241161113_-1801963913/warehouse/comp3/delta_0000002_0000002_0000
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPairToCompact.<init>(OrcRawRecordMerger.java:602)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:1154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:2462)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:811)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:787)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
