Istvan Fajth created HDDS-2914:
----------------------------------
Summary: Certain Hive queries started to fail on generating splits
Key: HDDS-2914
URL: https://issues.apache.org/jira/browse/HDDS-2914
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Istvan Fajth
After updating a cluster where I am running TPCDS queries, some queries started
to fail.
The update happened from an early dec state to the jan 10 state of master.
Most likely the addition of HDDS-2188 is related to the problem, but it is
still under investigation.
The exception I see in the queries:
{code}
[ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input:
inventory initializer failed, vertex=vertex_XXXX [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
java.lang.RuntimeException: ORC split generation failed with exception:
java.io.IOException: File
o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
should have had overlap on block starting at 0
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
at
com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: ORC split generation failed with
exception: java.io.IOException: File
o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
should have had overlap on block starting at 0
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
... 5 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File
o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
should have had overlap on block starting at 0
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1909)
... 17 more
Caused by: java.io.IOException: File
o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
should have had overlap on block starting at 0
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:1506)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateOrUpdateSplit(OrcInputFormat.java:1678)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateSplitsFromStripes(OrcInputFormat.java:1661)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1609)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
{code}
I haven't found any more specific stack traces so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
