[
https://issues.apache.org/jira/browse/HDDS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020673#comment-17020673
]
Aravindan Vijayan edited comment on HDDS-2914 at 1/22/20 12:12 AM:
-------------------------------------------------------------------
Based on looking at the way Hive OrcInputFormat uses the block locations, there
seems to be a misunderstanding in the way the block offset was handled in
HDDS-2188.
The following is the method used by Hive to create a map of offset -> locations
{code}
public TreeMap<Long, BlockLocation> getLocationsWithOffset(FileSystem fs,
FileStatus status)
throws IOException {
TreeMap<Long, BlockLocation> offsetBlockMap = new TreeMap<Long,
BlockLocation>();
BlockLocation[] locations = getLocations(fs, status);
for (BlockLocation location : locations) {
offsetBlockMap.put(location.getOffset(), location);
}
return offsetBlockMap;
}
{code}
It looks like the offset field in BlockLocation refers to the "offset of the
block in the set of blocks that are part of the file" rather than the offset of
the block in the physical file that contains the block. In our case, the latter
is always 0, and when we use 0 in the Block Locations, the above code
overwrites the same entry in the Map. I am planning on testing this theory out
on a cluster where the hive queries are failing.
[~msingh] Please let me know if this makes sense.
was (Author: avijayan):
Based on looking at the way org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
uses the block locations, there seems to be a misunderstanding in the way the
block offset was handled in HDDS-2188.
The following is the method used by Hive to create a map of offset -> locations
{code}
public TreeMap<Long, BlockLocation> getLocationsWithOffset(FileSystem fs,
FileStatus status)
throws IOException {
TreeMap<Long, BlockLocation> offsetBlockMap = new TreeMap<Long,
BlockLocation>();
BlockLocation[] locations = getLocations(fs, status);
for (BlockLocation location : locations) {
offsetBlockMap.put(location.getOffset(), location);
}
return offsetBlockMap;
}
{code}
It looks like the offset field in BlockLocation refers to the "offset of the
block in the set of blocks that are part of the file" rather than the offset of
the block in the physical file that contains the block. In our case, the latter
is always 0, and when we use 0 in the Block Locations, the above code
overwrites the same entry in the Map. I am planning on testing this theory out
on a cluster where the hive queries are failing.
[~msingh] Please let me know if this makes sense.
> Certain Hive queries started to fail on generating splits
> ---------------------------------------------------------
>
> Key: HDDS-2914
> URL: https://issues.apache.org/jira/browse/HDDS-2914
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Istvan Fajth
> Assignee: Aravindan Vijayan
> Priority: Critical
>
> After updating a cluster where I am running TPCDS queries, some queries
> started to fail.
> The update happened from an early dec state to the jan 10 state of master.
> Most likely the addition of HDDS-2188 is related to the problem, but it is
> still under investigation.
> The exception I see in the queries:
> {code}
> [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input:
> inventory initializer failed, vertex=vertex_XXXX [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.RuntimeException: ORC split generation failed with exception:
> java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
> should have had overlap on block starting at 0
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
> at
> com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
> should have had overlap on block starting at 0
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> ... 5 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
> should have had overlap on block starting at 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1909)
> ... 17 more
> Caused by: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000
> should have had overlap on block starting at 0
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:1506)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateOrUpdateSplit(OrcInputFormat.java:1678)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateSplitsFromStripes(OrcInputFormat.java:1661)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1609)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> I haven't found any more specific stack traces so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
