Rajesh Balamohan created HDDS-8085:
--------------------------------------
Summary: Reduce network traffic in
XceiverClientGrpc::sendCommandWithRetry
Key: HDDS-8085
URL: https://issues.apache.org/jira/browse/HDDS-8085
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Rajesh Balamohan
Context: simple "select count(*) from store_sales" causes lot of network
traffic in Ozone than in HDFS.
Probing revealed good amount of time being spent in splits, which needs to be
fixed separately.
Another potential fix is that, XceiverClientGrpc gets a list of DataNodes which
it can communicate with in the task.
{color:#0747A6}If DN matches that of localhost, it should prefer localhost
rather than communicating with another node.
Request is get "InetAddress.getLocalHost().getHostName().trim()", and compare
with the set of hosts available in "DatanodeDetails". If found, prefer that DN
rather than remote node.
{color}
https://github.com/apache/ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java#L397-L440
Stacktrace below many not match with apache ozone, but pasting it to provide
codepath details.
{noformat}
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:398)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:330)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc$$Lambda$71/1551468933.get(Unknown
Source)
at
org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:174)
at
org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:148)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:324)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:305)
at
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:283)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:432)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkDataIntoBuffers(ChunkInputStream.java:404)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:389)
- eliminated <0x00000007823965b0> (a
org.apache.hadoop.hdds.scm.storage.ChunkInputStream)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:321)
- locked <0x00000007823965b0> (a
org.apache.hadoop.hdds.scm.storage.ChunkInputStream)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:207)
- locked <0x00000007823965b0> (a
org.apache.hadoop.hdds.scm.storage.ChunkInputStream)
at
org.apache.hadoop.hdds.scm.storage.ByteBufferReader.readFromBlock(ByteBufferReader.java:56)
at
org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:303)
- locked <0x0000000783646d30> (a
org.apache.hadoop.hdds.scm.storage.BlockInputStream)
at
org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:64)
- locked <0x0000000783646d30> (a
org.apache.hadoop.hdds.scm.storage.BlockInputStream)
at
org.apache.hadoop.hdds.scm.storage.ByteBufferReader.readFromBlock(ByteBufferReader.java:56)
at
org.apache.hadoop.ozone.client.io.KeyInputStream.readWithStrategy(KeyInputStream.java:273)
- locked <0x0000000783646ca0> (a
org.apache.hadoop.ozone.client.io.KeyInputStream)
at
org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:250)
- locked <0x0000000783646ca0> (a
org.apache.hadoop.ozone.client.io.KeyInputStream)
at
org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:110)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:152)
at
org.apache.parquet.hadoop.util.H2SeekableInputStream$H2Reader.read(H2SeekableInputStream.java:81)
at
org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:90)
at
org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:75)
at
org.apache.parquet.hadoop.ParquetFileReader$ConsecutivePartList.readAll(ParquetFileReader.java:1427)
at
org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:846)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:447)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:425)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:377)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:99)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:118)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecor
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
