z-bb opened a new pull request, #4703:
URL: https://github.com/apache/ozone/pull/4703
## What changes were proposed in this pull request?
Hadoop client write slowly when stream enabled
code stack
```
at
org.apache.hadoop.hdds.scm.storage.BlockDataStreamOutput.doFlushIfNeeded(BlockDataStreamOutput.java:321)
at
org.apache.hadoop.hdds.scm.storage.BlockDataStreamOutput.write(BlockDataStreamOutput.java:260)
at
org.apache.hadoop.ozone.client.io.BlockDataStreamOutputEntry.write(BlockDataStreamOutputEntry.java:108)
at
org.apache.hadoop.ozone.client.io.KeyDataStreamOutput.writeToDataStreamOutput(KeyDataStreamOutput.java:201)
at
org.apache.hadoop.ozone.client.io.KeyDataStreamOutput.handleWrite(KeyDataStreamOutput.java:179)
at
org.apache.hadoop.ozone.client.io.KeyDataStreamOutput.write(KeyDataStreamOutput.java:159)
at
org.apache.hadoop.hdds.scm.storage.ByteBufferStreamOutput.write(ByteBufferStreamOutput.java:37)
at
org.apache.hadoop.fs.ozone.OzoneFSDataStreamOutput.write(OzoneFSDataStreamOutput.java:72)
at java.io.OutputStream.write(OutputStream.java:116)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
- locked <0x0000000080670800> (a
org.apache.hadoop.fs.FSDataOutputStream)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:88)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:60)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:120)
at
org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:466)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
at
org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
at
org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
```
Because OzoneFSDataStreamOutput does not override the OutputStream write
method
it will write out a singleBytes
```
OutputStream.class
public void write(byte b[], int off, int len) throws IOException {
Objects.checkFromIndexSize(off, len, b.length);
// len == 0 condition implicitly handled by loop bounds
for (int i = 0 ; i < len ; i++) {
write(b[off + i]);
}
}
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/projects/HDDS/issues/HDDS-8584
## How was this patch tested?
```
# before fix:
[[email protected] ~]$ time
~/hadoop-2.7.2-5504-ozone-client/bin/hadoop fs -put file_1.5g
/vol1/buk1/key_test11
^C
real 13m4.849s
user 3m43.157s
sys 0m9.274s
# after fix
[[email protected] ~]$ time
~/hadoop-2.7.2-5504-ozone-client/bin/hadoop fs -put file_1.5g
/vol1/buk1/key_test22
real 0m12.368s
user 0m36.840s
sys 0m10.346s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
