[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314013#comment-16314013
]
Xiaoyu Yao commented on HDFS-12794:
-----------------------------------
Thanks [~shashikant] for the patch and all for the reviews/discussion. The
latest patch looks good to me overall. I just have a few minor issues.
ChunkGroupOutputStream.java
Suggest rename the blockSize to streamBufferSize here and use a separate
configuration key to allow customization.
We still can set it the same as SCM block size by DistributedStorageHandler by
default.
ChunkOutputStream.java
Line 74: NIT: chukIndex can be removed as we are replacing it with a timestamp.
Line 87: should we rename the parameter to avoid confusion. It seems to be more
like a maxBufferSize. (Same applies to the change in
ChunkGroupOutputStream.java that plumbs the blockSize parameter.
Line 139-166: in the new code, the rollback will position the buffer to the
very beginning of all async operations even though some of them may succeed
while the previous sequential write allows partial rollback. Any thoughts on
the tradeoff here? The parallel steam write is beneficial only for certain use
cases (e.g., large files)?
Line 196: NIT: Need to clarify the comments
Line 337: should we put offset here or remove the offset parameter assuming
this always starts with ByteBuffer offset 0.
ContainerProtocolCalls.java
Line 56: NIT: unused imports
> Ozone: Parallelize ChunkOutputSream Writes to container
> -------------------------------------------------------
>
> Key: HDFS-12794
> URL: https://issues.apache.org/jira/browse/HDFS-12794
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Affects Versions: HDFS-7240
> Reporter: Shashikant Banerjee
> Assignee: Shashikant Banerjee
> Fix For: HDFS-7240
>
> Attachments: HDFS-12794-HDFS-7240.001.patch,
> HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch,
> HDFS-12794-HDFS-7240.004.patch, HDFS-12794-HDFS-7240.005.patch,
> HDFS-12794-HDFS-7240.006.patch
>
>
> The chunkOutPutStream Write are sync in nature .Once one chunk of data gets
> written, the next chunk write is blocked until the previous chunk is written
> to the container.
> The ChunkOutputWrite Stream writes should be made async and Close on the
> OutputStream should ensure flushing of all dirty buffers to the container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
