[
https://issues.apache.org/jira/browse/HDFS-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319468#comment-14319468
]
Kai Zheng commented on HDFS-7729:
---------------------------------
Thanks [~jingzhao] and [~szetszwo] for the great thoughts, and [~zhz] for the
meaningful analysis at this phase ! It's never late and it's good to experiment
in this direction as well, so we will be more clear about which way is better.
bq.whether we can/should now define DataStreamer as a static class and move it
out of DFSOutputStream....It may be better and more clear to make DataStreamer
as a standalone class only handling the logic about transferring packets that
is assigned to it from outside.
It's surely doable, sounds much elegant and clean. I discussed with
[~libo-intel], actually initially when he worked on this, he wanted to go this
way. As he's not sure if the community would like that big refactoring he first
experimented to do small changes as current patch, as we thought it's important
to have an initial workable patch for the community experts' review. It's great
we have [~zhz]'s review and also your feedback now. We're open and would like
to adapt the patch as you suggested. As [~libo-intel] has worked on this part
for quite much time, he's familiar with it now and has much confident in the
bigger refactoring. He will provide a new revision according to this thought.
Thanks !
bq.After this refactoring, we can extend the current DFSOutputStream to
DFSOutputStreamStriped, which contains the striping logic
Again, very good idea and design thoughts. [~libo-intel], please consider this
as well in your new update. Thanks.
bq.Hope the following analysis helps exploring the DataStreamer refactor.
[~zhz] you made good points about what's doing in current codes. I thought you
would also agree with the refactoring. Glad we have a strong basis !
As [~libo-intel] is working on the new revision in this approach, please kindly
let we know if you have other concerns. Thanks.
> Add logic to DFSOutputStream to support writing a file in striping layout
> --------------------------------------------------------------------------
>
> Key: HDFS-7729
> URL: https://issues.apache.org/jira/browse/HDFS-7729
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Li Bo
> Assignee: Li Bo
> Attachments: Codec-tmp.patch, HDFS-7729-001.patch,
> HDFS-7729-002.patch, HDFS-7729-003.patch, HDFS-7729-004.patch,
> HDFS-7729-005.patch, HDFS-7729-006.patch, HDFS-7729-007.patch,
> HDFS-7729-008.patch
>
>
> If client wants to directly write a file striping layout, we need to add some
> logic to DFSOutputStream. DFSOutputStream needs multiple DataStreamers to
> write each cell of a stripe to a remote datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
