[
https://issues.apache.org/jira/browse/HDFS-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056712#comment-16056712
]
Ewan Higgs commented on HDFS-11125:
-----------------------------------
The Provided Storage system is looking to use the SPS for asynchronously
writing files to a remote system ({{hdfs://}}, {{s3a://}}, {{wasb://}}). The
way the SPS currently works is file by file which suits the Provided Storage
system very well since it works on a file by file basis as well. Moving to use
smaller batches unlinked to the actual files would break current plans
discussed offline with [~umamaheswararao].
This could be worked through by offering the interface as
{{BlockStorageMovementCommandSchedulerSpi}} (or something less of a mouthful)
and implementing it one way for normal SPS and another way for Provided Storage.
> [SPS]: Use smaller batches of BlockMovingInfo into the block storage movement
> command
> -------------------------------------------------------------------------------------
>
> Key: HDFS-11125
> URL: https://issues.apache.org/jira/browse/HDFS-11125
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Rakesh R
> Assignee: Rakesh R
>
> This is a follow-up task of HDFS-11068, where it sends all the blocks under a
> trackID over single heartbeat response(DNA_BLOCK_STORAGE_MOVEMENT command).
> If blocks are many under a given trackID(For example: a file contains many
> blocks) then those requests go across a network and come with a lot of
> overhead. In this jira, we will discuss and implement a mechanism to limit
> the list of items into smaller batches with in trackID.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
