[
https://issues.apache.org/jira/browse/HDFS-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-12818:
-------------------------------
Attachment: HDFS-12818.000.patch
Attaching v000 patch. {{SimulatedFSDataset}} now stores a list of
{{SimulatedStorage}} instead of a single one, and the block ID is hashed to
determine which storage the block belongs in. Each storage now stores its own
block map as opposed to a single one for the entire {{FSDataset}}. Modified
{{TestSimulatedFSDataset}} to accommodate the possibility of having multiple
storages and created a subclass {{TestSimulatedFSDatasetWithMultipleStorages}}
which runs the same tests against a {{SimulatedFSDataset}} that has 2 storages.
I had to make a minor modification to {{DataNode}} to support this change. If a
{{DataStorage}} is passed in to the {{SimulatedFSDataset}} factory, that
storage's volume count will be used. Previously, {{DataNode}} would pass in a
{{DataStorage}} to the factory, but it does not call
{{recoverTransitionRead()}} on that storage if the {{FSDataset}} is simulated,
so the storage that gets passed in always has 0 volumes. I added a check to
just pass a {{null}} storage down if the {{FSDataset}} is simulated. This works
fine but I am also open to other suggestions on the best way to handle this.
> Support multiple storages in DataNodeCluster / SimulatedFSDataset
> -----------------------------------------------------------------
>
> Key: HDFS-12818
> URL: https://issues.apache.org/jira/browse/HDFS-12818
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, test
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Minor
> Attachments: HDFS-12818.000.patch
>
>
> Currently {{SimulatedFSDataset}} (and thus, {{DataNodeCluster}} with
> {{-simulated}}) only supports a single storage per {{DataNode}}. Given that
> the number of storages can have important implications on the performance of
> block report processing, it would be useful for these classes to support a
> multiple storage configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
