[
https://issues.apache.org/jira/browse/HDFS-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108197#comment-15108197
]
Vinayakumar B commented on HDFS-9608:
-------------------------------------
Changes looks good.
I have some comments,
1. To reduce the number of lines changed, Instead of moving entire while loop
inside synchronized block, just extract that part to a method and call that
method from synchronized block.
As below.
{code}
@Override
public synchronized V chooseVolume(final List<V> volumes, long blockSize)
@@ -40,13 +51,24 @@ public synchronized V chooseVolume(final List<V> volumes,
long blockSize)
if(volumes.size() < 1) {
throw new DiskOutOfSpaceException("No more available volumes");
}
-
+
+ // As all the items in volumes are with the same storage type,
+ // so only need to get the storage type index of the first item in volumes
+ StorageType storageType = volumes.get(0).getStorageType();
+ int index = storageType != null ?
+ storageType.ordinal() : StorageType.DEFAULT.ordinal();
+ synchronized (syncLocks[index]) {
+ return chooseVolume(index, volumes, blockSize);
+ }
+ }
+
+ private V chooseVolume(final int curVolumesIndex, final List<V> volumes,
+ long blockSize) throws IOException {
// since volumes could've been removed because of the failure
// make sure we are not out of bounds
- if(curVolume >= volumes.size()) {
- curVolume = 0;
- }
-
+ int curVolume = curVolumes[curVolumesIndex] < volumes.size()
+ ? curVolumes[curVolumesIndex] : 0;
+
int startVolume = curVolume;
long maxAvailable = 0;
@@ -55,6 +77,7 @@ public synchronized V chooseVolume(final List<V> volumes,
long blockSize)
curVolume = (curVolume + 1) % volumes.size();
long availableVolumeSize = volume.getAvailable();
if (availableVolumeSize > blockSize) {
+ curVolumes[curVolumesIndex] = curVolume;
return volume;
}
{code}
2. Is this idea ( separate lock for each storage type) applicable for
{{AvailableSpaceVolumeChoosingPolicy}} as well?
> Disk IO imbalance in HDFS with heterogeneous storages
> -----------------------------------------------------
>
> Key: HDFS-9608
> URL: https://issues.apache.org/jira/browse/HDFS-9608
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Wei Zhou
> Assignee: Wei Zhou
> Attachments: HDFS-9608.01.patch, HDFS-9608.02.patch,
> HDFS-9608.03.patch, HDFS-9608.04.patch, HDFS-9608.05.patch
>
>
> Currently RoundRobinVolumeChoosingPolicy use a shared index to choose volumes
> in HDFS with heterogeneous storages, this leads to non-RR choosing mode for
> certain type of storage.
> Besides, it uses a shared lock for synchronization which limits the
> concurrency of volume choosing process. Volume choosing threads that
> operating on different storage types should be able to run concurrently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
