[
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410971#comment-15410971
]
Chris Nauroth commented on HADOOP-13403:
----------------------------------------
[~pattipaka], patch 005 looks great. Thank you for the update.
Unfortunately, it looks like we'll need one more patch revision. I'll want to
commit this to both trunk and branch-2. The patch works fine on trunk, but the
test code fails to compile when applied to branch-2. (See below.)
A potential explanation for this difference is that we build trunk with Java 8
and branch-2 with Java 7. You might need to put explicit type bounds on the
Mockito calls to fix this.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-azure: Compilation failure: Compilation
failure:
[ERROR]
/Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[376,99]
incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
[ERROR]
/Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[424,118]
incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
[ERROR]
/Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[452,69]
incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
[ERROR]
/Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[571,99]
incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
[ERROR]
/Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[618,118]
incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
[ERROR]
/Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[649,69]
incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
{code}
> AzureNativeFileSystem rename/delete performance improvements
> ------------------------------------------------------------
>
> Key: HADOOP-13403
> URL: https://issues.apache.org/jira/browse/HADOOP-13403
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure
> Affects Versions: 2.7.2
> Reporter: Subramanyam Pattipaka
> Assignee: Subramanyam Pattipaka
> Fix For: 2.9.0
>
> Attachments: HADOOP-13403-001.patch, HADOOP-13403-002.patch,
> HADOOP-13403-003.patch, HADOOP-13403-004.patch, HADOOP-13403-005.patch
>
>
> WASB Performance Improvements
> Problem
> -----------
> Azure Native File system operations like rename/delete which has large number
> of directories and/or files in the source directory are experiencing
> performance issues. Here are possible reasons
> a) We first list all files under source directory hierarchically. This is
> a serial operation.
> b) After collecting the entire list of files under a folder, we delete or
> rename files one by one serially.
> c) There is no logging information available for these costly operations
> even in DEBUG mode leading to difficulty in understanding wasb performance
> issues.
> Proposal
> -------------
> Step 1: Rename and delete operations will generate a list all files under the
> source folder. We need to use azure flat listing option to get list with
> single request to azure store. We have introduced config
> fs.azure.flatlist.enable to enable this option. The default value is 'false'
> which means flat listing is disabled.
> Step 2: Create thread pool and threads dynamically based on user
> configuration. These thread pools will be deleted after operation is over.
> We are introducing introducing two new configs
> a) fs.azure.rename.threads : Config to set number of rename
> threads. Default value is 0 which means no threading.
> b) fs.azure.delete.threads: Config to set number of delete
> threads. Default value is 0 which means no threading.
> We have provided debug log information on number of threads not used
> for the operation which can be useful .
> Failure Scenarios:
> If we fail to create thread pool due to ANY reason (for example trying
> create with thread count with large value such as 1000000), we fall back to
> serialization operation.
> Step 3: Bob operations can be done in parallel using multiple threads
> executing following snippet
> while ((currentIndex = fileIndex.getAndIncrement()) < files.length) {
> FileMetadata file = files[currentIndex];
> Rename/delete(file);
> }
> The above strategy depends on the fact that all files are stored in a
> final array and each thread has to determine synchronized next index to do
> the job. The advantage of this strategy is that even if user configures large
> number of unusable threads, we always ensure that work doesn’t get serialized
> due to lagging threads.
> We are logging following information which can be useful for tuning
> number of threads
> a) Number of unusable threads
> b) Time taken by each thread
> c) Number of files processed by each thread
> d) Total time taken for the operation
> Failure Scenarios:
> Failure to queue a thread execute request shouldn’t be an issue if we
> can ensure at least one thread has completed execution successfully. If we
> couldn't schedule one thread then we should take serialization path.
> Exceptions raised while executing threads are still considered regular
> exceptions and returned to client as operation failed. Exceptions raised
> while stopping threads and deleting thread pool shouldn't can be ignored if
> operation all files are done with out any issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
