[
https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277750#comment-14277750
]
Ted Yu commented on HADOOP-11463:
---------------------------------
Without the patch, I observed the following exception in test
(fs.s3a.multipart.purge being true):
{code}
testRenameFileAsExistingFile(org.apache.hadoop.fs.s3a.TestS3AFileSystemContract)
Time elapsed: 1.944 sec <<< ERROR!
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@14b43b9c rejected from
java.util.concurrent.ThreadPoolExecutor@44b58962[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.<init>(UploadMonitor.java:129)
at
com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:449)
at
com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:382)
at
org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:125)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.createFile(FileSystemContractBaseTest.java:484)
at
org.apache.hadoop.fs.s3a.TestS3AFileSystemContract.testRenameFileAsExistingFile(TestS3AFileSystemContract.java:68)
{code}
The above is fixed by patch.
> Replace method-local TransferManager object with S3AFileSystem#transfers
> ------------------------------------------------------------------------
>
> Key: HADOOP-11463
> URL: https://issues.apache.org/jira/browse/HADOOP-11463
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: hadoop-11463-001.patch
>
>
> This is continuation of HADOOP-11446.
> The following changes are made according to Thomas Demoor's comments:
> 1. Replace method-local TransferManager object with S3AFileSystem#transfers
> 2. Do not shutdown TransferManager after purging existing multipart file -
> otherwise the current transfer is unable to proceed
> 3. Shutdown TransferManager instance in the close method of S3AFileSystem
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
