[
https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886240#comment-16886240
]
Steve Loughran edited comment on HADOOP-16430 at 7/16/19 3:58 PM:
------------------------------------------------------------------
Managed to create an NPE in test teardown here in a test run:
-Dit.test=ITestS3ADeleteManyFiles -Dscale -Ds3guard -Ddynamo
I'm assuming this is my new patch for now, but it could be a sign of something
which has been lurking for a bit and its only with one of us bumping up the
scale count it surfaces.
{code}
<property>
<name>scale.test.operation.count</name>
<value>2000</value>
</property>
{code}
{code}
Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
0:01.605s
2019-07-16 16:52:14,179 [teardown] ERROR contract.ContractTestUtils
(ContractTestUtils.java:cleanup(380)) - Error deleting in TEARDOWN - /test:
java.lang.NullPointerException: Path
/test/testBulkRenameAndDelete/final/file-1899 missing scheme
java.lang.NullPointerException: Path
/test/testBulkRenameAndDelete/final/file-1899 missing scheme
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:980)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:2092)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerDelete(DynamoDBMetadataStore.java:577)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$deleteEntries$2(DynamoDBMetadataStore.java:676)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:61)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-07-16 16:52:14,182 [teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
{code}
was (Author: [email protected]):
Managed to create an NPE in test teardown here in a test run:
-Dit.test=ITestS3ADeleteManyFiles -Dscale -Ds3guard -Ddynamo
I'm assuming this is my new patch for now, but it could be a sign of something
which has been lurking for a bit and its only with one of us bumping up the
scale count it surfaces.
{code}
<property>
<name>scale.test.operation.count</name>
<value>2000</value>
</property>
{code}
{code}
Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
0:01.605s
2019-07-16 16:52:14,179 [teardown] ERROR contract.ContractTestUtils
(ContractTestUtils.java:cleanup(380)) - Error deleting in TEARDOWN - /test:
java.lang.NullPointerException: Path
/test/testBulkRenameAndDelete/final/file-1899 missing scheme
java.lang.NullPointerException: Path
/test/testBulkRenameAndDelete/final/file-1899 missing scheme
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:980)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:2092)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerDelete(DynamoDBMetadataStore.java:577)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$deleteEntries$2(DynamoDBMetadataStore.java:676)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:61)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-07-16 16:52:14,182 [teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
> S3AFilesystem.delete to incrementally update s3guard with deletions
> -------------------------------------------------------------------
>
> Key: HADOOP-16430
> URL: https://issues.apache.org/jira/browse/HADOOP-16430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> Currently S3AFilesystem.delete() only updates the delete at the end of a
> paged delete operation. This makes it slow when there are many thousands of
> files to delete ,and increases the window of vulnerability to failures
> Preferred
> * after every bulk DELETE call is issued to S3, queue the (async) delete of
> all entries in that post.
> * at the end of the delete, await the completion of these operations.
> * inside S3AFS, also do the delete across threads, so that different HTTPS
> connections can be used.
> This should maximise DDB throughput against tables which aren't IO limited.
> When executed against small IOP limited tables, the parallel DDB DELETE
> batches will trigger a lot of throttling events; we should make sure these
> aren't going to trigger failures
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
