steveloughran commented on issue #1246: HADOOP-16499. S3A retry policy to be exponential URL: https://github.com/apache/hadoop/pull/1246#issuecomment-519274539 Turns out there are even more tests which have expectations that connectivity/read exceptions retry for a time less than the test timeout, i.e. the default of 20s. Shrink the retry time to a couple of seconds *and the tests are faster* ``` [INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.465 s - in org.apache.hadoop.fs.s3a.select.ITestS3Select[INFO] Tests run: 135, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 387.994 s - in org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged[ERROR] Tests run: 28, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1,208.394 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration[ERROR] testProxyConnection(org.apache.hadoop.fs.s3a.ITestS3AConfiguration) Time elapsed: 600.012 s <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:433) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:310) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:285) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:468) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:363) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:176) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:138) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.lambda$expectFSCreateFailure$0(ITestS3AConfiguration.java:149) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration$$Lambda$8/1830464825.call(Unknown Source) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.expectFSCreateFailure(ITestS3AConfiguration.java:147) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testProxyConnection(ITestS3AConfiguration.java:132) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.ITestS3AConfiguration) Time elapsed: 599.994 s <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:433) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:310) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:285) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:468) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:363) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:176) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:138) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.lambda$expectFSCreateFailure$0(ITestS3AConfiguration.java:149) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration$$Lambda$8/1830464825.call(Unknown Source) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.expectFSCreateFailure(ITestS3AConfiguration.java:147) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testAutomaticProxyPortSelection(ITestS3AConfiguration.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] Tests run: 22, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 2,492.747 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations[ERROR] testOutOfBandDeletes[auth=true](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations) Time elapsed: 599.995 s <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:433) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:342) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:220) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:264) at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.lambda$expectExceptionWhenReading$7(ITestS3GuardOutOfBandOperations.java:925) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations$$Lambda$53/1880629242.call(Unknown Source) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.expectExceptionWhenReading(ITestS3GuardOutOfBandOperations.java:923) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.outOfBandDeletes(ITestS3GuardOutOfBandOperations.java:609) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testOutOfBandDeletes(ITestS3GuardOutOfBandOperations.java:275) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testListingDelete[auth=true](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations) Time elapsed: 599.993 s <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:433) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:342) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:220) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:264) at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.lambda$expectExceptionWhenReading$7(ITestS3GuardOutOfBandOperations.java:925) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations$$Lambda$53/1880629242.call(Unknown Source) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.expectExceptionWhenReading(ITestS3GuardOutOfBandOperations.java:923) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.deleteFileInListing(ITestS3GuardOutOfBandOperations.java:907) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testListingDelete(ITestS3GuardOutOfBandOperations.java:290) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testOutOfBandDeletes[auth=false](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations) Time elapsed: 599.994 s <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:433) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:342) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:220) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:264) at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.lambda$expectExceptionWhenReading$7(ITestS3GuardOutOfBandOperations.java:925) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations$$Lambda$53/1880629242.call(Unknown Source) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.expectExceptionWhenReading(ITestS3GuardOutOfBandOperations.java:923) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.outOfBandDeletes(ITestS3GuardOutOfBandOperations.java:609) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testOutOfBandDeletes(ITestS3GuardOutOfBandOperations.java:275) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testListingDelete[auth=false](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations) Time elapsed: 599.994 s <<< ERROR!org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:433) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:342) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:220) at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:264) at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.lambda$expectExceptionWhenReading$7(ITestS3GuardOutOfBandOperations.java:925) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations$$Lambda$53/1880629242.call(Unknown Source) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.expectExceptionWhenReading(ITestS3GuardOutOfBandOperations.java:923) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.deleteFileInListing(ITestS3GuardOutOfBandOperations.java:907) at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testListingDelete(ITestS3GuardOutOfBandOperations.java:290) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
