steveloughran commented on issue #1962: HADOOP-16953. tuning s3guard disabled warnings URL: https://github.com/apache/hadoop/pull/1962#issuecomment-615194730 did a full release and storediag test run with the -D fail option; triggers an NPE in instrumentation shut down. Will fix ``` 2020-04-17 12:27:09,475 [main] ERROR s3a.S3AFileSystem (S3Guard.java:logS3GuardDisabled(1100)) - S3Guard is disabled on this bucket: stevel-london 2020-04-17 12:27:09,476 [main] DEBUG s3a.S3AFileSystem (HadoopExecutors.java:shutdown(118)) - Gracefully shutting down executor service. Waiting max 30 SECONDS 2020-04-17 12:27:09,476 [main] DEBUG s3a.S3AFileSystem (HadoopExecutors.java:shutdown(129)) - Succesfully shutdown executor service 2020-04-17 12:27:09,476 [main] DEBUG s3a.S3AFileSystem (HadoopExecutors.java:shutdown(118)) - Gracefully shutting down executor service. Waiting max 30 SECONDS 2020-04-17 12:27:09,476 [main] DEBUG s3a.S3AFileSystem (HadoopExecutors.java:shutdown(129)) - Succesfully shutdown executor service 2020-04-17 12:27:09,476 [main] DEBUG s3a.S3AInstrumentation (S3AInstrumentation.java:close(627)) - Shutting down metrics publisher 2020-04-17 12:27:09,477 [main] DEBUG delegation.S3ADelegationTokens (S3ADelegationTokens.java:serviceStop(225)) - Stopping delegation tokens 2020-04-17 12:27:09,480 [main] DEBUG auth.SignerManager (SignerManager.java:close(140)) - Unregistering fs from 0 initializers 2020-04-17 12:27:09,480 [main] DEBUG s3a.S3AFileSystem (S3AUtils.java:closeAutocloseables(1666)) - Closing AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@3d246ea3] 2020-04-17 12:27:09,480 [main] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:close(315)) - Closing AWSCredentialProviderList[refcount= 0: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@3d246ea3] 2020-04-17 12:27:09,481 [main] WARN fs.FileSystem (FileSystem.java:createFileSystem(3418)) - Failed to initialize fileystem s3a://stevel-london/: 49: S3Guard is disabled on this bucket: stevel-london 2020-04-17 12:27:09,481 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:close(3269)) - Filesystem s3a://stevel-london is closed 2020-04-17 12:27:09,482 [main] DEBUG s3a.S3AFileSystem (IOUtils.java:cleanupWithLogger(283)) - Exception in closing org.apache.hadoop.fs.s3a.S3AInstrumentation@f415a95 java.lang.NullPointerException at org.apache.hadoop.fs.s3a.S3AInstrumentation.close(S3AInstrumentation.java:624) at org.apache.hadoop.io.IOUtils.cleanupWithLogger(IOUtils.java:280) at org.apache.hadoop.fs.s3a.S3AFileSystem.stopAllServices(S3AFileSystem.java:3299) at org.apache.hadoop.fs.s3a.S3AFileSystem.close(S3AFileSystem.java:3273) at org.apache.hadoop.io.IOUtils.cleanupWithLogger(IOUtils.java:280) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3423) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:860) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:409) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:353) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:1166) at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:1175) at storediag.main(storediag.java:25) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) 2020-04-17 12:27:09,484 [main] INFO diag.StoreDiag (DurationInfo.java:close(100)) - Creating filesystem s3a://stevel-london/: duration 0:01:715 49: S3Guard is disabled on this bucket: stevel-london at org.apache.hadoop.fs.s3a.s3guard.S3Guard.logS3GuardDisabled(S3Guard.java:1101) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:446) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:860) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:409) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:353) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:1166) at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:1175) at storediag.main(storediag.java:25) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) 2020-04-17 12:27:09,487 [main] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status -1: 49: S3Guard is disabled on this bucket: stevel-london ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
