[
https://issues.apache.org/jira/browse/HADOOP-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930484#comment-15930484
]
John Zhuge commented on HADOOP-14195:
-------------------------------------
More and more applications start to access cloud filesystem directly using URI
{{s3a://}} or {{adl://}}. {{Configuration.getPassword}} is called during
construction of each fs instance to get credentials, thus more likely in
multiple threads. {{Configuration.getPassword}} uses service loader to load
credential providers.
Here is a real-world Hive exception backtrace:
{noformat}
Caused by: java.util.NoSuchElementException
at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59)
at
java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
at
org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:374)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:175)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2696)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2733)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:372)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:252)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)
{noformat}
SLIDER-888 reported a similar issue.
So somewhere along the stack, we need to serialize the call.
> CredentialProviderFactory is not thread-safe
> --------------------------------------------
>
> Key: HADOOP-14195
> URL: https://issues.apache.org/jira/browse/HADOOP-14195
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Affects Versions: 2.7.0
> Reporter: Vihang Karajgaonkar
> Assignee: John Zhuge
> Attachments: TestCredentialProvider.java
>
>
> Multi-threaded access to CredentialProviderFactory is not thread-safe because
> {{java.util.ServiceLoader}} is not thread-safe (as noted in its Java doc).
> Thanks to [~jzhuge] I was able to reproduce this issue but creating a simple
> multi-threaded application which executes the following code in parallel.
> {code:java}
> for (int i = 0; i < ITEMS; i++) {
> futures.add(executor.submit(new Callable<Void>() {
> @Override
> public Void call() throws Exception {
> boolean found = false;
> for (CredentialProviderFactory factory : serviceLoader) {
> CredentialProvider kp = factory.createProvider(uri,
> conf);
> if (kp != null) {
> result.add(kp);
> found = true;
> break;
> }
> }
> if (!found) {
> throw new IOException(Thread.currentThread() + "No
> CredentialProviderFactory for " + uri);
> } else {
> System.out.println(Thread.currentThread().getName() + "
> found credentialProvider for " + path);
> }
> return null;
> }
> }));
> }
> {code}
> I see the following exception trace when I execute the above code.
> {code:java}
> java.util.concurrent.ExecutionException: java.util.NoSuchElementException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.util.NoSuchElementException
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615)
> at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590)
> at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I also see a NPE sometimes
> {code:java}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at TestCredentialProvider.main(TestCredentialProvider.java:58)
> Caused by: java.lang.NullPointerException
> at java.util.ServiceLoader.parse(ServiceLoader.java:304)
> at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:38)
> at TestCredentialProvider$1.call(TestCredentialProvider.java:1)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
