Bharat Viswanadham created HADOOP-15924:
-------------------------------------------
Summary: Hadoop aws cannot be used with shaded jars
Key: HADOOP-15924
URL: https://issues.apache.org/jira/browse/HADOOP-15924
Project: Hadoop Common
Issue Type: Bug
Reporter: Bharat Viswanadham
Issue is hadoop-aws cannot be used with shaded jars.
The recommended client side jars for hadoop 3 are client-api/runtime shaded
jars.
They shade guava etc. So something like SemaphoredDelegatingExecutor refers to
shaded guava classes.
hadoop-aws has S3AFileSystem implementation which refers to
SemaphoredDelegatingExecutor with unshaded guava ListeningService in the
constructor. When S3AFileSystem is created then it uses the hadoop-api jar and
finds SemaphoredDelegatingExecutor but not the right constructor because in
client-api jar SemaphoredDelegatingExecutor constructor has the shaded guava
ListenerService.
So essentially none of the aws/azure/adl hadoop FS implementations will work
with the shaded Hadoop client runtime jars.
This Jira is created to track the work required to make hadoop-aws work with
hadoop shaded client jars.
The solution for this can be, hadoop-aws depends on hadoop shaded jars. In this
way, we shall not see the issue. Currently, hadoop-aws depends on
aws-sdk-bundle and all other remaining jars are provided dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
