[
https://issues.apache.org/jira/browse/FLINK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian J. updated FLINK-13044:
---------------------------------
Description:
The current build configuration shades the AWS SDK into
{{flink-s3-fs-hadoop.jar}}
Due to [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156] the shading
itself does not only add the classes to the JAR but also alters the string
literal {{PACKAGE_PREFIXES_TO_SHADE}} in
{{org.apache.flink.fs.s3hadoop.S3FileSystemFactory}} from {{com.amazonaws.}} to
{{org.apache.flink.fs.s3base.shaded.com.amazonaws}} - which causes settings
like
{noformat}fs.s3a.aws.credentials.provider:
com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
to not be remapped properly as the {{startsWith}} in
{{org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig()}} doesn't
match the beginning of the FQCN.
This is a known issue in the Maven shade plugin (see
[MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156]) - a possible
workaround is named in the ticket, too.
This issue is only visible looking into the compiled {{class}} file after the
shading happened!
Fixing the {{PACKAGE_PREFIXES_TO_SHADE}} value alone doesn't help as based on
{{FLINK_SHADING_PREFIX}} the FQCN is assembled as
{noformat}org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
which can't be found either;
the class is located at
{noformat}org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
was:
The current build configuration shades the AWS SDK into
{{flink-s3-fs-hadoop.jar}}
Due to MSHAD-156 the shading itself does not only add the classes to the JAR
but also alters the string literal {{PACKAGE_PREFIXES_TO_SHADE}} in
{{org.apache.flink.fs.s3hadoop.S3FileSystemFactory}} from {{com.amazonaws.}} to
{{org.apache.flink.fs.s3base.shaded.com.amazonaws}} - which causes settings
like
{noformat}fs.s3a.aws.credentials.provider:
com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
to not be remapped properly as the {{startsWith}} in
{{org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig()}} doesn't
match the beginning of the FQCN.
This is a known issue in the Maven shade plugin (see MSHAD-156) - a possible
workaround is named in the ticket, too.
This issue is only visible looking into the compiled {{class}} file after the
shading happened!
Fixing the {{PACKAGE_PREFIXES_TO_SHADE}} value alone doesn't help as based on
{{FLINK_SHADING_PREFIX}} the FQCN is assembled as
{noformat}org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
which can't be found either;
the class is located at
{noformat}org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
> Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions
> ---------------------------------------------------------------------------
>
> Key: FLINK-13044
> URL: https://issues.apache.org/jira/browse/FLINK-13044
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded, Connectors / Hadoop Compatibility
> Affects Versions: 1.8.0
> Reporter: Sebastian J.
> Priority: Critical
>
> The current build configuration shades the AWS SDK into
> {{flink-s3-fs-hadoop.jar}}
> Due to [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156] the
> shading itself does not only add the classes to the JAR but also alters the
> string literal {{PACKAGE_PREFIXES_TO_SHADE}} in
> {{org.apache.flink.fs.s3hadoop.S3FileSystemFactory}} from {{com.amazonaws.}}
> to {{org.apache.flink.fs.s3base.shaded.com.amazonaws}} - which causes
> settings like
> {noformat}fs.s3a.aws.credentials.provider:
> com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
> to not be remapped properly as the {{startsWith}} in
> {{org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig()}}
> doesn't match the beginning of the FQCN.
> This is a known issue in the Maven shade plugin (see
> [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156]) - a possible
> workaround is named in the ticket, too.
> This issue is only visible looking into the compiled {{class}} file after the
> shading happened!
> Fixing the {{PACKAGE_PREFIXES_TO_SHADE}} value alone doesn't help as based on
> {{FLINK_SHADING_PREFIX}} the FQCN is assembled as
> {noformat}org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
> which can't be found either;
> the class is located at
> {noformat}org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
