[
https://issues.apache.org/jira/browse/FLINK-14896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980053#comment-16980053
]
Michel Davit commented on FLINK-14896:
--------------------------------------
The bug triggers only when user code access part of the AWS SDK which relies on
jackson (in my case the PartitionsLoader).
In our case, we were simply using the SDK in the main function to figure out
the DynamoDb stream ARN from the table name. Seeing that flink includes the
aws-sdk-dynamodb, I though it was safe to use it. It actually worked for some
time until the classpath ordering changed when updating to 1.9. At this point,
the jackson lib used was the one pulled by hadoop which is not compatible with
the SDK.
Even if this does not impact the flink framework itself, I'considering this as
a *minor* bug since this problem can be avoided without costs (flink already
contains a compatible shaded version of jackson)
> Kinesis connector doesn't shade jackson dependency
> --------------------------------------------------
>
> Key: FLINK-14896
> URL: https://issues.apache.org/jira/browse/FLINK-14896
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
> Affects Versions: 1.9.0
> Environment: AWS EMR 5.28.0
> Reporter: Michel Davit
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> flink-kinesis-connector depends on aws java sdk which is shaded to
> {{org.apache.flink.kinesis.shaded.com.amazonaws.}}
>
> {{However, the aws sdk has a transitive dependency to jackson wich is not
> shaded in the artifact.}}
>
> {{This creates problem when running flink on YARN: }}{{The aws sdk requires
> jackson-core v2.6 but hadoop pulls in 2.3. See
> [here|https://github.com/apache/flink/blob/e7c11ed672013512e5b159e7e892b27b1ef60a1b/flink-yarn/pom.xml#L133].}}
>
> {{If YARN uses the loads wrong jackson version from classpath. Jod fails
> with}}
> {code:java}
> 2019-11-20 17:23:11,563 ERROR
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Unhandled
> exception.org.apache.flink.client.program.ProgramInvocationException: The
> program caused an error: at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)
> at
> org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:80)
> at
> org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:126)
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$6(JarRunHandler.java:142)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
> at
> com.amazonaws.partitions.PartitionsLoader.<clinit>(PartitionsLoader.java:54)
> at
> com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30)
> at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:65)
> at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:53)
> at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:107) at
> com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256)
> at
> com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:460)
> at
> com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
> at
> com.amazonaws.client.builder.AwsAsyncClientBuilder.build(AwsAsyncClientBuilder.java:80)
> ...
> {code}
> The flink-kinesis-connector should do as other connectors: shade jackson or
> use the flink-shaded-jackson core dependency
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
