[
https://issues.apache.org/jira/browse/SPARK-45720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786158#comment-17786158
]
junyuc25 edited comment on SPARK-45720 at 11/15/23 4:08 AM:
------------------------------------------------------------
The upgrade also requires migrating Kinesis-client-library from 1.x to 2.x. But
KCL 2.x pulls in transitive dependencies that are compiled with Scala 2.12,
which is considered banned dependencies in Spark 4.0. We should probably
consider updating these dependencies before working on this ticket.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:3.3.0:enforce (enforce-versions)
on project spark-streaming-kinesis-asl_2.13:
[ERROR] Rule 2: org.apache.maven.enforcer.rules.dependency.BannedDependencies
failed with message:
[ERROR] org.apache.spark:spark-streaming-kinesis-asl_2.13:jar:4.0.0-SNAPSHOT
[ERROR] software.amazon.kinesis:amazon-kinesis-client:jar:2.5.2
[ERROR] software.amazon.glue:schema-registry-serde:jar:1.1.14
[ERROR] com.kjetland:mbknor-jackson-jsonschema_2.12:jar:1.0.39 <---
banned via the exclude/include list
was (Author: junyu chen):
The upgrade also requires migrating Kinesis-client-library from 1.x to 2.x. But
KCL 2.x pulls in transitive dependencies that are compiled with Scala 2.12,
which is considered banned dependencies in Spark 4.0. We should probably
consider updating these dependencies before working on this ticket.
{quote}[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:3.3.0:enforce (enforce-versions)
on project spark-streaming-kinesis-asl_2.13:
[ERROR] Rule 2: org.apache.maven.enforcer.rules.dependency.BannedDependencies
failed with message:
[ERROR] org.apache.spark:spark-streaming-kinesis-asl_2.13:jar:4.0.0-SNAPSHOT
[ERROR] software.amazon.kinesis:amazon-kinesis-client:jar:2.5.2
[ERROR] software.amazon.glue:schema-registry-serde:jar:1.1.14
[ERROR] com.kjetland:mbknor-jackson-jsonschema_2.12:jar:1.0.39 <--- banned via
the exclude/include list
{quote}
> Upgrade AWS SDK to v2 for Spark Kinesis connector
> -------------------------------------------------
>
> Key: SPARK-45720
> URL: https://issues.apache.org/jira/browse/SPARK-45720
> Project: Spark
> Issue Type: Sub-task
> Components: Connect Contrib
> Affects Versions: 3.5.0
> Reporter: Lantao Jin
> Priority: Major
>
> Sub-task of [SPARK-44124|https://issues.apache.org/jira/browse/SPARK-44124].
> In this issue, we focus on the AWS SDK v2 upgrade in Kinesis connector
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
