[
https://issues.apache.org/jira/browse/SPARK-49508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879818#comment-17879818
]
Kent Yao commented on SPARK-49508:
----------------------------------
I don't see this in tar.gz files. Do they only pulled when you turn on
hadoop-cloud file
> Optimized hadoop-aws dependency, aws-java-sdk-bundle jar is too large
> ---------------------------------------------------------------------
>
> Key: SPARK-49508
> URL: https://issues.apache.org/jira/browse/SPARK-49508
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 4.0.0, 3.5.2
> Reporter: melin
> Priority: Major
>
> aws-java-sdk-bundle jar is too large，The size of the spark image will
> double。hadoop aws only requires the use of aws-java-sdk-s3 and
> aws-java-sdk-dynamodb
>
> {code:java}
> // code placeholder
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-aws</artifactId>
> <version>${hadoop.version}</version>
> <exclusions>
> <exclusion>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-bundle</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-s3</artifactId>
> <version>${awssdk.v1.version}</version>
> </dependency>
> <dependency>
> <groupId>com.amazonaws</groupId>
> <artifactId>aws-java-sdk-dynamodb</artifactId>
> <version>${awssdk.v1.version}</version>
> </dependency> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
