Sebastian Nagel created BIGTOP-4548:
---------------------------------------
Summary: Accessing S3A file system fails because of missing AWS SDK
Key: BIGTOP-4548
URL: https://issues.apache.org/jira/browse/BIGTOP-4548
Project: Bigtop
Issue Type: Bug
Components: hadoop
Affects Versions: 3.6.0
Environment: Ubuntu 24.04, ARM64, AWS
Reporter: Sebastian Nagel
(seen while testing Bigtop 3.6.0 RC0 on Ubuntu 24.04, ARM64, AWS EC2)
On Bigtop 3.5.0 the hadoop-aws.jar and the required AWS SDK are contained in
the package "hadoop" and are installed into the tools lib dir:
{noformat}
/usr/lib/hadoop/tools/lib/hadoop-aws-3.3.6.jar
/usr/lib/hadoop/tools/lib/aws-java-sdk-bundle-1.12.367.jar
{noformat}
On Bigtop 3.6.0 (RC0)
- the hadoop-aws jar is moved into the lib dir:
{noformat}
/usr/lib/hadoop/lib/hadoop-aws-3.4.3.jar
{noformat}
- the AWS SDK bundle is missing
Access to S3 via S3A fails:
{noformat}
$> hadoop fs -ls s3a://mybucket/
Exception in thread "main" java.lang.NoClassDefFoundError:
software/amazon/awssdk/core/exception/SdkException
{noformat}
Related issues:
- HADOOP-19083 removed the AWS SDK from the binary distribution because of its
large size.
- HADOOP-19696 has moved the cloud connectors to the common lib dir to avoid
extra setup when accessing data on cloud storage.
- See BIGTOP-3495 which moved the cloud packages to the tools lib dir.
- hadoop-aws.sh in the Hadoop binary tarball also still references the tools
lib dir (that's a Hadoop issue, not a Bigtop one)
The problem can be easily worked around by
- Determine the correct version used with Hadoop 3.4.3:
- see
<https://github.com/apache/hadoop/blob/branch-3.4.3/hadoop-project/pom.xml#L207>
- => 2.35.4
- Download and install the AWS SDK bundle:
{noformat}
mvn dependency:get -Dartifact=software.amazon.awssdk:bundle:2.35.4
sudo cp ~/.m2/repository/software/amazon/awssdk/bundle/2.35.4/bundle-2.35.4.jar
/usr/lib/hadoop/lib/
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
