EricJoy2048 commented on PR #5745:
URL: https://github.com/apache/seatunnel/pull/5745#issuecomment-2084303699
> > I think we can create a minio container in e2e test case for s3a file
read.
>
> I created minio test container and added e2e test case. It passes on
spark3, but fails on flink/seatunnel/spark2.
>
> I think I need to add hadoop-aws/aws-java-sdk jar file to the test
container docker image and need your help.
>
> * spark2 : hadoop-aws-2.7.5, aws-java-sdk-1.7.4
>
> I'm also looking to see if there are other workarounds.
Like the driver, you can add `hadoop-aws` and `aws-java-sdk` to the test
container docker like
`seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-oss-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/oss/OssFileIT.java`
```
@TestContainerExtension
private final ContainerExtendedFactory extendedFactory =
container -> {
Container.ExecResult extraCommands =
container.execInContainer(
"bash",
"-c",
"mkdir -p /tmp/seatunnel/plugins/oss/lib &&
cd /tmp/seatunnel/plugins/oss/lib && curl -O "
+ OSS_SDK_DOWNLOAD);
Assertions.assertEquals(0, extraCommands.getExitCode());
extraCommands =
container.execInContainer(
"bash",
"-c",
"cd /tmp/seatunnel/plugins/oss/lib && curl
-O " + JDOM_DOWNLOAD);
Assertions.assertEquals(0, extraCommands.getExitCode());
extraCommands =
container.execInContainer(
"bash",
"-c",
"cd /tmp/seatunnel/plugins/oss/lib && curl
-O "
+ HADOOP_ALIYUN_DOWNLOAD);
Assertions.assertEquals(0, extraCommands.getExitCode());
extraCommands =
container.execInContainer(
"bash",
"-c",
"cd /tmp/seatunnel/lib && curl -O " +
OSS_SDK_DOWNLOAD);
Assertions.assertEquals(0, extraCommands.getExitCode());
extraCommands =
container.execInContainer(
"bash", "-c", "cd /tmp/seatunnel/lib && curl
-O " + JDOM_DOWNLOAD);
Assertions.assertEquals(0, extraCommands.getExitCode());
extraCommands =
container.execInContainer(
"bash",
"-c",
"cd /tmp/seatunnel/lib && curl -O " +
HADOOP_ALIYUN_DOWNLOAD);
Assertions.assertEquals(0, extraCommands.getExitCode());
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
