This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 64ec5f1017e1 [SPARK-32246][BUILD][INFRA] Enable
`streaming-kinesis-asl` tests in Github Action CI
64ec5f1017e1 is described below
commit 64ec5f1017e1f2ca479060ca76f18b1c4a803b81
Author: Junyu Chen <junyu...@amazon.com>
AuthorDate: Mon Dec 4 12:44:11 2023 -0800
[SPARK-32246][BUILD][INFRA] Enable `streaming-kinesis-asl` tests in Github
Action CI
### What changes were proposed in this pull request?
This PR attempts to set up Kinesis tests in one of the existing Github
Actions. Note that currently there are totally 57 tests in the Kinesis-asl
module, and this PR enabled 35 of them. The remaining tests requires
interaction with Amazon Kinesis service which would incur billing costs to
users. Hence they are not included in the Github Action.
### Why are the changes needed?
Addressing the comments in this PR:
https://github.com/apache/spark/pull/42581#issuecomment-1685925739 which
attempts to upgrade the AWS SDK to v2 for Spark Kinesis connector. Since
Kinesis tests are not being run in the Github Actions, there is no automated
mechanism to verify the SDK v2 upgrade changes in this module.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
1. All existing Github Actions passed.
2. All Kinesis tests passed when running locally: `export
ENABLE_KINESIS_TESTS=1 && mvn test -Pkinesis-asl -pl connector/kinesis-asl`
```
Tests: succeeded 57, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 13:25 min
[INFO] Finished at: 2023-11-12T00:15:49+08:00
[INFO]
------------------------------------------------------------------------
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43736 from junyuc25/junyuc25/kinesis-test.
Authored-by: Junyu Chen <junyu...@amazon.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.github/workflows/build_and_test.yml | 2 +-
dev/sparktestsupport/modules.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 4612b504ccdf..d58356ec1c5d 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -150,7 +150,7 @@ jobs:
- >-
mllib-local, mllib, graphx
- >-
- streaming, sql-kafka-0-10, streaming-kafka-0-10,
+ streaming, sql-kafka-0-10, streaming-kafka-0-10,
streaming-kinesis-asl,
yarn, kubernetes, hadoop-cloud, spark-ganglia-lgpl,
connect, protobuf
# Here, we split Hive and SQL tests into some of slow ones and the
rest of them.
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 8995b7de0df9..15b2e8f186e5 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -359,7 +359,7 @@ streaming_kinesis_asl = Module(
build_profile_flags=[
"-Pkinesis-asl",
],
- environ={"ENABLE_KINESIS_TESTS": "1"},
+ environ={"ENABLE_KINESIS_TESTS": "0"},
sbt_test_goals=[
"streaming-kinesis-asl/test",
],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
