This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 809cec01205 [SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module
from `MODULES_TO_TEST` for branch-3.x daily tests
809cec01205 is described below
commit 809cec012055d6f15987f338122d2fdb5bdd5c92
Author: yangjie01 <[email protected]>
AuthorDate: Wed Dec 6 10:50:27 2023 -0800
[SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from
`MODULES_TO_TEST` for branch-3.x daily tests
### What changes were proposed in this pull request?
After the merge of https://github.com/apache/spark/pull/43736, the master
branch began testing the `streaming-kinesis-asl` module.
At the same time, because the daily test will reuse `build_and_test.yml`,
the daily test of branch-3.x also began testing `streaming-kinesis-asl`.
However, in branch-3.x, the env `ENABLE_KINESIS_TESTS` is hard-coded as 1
in `dev/sparktestsupport/modules.py`:
https://github.com/apache/spark/blob/1321b4e64deaa1e58bf297c25b72319083056568/dev/sparktestsupport/modules.py#L332-L346
which leads to the failure of the daily test of branch-3.x:
- branch-3.3: https://github.com/apache/spark/actions/runs/7111246311
- branch-3.4: https://github.com/apache/spark/actions/runs/7098435892
- branch-3.5: https://github.com/apache/spark/actions/runs/7099811235
```
[info]
org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite ***
ABORTED *** (1 second, 14 milliseconds)
[info] java.lang.Exception: Kinesis tests enabled using environment
variable ENABLE_KINESIS_TESTS
[info] but could not find AWS credentials. Please follow instructions in
AWS documentation
[info] to set the credentials in your system such that the
DefaultAWSCredentialsProviderChain
[info] can find the credentials.
[info] at
org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258)
[info] at
org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at
org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at
org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at
org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at
org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at
org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info] at
org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at
org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at
org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info] at
org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info] at
org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at
org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at
org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info] at
org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] at java.lang.Thread.run(Thread.java:750)
[info] Test run
org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started
[info] Test
org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi
started
[info] Test
org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder
started
[info] Test run
org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite
finished: 0 failed, 0 ignored, 2 total, 0.244s
[info] ScalaTest
[info] Run completed in 8 seconds, 542 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 4, aborted 4
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] *** 4 SUITES ABORTED ***
[error] Error: Total 37, Failed 0, Errors 4, Passed 33
[error] Error during tests:
[error]
org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
[error]
org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
[error]
org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
[error]
org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
[error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException:
Tests unsuccessful
[error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM
```
This PR adds a conditional judgment for the build task:
```shell
if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{
inputs.branch }}" =~ ^branch-3 ]]; then
MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /}
fi
```
When `MODULES_TO_TEST` contains `streaming-kinesis-asl` and the test branch
is not master, it removes the `streaming-kinesis-asl, ` substring from
`MODULES_TO_TEST` and reassigns it to `MODULES_TO_TEST`. This avoids the
testing of the `streaming-kinesis-asl` module in branch-3.x.
### Why are the changes needed?
Prevent the daily tests of branch-3.x from testing the
streaming-kinesis-asl module.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Monitor GA after merged
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #44204 from LuciferYang/SPARK-46283.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index d58356ec1c5..920ac6a38c8 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -268,6 +268,10 @@ jobs:
export TERM=vt100
# Hive "other tests" test needs larger metaspace size based on
experiment.
if [[ "$MODULES_TO_TEST" == "hive" ]] && [[ "$EXCLUDED_TAGS" ==
"org.apache.spark.tags.SlowHiveTest" ]]; then export METASPACE_SIZE=2g; fi
+ // SPARK-46283: should delete the following env replacement after
SPARK 3.x EOL
+ if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{
inputs.branch }}" =~ ^branch-3 ]]; then
+ MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /}
+ fi
export SERIAL_SBT_TESTS=1
./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
--included-tags "$INCLUDED_TAGS" --excluded-tags "$EXCLUDED_TAGS"
- name: Upload test results to report
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
