This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 809cec01205 [SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests 809cec01205 is described below commit 809cec012055d6f15987f338122d2fdb5bdd5c92 Author: yangjie01 <yangji...@baidu.com> AuthorDate: Wed Dec 6 10:50:27 2023 -0800 [SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests ### What changes were proposed in this pull request? After the merge of https://github.com/apache/spark/pull/43736, the master branch began testing the `streaming-kinesis-asl` module. At the same time, because the daily test will reuse `build_and_test.yml`, the daily test of branch-3.x also began testing `streaming-kinesis-asl`. However, in branch-3.x, the env `ENABLE_KINESIS_TESTS` is hard-coded as 1 in `dev/sparktestsupport/modules.py`: https://github.com/apache/spark/blob/1321b4e64deaa1e58bf297c25b72319083056568/dev/sparktestsupport/modules.py#L332-L346 which leads to the failure of the daily test of branch-3.x: - branch-3.3: https://github.com/apache/spark/actions/runs/7111246311 - branch-3.4: https://github.com/apache/spark/actions/runs/7098435892 - branch-3.5: https://github.com/apache/spark/actions/runs/7099811235 ``` [info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 14 milliseconds) [info] java.lang.Exception: Kinesis tests enabled using environment variable ENABLE_KINESIS_TESTS [info] but could not find AWS credentials. Please follow instructions in AWS documentation [info] to set the credentials in your system such that the DefaultAWSCredentialsProviderChain [info] can find the credentials. [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61) [info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41) [info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42) [info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273) [info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [info] at java.lang.Thread.run(Thread.java:750) [info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started [info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi started [info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder started [info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite finished: 0 failed, 0 ignored, 2 total, 0.244s [info] ScalaTest [info] Run completed in 8 seconds, 542 milliseconds. [info] Total number of tests run: 31 [info] Suites: completed 4, aborted 4 [info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0 [info] *** 4 SUITES ABORTED *** [error] Error: Total 37, Failed 0, Errors 4, Passed 33 [error] Error during tests: [error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite [error] org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite [error] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite [error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite [error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM ``` This PR adds a conditional judgment for the build task: ```shell if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{ inputs.branch }}" =~ ^branch-3 ]]; then MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /} fi ``` When `MODULES_TO_TEST` contains `streaming-kinesis-asl` and the test branch is not master, it removes the `streaming-kinesis-asl, ` substring from `MODULES_TO_TEST` and reassigns it to `MODULES_TO_TEST`. This avoids the testing of the `streaming-kinesis-asl` module in branch-3.x. ### Why are the changes needed? Prevent the daily tests of branch-3.x from testing the streaming-kinesis-asl module. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Monitor GA after merged ### Was this patch authored or co-authored using generative AI tooling? No Closes #44204 from LuciferYang/SPARK-46283. Lead-authored-by: yangjie01 <yangji...@baidu.com> Co-authored-by: YangJie <yangji...@baidu.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .github/workflows/build_and_test.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index d58356ec1c5..920ac6a38c8 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -268,6 +268,10 @@ jobs: export TERM=vt100 # Hive "other tests" test needs larger metaspace size based on experiment. if [[ "$MODULES_TO_TEST" == "hive" ]] && [[ "$EXCLUDED_TAGS" == "org.apache.spark.tags.SlowHiveTest" ]]; then export METASPACE_SIZE=2g; fi + // SPARK-46283: should delete the following env replacement after SPARK 3.x EOL + if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{ inputs.branch }}" =~ ^branch-3 ]]; then + MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /} + fi export SERIAL_SBT_TESTS=1 ./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST" --included-tags "$INCLUDED_TAGS" --excluded-tags "$EXCLUDED_TAGS" - name: Upload test results to report --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org