Yikun opened a new pull request #35830: URL: https://github.com/apache/spark/pull/35830
### What changes were proposed in this pull request? Enable K8S integration tests in Github Action ### Why are the changes needed? I have previously explored to enabe Spark on K8S integration test on GitHub action, but it failed due to github hosted vm, there are some [hardware limit (2 CPU, 7G Memory)](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources). For Spak on K8S IT, 2 CPU is very limited for case in test, so we can't make all jobs passed. Recently, I found we can reduce limit for each jobs `request.cores` to make most of jobs passed. I test it in my local env with 2U 7G cluster, all test passed, so all IT can also passed in Github-hosted runner. In theory, there are no more influence of changing cpu a minor to validate spark on K8s features. This will also improve the efficiency of K8S development and guaranteed the quality of K8S. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ``` [info] KubernetesSuite: [info] - Run SparkPi with no resources (22 seconds, 560 milliseconds) [info] - Run SparkPi with no resources & statefulset allocation (19 seconds, 747 milliseconds) [info] - Run SparkPi with a very long application name. (20 seconds, 16 milliseconds) [info] - Use SparkLauncher.NO_RESOURCE (19 seconds, 445 milliseconds) [info] - Run SparkPi with a master URL without a scheme. (19 seconds, 95 milliseconds) [info] - Run SparkPi with an argument. (19 seconds, 271 milliseconds) [info] - Run SparkPi with custom labels, annotations, and environment variables. (18 seconds, 295 milliseconds) [info] - All pods have the same service account by default (18 seconds, 949 milliseconds) [info] - Run extraJVMOptions check on driver (9 seconds, 882 milliseconds) **[info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 7 seconds)** [info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (34 seconds, 169 milliseconds) [info] - Run SparkPi with env and mount secrets. (38 seconds, 594 milliseconds) [info] - Run PySpark on simple pi.py example (20 seconds, 116 milliseconds) [info] - Run PySpark to test a pyfiles example (24 seconds, 445 milliseconds) [info] - Run PySpark with memory customization (20 seconds, 272 milliseconds) [info] - Run in client mode. (37 seconds, 757 milliseconds) [info] - Start pod creation from template (19 seconds, 336 milliseconds) [info] - SPARK-38398: Schedule pod creation from template (19 seconds, 327 milliseconds) **[info] - Test basic decommissioning *** FAILED *** (3 minutes, 13 seconds)** [info] - Test basic decommissioning with shuffle cleanup (1 minute, 20 seconds) **[info] - Test decommissioning with dynamic allocation & shuffle cleanups *** FAILED *** (2 minutes, 14 seconds)** [info] - SPARK-37576: Rolling decommissioning (1 minute, 12 seconds) **[info] - Test decommissioning timeouts *** FAILED *** (3 minutes, 14 seconds)** [info] - SPARK-37576: Rolling decommissioning (1 minute, 12 seconds) [info] Run completed in 23 minutes, 20 seconds. [info] Total number of tests run: 23 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 19, failed 4, canceled 0, ignored 0, pending 0 [info] *** 4 TESTS FAILED *** ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
