[
https://issues.apache.org/jira/browse/SPARK-37743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-37743:
---------------------------------
Summary: Revisit codes in Scala tests (was: Remove/update obsolete codes
in Scala tests)
> Revisit codes in Scala tests
> ----------------------------
>
> Key: SPARK-37743
> URL: https://issues.apache.org/jira/browse/SPARK-37743
> Project: Spark
> Issue Type: Sub-task
> Components: Tests
> Affects Versions: 3.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> {code}
> core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala: //
> TODO(SPARK-9603): Building a package is flaky on Jenkins Maven builds.
> core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala:
> // This test suite has some weird behaviors when executed on Jenkins:
> core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala:
> // 1. Sometimes it gets extremely slow out of unknown reason on Jenkins.
> Here we add a
> core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala: //
> is only 2, while on Jenkins it's 32. For this specific test, 2 available
> processors, which
> external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala:
> * when running on a slow Jenkins machine) before records start to be
> removed. To make sure a test
> project/SparkBuild.scala: // with Jenkins flakiness.
> repl/src/test/scala/org/apache/spark/repl/SparkShellSuite.scala: // This
> test suite sometimes gets extremely slow out of unknown reason on Jenkins.
> Here we
> sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
> // TODO: Why fs.getContentSummary returns wrong size on Jenkins?
> sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
> // Seems fs.getContentSummary returns wrong table size on Jenkins. So we
> use
> sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala: //
> the last dot due to a bug in SBT. This makes easier to debug via Jenkins test
> result
> sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreIntegrationSuite.scala:
> // Should emit new progresses every 10 ms, but we could be facing a
> slow Jenkins
> sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala:
> // Should emit new progresses every 10 ms, but we could be facing a
> slow Jenkins
> sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala:
> // This test suite sometimes gets extremely slow out of unknown reason
> on Jenkins. Here we
> sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala:
> // started at a time, which is not Jenkins friendly.
> sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientBuilder.scala:
> // In order to speed up test execution during development or in Jenkins,
> you can specify the path
> sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala:
> // more cores, the issue can be reproduced steadily. Fortunately our
> Jenkins builder meets this
> streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala:
> // If Jenkins is slow, we may not have a chance to run many threads
> simultaneously. Having
> core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
> // This is to avoid any potential flakiness in the test because of large
> pauses in jenkins
> sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java:
> // (e.g., /home/jenkins/workspace/SparkPullRequestBuilder@2)
> sql/core/src/test/resources/sql-tests/inputs/datetime-parsing-invalid.sql:--
> in java 8 this case is invalid, but valid in java 11, disabled for jenkins
> sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala:
> // TODO: Figure out why StringType doesn't work on jenkins.
> sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala:
> // Weird DDL differences result in failures on jenkins.
> project/SparkBuild.scala: // with Jenkins flakiness.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
