jiayuasu opened a new issue, #3316: URL: https://github.com/apache/sedona/issues/3316
## Expected behavior `SpiderTests` should be deterministic and should not fail when the diagonal generator produces a statistically valid random sample. ## Actual behavior The `generate data with diagonal distribution` test failed in the Spark 4.1.1 / Scala 2.13.17 / JRE 17 CI job for PR #3310: ```text generate data with diagonal distribution *** FAILED *** 649 was not greater than or equal to 650 (SpiderTests.scala:197) ``` Failed job: https://github.com/apache/sedona/actions/runs/33615369126/job/100547139466 The test requests 1,000 records with `percentage=0.7`, but does not set the Spider `seed`. The data source therefore defaults to `System.currentTimeMillis()`, and `DiagonalGenerator` independently uses a Bernoulli trial for each record. The number of diagonal points is random, so a legitimate sample can fall outside the asserted inclusive range of 650 to 750. In this run it missed the lower bound by one. This unrelated failure also triggered matrix fail-fast and canceled four other JVM compatibility jobs on PR #3310. Relevant code: - Test assertion: https://github.com/apache/sedona/blob/0fdb69d4391da3dd34ebe2d6d0bdd88f985611f9/spark/common/src/test/scala/org/apache/sedona/sql/SpiderTests.scala#L181-L197 - Time-based default seed: https://github.com/apache/sedona/blob/0fdb69d4391da3dd34ebe2d6d0bdd88f985611f9/spark/common/src/main/scala/org/apache/sedona/sql/datasources/spider/SpiderDataSource.scala#L53-L57 - Bernoulli sampling: https://github.com/apache/sedona/blob/0fdb69d4391da3dd34ebe2d6d0bdd88f985611f9/common/src/main/java/org/apache/sedona/common/spider/DiagonalGenerator.java#L67-L79 ## Steps to reproduce Run the focused suite repeatedly without adding a `seed` option: ```shell mvn -pl spark/common -Dspark=4.1 -Dscala=2.13 -Dspark.version=4.1.1 -DwildcardSuites=org.apache.sedona.sql.SpiderTests test ``` The failure is probabilistic, so individual runs will usually pass. ## Suggested fix Set an explicit seed in this test and assert the deterministic result for that seed. This preserves coverage of the `percentage` behavior while eliminating random CI outcomes. Widening the statistical bounds alone would still leave a nonzero failure probability. ## Environment - Sedona version: `2.0.0-SNAPSHOT` (PR #3310 merge commit `4c7585b9cd7eea2d3ae0eead87864fc1a973ed1c`) - Apache Spark version: `4.1.1` - Scala version: `2.13.17` - JRE version: `17` - API type: Scala - Environment: GitHub Actions, Ubuntu 22.04 ## Existing issues - [x] I searched the existing issues and did not find a duplicate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
