HeartSaVioR commented on a change in pull request #25048: [SPARK-28247][SS] Fix flaky test "query without test harness" on ContinuousSuite URL: https://github.com/apache/spark/pull/25048#discussion_r300851488
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala ########## @@ -36,6 +38,9 @@ class RateStreamContinuousStream(rowsPerSecond: Long, numPartitions: Int) extend val perPartitionRate = rowsPerSecond.toDouble / numPartitions.toDouble + private[sql] val highestCommittedValue = new AtomicLong(Long.MinValue) + private[sql] val firstCommittedTime = new AtomicLong(Long.MinValue) Review comment: As I commented earlier, `creationTime` in this class above is one of fields just for testing. Unlike microbatch mode, it's not easy to track and capture the progress. That might be the reason we were relying on wait time. I totally agree the change is not beauty, but given rate source is for testing/benchmarking purpose, and end users don't deal with RateStreamContinuousStream directly, we might be just OK with it. If it's still a thing for us to be concerned, please let me know, and we can rollback the change and just add more seconds in wait time to make test be higher rate to be passed. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
