Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20097#discussion_r159492891
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamSourceV2.scala
---
@@ -111,7 +112,7 @@ class RateStreamV2Reader(options: DataSourceV2Options)
val packedRows = mutable.ListBuffer[(Long, Long)]()
var outVal = startVal + numPartitions
- var outTimeMs = startTimeMs + msPerPartitionBetweenRows
+ var outTimeMs = startTimeMs
--- End diff --
The original behavior was an off-by-one error. With 1 partition and 1 row
per second, for example, every row would come timestamped 1 second after it was
actually generated.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
