Iván Morales created SPARK-57365:
------------------------------------
Summary: Fix flaky KafkaRelationSuite resolved-offset test by
using distinct message timestamps
Key: SPARK-57365
URL: https://issues.apache.org/jira/browse/SPARK-57365
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.1.0
Reporter: Iván Morales
The test "resolved start offset greater than end offset (without latest)" in
KafkaRelationSuite is flaky. It produces two messages per partition without
explicit timestamps and expects offsetsForTimes to resolve to offset 1.
When both messages land in the same millisecond they share a CreateTime
timestamp, offsetsForTimes resolves to offset 0, and the hardcoded "resolved
end offset 1" assertion fails. he existing eventually(60.seconds) does not
help: produced timestamps are fixed at produce time, so retrying always
resolves the same offset.
Introduced by SPARK-52096 (commit 2b5cd7dc59, 2025-05-15).
Affects master, 4.1.x, 4.2.x.
Fix: produce the two messages with explicit increasing timestamps (base, base+1)
via RecordBuilder so the second message deterministically resolves to offset 1.
Test-only change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
