eason-yuchen-liu opened a new pull request, #57860:
URL: https://github.com/apache/spark/pull/57860
Backport of #57580 to branch-4.2. Cherry-picked cleanly from the merged
master commit 041ed937785 (no conflicts).
### What changes were proposed in this pull request?
Two changes for the public Real-Time Mode (RTM) connector SPI:
1. Restore the single-argument `nextWithTimeout(Long timeoutMs)` method on
the `@Evolving` `SupportsRealTimeRead` interface, which SPARK-55699 replaced
with `nextWithTimeout(Long startTimeMs, Long timeoutMs)`. Both overloads are
now `default` methods:
- `nextWithTimeout(Long)` throws `UnsupportedOperationException` by
default and is the method a third-party source is expected to implement.
- `nextWithTimeout(Long, Long)` is the overload the engine invokes; by
default it delegates to `nextWithTimeout(Long)`, ignoring `startTimeMs`. It is
meant for engine-internal sources that need the engine's reference start time.
A source overrides whichever one it needs. The engine always invokes the
two-argument overload, so a source that overrides only the single-argument
variant (as external Spark 4.1-era sources do) is driven through the
delegation. Internal implementors (`LowLatencyMemoryStream`, Kafka) already
override the two-argument variant and are unchanged.
2. Add `StreamingRealTimeModeSourceCompatSuite`, a backward-compatibility
guard. It defines a self-contained RTM streaming source written exclusively
against the public `@Evolving` connector APIs (`TableProvider`,
`Table`/`SupportsRead`, `ScanBuilder`/`Scan`, `MicroBatchStream` +
`SupportsRealTimeMode`, `PartitionReaderFactory`, `SupportsRealTimeRead`, and
custom `Offset`/`PartitionOffset`) and runs it end-to-end for both
`nextWithTimeout` entry points.
### Why are the changes needed?
SPARK-55699 changed the sole abstract method on the public interface rather
than adding an overload, which is a source- and binary-incompatible break. An
external source (a custom MQTT Real-Time Mode connector) that implemented
`nextWithTimeout(Long)` no longer compiles or links after upgrading from Spark
4.1 to 4.2. The `startTimeMs` parameter is only needed for the engine-internal
manual-clock path, so it should not be forced on external implementors.
### Does this PR introduce _any_ user-facing change?
Yes. It restores the Spark 4.1 `nextWithTimeout(Long)` method on the
`SupportsRealTimeRead` interface, so external sources written against Spark 4.1
compile and link again.
### How was this patch tested?
`StreamingRealTimeModeSourceCompatSuite`, which drives a public-API-only RTM
source end-to-end for both `nextWithTimeout` entry points. Verified it compiles
and passes on branch-4.2.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
