Eric Marnadi created SPARK-61309:
------------------------------------
Summary: Check streaming source names against the checkpoint
before creating sources
Key: SPARK-61309
URL: https://issues.apache.org/jira/browse/SPARK-61309
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Eric Marnadi
MicroBatchExecution should check the streaming source names in the analyzed
plan against the source tracking recorded in the checkpoint before it creates
any source, so that a rejected run leaves nothing behind in the checkpoint.
- A checkpoint that tracks sources by name, restarted with no named source,
fails with a new error,
STREAMING_QUERY_EVOLUTION_ERROR.CANNOT_DISABLE_ON_EXISTING_CHECKPOINT. Today
this hits an assert after the sources are created.
- A positional checkpoint (offset log v1 or v2) restarted with named sources
fails with CANNOT_ENABLE_ON_EXISTING_CHECKPOINT. Today v1 fails only after the
sources are created, and v2 runs with positional offsets while reading source
metadata from an empty sources/<name> directory. A checkpoint that already
keeps every named source's metadata in sources/<name> keeps running by
position: Spark 4.2 wrote checkpoints that way for catalog tables named with
IDENTIFIED BY or readStream.name().table().
- A query with both named and unnamed sources fails with
UNNAMED_STREAMING_SOURCES_WITH_ENFORCEMENT, including a union of DataFrames
analyzed separately, which has no NamedStreamingRelation left for the analyzer
to check.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
