[
https://issues.apache.org/jira/browse/SPARK-61309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61309:
-----------------------------------
Labels: pull-request-available (was: )
> Check streaming source names against the checkpoint before creating sources
> ---------------------------------------------------------------------------
>
> Key: SPARK-61309
> URL: https://issues.apache.org/jira/browse/SPARK-61309
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Eric Marnadi
> Priority: Major
> Labels: pull-request-available
>
> MicroBatchExecution should check the streaming source names in the analyzed
> plan against the source tracking recorded in the checkpoint before it creates
> any source, so that a rejected run leaves nothing behind in the checkpoint.
> - A checkpoint that tracks sources by name, restarted with no named source,
> fails with a new error,
> STREAMING_QUERY_EVOLUTION_ERROR.CANNOT_DISABLE_ON_EXISTING_CHECKPOINT. Today
> this hits an assert after the sources are created.
> - A positional checkpoint (offset log v1 or v2) restarted with named sources
> fails with CANNOT_ENABLE_ON_EXISTING_CHECKPOINT. Today v1 fails only after
> the sources are created, and v2 runs with positional offsets while reading
> source metadata from an empty sources/<name> directory. A checkpoint that
> already keeps every named source's metadata in sources/<name> keeps running
> by position: Spark 4.2 wrote checkpoints that way for catalog tables named
> with IDENTIFIED BY or readStream.name().table().
> - A query with both named and unnamed sources fails with
> UNNAMED_STREAMING_SOURCES_WITH_ENFORCEMENT, including a union of DataFrames
> analyzed separately, which has no NamedStreamingRelation left for the
> analyzer to check.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
