Eric Marnadi created SPARK-54867:
------------------------------------
Summary: Introduce NamedStreamingRelation wrapper for source
identification during analysis
Key: SPARK-54867
URL: https://issues.apache.org/jira/browse/SPARK-54867
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Eric Marnadi
Streaming sources need stable, predictable names for:
- Checkpoint location stability - Schema evolution and offset tracking require
consistent source identification
- Schema lookup at specific offsets - Analysis-time operations need to
reference sources by name
- Flow integration - DLT and similar systems need per-source metadata paths
- User control - Allow users to explicitly name sources via the `.name()` API
By introducing this wrapper during analysis (rather than at execution
planning), we enable these capabilities while maintaining a clean separation
between parsing, analysis, and execution phases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
