Eric Marnadi created SPARK-61311:
------------------------------------
Summary: Enable streaming source evolution for queries that name a
source
Key: SPARK-61311
URL: https://issues.apache.org/jira/browse/SPARK-61311
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Eric Marnadi
DataStreamReader.name() and SQL IDENTIFIED BY are public, but using them fails
with STREAMING_QUERY_EVOLUTION_ERROR.SOURCE_NAMING_NOT_SUPPORTED unless the
internal config spark.sql.streaming.queryEvolution.enableSourceEvolution is set.
Naming a source should enable source evolution for the query it belongs to. A
new query with a named source starts on offset log v2 and records the mode in
its offset metadata, a restart keeps the recorded mode, and other queries in
the session are unaffected. Every source of such a query must be named. The
config stays and forces source evolution for every streaming query in the
session.
Error messages that tell users to set the config to name sources are reworded.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
