Jungtaek Lim created SPARK-39564:
------------------------------------
Summary: Expose the information of catalog table to the logical
plan in streaming query
Key: SPARK-39564
URL: https://issues.apache.org/jira/browse/SPARK-39564
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.4.0
Reporter: Jungtaek Lim
In Spark 3.1.0, we added ability to read and write from/to catalog table for
streaming query. (Read:
[SPARK-32885|https://issues.apache.org/jira/browse/SPARK-32885] / Write:
[SPARK-32896|https://issues.apache.org/jira/browse/SPARK-32896])
When the streaming query is being executed with such API, the table identifier
for source and sink is "conditionally" exposed to the logical plan, depending
on the DSv1 vs DSv2 and source vs sink.
Since this is the only way to get the information of table, we would like to
make some changes on relevant logical nodes to carry out and expose the
information of table, specifically, table identifier.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
