Yuheng Chang created SPARK-53995:
------------------------------------
Summary: Support to track `spark_conf` at the dataset level
Key: SPARK-53995
URL: https://issues.apache.org/jira/browse/SPARK-53995
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.1
Reporter: Yuheng Chang
We need to add {{sql_confs}} to {{DefineOutput}} in SDP to support following
use caseSQL Notebook #1
{code:java}
SET some_conf = some_value;
CREATE STREAMING TABLE st; {code}
SQL Notebook #2
APPLY CHANGES INTO st FROM ....
Since we don't store {{sql_confs}} at the Dataset level, the conf would be
ignored. We should store it at dataset level and will merge the {{sql_conf}}
stored at the dataset level to its corresponding flow when we construct the
DataflowGraph.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
