Jacky Wang created SPARK-52714:
----------------------------------
Summary: [SDP] Remove unused "comment" arg on @append_flow
decorator
Key: SPARK-52714
URL: https://issues.apache.org/jira/browse/SPARK-52714
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.1.0
Reporter: Jacky Wang
In Spark Declarative Pipelines (SDP), users can define append flows in Python
using the
[{{append_flow}}|https://github.com/apache/spark/blob/e3321aa44ea255365222c491657b709ef41dc460/python/pyspark/pipelines/api.py#L34-L41]
decorator. The {{append_flow}} decorator currently accepts a comment arg,
which were copied over when porting over DLT’s equivalent decorator.
SDP however does not actually have a UI that shows comments, so this field
should be unused today. As a result we should remove it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
