This is an automated email from the ASF dual-hosted git repository.
sandy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9e17cbafbce8 [SPARK-52714][SDP] Remove unused comment arg in
append_flow decorator
9e17cbafbce8 is described below
commit 9e17cbafbce848dffd05267f25189328de739e57
Author: Jacky Wang <[email protected]>
AuthorDate: Tue Jul 8 16:20:21 2025 -0700
[SPARK-52714][SDP] Remove unused comment arg in append_flow decorator
### What changes were proposed in this pull request?
In Spark Declarative Pipelines (SDP), users can define append flows in
Python using the
[append_flow](https://github.com/apache/spark/blob/e3321aa44ea255365222c491657b709ef41dc460/python/pyspark/pipelines/api.py#L34-L41)
decorator. The append_flow decorator currently accepts a `comment` arg.
However, there is no way for user to see flow comments as of now. Therefore,
this argument is unused and not referenced in function body.
```py
def append_flow(
*,
target: str,
name: Optional[str] = None,
comment: Optional[str] = None, # <--- Removing
spark_conf: Optional[Dict[str, str]] = None,
once: bool = False,
) -> Callable[[QueryFunction], None]:
```
This PR removes the field.
### Why are the changes needed?
The `comment` arg is not being used anywhere and having it in the API will
confuse the user thinking they can see flow comments somewhere.
### Does this PR introduce _any_ user-facing change?
Yes, the previously optional `comment` arg is removed from the
`append_flow` API. However, SDP has not been released yet (pending release in
v4.1), so no user should be impacted by this change.
### How was this patch tested?
Examined all testcases to make sure none of the current append_flow usage
is supplying this argument
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #51404 from JiaqiWang18/SPARK-52714-remove-append_flow-comment.
Authored-by: Jacky Wang <[email protected]>
Signed-off-by: Sandy Ryza <[email protected]>
---
python/pyspark/pipelines/api.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/python/pyspark/pipelines/api.py b/python/pyspark/pipelines/api.py
index 57de64e97035..bcc72a4a04e1 100644
--- a/python/pyspark/pipelines/api.py
+++ b/python/pyspark/pipelines/api.py
@@ -35,7 +35,6 @@ def append_flow(
*,
target: str,
name: Optional[str] = None,
- comment: Optional[str] = None,
spark_conf: Optional[Dict[str, str]] = None,
once: bool = False,
) -> Callable[[QueryFunction], None]:
@@ -44,7 +43,6 @@ def append_flow(
:param name: The name of the flow. If unspecified, the query function's
name will be used.
:param target: The name of the dataset this flow writes to. Must be
specified.
- :param comment: Description of the flow. If unspecified, the dataset's
comment will be used.
:param spark_conf: A dict whose keys are the conf names and values are the
conf values. \
These confs will be set when the flow is executed; they can override
confs set for the \
destination, for the pipeline, or on the cluster.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
