JiaqiWang18 commented on code in PR #51507:
URL: https://github.com/apache/spark/pull/51507#discussion_r2213860432
##########
python/pyspark/pipelines/cli.py:
##########
@@ -217,8 +217,30 @@ def change_dir(path: Path) -> Generator[None, None, None]:
os.chdir(prev)
-def run(spec_path: Path) -> None:
- """Run the pipeline defined with the given spec."""
+def run(
+ spec_path: Path,
+ full_refresh: Optional[Sequence[str]] = None,
+ full_refresh_all: bool = False,
+ refresh: Optional[Sequence[str]] = None,
+) -> None:
+ """Run the pipeline defined with the given spec.
+
+ :param spec_path: Path to the pipeline specification file.
+ :param full_refresh: List of datasets to reset and recompute.
+ :param full_refresh_all: Perform a full graph reset and recompute.
+ :param refresh: List of datasets to update.
+ """
+ # Validate conflicting arguments
+ if full_refresh_all:
+ if full_refresh:
+ raise PySparkException(
+ errorClass="CONFLICTING_PIPELINE_REFRESH_OPTIONS",
messageParameters={}
Review Comment:
Added
[logic](https://github.com/apache/spark/pull/51507/commits/1693ac546225c8a6be1d96eb5e64fcf03f77a344#diff-273b1e23d9cb13f05f8167735c9571b549972f8b5270ad2aba3bc4a92e8b9766R179)
to pass along the conflicting option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
