jaceklaskowski commented on code in PR #53671: URL: https://github.com/apache/spark/pull/53671#discussion_r2678791654
########## docs/declarative-pipelines-programming-guide.md: ########## @@ -453,22 +476,24 @@ SELECT * FROM STREAM(customers_us_east); ### Python Considerations -- SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. Python functions that define datasets should include only the code required to define the table or view. -- The function used to define a dataset must return a `pyspark.sql.DataFrame`. -- Never use methods that save or write to files or tables as part of your SDP dataset code. -- When using the `for` loop pattern to define datasets in Python, ensure that the list of values passed to the `for` loop is always additive. +* SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. + Python functions that define datasets should include only the code required to define the table or view. +* The function used to define a dataset must return a `pyspark.sql.DataFrame`. +* Never use methods that save or write to files or tables as part of your SDP dataset code. +* When using the `for` loop pattern to define datasets in Python, + ensure that the list of values passed to the `for` loop is always additive. Examples of Spark SQL operations that should never be used in SDP code: -- `collect()` -- `count()` -- `pivot()` -- `toPandas()` -- `save()` -- `saveAsTable()` -- `start()` -- `toTable()` +* `collect()` Review Comment: Not sure, but asterisks seemed recommended (over dashes) by `rumdl`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
