jaceklaskowski commented on code in PR #53346: URL: https://github.com/apache/spark/pull/53346#discussion_r2620794362
########## docs/declarative-pipelines-programming-guide.md: ########## @@ -409,12 +454,15 @@ SELECT * FROM STREAM(customers_us_east); ### Python Considerations - SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. Python functions that define datasets should include only the code required to define the table or view. -- The function used to define a dataset must return a Spark DataFrame. +- The function used to define a dataset must return a Spark `pyspark.sql.DataFrame`. - Never use methods that save or write to files or tables as part of your SDP dataset code. +- When using the `for` loop pattern to define datasets in Python, ensure that the list of values passed to the `for` loop is always additive. Review Comment: We do, in `Creating Tables in For Loop in Python` ealier ########## docs/declarative-pipelines-programming-guide.md: ########## @@ -409,12 +454,15 @@ SELECT * FROM STREAM(customers_us_east); ### Python Considerations - SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. Python functions that define datasets should include only the code required to define the table or view. -- The function used to define a dataset must return a Spark DataFrame. +- The function used to define a dataset must return a Spark `pyspark.sql.DataFrame`. - Never use methods that save or write to files or tables as part of your SDP dataset code. +- When using the `for` loop pattern to define datasets in Python, ensure that the list of values passed to the `for` loop is always additive. Review Comment: We do, in `Creating Tables in For Loop in Python` earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
