sryza commented on code in PR #53671: URL: https://github.com/apache/spark/pull/53671#discussion_r2677460744
########## docs/declarative-pipelines-programming-guide.md: ########## @@ -24,22 +24,29 @@ license: | ## What is Spark Declarative Pipelines (SDP)? -Spark Declarative Pipelines (SDP) is a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark. SDP simplifies ETL development by allowing you to focus on the transformations you want to apply to your data, rather than the mechanics of pipeline execution. +<!-- rumdl-disable MD013 --> Review Comment: Do we follow this convention in other md files? Given that the target for this content is the docs site, and the docs site wraps text, I think in general it would be better to use a ruleset that allows docs authors to write paragraphs than to try to fit inside arbitrary line length boundaries. ########## docs/declarative-pipelines-programming-guide.md: ########## @@ -453,22 +476,24 @@ SELECT * FROM STREAM(customers_us_east); ### Python Considerations -- SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. Python functions that define datasets should include only the code required to define the table or view. -- The function used to define a dataset must return a `pyspark.sql.DataFrame`. -- Never use methods that save or write to files or tables as part of your SDP dataset code. -- When using the `for` loop pattern to define datasets in Python, ensure that the list of values passed to the `for` loop is always additive. +* SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. + Python functions that define datasets should include only the code required to define the table or view. +* The function used to define a dataset must return a `pyspark.sql.DataFrame`. +* Never use methods that save or write to files or tables as part of your SDP dataset code. +* When using the `for` loop pattern to define datasets in Python, + ensure that the list of values passed to the `for` loop is always additive. Examples of Spark SQL operations that should never be used in SDP code: -- `collect()` -- `count()` -- `pivot()` -- `toPandas()` -- `save()` -- `saveAsTable()` -- `start()` -- `toTable()` +* `collect()` Review Comment: Are asterixes the convention that's used most widely in other docs pages? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
