HyukjinKwon commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6097224794
@sarutak Thanks, updated. The Internals section now says writes convert rows to Arrow batches "that end at `spark.sql.execution.arrow.maxRecordsPerBatch` rows or once they hold `spark.sql.execution.arrow.maxBytesPerBatch` bytes, whichever is reached first, as for Python UDFs". I worded it this way because the byte limit is checked after each row, so a batch can go slightly over it, as in `PythonArrowInput`. While there, I went through the rest of the description against 9ac5a49 and updated the other parts that changed during the review: - the per-micro-batch stream writers; - the precedence of usable packages over invalid or unsupported ones; - the checksum-named copies of configured packages; - the single lookup per resolution and the Python data source precedence on registration; - the changes to existing classes (`DataFrameWriter`/`DataStreamWriter` under `withActive`, `ArtifactManager`, `DataSourceRegistration`, PySpark `addArtifacts`, the Connect compatibility exclusion); - the user-facing fix for writes from a non-active session; - the tests added during the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
