sarutak commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6097134373
One small thing before merge: could you update the PR description to mention `spark.sql.execution.arrow.maxBytesPerBatch`? The Internals section still says > Writes convert rows to Arrow batches of at most > `spark.sql.execution.arrow.maxRecordsPerBatch` rows. Since 9ac5a49, batches are also bounded by `maxBytesPerBatch`, so a line like "...at most `maxRecordsPerBatch` rows or `maxBytesPerBatch` bytes, whichever is reached first" would match the code. The description already names `maxRecordsPerBatch` explicitly, so adding the byte limit keeps the two consistent. Not blocking, just so the description stays accurate for the record. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
