wgtmac commented on PR #48468: URL: https://github.com/apache/arrow/pull/48468#issuecomment-5067810580
Thanks for bringing this up, @mbasmanova. Some historical context: The unresolved issue was how to define and estimate the byte limit. The encoded/compressed size is only known after data has been written. Using compressed pages alone underestimates the size; including buffered values, dictionaries, and levels mixes compressed and uncompressed measurements and may significantly overestimate it. The main implementation difficulty is providing consistent behavior across the buffered WriteRecordBatch path and the column-oriented WriteTable path, especially for the first row group where no historical compression ratio exists. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
