fallintoplace opened a new pull request, #1517:
URL: https://github.com/apache/iceberg-go/pull/1517
## Summary
* honor `write.parquet.row-group-size-bytes` when writing buffered Parquet
row groups
* start a new row group before the next non-empty batch after the byte
target is reached
* avoid creating an empty trailing row group when the file is closed
* reject invalid and non-positive target values, matching Iceberg Java
behavior
* preserve `write.parquet.row-group-limit` as an independent row-count
ceiling
## Why
`write.parquet.row-group-size-bytes` is exported and documented, but the Go
Parquet writer did not read it. Changing the property therefore had no effect
on generated row groups.
Arrow Go v18.7 provides `RowGroupTotalBytesWritten` and
`NewBufferedRowGroupChecked`, allowing the writer to apply the configured
uncompressed byte target safely.
Rotation happens before the next non-empty batch instead of immediately
after reaching the target. This prevents closing the writer from producing an
empty trailing row group.
## Testing
* added coverage for the default and configured byte targets
* added coverage for malformed, zero, and negative values
* added a Parquet metadata regression test verifying that:
* a small target creates multiple row groups
* each row group contains data
* no empty trailing row group is produced
* `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
