garyanaplan commented on a change in pull request #443:
URL: https://github.com/apache/arrow-rs/pull/443#discussion_r650750468
##########
File path: parquet/src/data_type.rs
##########
@@ -661,8 +661,15 @@ pub(crate) mod private {
_: &mut W,
bit_writer: &mut BitWriter,
) -> Result<()> {
+ if bit_writer.bytes_written() + values.len() >=
bit_writer.capacity() {
+ bit_writer.extend(256);
+ }
for value in values {
- bit_writer.put_value(*value as u64, 1);
+ if !bit_writer.put_value(*value as u64, 1) {
Review comment:
I found it hard to think of a good way to test this with the fix in
place.
I preferred the "don't auto expand memory at the point of failure" approach
because I'm fairly conservative and didn't want to make a change that was too
wide in impact without a better understanding of the code. i.e.: my fix
specifically targeted the error I reported and made it possible to detect in
other locations.
I think a better fix would be to (somehow) pre-size the vector or avoid
having to size a vector for all the bytes that could be written, but that would
be a much bigger scope to the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
