ogrman commented on issue #5150: URL: https://github.com/apache/arrow-rs/issues/5150#issuecomment-1833963642
> > we will no longer get a complete batch which means that sometimes we will start writing a new batch while rep_levels is still 1. > > I'm confused by this, read_records was added and read_batch deprecated precisely because read_batch didn't guarantee this, whereas read_records does You shouldn't be, I'm aware that I'm most likely the confused one here. We've been using the above code and it's been working fine for years. When we upgraded the parquet dep this code started crashing due with the naive implementation and I'm aware that the answer to this question might be to rewrite it. In this case we know that the input is well-formed, and we expected there to be something akin to `write_records` for the write case. If we had a function that just let us write more records that would also be fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
