davidhewitt commented on issue #4658: URL: https://github.com/apache/datafusion/issues/4658#issuecomment-2678487900
@tustvold we're hitting this issue in present-day at Pydantic. It's possibly been made more relevant by the [recent fixes to spill heuristics](https://github.com/apache/datafusion/pull/14644). Is your preferred solution still to use row format? I assume by row format you mean [`arrow-row`](https://arrow.apache.org/rust/arrow_row/index.html), however it's not clear to me if there's a standard way to serialize these to a file. I could create something simple which just writes the streams of rows as binary (probably a length then the bytes, repeated?). We would benefit from this fairly urgently so I'm willing to put in the work to fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
