frbvianna commented on issue #37976: URL: https://github.com/apache/arrow/issues/37976#issuecomment-1755540115
Thanks @zeroshade for your detailed explanation. Since the alternative of chunking based on the IPC encoded bytes like I suggested is a bit more grey area to me, mostly concerning its performance, I am convinced that the row size estimation proposed in this issue might be a good place to start. I am a bit confused though around the null bitmaps and variable data type width estimation. Would it not make sense for the Arrow library to provide that sort of calculation? I find myself reaching deep into some internal Arrow concepts to understand how to calculate the total row size from the Record column buffers. @polestar1988, would you mind sharing your complete calculation code, please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
