arthurpassos commented on issue #32723: URL: https://github.com/apache/arrow/issues/32723#issuecomment-1564693573
Hi @emkornfield. I am the OP of this issue and I came across this again, so I think it's time work on this. I have re-implemented your suggestion to use LARGE* variants of String and Binary types in order to avoid chunking, see https://github.com/arthurpassos/arrow/pull/1/files. Ofc this is just a hard-coded version to validate it fixes the issue, apparently it did. Based on that, I have a couple of questions: 1. Is that still your recommendation to address this issue? 2. You have mentioned this should be backed by a setting, can you point to any examples on how to create a setting? 3. Is memory limit the only thing that would lead to chunking? I wonder if there are other reasons. In that case, this fix wouldn't completely solve the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
