zeroshade commented on issue #195: URL: https://github.com/apache/arrow-go/issues/195#issuecomment-2492471706
There are a couple ways we can adjust to do this that I can think of, depending on your use case: 1. We could add an option to the `ArrowReaderProperties` to explicitly always use LargeString/LargeBinary for strings (eliminating the need for the workaround). This requires a user to know ahead of time that they need to use LargeString/LargeBinary, which may not be feasible or the best route. 2. Are you using ReadTable? Or are you streaming the records? We could check the size of the column data ahead of time and force it to split records based on the column size, that would avoid this problem. This would require a bit of extra up-front work to do the checking, but allows seamless record reading without the user needing to know ahead of time. Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
