miguelpragier opened a new issue, #40089: URL: https://github.com/apache/arrow/issues/40089
### Describe the bug, including details regarding any error messages, version, and platform. ### Describe the bug, including details regarding any error messages, version, and platform. We work with large volumes of data, and we have observed that the results of queries (db.Query()) are downloaded at once into an array in memory, which in these cases always results in containers being OOM-Killed. We have developed a patch, where we introduced an asynchronous/concurrent approach, using goroutines, which reads only the necessary blocks, as calls to Rows.Next() are made. All the relevant routines are in file https://github.com/apache/arrow/blob/main/go/arrow/flight/flightsql/driver/driver.go. ### Component(s) Go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
