Dandandan opened a new issue, #577: URL: https://github.com/apache/arrow-ballista/issues/577
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Currently we are writing to the (deprecated) Arrow format, but when sending the data to workers, we encode it to `RecordBatch` and re-encode data in the IPC format. The benefit of writing to the IPC format directly, is that we can stream the data from disk and don't have to de-encode / re-encode the data. We'll also have to compress the data once (and benefit from reduced file-sizes too). **Describe the solution you'd like** Use the Arrow IPC stream format rather than the (old) file format. **Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. **Additional context** Add any other context or screenshots about the feature request here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
