Jefffrey commented on PR #4661: URL: https://github.com/apache/arrow-datafusion/pull/4661#issuecomment-1363824177
> Hi @Jefffrey -- thank you for this contribution > > Is there any way to test this code ? Perhaps with a mocked out object store and verify that only the first file is fetched? > > Also, I can't help but wonder if we could implement the logic using future::Stream more efficiently , but I haven't studied the logic in detail to really know for sure Hi @alamb, I've pushed a test which uses a mocked ObjectStore which returns an infinite stream of bytes, with the test proving that by finishing it won't try to consume the entire stream first before inferring (which was the previous behaviour) Could you elaborate on the future::Stream part? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
