alamb commented on PR #8021: URL: https://github.com/apache/arrow-datafusion/pull/8021#issuecomment-1802591933
> We anticipate to have @metesynnada's design proposal, which aims to embody the discussions in Issue #7994, in a weeks time. The proposed design will provide an API for streaming sources that will decouple them from `ListingTable`. We propose to not merge this draft, iterate on the upcoming proposal to converge on a design as a community, and hold off on upgrading to arrow-rs 49.0.0 during this process. We do so as this upgrade removes the append functionality from object_store, which is critical to the current implementation of FIFO functionality in Datafusion. > > We aim to resolve the situation in approximately two weeks, with the following end state: (1) We will have merged the converged-on design decoupling ListingTable from streaming sources, (2) We will proceed with the upgrade to arrow-rs 49.0.0. > > In the meantime, we may make a serde-fix-backported 48.1.0 release of arrow-rs to get Datafusion benefit from this fix. I think this makes a lot of sense -- thank you @ozankabak . I will prepare a 48.0.1 patch release candidate -- see https://github.com/apache/arrow-rs/issues/5050#issuecomment-1802574764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
