camuel commented on issue #1797: URL: https://github.com/apache/iceberg-rust/issues/1797#issuecomment-3459626314
> [@camuel](https://github.com/camuel) I think that's a bit beyond the scope of what DataFusion as a project wants to pick up and would fragment the ecosystem even more. But I do get your point that Iceberg can be complex, etc. Something that is likely much lower lift and would be a fun project would be to build a DuckLake + DataFusion system where there's a `DuckLakeSchemaCatalog` type thing so that DuckLake / DuckDB can control the schema catalog and table scans but DataFusion does the rest. Thanks @adriangb for the elaboration. That would be a fun project for sure and I think the utility could be in making DuckLake extensible in rust something not possible with Ducklake. May be it isn't too much effort just reimplement Ducklake in rust, not necessarily as part of DataFusion project but using DataFusion in the way Ducklake uses DuckDB, with a roundtrip metadata only interop, it is a full iceberg implementation, not a canonical one but still a fully compatible one. The bulk of the scan planning in DuckLake is done by SQL statements, not by DuckDB C++ code anyway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
