qzyu999 commented on issue #3554: URL: https://github.com/apache/iceberg-python/issues/3554#issuecomment-4861618906
> Thank you for putting this together... Hi @abnobdoss, thanks for your insight, there are definitely some issues/PRs that I hadn't considered and will analyze further. Not sure if you were present, but we were able to discuss this topic at length at the previous 6/30 [PyIceberg sync](https://docs.google.com/document/d/1oMKodaZJrOJjPfc8PDVAoTdl02eGQKHlhwuggiw7s9U/edit?tab=t.0#heading=h.odmna4i5n63u) (I would encourage you to attend if you have time). After discussion and input from various community members the direction to take was to set the boundary at plan scanning to remain within PyIceberg while other ops like read/write/compute can migrate towards a pluggable interface. I'm working on it now and will create an issue and an initial PR (starting with just one engine, DataFusion) to explore the potential of a pluggable backend architecture which would help with issues like the 3k+ line `/pyiceberg/io/pyarrow.py` monolith. I'll tag this issue so all community members watching this will can get looped in. Will definite ly appreciate your help with review and follow-up PRs after that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
