qzyu999 commented on PR #3738: URL: https://github.com/apache/iceberg-python/pull/3738#issuecomment-5184076762
> Given that #3737 plans several PyArrow-specific extractions, should we first convert `pyiceberg/io/pyarrow.py` into a package with a pure rename to `pyiceberg/io/pyarrow/__init__.py`, then extract concerns into submodules such as `pyarrow/fileio.py`? Hi @abnobdoss, that's a good idea, a `pyarrow/` package with submodules (`fileio.py`, `schema.py`, etc.) is cleaner than underscore-prefixed flat files. The `__init__.py` would handle re-exports and the import path from `pyiceberg.io.pyarrow import PyArrowFileIO` stays stable. > Is this really needed? I'm not sure I find this an improvement, since I'd now have to go back and forth between two highly coupled files. Hi @Fokko, I agree that on its own, this extraction doesn't add much, ` _pyarrow_fileio.py` and `pyarrow.py` are still coupled through the re-exports. The value comes from the end state: once all six concerns are extracted, the remaining `pyarrow.py` is just a re-export shim, and each concern (`FileIO`, `schema`, `scan`, `write`, etc.) becomes independently testable and substitutable. The immediate goal is enabling a `ComputeEngine` protocol at the scan/write boundary so DataFusion can slot in for bounded-memory operations (e.g., #271, #1210, #1092) without touching a 3K-line file. Would a single PR that converts pyarrow.py into a package (as @abnobdoss suggests) and splits all concerns at once be more palatable? This current direction is based on conversations (recently with @kevinjqliu, @rambleraptor, and others previously) during the past two Python sync calls. It started with #3554, then #3715 / #3716 which has been superseded by #3737 and this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
