nssalian commented on PR #3381: URL: https://github.com/apache/iceberg-python/pull/3381#issuecomment-4589335476
Thanks for taking a look @Kurtiscwright. PyIceberg is Arrow-only today. `FileFormatWriter.write(table: pa.Table)` takes Arrow, so Spark would convert to Arrow first. I had proposed a broader engine abstraction and Fokko covered on that thread (#3219) that PyIceberg's role is metadata and planning, not execution. Engines wanting their own format (Polars, Daft, DataFusion) skip our read/write and use `plan_files()`. The Rust RFC's three layers fit iceberg-rust because it's a kernel. PyIceberg sits at a different layer. Long-term path for non-Arrow is delegating to iceberg-rust (#2396). Out of scope here. My next focus would be #20 following the same model here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
