VVKot commented on issue #2864: URL: https://github.com/apache/iceberg-rust/issues/2864#issuecomment-5083588815
@dannycjones seems that pyiceberg-core was published NOT from the v0.10.0 tag (latest commit b882e63), but rather from a 633b8e0 commit main - https://github.com/apache/iceberg-rust/actions/runs/29818317581. I assume this is not expected? Would it be worth to have the workflow reject non-tag refs? Short-term, should we cut v0.10.1 rectifying this? There were quite a few changes between the two commits https://github.com/apache/iceberg-rust/compare/b882e63..633b8e0, most importantly DataFusion 53 -> 54 bump. A quick bash to confirm: ```bash $ curl -sL "$(python3 -c "import json,urllib.request as u;print([f['url'] for f in json.load(u.urlopen('https://pypi.org/pypi/pyiceberg-core/0.10.0/json'))['urls'] if f['filename'].endswith('.tar.gz')][0])")" \ | tar xzO pyiceberg_core-0.10.0/Cargo.toml | grep -E '^datafusion' datafusion = "54.0.0" datafusion-cli = "54.0.0" datafusion-ffi = "54.0.0" datafusion-sqllogictest = "54.0.0" ``` I have confirmed that crates were definitely published from the right tag: ``` $ curl -sL https://static.crates.io/crates/iceberg-datafusion/iceberg-datafusion-0.10.0.crate \ | tar xzO iceberg-datafusion-0.10.0/.cargo_vcs_info.json { "git": { "sha1": "b882e63652f4d4a172812994679a1dbc0c64cbd0" }, "path_in_vcs": "crates/integrations/datafusion" } ``` ``` $ curl -sS -H 'User-Agent: version-check' \ https://crates.io/api/v1/crates/iceberg-datafusion/0.10.0/dependencies \ | jq -r '.dependencies[] | select(.crate_id=="datafusion") | .req' ^53.1.0 ``` Importantly, this is not just confusing - it out right fails if trying to use pyiceberg-core with DataFusion 53: ``` $ cat > /tmp/repro.py <<'PY' import tempfile import pyarrow as pa import datafusion, importlib.metadata as md from datafusion import SessionContext from pyiceberg.catalog.sql import SqlCatalog from pyiceberg_core.datafusion import IcebergDataFusionTable print(f"datafusion={datafusion.__version__} pyiceberg-core={md.version('pyiceberg-core')}", flush=True) wh = tempfile.mkdtemp() cat = SqlCatalog("d", uri=f"sqlite:///{wh}/c.db", warehouse=f"file://{wh}") cat.create_namespace("default") data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]}) tbl = cat.create_table("default.test", schema=data.schema) tbl.append(data) provider = IcebergDataFusionTable( identifier=tbl.name(), metadata_location=tbl.metadata_location, file_io_properties=tbl.io.properties, ) ctx = SessionContext() print("registering...", flush=True) ctx.register_table("test", provider) # <-- segfaults on a mismatched pair print("registered OK", flush=True) print("QUERY OK:", ctx.table("test").to_arrow_table().to_pylist(), flush=True) PY for DF in 53.0.0 54.0.0; do python3 -m venv "/tmp/v$DF" # pyarrow is installed directly rather than via pyiceberg[pyarrow]: the released # pyiceberg 0.11.x pyarrow extra pins pyiceberg-core<0.9.0, which cannot co-resolve # with pyiceberg-core==0.10.0. "/tmp/v$DF/bin/pip" -q install "pyiceberg-core==0.10.0" "datafusion==$DF" \ "pyiceberg[sql-sqlite]>=0.11" "pyarrow>=18" "/tmp/v$DF/bin/python" /tmp/repro.py; echo " datafusion $DF -> exit=$?" done datafusion=53.0.0 pyiceberg-core=0.10.0 registering... Segmentation fault (core dumped) datafusion 53.0.0 -> exit=139 datafusion=54.0.0 pyiceberg-core=0.10.0 registering... registered OK QUERY OK: [{'x': 1, 'y': 4}, {'x': 2, 'y': 5}, {'x': 3, 'y': 6}] datafusion 54.0.0 -> exit=0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
