kevinjqliu commented on issue #2372: URL: https://github.com/apache/iceberg-python/issues/2372#issuecomment-3217072185
i dug into this a bit more. the issue is not with pyiceberg's internal uuid representation, its with pushing down the pyarrow filter to pyarrow's `ds.Scanner` https://github.com/apache/iceberg-python/blob/19efd2d1d5f1802f33c3db77eaf426323b28df97/pyiceberg/io/pyarrow.py#L1518-L1525 breakpointing here shows ``` (Pdb) pyarrow_filter <pyarrow.compute.Expression (uuid == "00000000000000000000000000000000")> ``` Heres the script to repro ``` from pyiceberg.catalog import load_catalog import uuid import pyarrow as pa from pyiceberg.expressions import EqualTo catalog = load_catalog("default", type="in-memory") catalog.create_namespace_if_not_exists("default") TEST_DATA_WITH_NULL = { "uuid": [ uuid.UUID("00000000-0000-0000-0000-000000000000").bytes, None, uuid.UUID("11111111-1111-1111-1111-111111111111").bytes, uuid.UUID("22222222-2222-2222-2222-222222222222").bytes, ] } # schema = pa.schema([("uuid", pa.binary(16))]) schema = pa.schema([("uuid", pa.uuid())]) # Create table arrow_table = pa.table(TEST_DATA_WITH_NULL, schema=schema) table = catalog.create_table("default.test", schema=schema) table.append(arrow_table) # print(table.schema()) test_cases = [ uuid.UUID("00000000-0000-0000-0000-000000000000").bytes, uuid.UUID("00000000-0000-0000-0000-000000000000"), # "00000000-0000-0000-0000-000000000000", ] for test_case in test_cases: result = table.scan(row_filter=EqualTo("uuid", test_case)).to_arrow() assert len(result) == 1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
