kevinjqliu commented on issue #953: URL: https://github.com/apache/iceberg-python/issues/953#issuecomment-2251045032
Here's something interesting. I think the `pyarrow_filter` [here](https://github.com/apache/iceberg-python/blob/9b6503def4f07a70149da07378f4795c3a4b113b/pyiceberg/io/pyarrow.py#L1185-L1188) is wrong. ``` (Pdb) bound_row_filter BoundEqualTo(term=BoundReference(field=NestedField(field_id=13, name='status', field_type=StringType(), required=False), accessor=Accessor(position=5,inner=Accessor(position=0,inner=None))), literal=literal('Employed')) (Pdb) translated_row_filter EqualTo(term=Reference(name='employment.status'), literal=literal('Employed')) (Pdb) bound_file_filter BoundEqualTo(term=BoundReference(field=NestedField(field_id=13, name='status', field_type=StringType(), required=False), accessor=Accessor(position=5,inner=Accessor(position=0,inner=None))), literal=literal('Employed')) (Pdb) pyarrow_filter <pyarrow.compute.Expression (status == "Employed")> ``` The `pyarrow_filter` expression dropped its parent reference. Setting `pyarrow_filter = None` allows pyarrow to scan successfully. ``` (Pdb) pyarrow_filter = None (Pdb) c age contact employment 0 28 {'email': '[email protected]'} {'status': 'Employed', 'position': 'Software E... 1 35 {'email': '[email protected]'} {'status': 'Self-employed', 'position': 'Consu... ``` ### Code to repro ``` from pyiceberg.catalog.sql import SqlCatalog import pyarrow as pa data = [{'id': 1, 'name': 'Alice', 'age': 28, 'address': {'street': '123 Maple St', 'city': 'Springfield', 'postal_code': '12345'}, 'contact': {'email': '[email protected]', 'phone': '555-1234'}, 'employment': {'status': 'Employed', 'position': 'Software Engineer', 'company': {'name': 'Tech Corp', 'location': 'Silicon Valley'}}, 'preferences': {'newsletter': True, 'notifications': {'email': True, 'sms': False}}}, {'id': 2, 'name': 'Bob', 'age': 35, 'address': {'street': '456 Oak St', 'city': 'Metropolis', 'postal_code': '67890'}, 'contact': {'email': '[email protected]', 'phone': '555-5678'}, 'employment': {'status': 'Self-employed', 'position': 'Consultant', 'company': {'name': 'Freelance', 'location': 'Remote'}}, 'preferences': {'newsletter': False, 'notifications': {'email': True, 'sms': True}}}] schema = pa.schema([ ('id', pa.int32()), ('name', pa.string()), ('age', pa.int32()), ('address', pa.struct([ ('street', pa.string()), ('city', pa.string()), ('postal_code', pa.string()) ])), ('contact', pa.struct([ ('email', pa.string()), ('phone', pa.string()) ])), ('employment', pa.struct([ pa.field('status', pa.string(), nullable=True), pa.field('position', pa.string(), nullable=True), pa.field('company', pa.struct([ ('name', pa.string()), ('location', pa.string()) ]), nullable=True) ])), ('preferences', pa.struct([ ('newsletter', pa.bool_()), ('notifications', pa.struct([ ('email', pa.bool_()), ('sms', pa.bool_()) ])) ])) ]) pyarrow_table = pa.Table.from_pylist(data, schema=schema) catalog = SqlCatalog("default", **{"uri": "sqlite:///:memory:", "warehouse": "."}) catalog.create_namespace("foo") table = catalog.create_table("foo.bar", schema=schema) table.overwrite(pyarrow_table) table.scan(row_filter="employment.status = 'Employed'", selected_fields=["age", "employment", 'contact.email', 'employment.status']).to_pandas() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
