skyyyy955 opened a new issue, #11305:
URL: https://github.com/apache/paimon/issues/11305

   ### Search before asking
   
   - [x] I searched for existing reports. #11219 concerns the JVM time zone. 
The result below changes within a single Python process when an always-true 
condition is added.
   
   ### Paimon version
   
   `master` at `35e7aa8be8ad106650dab53fe44232535f017a90` (2.2-SNAPSHOT).
   
   ### Compute Engine
   
   PyPaimon, Python backend, Windows, Python 3.10.11, PyArrow 19.0.1, pandas 
2.3.3. The table uses a local filesystem catalog and Parquet.
   
   ### Minimal reproduce step
   
   Write one row to get a single entry in `db.t$files`. Take the entry's actual 
`creation_time` and set the query bound one nanosecond later. The two output 
columns count matches for the same predicate, first on its own and then with 
`AND contains(file_format, '')`. The file format is `parquet`, so that extra 
condition is true for this row.
   
   ```python
   from tempfile import TemporaryDirectory
   
   import pandas as pd
   import pyarrow as pa
   from pypaimon import CatalogFactory, Schema
   
   
   def read(table, predicate=None):
       builder = table.new_read_builder()
       if predicate is not None:
           builder.with_filter(predicate)
       return builder.new_read().to_arrow(builder.new_scan().plan().splits())
   
   
   with TemporaryDirectory() as warehouse:
       catalog = CatalogFactory.create({"warehouse": warehouse})
       catalog.create_database("db", False)
       row = pa.table({"id": [1]})
       options = {"file.format": "parquet", "bucket": "-1", "write-only": 
"true",
                  "scan.native-plan.enabled": "false", "read.native.enabled": 
"false",
                  "write.native.enabled": "false", "commit.native.enabled": 
"false"}
       catalog.create_table("db.t", Schema.from_pyarrow_schema(row.schema, 
options=options), False)
       batch = catalog.get_table("db.t").new_batch_write_builder()
       writer, commit = batch.new_write(), batch.new_commit()
       try:
           writer.write_arrow(row)
           commit.commit(writer.prepare_commit())
       finally:
           writer.close()
           commit.close()
   
       files = catalog.get_table("db.t$files")
       data = read(files)
       assert data.num_rows == 1 and data["file_format"][0].as_py() == "parquet"
       t = pd.Timestamp(data["creation_time"][0].as_py())
       bound = t + pd.Timedelta(1, unit="ns")
       pb = files.new_read_builder().new_predicate_builder()
       always_true = pb.contains("file_format", "")
       for predicate in (pb.less_than("creation_time", bound),
                         pb.equal("creation_time", bound),
                         pb.is_in("creation_time", [bound])):
           print(read(files, predicate).num_rows,
                 read(files, pb.and_predicates([predicate, 
always_true])).num_rows)
   ```
   
   ### What doesn't meet your expectations?
   
   | Predicate | Expected rows | Predicate alone | With the always-true 
condition |
   | --- | ---: | ---: | ---: |
   | `creation_time < bound` | 1 | 0 | 1 |
   | `creation_time = bound` | 0 | 1 | 0 |
   | `creation_time IN (bound)` | 0 | 1 | 0 |
   
   With the bound set to exactly `t` or to `t + 1 µs`, both forms agree. I also 
wrote the same millisecond timestamp to an ordinary Parquet table. There, the 
three predicates at `t + 1 ns` returned the expected counts of 1, 0 and 0 on 
their own.
   
   ### Anything else?
   
   The system table reader calls `predicate.to_arrow()` without the column 
schema. That skips the timestamp comparison logic that handles sub-microsecond 
literals. Adding a string condition selects the row-wise evaluation path, which 
explains the different result. [System table 
filter](https://github.com/apache/paimon/blob/35e7aa8be8ad106650dab53fe44232535f017a90/paimon-python/pypaimon/table/system/system_table_read.py#L123-L146)
 · [Timestamp 
comparison](https://github.com/apache/paimon/blob/35e7aa8be8ad106650dab53fe44232535f017a90/paimon-python/pypaimon/common/predicate.py#L227-L241)
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR.
   
   Could you assign this to me? I'd like to pass the full Arrow schema when 
building the system table filter and add regression tests for range, equality 
and membership predicates, including a filter on a column omitted from the 
projection.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to