skyyyy955 opened a new issue, #11347:
URL: https://github.com/apache/paimon/issues/11347

   ### Search before asking
   
   - [x] I searched the issues and pull requests for doubled quotes and the 
PyPaimon WHERE parser.
   
   ### Paimon version
   
   `master` at `ad4f6e09ef51c6b9932cbf22f0be563c6dde683f` (2.2-SNAPSHOT).
   
   ### Compute Engine
   
   PyPaimon on Windows, using Python 3.10.11, PyArrow 19.0.1, the Python 
backend, and a local Parquet table.
   
   ### Minimal reproduce step
   
   Write three names to a Parquet table and run the same string filter through 
the public read builder and the CLI.
   
   ```python
   import json
   from pathlib import Path
   import subprocess
   import sys
   from tempfile import TemporaryDirectory
   
   import pyarrow as pa
   from pypaimon import CatalogFactory, Schema
   
   with TemporaryDirectory() as directory:
       root = Path(directory)
       options = {"warehouse": (root / "warehouse").as_uri()}
       catalog = CatalogFactory.create(options)
       catalog.create_database("audit", False)
       rows = pa.table({"id": [1, 2, 3], "name": ["O'Brien", "O", "Brien"]})
       settings = {
           "file.format": "parquet", "bucket": "-1", "write-only": "true",
           "scan.native-plan.enabled": "false", "read.native.enabled": "false",
           "write.native.enabled": "false", "commit.native.enabled": "false",
       }
       catalog.create_table("audit.names", 
Schema.from_pyarrow_schema(rows.schema, options=settings), False)
       table = catalog.get_table("audit.names")
       batch = table.new_batch_write_builder()
       writer, commit = batch.new_write(), batch.new_commit()
       try:
           writer.write_arrow(rows)
           commit.commit(writer.prepare_commit())
       finally:
           writer.close()
           commit.close()
   
       where = "name IN ('O''Brien')"
   
       def ids(predicate):
           builder = table.new_read_builder().with_filter(predicate)
           data = 
builder.new_read().to_arrow(builder.new_scan().plan().splits())
           return sorted(data["id"].to_pylist())
   
       pb = table.new_read_builder().new_predicate_builder()
       print("String filter:", ids(where))
       print("PredicateBuilder:", ids(pb.is_in("name", ["O'Brien"])))
   
       config = root / "paimon.yaml"
       config.write_text(json.dumps(options), encoding="utf-8")
       result = subprocess.run(
           [sys.executable, "-m", "pypaimon.cli.cli", "-c", str(config), 
"table", "read",
            "audit.names", "-s", "id,name", "--format", "json", "--where", 
where],
           check=True, capture_output=True, text=True,
       )
       print("CLI:", sorted(row["id"] for row in json.loads(result.stdout)))
   ```
   
   ### What doesn't meet your expectations?
   
   The doubled single quote in `name IN ('O''Brien')` represents one 
apostrophe, so the filter should match only `O'Brien` (ID 1). Both 
string-filter entry points complete successfully and return the rows for `O` 
and `Brien` instead.
   
   ```text
   String filter: [2, 3]
   PredicateBuilder: [1]
   CLI: [2, 3]
   ```
   
   ### Anything else?
   
   The tokenizer splits `'O''Brien'` into two quoted tokens. The IN-list parser 
accepts both tokens as separate values even though there is no comma between 
them. 
[Tokenizer](https://github.com/apache/paimon/blob/ad4f6e09ef51c6b9932cbf22f0be563c6dde683f/paimon-python/pypaimon/common/where_parser.py#L214-L232)
 ยท [IN-list 
parser](https://github.com/apache/paimon/blob/ad4f6e09ef51c6b9932cbf22f0be563c6dde683f/paimon-python/pypaimon/common/where_parser.py#L419-L435)
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR.
   
   Could you assign this to me? I'd like to parse the doubled quote as part of 
a single literal and add a regression that checks the rows returned by the 
string filter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to