skyyyy955 opened a new issue, #11347:
URL: https://github.com/apache/paimon/issues/11347
### Search before asking
- [x] I searched the issues and pull requests for doubled quotes and the
PyPaimon WHERE parser.
### Paimon version
`master` at `ad4f6e09ef51c6b9932cbf22f0be563c6dde683f` (2.2-SNAPSHOT).
### Compute Engine
PyPaimon on Windows, using Python 3.10.11, PyArrow 19.0.1, the Python
backend, and a local Parquet table.
### Minimal reproduce step
Write three names to a Parquet table and run the same string filter through
the public read builder and the CLI.
```python
import json
from pathlib import Path
import subprocess
import sys
from tempfile import TemporaryDirectory
import pyarrow as pa
from pypaimon import CatalogFactory, Schema
with TemporaryDirectory() as directory:
root = Path(directory)
options = {"warehouse": (root / "warehouse").as_uri()}
catalog = CatalogFactory.create(options)
catalog.create_database("audit", False)
rows = pa.table({"id": [1, 2, 3], "name": ["O'Brien", "O", "Brien"]})
settings = {
"file.format": "parquet", "bucket": "-1", "write-only": "true",
"scan.native-plan.enabled": "false", "read.native.enabled": "false",
"write.native.enabled": "false", "commit.native.enabled": "false",
}
catalog.create_table("audit.names",
Schema.from_pyarrow_schema(rows.schema, options=settings), False)
table = catalog.get_table("audit.names")
batch = table.new_batch_write_builder()
writer, commit = batch.new_write(), batch.new_commit()
try:
writer.write_arrow(rows)
commit.commit(writer.prepare_commit())
finally:
writer.close()
commit.close()
where = "name IN ('O''Brien')"
def ids(predicate):
builder = table.new_read_builder().with_filter(predicate)
data =
builder.new_read().to_arrow(builder.new_scan().plan().splits())
return sorted(data["id"].to_pylist())
pb = table.new_read_builder().new_predicate_builder()
print("String filter:", ids(where))
print("PredicateBuilder:", ids(pb.is_in("name", ["O'Brien"])))
config = root / "paimon.yaml"
config.write_text(json.dumps(options), encoding="utf-8")
result = subprocess.run(
[sys.executable, "-m", "pypaimon.cli.cli", "-c", str(config),
"table", "read",
"audit.names", "-s", "id,name", "--format", "json", "--where",
where],
check=True, capture_output=True, text=True,
)
print("CLI:", sorted(row["id"] for row in json.loads(result.stdout)))
```
### What doesn't meet your expectations?
The doubled single quote in `name IN ('O''Brien')` represents one
apostrophe, so the filter should match only `O'Brien` (ID 1). Both
string-filter entry points complete successfully and return the rows for `O`
and `Brien` instead.
```text
String filter: [2, 3]
PredicateBuilder: [1]
CLI: [2, 3]
```
### Anything else?
The tokenizer splits `'O''Brien'` into two quoted tokens. The IN-list parser
accepts both tokens as separate values even though there is no comma between
them.
[Tokenizer](https://github.com/apache/paimon/blob/ad4f6e09ef51c6b9932cbf22f0be563c6dde683f/paimon-python/pypaimon/common/where_parser.py#L214-L232)
ยท [IN-list
parser](https://github.com/apache/paimon/blob/ad4f6e09ef51c6b9932cbf22f0be563c6dde683f/paimon-python/pypaimon/common/where_parser.py#L419-L435)
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR.
Could you assign this to me? I'd like to parse the doubled quote as part of
a single literal and add a regression that checks the rows returned by the
string filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]