sundapeng opened a new pull request, #11349:
URL: https://github.com/apache/paimon/pull/11349
### Purpose
System tables (`t$files`, `t$buckets`, `t$fileindexes`, `t$filekeyranges`)
render the partition as a single string column and parse a pushed-down
Equal/In/range literal back into a partition spec with
`PartitionPredicateHelper.parsePartitionSpec`, which splits on ", " without a
limit. A partition value that itself contains ", " (e.g. the STRING partition
`hello, world`) splits into more fields than there are partition keys, the
parse returns null, and `applyPartitionFilter` reports no results: the query
silently returns zero rows for a partition that does have files. In the IN
branch the same mismatch silently drops the affected literals, so those
partitions vanish from the results. A trailing empty component (`{a, }` for two
keys) is also lost, because `String.split` without a limit discards trailing
empty strings.
This splits with `partitionKeys.size()` as the limit, so a value containing
", " survives when it is the last partition field, and a trailing empty
component round-trips. Inputs with at most keys-1 separators are parsed
identically to before, so no working case changes.
The format stays ambiguous with multiple keys and a separator inside a
non-last value; such filters can still misparse, but they already return an
empty result today. #10207 previously reported the same round-trip failure and
proposed degrading to a full scan; this fixes the parse itself so the filter
keeps being pushed down.
### Tests
- `PartitionPredicateHelperTest#testParseValueContainingCommaSpace`: single
key, value `hello, world` — returns null on master, the spec with fix.
-
`PartitionPredicateHelperTest#testParseMultiPartitionLastValueContainsComma`:
two keys, last value contains the separator — null on master, correct spec with
fix.
- `PartitionPredicateHelperTest#testParseTrailingEmptyComponent`: two keys,
trailing empty component `{a, }` — null on master, `pt1=a, pt2=""` with fix.
- `FilesTableTest#testReadWithPartitionValueContainingSeparator`: end-to-end
— a table with the STRING partition `hello, world`, `WHERE partition = '{hello,
world}'` returns its files with the fix and an empty list on master.
-
`FilesTableTest#testReadWithMultiKeyPartitionAndInFilterContainingSeparator`:
two partition keys whose last value contains the separator — Equal returns the
partition's files (empty on master), and an IN containing both the
separator-bearing and a plain literal returns both partitions (the
separator-bearing literal is silently dropped on master).
On master production code the parser tests and both end-to-end assertions
fail (`Expecting actual not to be null`, empty list `[]` instead of the
partition's files, and the IN result missing `{a, b, c}-0-0`); with the fix
`FilesTableTest` (14/14) and `PartitionPredicateHelperTest` (8/8) pass and
spotless:check is clean.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]