ryanmeowy opened a new pull request, #13145:
URL: https://github.com/apache/seatunnel/pull/13145

   ### Purpose of this pull request
   
   Fixes #13144
   
   In the default ZETA SQL transform engine, a LIKE pattern whose `%` or `_` is 
not at the very
   beginning or the very end matches nothing, silently: rows are dropped with 
no error and no
   warning, and `NOT LIKE` with the same pattern keeps every row. Regex 
metacharacters in the
   pattern are interpreted as regex, and `LIKE NULL` throws 
`NullPointerException`.
   
   This is the out-of-the-box behaviour: `SQLTransform.java:68` defaults the 
engine to ZETA, and
   `ZetaSQLEngine.java:338` sends `where` predicates through `ZetaSQLFilter`.
   
   **Root cause.** `likeExpr` (`:191-232`) and `notLikeExpr` (`:235-273`) 
rewrite the pattern with
   four positional checks — `startsWith("%")`, `endsWith("%")`, 
`startsWith("_")`, `endsWith("_")`
   plus one `"_.*"` special case — and feed the result to `Pattern.compile`. 
Only the boundaries are
   considered, so an interior wildcard stays in the string and is compiled as a 
literal character;
   and nothing is quoted, so user characters become regex syntax. 
`rightVal.toString()` also runs
   before the null check, which made that check dead.
   
   **Fix.** Replace the positional rewriting with `likeMatches(pattern, 
value)`, a single iterative
   matcher shared by both methods: `%` consumes any sequence, `_` consumes 
exactly one code point,
   every other code point is compared literally. Both methods keep their 
existing dispatch and now
   check `rightVal` for null before using it.
   
   Two deliberate properties of this matcher:
   
   - *No regex escaping layer.* Matching characters directly means `'1.5'`, 
`'a(b'` and `'$100^'`
     are literal without any quoting, so the metacharacter subclass of the bug 
cannot come back.
   - *No catastrophic backtracking.* Translating `%` to `.*` — the obvious fix, 
and what this change
     did in its first revision — turns a pattern like `%a%a%a%a%a%a%a%a%a%a%b` 
into an exponential
     regex. Measured on a prototype of that approach: 11 wildcards against a 
40-character value took
     16 s, and the exact pattern used by the new unit test (10 wildcards, 
64-character value) was
     still running after 5 minutes. The iterative matcher keeps one backtrack 
point per `%` and runs
     in O(pattern × value): both cases finish in under 1 ms. A regex-based 
translation would have
     traded a silent wrong-answer bug for a hang, which is worse.
   
   Behaviour kept from before:
   
   - `Pattern.DOTALL` is still not applied, so `%` does not cross a newline 
(`"a\nb" LIKE 'a%b'` is
     false, as today). Aligning that with the SQL standard is a separate change.
   - LIKE ... ESCAPE remains unsupported, so a literal `%` or `_` still cannot 
be expressed. Also
     separate; there is a `shortcut:` comment at the site.
   - A pattern that legitimately starts and ends with a quote is no longer 
stripped. The previous
     `regex.startsWith("'") && regex.endsWith("'")` defence was dead code 
(`StringValue` is already
     unquoted by `getNotExcapedValue()`), and when it did fire it corrupted the 
pattern.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, within the unreleased `dev` branch — no released version is affected 
differently, since the
   bug is present in every version that has ZETA LIKE support.
   
   Previous behaviour versus behaviour after this PR, for a row `(1, "Alice", 
20)` unless stated
   otherwise. "SQL" is the expected SQL semantics.
   
   | predicate | before | after | SQL |
   |---|---|---|---|
   | `name LIKE 'Al%'` | true | true | true |
   | `name LIKE '%lic%'` | true | true | true |
   | `name LIKE '_li%'` | true | true | true |
   | `name LIKE 'A%ce'` | **false** | true | true |
   | `name LIKE 'A%ic%e'` | **false** | true | true |
   | `name LIKE 'A_i_e'` | **false** | true | true |
   | `name LIKE '%li%ce'` | **false** | true | true |
   | `name LIKE 'A%e'` | **false** | true | true |
   | `name LIKE '%%%%'` | **false** | true | true |
   | `name NOT LIKE 'A%ce'` | **true** | false | false |
   | `'125' LIKE '1.5'` | **true** | false | false |
   | `'axb' LIKE 'a.b'` | **true** | false | false |
   | `name LIKE '''quoted'''` on `'quoted'` | **false** | true | true |
   | `name LIKE 'a(b%'` | **PatternSyntaxException** | false | false |
   | `name LIKE NULL` | **NullPointerException** | false | false |
   
   Every change in that table moves the result toward the SQL truth value. In 
particular the three
   silent classes — interior wildcards, regex metacharacters, `LIKE NULL` — 
stop producing wrong
   answers or exceptions, and `NOT LIKE` stops inverting the result.
   
   The SQL transform documentation does not spell out LIKE wildcard semantics 
for the ZETA engine —
   there is no `LIKE` section in `docs/en/transforms/sql-functions.md` to 
contradict or to update, so
   no documentation change is included here. No configuration option, public 
API, data format or
   default value is touched, so `incompatible-changes.md` is not affected: the 
observable change is a
   bug fix, not a contract change. The two known gaps (`%` not crossing 
newlines, no
   `LIKE ... ESCAPE`) are listed above and left as separate changes.
   
   ### How was this patch tested?
   
   Unit tests, extending the existing `ZetaSQLFilterTest` (it already provides 
`createFilter()`, so no
   new test class). The two pre-existing LIKE tests only used `'Al%'` and 
`'_li%'` — the only shapes
   that worked before this PR. Six tests are added:
   
   - `testLikeWithWildcardInTheMiddle` — `A%ce`, `A%ic%e`, `A_i_e`, `%li%ce`, 
`A%e` match `Alice`
     and not `Bob`; `NOT LIKE 'A%ce'` inverts.
   - `testLikePatternsThatStillWork` — `Al%`, `%lic%`, `_li%` regression guard.
   - `testLikeTreatsRegexMetacharactersAndBrokenGroupsAsLiterals` — `'125' LIKE 
'1.5'` is false,
     `'a(b%'` returns false instead of throwing, and a value that starts and 
ends with a quote is not
     stripped.
   - `testLikeNullDoesNotThrow` — `LIKE NULL` and `NOT LIKE NULL` return false.
   - `testLikeMatchesOneCodePointPerUnderscore` — `_` matches one code point, 
so an astral character
     (emoji) counts as one and `a__b` does not match `a😀b`.
   - `testLikeWithManyWildcardsDoesNotBacktrackExponentially` — 10 interior 
wildcards against a
     64-character value complete within 5 s (`assertTimeoutPreemptively`). A 
real guard, not
     decoration: a regex-based implementation of the same pattern was still 
running after 5 minutes.
   
   If you want to check the behaviour by hand, the two-line version is:
   
   ```bash
   ./mvnw -pl seatunnel-transforms-v2 test -Dtest='ZetaSQLFilterTest'
   ```
   
   Commands run locally:
   
   ```bash
   # format (JDK 17)
   ./mvnw -pl seatunnel-transforms-v2 spotless:check
   # BUILD SUCCESS
   
   # the fix and its neighbouring tests
   ./mvnw -pl seatunnel-transforms-v2 test \
     -Dtest='ZetaSQLFilterTest,DateTimeFunctionsTest,SQLDateTimeFunctionsTest'
   # Tests run: 47, Failures: 0, Errors: 0
   #   ZetaSQLFilterTest           11  (5 before + 6 new)
   #   DateTimeFunctionsTest       21
   #   SQLDateTimeFunctionsTest    15
   ```
   
   On the parent commit the new behaviour tests fail with `3 failures + 1 NPE`: 
`A%ce` returns
   false, `'125' LIKE '1.5'` returns true, the emoji case returns false, and 
`LIKE NULL` throws —
   so these tests do detect the bug rather than passing vacuously.
   
   Full module (`seatunnel-transforms-v2`, 1901 tests) reports 2 failures / 16 
errors, all in
   `PythonTransformTest`, and the identical 2F/16E occur on an unmodified `dev` 
checkout on this
   machine: `python_executable=python3` resolves to `/usr/bin/python3`, which 
is not listed in the
   local `seatunnel.transform.python.allowed-executables` server property. 
Environment only,
   unrelated to this change.
   
   Beyond the unit tests, the fix was cross-checked against an independently 
written dynamic
   programming LIKE matcher on 61,609 random pattern/value pairs: zero 
mismatches (the only two
   divergences involve `_` against an astral character, where the 
code-point-aware implementation is
   the correct one).
   
   ### Check list
   
   - [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
         — not applicable, no new dependency or binary.
   - [ ] If necessary, please update the documentation to describe the new 
feature.
         — not applicable, no new option or behaviour surface; this restores 
documented-standard
         LIKE semantics rather than adding a feature. Note that the ZETA LIKE 
semantics are not
         currently documented at all (no `LIKE` section in 
`docs/en/transforms/sql-functions.md`),
         which could be a separate docs improvement if maintainers want it.
   - [ ] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility
         caused by this PR. — not applicable, no contract change (see above).
   - [ ] Connector-specific checks (`plugin-mapping.properties`, 
`seatunnel-dist` pom, label scope,
         e2e, `plugin_config`) — not applicable, this is not a connector change.
   
   Diff is limited to `ZetaSQLFilter.java` and `ZetaSQLFilterTest.java`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to