kz930 opened a new pull request, #8990: URL: https://github.com/apache/texera/pull/8990
### What changes were proposed in this PR? The exported Python for Regular Expression and Substring Search disagreed with the engine on non-ASCII text. Regular Expression: Java's Pattern ignores case only for ASCII letters, and its \w, \d and \s match only ASCII. Python's re is Unicode by default. The export now passes `flags=re.ASCII`. Java 17's \b still treats any letter or digit as part of a word, so each \b and \B outside a character class is rewritten as `(?u:\b)` and `(?u:\B)`. Substring Search: without Case Sensitive, the engine compares `toLowerCase` of both sides, and pandas' `case=False` compares `upper()`, which folds "ß" to "SS" and "ς" to "Σ". The export now lowers the substring with the same Java call and the column with Python's `lower()`. ### Any related issues, documentation, discussions? Closes #8989 ### How was this PR tested? Each operator's spec gets a test that runs a set of patterns or substrings over non-ASCII values through the engine's executor and through the export, and compares the kept rows. The Substring Search test fails on main. A unit test covers the \b rewrite. The verify runs for both operators pass. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
