sam-1112 opened a new issue, #5812: URL: https://github.com/apache/datafusion-comet/issues/5812
### What is the problem the feature request solves? The initial `CometRegex` whitelist introduced by PR #5415 is deliberately conservative. It rejects commonly used constructs such as `.`, `^`, `$`, `\d`, `\w`, and `\s`. We currently do not know what fraction of representative real-world `rlike` patterns is admitted for automatic native execution. TPC-DS does not contain useful `rlike` coverage, so it cannot answer this question. Without an admission-rate measurement, it is difficult to prioritize follow-up compatibility work or estimate how much real-world workload coverage the whitelist provides. ### Describe the potential solution Identify one or more representative regex corpora, such as public SQL workloads, open-source application queries, or safely shareable production-derived patterns. Run every literal pattern through `CometRegex` and report: - the corpus source and collection method; - total, admitted, and rejected pattern counts; - the admission percentage; - duplicate-handling methodology; - rejection counts grouped by unsupported construct or analyzer rule; - useful pattern-length and complexity distributions. Use the results to prioritize follow-up work such as support for leading `^` or safe normalization of selected Java regex constructs. The whitelist should not be weakened solely to increase the reported admission percentage. ### Additional context Follow-up from #5415 and #5351. This tracks the representative admission-rate measurement requested during review of #5415: https://github.com/apache/datafusion-comet/pull/5415#pullrequestreview-5154502402 The original issue notes that the TPC query corpus does not contain `rlike` usage: https://github.com/apache/datafusion-comet/issues/5351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
