kazantsev-maksim opened a new pull request, #5416: URL: https://github.com/apache/datafusion-comet/pull/5416
## Which issue does this PR close? - N/A ## Rationale for this change Optimize existing expression. ## What changes are included in this PR? This PR improves the performance and memory efficiency of spark_split and related string split expressions by optimizing buffer allocations, avoiding unnecessary regex compilation for literal delimiters, and reusing scratch buffers. 1. Fast-path for literal / single-char delimiters: - Detects if the pattern is a literal string (without regex metacharacters) and bypasses the regex::Regex DFA engine. - For single-character delimiters (e.g. ',', '|'), uses fast std::str::split(char) pattern matching instead of general substring or regex searches. 3. Eliminated per-row heap allocations on limit = 0: - Reused a single scratch: Vec<&str> buffer across rows using .clear() instead of collecting an allocated vector for every single string in the batch. 4. Pre-allocated Arrow builders with capacity: - Pre-allocated string value and offset buffers based on the input batch size and total byte length (value_data().len()), eliminating frequent realloc spikes. 5. Optimized scalar branches: - Removed intermediate Vec<String> allocations for scalar inputs, building Arrow buffers directly from borrowed string slices. ## How are these changes tested? Existing tests. | Benchmark | Baseline | Optimized | Throughput Diff | Time Diff | | :--- | :---: | :---: | :---: | :---: | | `literal_char_default_limit` (1024) | 96.3 µs | **68.5 µs** | **+41.2%** | **-29.2%** | | `literal_char_default_limit` (8192) | 754 µs | **649 µs** | **+15.3%** | **-13.3%** | | `literal_char_limit_0` (1024) | 106 µs | **39.8 µs** | **+165.5%** | **-62.3%** | | `literal_char_limit_0` (8192) | 840 µs | **319 µs** | **+169.3%** | **-62.9%** | | `regex_pattern` (8192) | 3.16 ms | **3.18 ms** | Within noise | +1.8% | Benchmark (criterion): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
