mvzink opened a new pull request, #2729: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2729
Some callers of the tokenizer's `peeking_take_while` helper simply copied the newly-allocated `String` into an existing `String`. Refactor it to call a new `peeking_read_while` helper, and convert 4 `take` callers to `read`. This avoids an allocation and extraneous string copy at each callsite. This was motivated by finding one of the converted callers (`tokenize_word`) pretty high up when profiling some real workloads. Converting that caller gets most of the improvement in tokenizing time in the added benchmarks, but converting the rest seemed harmless. Add two benchmarks (results from a few runs on a Ryzen 5 3600): * `mixed_sql`: 22 TPC-H files plus eight short statements. ~9-11% improvement to tokenizing time and ~5% improvement to overall parse time. * `tokenize_identifiers` benchmark targeting `tokenize_word`, testing power-of-two identifier lengths from 1 to 128. Pretty contrived, mainly to check this doesn't regress small identifiers. 1-character identifiers showed ~4% reduction in tokenizing time, up to ~14% reduction for 128-character identifiers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
