eyalsatori commented on issue #2036:
URL:
https://github.com/apache/datafusion-sqlparser-rs/issues/2036#issuecomment-3597146931
Progress Update:
I've opened PR #2112 to add the foundational infrastructure for zero-copy
tokenization.
The PR introduces a lifetime parameter to the token enum (BorrowedToken<'a>)
while maintaining full backward compatibility through a type alias `Token =
BorrowedToken<'static>`. It also adds a tokenized_owned() method for consumers
who need owned tokens, implements visitor pattern support for Cow<'a, str>.
The tokenizer infrastructure changed to support lifetimes, though all string
allocations remain in place for now.
The next phase will focus on replacing String fields with Cow<'a, str> in
token variants and updating tokenizer methods to use borrowed strings where
possible, leveraging the borrowing functions added in previous commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
