LucaCappelletti94 commented on code in PR #2428:
URL:
https://github.com/apache/datafusion-sqlparser-rs/pull/2428#discussion_r4318316897
##########
src/parser/mod.rs:
##########
@@ -12548,40 +12548,50 @@ impl<'a> Parser<'a> {
Ok(s.chars().next().unwrap())
}
- /// Parse a tab separated values in
- /// COPY payload
- pub fn parse_tsv(&mut self) -> Vec<Option<String>> {
- self.parse_tab_value()
- }
-
- /// Parse a single tab-separated value row used by `COPY` payload parsing.
- pub fn parse_tab_value(&mut self) -> Vec<Option<String>> {
- let mut values = vec![];
- let mut content = String::new();
- while let Some(t) = self.next_token_no_skip().map(|t| &t.token) {
- match t {
- Token::Whitespace(Whitespace::Tab) => {
- values.push(Some(core::mem::take(&mut content)));
- }
- Token::Whitespace(Whitespace::Newline) => {
- values.push(Some(core::mem::take(&mut content)));
- }
+ /// Parse the inline payload of `COPY ... FROM STDIN` as written.
+ ///
+ /// The payload is data rather than SQL, so it is captured uninterpreted.
Review Comment:
Rendering tokens could never be exact, and data such as `O'Brien` did not
even tokenize.
The tokenizer now cuts the payload out of the source before tokenizing, so
it comes back byte for byte, including `0x1F`, `E'\x41'`, CRLF and `'a''b'`.
Objectively, the tokenizer needs a complete overhaul, but I will tackle that
after most of the fuzzer bugs are squashed or the new tokenizer will be a mess
to fuzz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
