AdityRajGupta commented on PR #54059: URL: https://github.com/apache/arrow/pull/54059#issuecomment-6503069979
> With the latest commit we are losing some performance so I'd like to ask you to go back few steps and find a different path. @HuaHuaY could we get a second opinion on the direction? I went back and looked at where the cost actually came from. It was the byte-at-a-time lexer running on every byte, unconditionally. But MakeChunker is only quote-aware when quoting and newlines_in_values are both on, and that's the only case where this bug can happen.So I gated the quote tracking on exactly that condition. I've pushed this update to the PR. The full CSV suite passes on my Mac, 298/298, including your quoted-CR case in all three reader modes. Happy to hear your thoughts, and HuaHuaY's too if they get a chance to look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
