Jackie-Jiang commented on PR #8878: URL: https://github.com/apache/pinot/pull/8878#issuecomment-1152592493
@richardstartin Thanks for the detailed comment. I have some questions regarding to the comments: > I’m not sure this is valid for UTF-8 anyway unless the text is normalized, because UTF-8 doesn’t guarantee the uniqueness of representation of characters. For example: both "\u00e9" and "\u0065\u0301" represent 'é'. Is the UTF-8 normalized if it is always encoded from a String object? > The size of this improvement depends on the data: how long the values are and on common prefixes within the set of strings. > > The best case for this optimisation relative to the baseline is uniformly random long strings (so the loop terminates on the first byte 255 times in 256, and the cost of materialising the string is exacerbated), but natural language text is never uniformly distributed and some sequences of bytes are very common. > > When there is more regularity in the data (imagine the strings are English book titles and a good percentage of them start with “The “ but others start with “Their” or other common words prefixed by “The” and the average length is ~16 bytes) you might get a different relative outcome. Comparing byte by byte would be even worse with URLs, which have very regular and long common prefixes. It would be better to read the unpadded bytes into the buffer and perform a vectorized comparison with Arrays.mismatch, this will likely regress on randomly generated data but will be much faster otherwise. Please compare with the existing baseline for URLs, book titles, people’s names written in English from a range of cultures, etc. too. Very good point. I think we can do batch read to reduce the byte access but also not reading every bytes for the long strings. What do you think if we read long values from the buffer, and perform unsigned long comparison? Similar to how we read unpadded strings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
