findepi commented on code in PR #12978:
URL: https://github.com/apache/datafusion/pull/12978#discussion_r1882188269
##########
datafusion/core/src/physical_optimizer/pruning.rs:
##########
@@ -3443,6 +3583,425 @@ mod tests {
);
}
+ #[test]
+ fn test_increment_utf8() {
+ // Basic ASCII
+ assert_eq!(increment_utf8("abc").unwrap(), "abd");
+ assert_eq!(increment_utf8("abz").unwrap(), "ab{");
+
+ // Test around ASCII 127 (DEL)
+ assert_eq!(increment_utf8("~").unwrap(), "\u{7f}"); // 126 -> 127
+ assert_eq!(increment_utf8("\u{7f}").unwrap(), "\u{80}"); // 127 -> 128
+
+ // Test 2-byte UTF-8 sequences
+ assert_eq!(increment_utf8("ß").unwrap(), "à"); // U+00DF -> U+00E0
+
+ // Test 3-byte UTF-8 sequences
+ assert_eq!(increment_utf8("℣").unwrap(), "ℤ"); // U+2123 -> U+2124
+
+ // Test at UTF-8 boundaries
+ assert_eq!(increment_utf8("\u{7FF}").unwrap(), "\u{800}"); // 2-byte
to 3-byte boundary
+ assert_eq!(increment_utf8("\u{FFFF}").unwrap(), "\u{10000}"); //
3-byte to 4-byte boundary
+
+ // Test that if we can't increment we return None
+ assert!(increment_utf8("").is_none());
+ assert!(increment_utf8("\u{10FFFF}").is_none()); // U+10FFFF is the
max code point
+
+ // Test that if we can't increment the last character we do the
previous one and truncate
+ assert_eq!(increment_utf8("a\u{10FFFF}").unwrap(), "b");
+
+ // Test surrogate pair range (0xD800..=0xDFFF)
+ assert_eq!(increment_utf8("a\u{D7FF}").unwrap(), "b");
+ assert!(increment_utf8("\u{D7FF}").is_none());
Review Comment:
thanks for the link.
@alamb i am not against reusing, but note that business logic for Parquet
stats and here is different.
For stats, it's enough to find _some_ value bigger than input.
Here, it's mandatory to find _next_ value bigger than input.
i have feeling that the routine in this PR is ready to be merged.
@adriangb can you please rebase the PR?
if a better version emerges, it's not a problem to improve later, but there
is no need to tie this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
