u70b3 commented on code in PR #4971:
URL: https://github.com/apache/datafusion-comet/pull/4971#discussion_r4055793241
##########
native/spark-expr/src/string_funcs/get_json_object.rs:
##########
@@ -246,68 +284,177 @@ fn parse_json_path(path: &str) -> Option<ParsedPath> {
}
}
- Some(ParsedPath {
- segments,
- has_wildcard,
- })
+ Some(ParsedPath { segments })
+}
+
+/// Jackson (and therefore Spark) rejects number tokens longer than 1000
+/// characters wherever they appear in the document — including values this
+/// evaluation skips — so `get_json_object` returns null. serde_json enforces
no
+/// such limit when skipping, so mirror it with a byte scan before parsing.
+fn has_oversized_number(json: &str) -> bool {
+ const MAX_NUMBER_LEN: usize = 1000;
+ let bytes = json.as_bytes();
+ let mut in_string = false;
+ let mut escaped = false;
+ let mut i = 0;
+ while i < bytes.len() {
+ let b = bytes[i];
+ if in_string {
+ if escaped {
+ escaped = false;
+ } else {
+ match b {
+ b'\\' => escaped = true,
+ b'"' => in_string = false,
+ _ => {}
+ }
+ }
+ i += 1;
+ continue;
+ }
+ if b == b'"' {
+ in_string = true;
+ } else if b.is_ascii_digit() || b == b'-' {
+ // A number token: digits, sign, decimal point and exponent marker
+ // all count towards Jackson's limit.
+ let start = i;
+ i += 1;
+ while i < bytes.len()
+ && matches!(bytes[i], b'0'..=b'9' | b'.' | b'e' | b'E' | b'+'
| b'-')
+ {
+ i += 1;
+ }
+ if i - start > MAX_NUMBER_LEN {
+ return true;
Review Comment:
Fixed in 38429ffb — the scan now counts digits exactly the way jackson-core
2.21.2 does, per `ParserBase.resetInt`/`resetFloat`: the sign and decimal point
do not count; integers are limited by their digit count; floats by the sum of
the integer-part, fraction and exponent digit counts.
Probing the pinned jackson-core directly (Spark 4.1.3's
`fasterxml.jackson.version` is 2.21.2) pinned down one more corner: a lone
leading-zero integer part counts as zero digits, except when both a fraction
and an exponent are present, where it counts as one — `0.5e` followed by 999
exponent digits is rejected with "Number value length (1001) exceeds the
maximum allowed (1000)", while `0.` + 1000 fraction digits and `0e` + 1000
exponent digits are accepted. The scan mirrors all of this and it is covered in
`get_json_object.sql` (signed, fractional and exponent boundary columns) and in
unit tests.
Validation: a 69-case battery of these boundary shapes (plus string-content
and escaped-quote controls) run through Spark 4.1.3's evaluator matches on
every case that is not already a pre-existing `$`-materialization divergence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
