raghav-reglobe commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4546125865
##########
gensrc/thrift/Types.thrift:
##########
@@ -136,6 +136,15 @@ enum TInvertedIndexFileStorageFormat {
SNII = 4 // SNII native inverted index storage format
}
+// One field of a VARIANT Schema Template, in declaration order, for the BE's
JSON parser: a number on a
+// path whose first matching field is DECIMAL (or ARRAY<DECIMAL>) is parsed
from its text instead of
+// through a binary64. The whole template rides so the BE matches a path the
way the shredder does.
+struct TVariantTemplatePath {
+ 1: required string pattern
+ 2: required bool is_glob
+ 3: required bool is_decimal
Review Comment:
Done. `TVariantTemplatePath.type` is now the field's `TPrimitiveType` (an
ARRAY field sends its item type) and `is_decimal` is gone. The BE decides how a
number on a path is read from that type, matching the first field in
declaration order as before:
- DECIMAL32/64/128I/256: the number is read from its token text into an
exact decimal, as in the previous revision (this push fixes that reader's
handling of a long exponent; see the other thread).
- LARGEINT: an integer simdjson cannot hold in 64 bits is now read from its
text into an int128. Before this, such a document left the typed parser and
failed. Up to 10^38−1 it is stored as a scale-0 decimal; above that it is
stored as its digit text. Either way the cast to LARGEINT is exact, and a value
outside the LARGEINT range ends up NULL through the cast. In a regular
(non-doc-mode) VARIANT a number now ends up the same as its JSON-string form:
same subcolumn value, same document text, which prints a LARGEINT longer than
38 digits quoted for the string form as well. In doc mode the document column
keeps the value as parsed, as it does for every typed path.
- Any other type: the native read, as before.
The template is sent when it declares a DECIMAL or LARGEINT field, or an
ARRAY of one; otherwise nothing is sent, as before. With a DECIMAL field every
document is read with the ondemand parser, because the DOM parser turns a
fraction into a double without an error and leaves nothing to fall back on.
With LARGEINT fields only, the DOM parser reads the document first, as without
a template, and every document it rejects is read again from its text; a
document holding an integer beyond 64 bits is therefore parsed twice.
I sent the primitive type rather than a full `TTypeDesc`. The struct sits
inside `TScalarType`, so a `TTypeDesc` there would make the definition
recursive. Precision and scale are applied by the cast to the declared type, so
the parser only needs the kind.
The struct is new in this PR, so changing field 3 from bool to an enum
affects no release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
