raghav-reglobe opened a new pull request, #69222:
URL: https://github.com/apache/doris/pull/69222
### What problem does this PR solve?
Issue Number: close #69220
Related PR: none
Problem Summary:
This PR implements the proposal in #69220 as a suggestion for how the V2
path can keep an exact number on a declared DECIMAL template path. The
alternative discussed there, an ondemand-based collector that keeps the raw
token, is open: if the maintainers prefer that direction I will rework this PR
to it.
With a Schema Template that declares a DECIMAL path, a JSON number on that
path was parsed to a binary64 before the
template was applied, so digits beyond what a double carries were lost
(`0.123456789012345678` stored as
`0.123456789012345680`; the regression suite pins `188118222.011121933` →
`188118222.011121920`), while the same value
as a JSON string was exact. The parser is schema-blind by design; this
change keeps it that way and carries the
template's DECIMAL paths to the parse site:
- `TScalarType.variant_decimal_paths` (optional): the DECIMAL /
ARRAY<DECIMAL> predefined fields of a VARIANT type,
filled by `VariantType.toThrift`. An old BE ignores the field; a BE that
receives none behaves as before.
- `DataTypeVariantV2` carries the paths as an ingestion hint (never part of
the type's identity) and hands them to its
serde; `parse_to_variant` / `try_parse_to_variant` read them from their
result type.
- Before parsing, one pass over the text quotes every number token with a
fraction and no exponent that sits on a
declared DECIMAL path. The unchanged parser then takes the string route it
already has and the shredder casts the
text to the template's DECIMAL exactly. Integers, exponent forms and
undeclared paths are untouched (inferred types
do not move); overflow still yields NULL; malformed text falls through to
the parser's own verdict; rows without such
a token are parsed from the original buffer.
- `INSERT … SELECT`: the sink casts the slot that carries the parse call;
the projects merge in the rewrite phase, where
`SimplifyCastRule` rewrites `CAST(parse_to_variant(x) AS
variant<template>)` into the call typed with the template.
Stream Load already types the call with the column's VARIANT type.
### Release note
A JSON number on a DECIMAL path of a VARIANT Schema Template is now stored
exactly (previously it went through a double
first; only the string form was exact).
### Check List (For Author)
- Test
- [x] Unit Test — BE: `VariantDecimalPathsTest` (pre-pass + path set),
`DataTypeVariantV2SerdeInputTest`,
`FunctionVariantParseTest`; FE: `VariantTypeDecimalPathsThriftTest`,
`SimplifyCastRuleTest`.
- [x] Regression test — the pinned lossy rendering `188118222.011121920`
in `variant_p0/predefine` and
`variant_p0/v2/predefine` (load.out + sql/q02.out, 42 lines) becomes
the exact `188118222.011121933` (updated in
this PR).
- [x] Manual test — Stream Load JSON and CSV, and `INSERT … SELECT
parse_to_variant(text)` in plain, UNION ALL, CTE
and LIMIT-subquery shapes, on a two-FE / seven-BE cluster: number form
exact on declared DECIMAL paths, string form
unchanged, undeclared paths still DOUBLE.
- Behavior changed:
- [x] Yes. The number form on a declared DECIMAL template path keeps all
its digits; the FAQ paragraph that
recommends the string form becomes advice rather than a requirement.
- Does this need documentation?
- [x] Yes. The FAQ entry on DECIMAL template paths (doris-website) —
follow-up PR once this merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
