Ramin Gharib created FLINK-40218:
------------------------------------
Summary: Allow building a Variant from a token source, without a
String round-trip
Key: FLINK-40218
URL: https://issues.apache.org/jira/browse/FLINK-40218
Project: Flink
Issue Type: Improvement
Components: API / Type Serialization System
Reporter: Ramin Gharib
Assignee: Ramin Gharib
*Description*
The only way to build a {{Variant}} from JSON today is from a {{{}String{}}}.
{{PARSE_JSON}} and the internal
{{BinaryVariantInternalBuilder.parseJson(String)}} both take a string, create a
Jackson {{JsonParser}} over it, and walk the tokens in {{{}buildJson{}}}.
A consumer that already holds parsed JSON cannot use that work. It must
serialize its data back to a {{String}} and hand it to Flink, which parses it a
second time. This shows up wherever a format or connector wants a VARIANT
column: the format has already parsed the record into a tree or a parser of its
own, yet it is forced to {{toString()}} and let flink-core re-parse.
Two things drive this:
# *A redundant pass.* The value is parsed by the caller, serialized to a
{{{}String{}}}, then parsed again by flink-core.
# *The Jackson type is not portable.* {{buildJson}} is written against Flink's
shaded {{{}org.apache.flink.shaded.jackson2...JsonParser{}}}. A caller that
uses a differently relocated or unshaded Jackson cannot pass its own parser in.
A {{String}} is the only type that crosses that boundary, which is why the
round-trip exists.
Concretely, for one VARIANT value read through a JSON-based format:
{code:java}
Current:
bytes ──(caller parses)──▶ tree ──(toString)──▶ String ──(flink re-parses)──▶
tokens ──▶ Variant
[needed] [wasted] [wasted] {code}
*Proposed change*
Introduce a small token-source abstraction in
{{org.apache.flink.types.variant}} and drive {{buildJson}} off it. The concrete
Jackson parser becomes one implementation of that abstraction rather than a
hard dependency.
{code:java}
// flink-core: org.apache.flink.types.variant
@Internal
public interface VariantJsonSource { /
** Advance to the next token and return it. Returns END_INPUT once input is
exhausted. */
Token next() throws IOException;
/** Field name of the current FIELD_NAME token. */
String fieldName() throws IOException;
/** Text of the current STRING token. */
String stringValue() throws IOException; /**
* Raw literal of the current NUMBER token, e.g. "1e5", "100000", "3.14".
* The builder decides long vs decimal vs double from this text, so numeric
* classification stays in one place and every source behaves identically.
*/
String numberText() throws IOException;
enum Token {
START_OBJECT, END_OBJECT,
START_ARRAY, END_ARRAY,
FIELD_NAME,
STRING, NUMBER, TRUE, FALSE, NULL,
END_INPUT
}
} {code}
{code:java}
// BinaryVariantInternalBuilderpublic static BinaryVariant
parseJson(VariantJsonSource source, boolean allowDuplicateKeys) throws
IOException {
// buildJson, expressed purely against VariantJsonSource
}
{code}
With the interface a JSON-based format wraps the value it already parsed and
builds the Variant in a single walk:
{code:java}
Proposed:
bytes ──(caller parses)──▶ tree ──(wrap as source)──▶ tokens ──▶ Variant
[needed] [no serialize] [no re-parse] {code}
*Design notes*
* Number classification stays in the builder. The source exposes the raw
number literal through {{{}numberText(){}}}, so the long-vs-decimal-vs-double
decision, including the rule that scientific notation goes to {{{}double{}}},
lives in one place. Sources do not re-implement it and cannot diverge.
* Boolean values are carried by the {{TRUE}} / {{FALSE}} tokens, so no
separate value accessor is needed.
* The interface uses only JDK types plus its own enum. It names no Jackson
type, shaded or otherwise, so any caller can implement it regardless of how its
Jackson is relocated.
* The exact signature is open. {{numberText}} versus typed number accessors is
the main choice, and the text form is preferred because it is the only shape
that preserves the current numeric routing without leaking policy into each
source. This can be finalized in the PR.
*Compatibility*
Additive and {{{}@Internal{}}}. {{parseJson(String)}} stays and is
reimplemented on top of the new method by adapting the existing Jackson parser
to {{{}VariantJsonSource{}}}. {{PARSE_JSON}} and {{TRY_PARSE_JSON}} behavior is
unchanged.
*Verifying this change*
* Existing {{PARSE_JSON}} / {{TRY_PARSE_JSON}} and
{{BinaryVariantInternalBuilder}} tests pass unchanged, proving the String path
still behaves identically.
* Add a test that builds a {{Variant}} from a non-Jackson
{{VariantJsonSource}} implementation and asserts it is byte-for-byte equal to
the Variant produced by {{parseJson(String)}} for the same document, across
objects, arrays, and all scalar and numeric forms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
