The idea would be to make ARRAY the default for all elements, so the
structure stays the same no matter how many elements are actually there.
The downside is that for simple XMLs, where most elements only ever appear
once, you'd need to add [1] everywhere just to access them.
Hi Moritz,
Nice FLIP. The # field is a neat solution, and I checked on master that
Variant does sort object keys, so it is doing necessary work.
One small question: what does PARSE_XML do with an xmlns declaration in
the input? The parser is not namespace-aware, so xmlns:xsi is an
attribute, and by the @name rule it would land as "@xmlns:xsi". If
XML_STRING writes that back and also adds its own xmlns:xsi declaration
for xsi:nil, the root would carry that attribute twice, which is not
well-formed, so that Variant would not round trip.
<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
<a xsi:nil="true"/>
</root>
Dropping xmlns at parse time would instead leave other prefixes
undeclared. Maybe I am misreading the mapping, but a line either way
might help.
Best,
Sai Krishna Sepuri
On Thu, Sep 17, 2026 at 7:51 PM Moritz Manner <[email protected]>
wrote:
Hi Fabian,
Good point, and I agree this is a real problem. A consistent VARIANT
structure across documents with the same schema is important, especially
for streaming SQL where you don't want CASE WHEN everywhere, or a job
failing because it tries to access an array that doesn't exist.
Initially,
the structure optimized for queryability, i.e. a single element stays a
single node instead of an array. But I think consistency should probably
win here.
The idea would be to make ARRAY the default for all elements, so the
structure stays the same no matter how many elements are actually there.
The downside is that for simple XMLs, where most elements only ever
appear
once, you'd need to add [1] everywhere just to access them.
One option would be to start with ARRAY by default and see how it goes
based on feedback. If it turns out to be too cumbersome for simple XMLs,
we
could add an opt-out parameter later, that would be a backward-compatible
addition so it doesn't need to be solved now.
Does this make sense, or is there a better way to approach this?
Best,
Moritz
On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <
[email protected]>
wrote:
Hi everyone and thanks for the proposal Moritz,
The FLIP reads very well and I agree (mostly) with the design.
I have only one concern, which is about the representation of repeated
elements as arrays.
I think it is very important that a batch of XML documents with the
same
structure converts into an identical VARIANT structure.
If I understand the current proposal correctly, this would not be the
case
for two documents: one with a single element and one with a repeated
element.
The first document would have a single node, while the second document
would have an array.
If you want to access the nodes, you would need a cumbersome CASE WHEN
construct to handle both cases.
It would be better to convert both documents into a VARIANT containing
an
ARRAY (one only holding a single element).
Since deriving the right conversion strategy from a single XML document
isn't possible, could we add an optional argument to pass conversion
hints?
There might be other solutions to this problem.
IMO, it would also be fine to continue with this FLIP and address this
issue later, but having a proposal for it would be good.
Do you have any thoughts on this, Moritz?
Best, Fabian
On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <[email protected]>
wrote:
Hi Moritz,
this is an excellent design document with the right level of detail.
I
really like the design, it is a nice mixture of concerns. The result
of
PARSE_XML looks and feels like a VARIANT while the XML content is
preserved in a lossless fashion and all attributes and text is
accessible. It is a nice combination of what other vendors offer
(modulo
the historical legacy they have to deal with).
Looking at various UDF implementations, XML handling is by far the
most
important reason for the need of a custom function. So +1 for this
FLIP.
Thanks,
Timo
On 11.09.26 14:54, Moritz Manner wrote:
Hey everyone,
I'd like to start a discussion on FLIP-612: Native XML Functions
for
Flink
SQL [1].
The goal is native XML parsing in Flink SQL, parsing XML into a
VARIANT
instead of requiring custom UDFs.
The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML
turn an
XML
string into a VARIANT, and XML_STRING turns a VARIANT back into an
XML
string.
This mirrors the existing JSON functions PARSE_JSON,
TRY_PARSE_JSON,
and
JSON_STRING.
Once XML is a VARIANT, you query it the same way as the output of
PARSE_JSON:
with the variant accessors (variant.key, variant['key'],
variant[index])
and a CAST to the type you need.
The main part is the XML→VARIANT mapping: attributes and text go
into
@/$
fields, and since Variant objects store fields sorted by key rather
than
in
document order, a # field is used to recover the original order
between
differently-named siblings. Details and examples are in the FLIP.
It's a small, additive API, no new SQL grammar or Table API methods
needed.
Looking forward to your thoughts!
Best,
Moritz
[1]
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
