raminqaf commented on code in PR #29520:
URL: https://github.com/apache/flink/pull/29520#discussion_r4357068233
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -236,6 +240,66 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### VARIANT to XML
Review Comment:
```suggestion
#### VARIANT to XML String
```
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -236,6 +240,66 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### VARIANT to XML
+
+The `XML_STRING` function converts a `VARIANT` into an XML string. It reverses
the mapping of
+`PARSE_XML`, so `PARSE_XML(XML_STRING(v))` gives `v` for any `v` from
`PARSE_XML`, with or without
+`force_array`, apart from the whitespace described below. The other way around,
+`XML_STRING(PARSE_XML(xml))` gives the normalized document, without the XML
declaration, comments,
+and whitespace between tags, and with the attributes in the order of their
keys. For example:
+
+```sql
+-- '<book pages="320"><title>Dune</title>This is a
description<price>12.50</price>Text in between<price
currency="EUR">13.50</price></book>'
+XML_STRING(PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>'))
+```
+
+`XML_STRING` writes a `VARIANT` by the following rules:
+
+- `@` fields are written as attributes, `$` fields as text, and the other
fields as child elements.
+- The content of an element is written in the order of its `#` field. Content
without a position in
+ the `#` field follows in the order of its keys, so without a `#` field, all
content is written in
+ key order.
+- Each item of an array is written as an element or a text, so an empty array
writes nothing.
+- An empty string is written as `<a></a>`. A `NULL`, and a `$` field with
`NULL`, are written with
+ `xsi:nil="true"`, for example `<a xsi:nil="true"/>` and `<a id="1"
xsi:nil="true"/>`.
+- A scalar other than a string gets the matching `xsi:type` from the table
above with the `xs`
+ prefix, for example `<a xsi:type="xs:int">5</a>`. A `DECIMAL` gets
`xs:decimal`, and a
+ `TIMESTAMP_LTZ` gets `xs:dateTime` and is written in UTC, for example
`2026-09-30T10:30:00Z`. Like in `PARSE_XML`, only the text of an
+ element without child elements is typed.
+- `NaN` and infinity are written as `NaN`, `INF`, and `-INF`, like in XML
Schema. `PARSE_XML` reads
+ them back as strings, like dates and timestamps after the year 9999.
+- XML Schema has no type for a `UUID`, so it is written as text without
`xsi:type`, and reads back
+ as a `STRING`.
+- If the result uses the `xsi` or `xs` prefix, the root element declares it as
+ `xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"` or
`xmlns:xs="http://www.w3.org/2001/XMLSchema"`,
+ unless the `VARIANT` declares the prefix on the root element itself, for
example with an
+ `@xmlns:xs` field.
+- `<`, `>`, and `&` are escaped, and so is `"` in attribute values. Other
characters are written as
+ they are, and XML parsers normalize some of them: tabs and line breaks in
attribute values come
+ back as spaces, and carriage returns in text come back as line feeds.
+
+`XML_STRING` fails if the `VARIANT` has no XML representation, which is the
case if:
+
+- it isn't an object with a single field for the root element, or the root
element is an array,
+- a key isn't an XML name that `PARSE_XML` accepts, for example `1a` or `a b`,
apart from the `@`
+ prefix of an attribute,
+- a string has a character that XML 1.0 doesn't allow, for example `U+0000`,
+- an array is nested in an array,
Review Comment:
can an object get nested in an array?
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -236,6 +240,66 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### VARIANT to XML
+
+The `XML_STRING` function converts a `VARIANT` into an XML string. It reverses
the mapping of
+`PARSE_XML`, so `PARSE_XML(XML_STRING(v))` gives `v` for any `v` from
`PARSE_XML`, with or without
+`force_array`, apart from the whitespace described below. The other way around,
+`XML_STRING(PARSE_XML(xml))` gives the normalized document, without the XML
declaration, comments,
+and whitespace between tags, and with the attributes in the order of their
keys. For example:
+
+```sql
+-- '<book pages="320"><title>Dune</title>This is a
description<price>12.50</price>Text in between<price
currency="EUR">13.50</price></book>'
+XML_STRING(PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>'))
+```
+
+`XML_STRING` writes a `VARIANT` by the following rules:
+
+- `@` fields are written as attributes, `$` fields as text, and the other
fields as child elements.
+- The content of an element is written in the order of its `#` field. Content
without a position in
+ the `#` field follows in the order of its keys, so without a `#` field, all
content is written in
+ key order.
+- Each item of an array is written as an element or a text, so an empty array
writes nothing.
+- An empty string is written as `<a></a>`. A `NULL`, and a `$` field with
`NULL`, are written with
+ `xsi:nil="true"`, for example `<a xsi:nil="true"/>` and `<a id="1"
xsi:nil="true"/>`.
+- A scalar other than a string gets the matching `xsi:type` from the table
above with the `xs`
+ prefix, for example `<a xsi:type="xs:int">5</a>`. A `DECIMAL` gets
`xs:decimal`, and a
+ `TIMESTAMP_LTZ` gets `xs:dateTime` and is written in UTC, for example
`2026-09-30T10:30:00Z`. Like in `PARSE_XML`, only the text of an
+ element without child elements is typed.
+- `NaN` and infinity are written as `NaN`, `INF`, and `-INF`, like in XML
Schema. `PARSE_XML` reads
+ them back as strings, like dates and timestamps after the year 9999.
+- XML Schema has no type for a `UUID`, so it is written as text without
`xsi:type`, and reads back
+ as a `STRING`.
+- If the result uses the `xsi` or `xs` prefix, the root element declares it as
+ `xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"` or
`xmlns:xs="http://www.w3.org/2001/XMLSchema"`,
+ unless the `VARIANT` declares the prefix on the root element itself, for
example with an
+ `@xmlns:xs` field.
+- `<`, `>`, and `&` are escaped, and so is `"` in attribute values. Other
characters are written as
+ they are, and XML parsers normalize some of them: tabs and line breaks in
attribute values come
+ back as spaces, and carriage returns in text come back as line feeds.
+
+`XML_STRING` fails if the `VARIANT` has no XML representation, which is the
case if:
+
+- it isn't an object with a single field for the root element, or the root
element is an array,
+- a key isn't an XML name that `PARSE_XML` accepts, for example `1a` or `a b`,
apart from the `@`
+ prefix of an attribute,
+- a string has a character that XML 1.0 doesn't allow, for example `U+0000`,
+- an array is nested in an array,
+- an `@` field holds an object, an array, or `NULL`, or a `$` field holds an
object,
Review Comment:
Split them into two. WDYT?
```suggestion
- an `@` field holds an object, an array, or `NULL`
- a `$` field holds an object
```
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -236,6 +240,66 @@ A document fails to parse if it:
In all of these cases, `TRY_PARSE_XML` returns `NULL`.
+#### VARIANT to XML
+
+The `XML_STRING` function converts a `VARIANT` into an XML string. It reverses
the mapping of
+`PARSE_XML`, so `PARSE_XML(XML_STRING(v))` gives `v` for any `v` from
`PARSE_XML`, with or without
+`force_array`, apart from the whitespace described below. The other way around,
+`XML_STRING(PARSE_XML(xml))` gives the normalized document, without the XML
declaration, comments,
+and whitespace between tags, and with the attributes in the order of their
keys. For example:
+
+```sql
+-- '<book pages="320"><title>Dune</title>This is a
description<price>12.50</price>Text in between<price
currency="EUR">13.50</price></book>'
+XML_STRING(PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>'))
+```
+
+`XML_STRING` writes a `VARIANT` by the following rules:
+
+- `@` fields are written as attributes, `$` fields as text, and the other
fields as child elements.
+- The content of an element is written in the order of its `#` field. Content
without a position in
+ the `#` field follows in the order of its keys, so without a `#` field, all
content is written in
+ key order.
+- Each item of an array is written as an element or a text, so an empty array
writes nothing.
+- An empty string is written as `<a></a>`. A `NULL`, and a `$` field with
`NULL`, are written with
+ `xsi:nil="true"`, for example `<a xsi:nil="true"/>` and `<a id="1"
xsi:nil="true"/>`.
+- A scalar other than a string gets the matching `xsi:type` from the table
above with the `xs`
+ prefix, for example `<a xsi:type="xs:int">5</a>`. A `DECIMAL` gets
`xs:decimal`, and a
+ `TIMESTAMP_LTZ` gets `xs:dateTime` and is written in UTC, for example
`2026-09-30T10:30:00Z`. Like in `PARSE_XML`, only the text of an
+ element without child elements is typed.
+- `NaN` and infinity are written as `NaN`, `INF`, and `-INF`, like in XML
Schema. `PARSE_XML` reads
+ them back as strings, like dates and timestamps after the year 9999.
+- XML Schema has no type for a `UUID`, so it is written as text without
`xsi:type`, and reads back
+ as a `STRING`.
+- If the result uses the `xsi` or `xs` prefix, the root element declares it as
+ `xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"` or
`xmlns:xs="http://www.w3.org/2001/XMLSchema"`,
+ unless the `VARIANT` declares the prefix on the root element itself, for
example with an
+ `@xmlns:xs` field.
+- `<`, `>`, and `&` are escaped, and so is `"` in attribute values. Other
characters are written as
+ they are, and XML parsers normalize some of them: tabs and line breaks in
attribute values come
+ back as spaces, and carriage returns in text come back as line feeds.
+
+`XML_STRING` fails if the `VARIANT` has no XML representation, which is the
case if:
+
+- it isn't an object with a single field for the root element, or the root
element is an array,
+- a key isn't an XML name that `PARSE_XML` accepts, for example `1a` or `a b`,
apart from the `@`
+ prefix of an attribute,
+- a string has a character that XML 1.0 doesn't allow, for example `U+0000`,
Review Comment:
Are we speaking about UTF-16 here?
##########
docs/data/sql_functions.yml:
##########
@@ -1434,6 +1434,28 @@ variant:
error instead of returning NULL, use the `PARSE_XML` function.
See `PARSE_XML` for the full documentation.
+ - sql: XML_STRING(variant)
+ table: VARIANT.xmlString()
+ description: |
+ Converts a Variant into an XML string. This is the inverse of
`PARSE_XML`.
+
+ Fields with an `@` prefix are written as attributes, the `$` field as
text, and the other
+ fields as child elements, in the order that the `#` field records. A
Variant NULL is written
+ with `xsi:nil="true"`. A value other than a string or a UUID gets an
`xsi:type` attribute,
+ e.g. `xsi:type="xs:int"`, so that it reads back as the same type. If the
XML string uses the
+ `xsi` or `xs` prefix, the root element declares it.
+
+ If the Variant has no XML representation, an error is thrown.
Review Comment:
Please also define the SQL null behavior
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/XmlFunctionsITCase.java:
##########
@@ -34,10 +34,11 @@
import static org.apache.flink.table.api.Expressions.nullOf;
/**
- * Tests for {@link BuiltInFunctionDefinitions#PARSE_XML} and {@link
- * BuiltInFunctionDefinitions#TRY_PARSE_XML}.
+ * Tests for {@link BuiltInFunctionDefinitions#PARSE_XML}, {@link
+ * BuiltInFunctionDefinitions#TRY_PARSE_XML}, and {@link
BuiltInFunctionDefinitions#XML_STRING}.
*
- * <p>The bulk of the XML mapping is covered by {@code XmlToVariantParserTest}.
+ * <p>The bulk of the XML mapping is covered by {@code XmlToVariantParserTest}
and {@code
Review Comment:
remove
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/XmlFunctionsITCase.java:
##########
@@ -194,6 +205,53 @@ private static TestSetSpec tryParseXmlSpec() {
STRING());
}
+ private static TestSetSpec xmlStringSpec() {
+ return TestSetSpec.forFunction(BuiltInFunctionDefinitions.XML_STRING)
+ .onFieldsWithData(BOOK, "[1]", null)
+ .andDataTypes(STRING().notNull(), STRING().notNull(), STRING())
+ .testResult(
+ $("f0").parseXml().xmlString(),
+ "XML_STRING(PARSE_XML(f0))",
+ BOOK_XML,
+ STRING().notNull())
+ // Parsing the result gives the same variant.
+ .testResult(
+ jsonString($("f0").parseXml().xmlString().parseXml()),
+ "JSON_STRING(PARSE_XML(XML_STRING(PARSE_XML(f0))))",
+ BOOK_JSON,
+ STRING().notNull())
+ .testResult(
+ $("f2").parseJson().xmlString(),
+ "XML_STRING(PARSE_JSON(f2))",
+ null,
+ STRING())
+ .testSqlRuntimeError(
+ "XML_STRING(PARSE_JSON(f1))",
+ TableRuntimeException.class,
+ "Failed to convert the variant to an XML string: The
variant must be an "
+ + "object with a single field for the root
element, but it is of "
+ + "type ARRAY.")
+ .testTableApiRuntimeError(
+ $("f1").parseJson().xmlString(),
+ IllegalArgumentException.class,
+ "The variant must be an object with a single field for
the root element, "
+ + "but it is of type ARRAY.");
+ }
+
+ private static TestSetSpec xmlStringValidationSpec() {
Review Comment:
Should we just move this one test into `xmlStringSpec`?
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/functions/VariantToXmlWriterTest.java:
##########
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.types.variant.BinaryVariantInternalBuilder;
+import org.apache.flink.types.variant.Variant;
+import org.apache.flink.types.variant.VariantBuilder;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.util.Arrays;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+/** Tests for {@link VariantToXmlWriter}. */
+class VariantToXmlWriterTest {
Review Comment:
One `PARSE_JSON` happy path in `XmlFunctionsITCase` would also help, since
it only has error cases today.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantToXmlWriter.java:
##########
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.runtime.functions.XmlVariantNames.SchemaPrefix;
+import org.apache.flink.types.variant.Variant;
+
+import org.w3c.dom.DOMException;
+import org.w3c.dom.Document;
+
+import javax.annotation.Nullable;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+
+import java.io.StringWriter;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ATTRIBUTE_PREFIX;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ORDER_KEY;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.TEXT_KEY;
+import static org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_NIL;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_TYPE;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_FORMATTER;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_LTZ_FORMATTER;
+import static org.apache.flink.types.variant.BinaryVariantUtil.TIME_FORMATTER;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Writes a {@link Variant} as an XML string for {@code XML_STRING}. It
reverses the mapping of
+ * {@link XmlToVariantParser}, so that parsing the XML string of a variant
from {@code PARSE_XML}
+ * gives the same variant. The mapping is described in the documentation of
{@code XML_STRING}.
+ *
+ * <p>The content of an element is written in the order of its {@code #}
field. Content without a
+ * position in the {@code #} field follows in the order of its keys. A variant
{@code NULL}, or a
+ * {@code NULL} text, is written as {@code xsi:nil="true"}, and a scalar other
than a string or a
+ * UUID gets an {@code xsi:type}, e.g. {@code xsi:type="xs:int"}. If the
document uses the xsi or xs
+ * prefix, the root element declares it with the namespace of XML Schema,
unless the variant
+ * declares it on the root element itself.
+ *
+ * <p>The XML writer of the JDK writes the tags and escapes the text. It
writes tabs and line breaks
+ * in attribute values, and carriage returns in text, as they are, so a parser
reads them back as
+ * spaces and line feeds.
+ *
+ * <p>A variant fails with an {@link IllegalArgumentException} if it has no
XML representation:
+ *
+ * <ul>
+ * <li>it isn't an object with a single field for the root element, or the
root is an array,
+ * <li>a key isn't an XML name that {@code PARSE_XML} accepts,
+ * <li>a text or an attribute value has a character that XML 1.0 doesn't
allow, e.g. U+0000,
+ * <li>an array is nested in an array,
+ * <li>an {@code @} field holds an object, an array, or a {@code NULL}, or a
{@code $} field holds
+ * an object,
+ * <li>a {@code NULL} text isn't the only content of its element, or an
element has an {@code
+ * @xsi:nil} field that is true and content,
+ * <li>a {@code #} field isn't an object, doesn't have one integer position
for each item of a
+ * key, or has a position more than once,
+ * <li>an element would need two {@code xsi:type} or two {@code xsi:nil}
attributes, e.g. an
+ * {@code @xsi:type} field and a typed text, or an {@code @xsi:nil}
field and a {@code NULL}
+ * text.
+ * </ul>
+ */
+@Internal
+public final class VariantToXmlWriter {
+
+ private static final Set<Variant.Type> INTEGER_TYPES =
+ EnumSet.of(
+ Variant.Type.TINYINT,
+ Variant.Type.SMALLINT,
+ Variant.Type.INT,
+ Variant.Type.BIGINT);
+
+ // The XML writer of the JDK, regardless of other StAX implementations on
the classpath. Its
+ // factory creates a new writer for each call and keeps no state of it, so
it can be shared.
+ private static final XMLOutputFactory OUTPUT_FACTORY =
XMLOutputFactory.newDefaultFactory();
+
+ // Any character outside of the Char production of XML 1.0, see
+ // https://www.w3.org/TR/xml/#charsets.
+ private static final Pattern NON_XML_CHAR =
+
Pattern.compile("[^\t\n\r\u0020-\uD7FF\uE000-\uFFFD\\x{10000}-\\x{10FFFF}]");
+
+ // The DOM document that names are checked with. It isn't thread-safe, so
each thread has its
+ // own.
+ private static final ThreadLocal<Document> NAME_DOCUMENT =
+ ThreadLocal.withInitial(VariantToXmlWriter::newDocument);
+
+ // The document that is written, and the schema prefixes that it uses.
+ private final StringWriter output = new StringWriter();
+ private final XMLStreamWriter xml;
+ private final Set<SchemaPrefix> usedSchemaPrefixes =
EnumSet.noneOf(SchemaPrefix.class);
+
+ private VariantToXmlWriter() {
+ try {
+ xml = OUTPUT_FACTORY.createXMLStreamWriter(output);
+ } catch (XMLStreamException e) {
+ // Only the message, since the location of the exception isn't
serializable.
+ throw new IllegalStateException(e.getMessage());
+ }
+ }
+
+ public static String write(Variant variant) {
+ final String rootName = rootName(variant);
+ final Variant root = variant.getField(rootName);
+ checkRootIsNotArray(rootName, root);
+ return new VariantToXmlWriter().writeDocument(rootName, root);
Review Comment:
Extract as const `private static final VariantToXmlWriter
VARIANT_TO_XML_WRITER = new VariantToXmlWriter();
`
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantToXmlWriter.java:
##########
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.runtime.functions.XmlVariantNames.SchemaPrefix;
+import org.apache.flink.types.variant.Variant;
+
+import org.w3c.dom.DOMException;
+import org.w3c.dom.Document;
+
+import javax.annotation.Nullable;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+
+import java.io.StringWriter;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ATTRIBUTE_PREFIX;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ORDER_KEY;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.TEXT_KEY;
+import static org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_NIL;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_TYPE;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_FORMATTER;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_LTZ_FORMATTER;
+import static org.apache.flink.types.variant.BinaryVariantUtil.TIME_FORMATTER;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Writes a {@link Variant} as an XML string for {@code XML_STRING}. It
reverses the mapping of
+ * {@link XmlToVariantParser}, so that parsing the XML string of a variant
from {@code PARSE_XML}
+ * gives the same variant. The mapping is described in the documentation of
{@code XML_STRING}.
+ *
+ * <p>The content of an element is written in the order of its {@code #}
field. Content without a
+ * position in the {@code #} field follows in the order of its keys. A variant
{@code NULL}, or a
+ * {@code NULL} text, is written as {@code xsi:nil="true"}, and a scalar other
than a string or a
+ * UUID gets an {@code xsi:type}, e.g. {@code xsi:type="xs:int"}. If the
document uses the xsi or xs
+ * prefix, the root element declares it with the namespace of XML Schema,
unless the variant
+ * declares it on the root element itself.
+ *
+ * <p>The XML writer of the JDK writes the tags and escapes the text. It
writes tabs and line breaks
+ * in attribute values, and carriage returns in text, as they are, so a parser
reads them back as
+ * spaces and line feeds.
+ *
+ * <p>A variant fails with an {@link IllegalArgumentException} if it has no
XML representation:
+ *
+ * <ul>
+ * <li>it isn't an object with a single field for the root element, or the
root is an array,
+ * <li>a key isn't an XML name that {@code PARSE_XML} accepts,
+ * <li>a text or an attribute value has a character that XML 1.0 doesn't
allow, e.g. U+0000,
+ * <li>an array is nested in an array,
+ * <li>an {@code @} field holds an object, an array, or a {@code NULL}, or a
{@code $} field holds
+ * an object,
+ * <li>a {@code NULL} text isn't the only content of its element, or an
element has an {@code
+ * @xsi:nil} field that is true and content,
+ * <li>a {@code #} field isn't an object, doesn't have one integer position
for each item of a
+ * key, or has a position more than once,
+ * <li>an element would need two {@code xsi:type} or two {@code xsi:nil}
attributes, e.g. an
+ * {@code @xsi:type} field and a typed text, or an {@code @xsi:nil}
field and a {@code NULL}
+ * text.
+ * </ul>
+ */
+@Internal
+public final class VariantToXmlWriter {
+
+ private static final Set<Variant.Type> INTEGER_TYPES =
+ EnumSet.of(
+ Variant.Type.TINYINT,
+ Variant.Type.SMALLINT,
+ Variant.Type.INT,
+ Variant.Type.BIGINT);
+
+ // The XML writer of the JDK, regardless of other StAX implementations on
the classpath. Its
+ // factory creates a new writer for each call and keeps no state of it, so
it can be shared.
+ private static final XMLOutputFactory OUTPUT_FACTORY =
XMLOutputFactory.newDefaultFactory();
+
+ // Any character outside of the Char production of XML 1.0, see
+ // https://www.w3.org/TR/xml/#charsets.
+ private static final Pattern NON_XML_CHAR =
+
Pattern.compile("[^\t\n\r\u0020-\uD7FF\uE000-\uFFFD\\x{10000}-\\x{10FFFF}]");
+
+ // The DOM document that names are checked with. It isn't thread-safe, so
each thread has its
+ // own.
+ private static final ThreadLocal<Document> NAME_DOCUMENT =
+ ThreadLocal.withInitial(VariantToXmlWriter::newDocument);
+
+ // The document that is written, and the schema prefixes that it uses.
+ private final StringWriter output = new StringWriter();
+ private final XMLStreamWriter xml;
+ private final Set<SchemaPrefix> usedSchemaPrefixes =
EnumSet.noneOf(SchemaPrefix.class);
+
+ private VariantToXmlWriter() {
+ try {
+ xml = OUTPUT_FACTORY.createXMLStreamWriter(output);
+ } catch (XMLStreamException e) {
+ // Only the message, since the location of the exception isn't
serializable.
+ throw new IllegalStateException(e.getMessage());
+ }
+ }
+
+ public static String write(Variant variant) {
+ final String rootName = rootName(variant);
+ final Variant root = variant.getField(rootName);
+ checkRootIsNotArray(rootName, root);
+ return new VariantToXmlWriter().writeDocument(rootName, root);
+ }
+
+ /** Returns the name of the root element, the single field of the variant.
*/
+ private static String rootName(Variant variant) {
+ checkArgument(
+ variant.isObject(),
+ "The variant must be an object with a single field for the
root element, but it "
+ + "is of type %s.",
+ variant.getType());
+ final List<String> fieldNames = variant.getFieldNames();
+ checkArgument(
+ fieldNames.size() == 1,
+ "The variant must be an object with a single field for the
root element, but it "
+ + "has %s fields.",
+ fieldNames.size());
+ return fieldNames.get(0);
+ }
+
+ private static void checkRootIsNotArray(String rootName, Variant root) {
+ checkArgument(
+ !root.isArray(),
+ "The root element '%s' can't be an array, since a document has
only one root "
+ + "element.",
+ rootName);
+ }
+
+ //
--------------------------------------------------------------------------------------------
+
+ private String writeDocument(String rootName, Variant root) {
+ try {
+ writeElement(rootName, root);
+ // Closes the tag of a nil root element, which an empty element
leaves open.
+ xml.writeEndDocument();
+ xml.close();
+ } catch (XMLStreamException e) {
+ // Only the message, since the location of the exception isn't
serializable.
+ throw new IllegalStateException(e.getMessage());
+ }
+ insertIntoRootTag(rootName, schemaPrefixDeclarations(root));
+ return output.toString();
+ }
+
+ /**
+ * Returns the declarations of the schema prefixes that the document uses,
without the ones that
+ * the root element declares itself. A declaration on another element only
applies within that
+ * element, so the root element needs its own.
+ */
+ private String schemaPrefixDeclarations(Variant root) {
+ final StringBuilder declarations = new StringBuilder();
+ for (SchemaPrefix schemaPrefix : usedSchemaPrefixes) {
+ if (!declaresOnRoot(root, schemaPrefix)) {
+ declarations.append(
+ String.format(
+ " %s=\"%s\"", schemaPrefix.declaration,
schemaPrefix.namespace));
+ }
+ }
+ return declarations.toString();
+ }
+
+ private static boolean declaresOnRoot(Variant root, SchemaPrefix
schemaPrefix) {
+ return root.isObject()
+ && root.getField(ATTRIBUTE_PREFIX + schemaPrefix.declaration)
!= null;
+ }
+
+ /**
+ * Inserts the attributes after the name of the root element. They are
inserted at the end,
+ * since only then it is known which prefixes the document uses. The
document starts with the
+ * start tag of the root element, since it has no XML declaration.
+ */
+ private void insertIntoRootTag(String rootName, String attributes) {
+ final StringBuffer document = output.getBuffer();
+ final String rootTagStart = "<" + rootName;
+ checkState(
+ document.indexOf(rootTagStart) == 0,
+ "The document doesn't start with %s.",
+ rootTagStart);
+ document.insert(rootTagStart.length(), attributes);
+ }
+
+ //
--------------------------------------------------------------------------------------------
+ // Elements
+ //
--------------------------------------------------------------------------------------------
+
+ /** Writes a single item of a key, which isn't an array. */
+ private void writeElement(String name, Variant value) throws
XMLStreamException {
+ if (value.isNull()) {
+ writeNilElement(name);
+ } else if (value.isObject()) {
+ writeObjectElement(name, value);
+ } else {
+ writeScalarElement(name, value);
+ }
+ }
+
+ /** A variant {@code NULL} is nil, since an empty element is an empty
string. */
+ private void writeNilElement(String name) throws XMLStreamException {
+ writeEmptyElement(name);
+ writeXsiNil();
+ }
+
+ private void writeScalarElement(String name, Variant scalar) throws
XMLStreamException {
+ writeStartElement(name);
+ writeXsiType(scalar);
+ writeText(name, scalar);
+ writeEndElement();
+ }
+
+ /**
+ * Writes an object as an element with the {@code @} fields as attributes,
the {@code $} field
+ * as text, and the other fields as child elements.
+ */
+ private void writeObjectElement(String name, Variant element) throws
XMLStreamException {
+ final ElementKeys keys = new ElementKeys(element);
+ final List<ContentItem> content = content(name, element,
keys.contentKeys);
+ if (hasNullText(content)) {
+ writeNilElementWithAttributes(name, element, keys.attributeKeys);
+ } else {
+ writeElementWithContent(name, element, keys.attributeKeys,
content);
+ }
+ }
+
+ /** Like in {@code PARSE_XML}, a nil element with attributes has a null
text and no content. */
+ private static boolean hasNullText(List<ContentItem> content) {
+ final @Nullable Variant text = onlyText(content);
+ return text != null && text.isNull();
+ }
+
+ private void writeNilElementWithAttributes(
+ String name, Variant element, List<String> attributeKeys) throws
XMLStreamException {
+ checkNoAttributeField(name, element, XSI_NIL, "a null text");
+ writeEmptyElement(name);
+ writeFields(name, element, attributeKeys);
+ writeXsiNil();
+ }
+
+ private void writeElementWithContent(
+ String name, Variant element, List<String> attributeKeys,
List<ContentItem> content)
+ throws XMLStreamException {
+ checkIsNotNil(name, element, content);
+ writeStartElement(name);
+ // Attributes come first, since the XML writer writes them into the
start tag.
+ writeFields(name, element, attributeKeys);
+ writeXsiTypeOfText(name, element, content);
+ for (ContentItem item : content) {
+ writeField(name, item.key, item.value);
+ }
+ writeEndElement();
+ }
+
+ /** Like in {@code PARSE_XML}, only the text of an element without child
elements is typed. */
+ private void writeXsiTypeOfText(String name, Variant element,
List<ContentItem> content)
+ throws XMLStreamException {
+ final @Nullable Variant text = onlyText(content);
+ if (text != null && xsiType(text) != null) {
+ checkNoAttributeField(name, element, XSI_TYPE, "a typed text");
+ writeXsiType(text);
+ }
+ }
+
+ /** {@code PARSE_XML} drops the content of a nil element, so it can't have
content. */
+ private static void checkIsNotNil(
+ String elementName, Variant element, List<ContentItem> content) {
+ final @Nullable Variant xsiNil = element.getField(ATTRIBUTE_PREFIX +
XSI_NIL);
+ checkArgument(
+ content.isEmpty() || xsiNil == null || !isTrue(xsiNil),
+ "The element '%s' has an '%s%s' field that is true and
content, but an element "
+ + "with %s can't have content.",
+ elementName,
+ ATTRIBUTE_PREFIX,
+ XSI_NIL,
+ XSI_NIL);
+ }
+
+ /** Like in {@code PARSE_XML}, xsi:nil is true for the values true and 1.
*/
+ private static boolean isTrue(Variant xsiNil) {
+ if (!isNonNullScalar(xsiNil)) {
+ return false;
+ }
+ final String value = lexicalForm(xsiNil).trim();
+ return value.equals("true") || value.equals("1");
+ }
+
+ /** An element can't have an {@code @} field for an attribute that the
writer adds as well. */
+ private static void checkNoAttributeField(
+ String elementName, Variant element, String attribute, String
reason) {
+ checkArgument(
+ element.getField(ATTRIBUTE_PREFIX + attribute) == null,
+ "The element '%s' has an '%s%s' field and %s, but an element
can have only one %s "
+ + "attribute.",
+ elementName,
+ ATTRIBUTE_PREFIX,
+ attribute,
+ reason,
+ attribute);
+ }
+
+ //
--------------------------------------------------------------------------------------------
+ // Fields
+ //
--------------------------------------------------------------------------------------------
+
+ private void writeFields(String elementName, Variant element, List<String>
keys)
+ throws XMLStreamException {
+ for (String key : keys) {
+ writeField(elementName, key, element.getField(key));
+ }
+ }
+
+ /**
+ * Writes a field of an element by its key: an {@code @} field as an
attribute, the {@code $}
+ * field as text, and any other field as a child element.
+ */
+ private void writeField(String elementName, String key, Variant value)
+ throws XMLStreamException {
+ if (key.startsWith(ATTRIBUTE_PREFIX)) {
+ writeAttributeField(elementName,
key.substring(ATTRIBUTE_PREFIX.length()), value);
+ } else if (key.equals(TEXT_KEY)) {
+ writeText(elementName, value);
+ } else {
+ writeElement(key, value);
+ }
+ }
+
+ private void writeAttributeField(String elementName, String name, Variant
value)
+ throws XMLStreamException {
+ checkAttributeName(name);
+ checkIsAttributeValue(elementName, name, value);
+ checkXmlChars(elementName, name, value);
+ writeAttribute(name, lexicalForm(value));
+ }
+
+ private static void checkIsAttributeValue(String elementName, String name,
Variant value) {
+ checkArgument(
+ isNonNullScalar(value),
+ "The attribute '%s' of the element '%s' can't be of type %s.",
+ name,
+ elementName,
+ value.getType());
+ }
+
+ private void writeXsiNil() throws XMLStreamException {
+ writeAttribute(XSI_NIL, "true");
+ }
+
+ private void writeXsiType(Variant scalar) throws XMLStreamException {
+ final @Nullable String xsiType = xsiType(scalar);
+ if (xsiType != null) {
+ writeAttribute(XSI_TYPE, SchemaPrefix.XS.prefix + xsiType);
+ }
+ }
+
+ //
--------------------------------------------------------------------------------------------
+ // Writing to the XML writer, which writes the tags and escapes the text
+ //
--------------------------------------------------------------------------------------------
+
+ private void writeStartElement(String name) throws XMLStreamException {
+ checkElementName(name);
+ noteSchemaPrefix(name);
+ xml.writeStartElement(name);
+ }
+
+ private void writeEmptyElement(String name) throws XMLStreamException {
+ checkElementName(name);
+ noteSchemaPrefix(name);
+ xml.writeEmptyElement(name);
+ }
+
+ private void writeEndElement() throws XMLStreamException {
+ xml.writeEndElement();
+ }
+
+ /** The name is checked by the caller, since the writer adds xsi:nil and
xsi:type itself. */
+ private void writeAttribute(String name, String value) throws
XMLStreamException {
Review Comment:
XML_STRING writes tab, LF and CR in attribute values as raw characters. It
does the same with CR in text. A parser reads them back as a space and a line
feed. So `PARSE_XML('<a b="x y"/>')` has `x\ny` in `@b`, but after
`XML_STRING` and `PARSE_XML` again it is `x y`.
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/functions/XmlToVariantParserTest.java:
##########
@@ -334,6 +347,16 @@ Stream<Arguments> typedValues() {
"2262-04-11T23:47:16.854775807Z",
Variant.Type.TIMESTAMP_LTZ_NS,
Instant.parse("2262-04-11T23:47:16.854775807Z")),
+ arguments("base64Binary", "AQID", Variant.Type.BYTES, new
byte[] {1, 2, 3}),
Review Comment:
How about a test case with `Base64.getEncoder().encodeToString(new
byte[3000])`
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantToXmlWriter.java:
##########
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.runtime.functions.XmlVariantNames.SchemaPrefix;
+import org.apache.flink.types.variant.Variant;
+
+import org.w3c.dom.DOMException;
+import org.w3c.dom.Document;
+
+import javax.annotation.Nullable;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+
+import java.io.StringWriter;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ATTRIBUTE_PREFIX;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ORDER_KEY;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.TEXT_KEY;
+import static org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_NIL;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_TYPE;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_FORMATTER;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_LTZ_FORMATTER;
+import static org.apache.flink.types.variant.BinaryVariantUtil.TIME_FORMATTER;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Writes a {@link Variant} as an XML string for {@code XML_STRING}. It
reverses the mapping of
+ * {@link XmlToVariantParser}, so that parsing the XML string of a variant
from {@code PARSE_XML}
+ * gives the same variant. The mapping is described in the documentation of
{@code XML_STRING}.
+ *
+ * <p>The content of an element is written in the order of its {@code #}
field. Content without a
+ * position in the {@code #} field follows in the order of its keys. A variant
{@code NULL}, or a
+ * {@code NULL} text, is written as {@code xsi:nil="true"}, and a scalar other
than a string or a
+ * UUID gets an {@code xsi:type}, e.g. {@code xsi:type="xs:int"}. If the
document uses the xsi or xs
+ * prefix, the root element declares it with the namespace of XML Schema,
unless the variant
+ * declares it on the root element itself.
+ *
+ * <p>The XML writer of the JDK writes the tags and escapes the text. It
writes tabs and line breaks
+ * in attribute values, and carriage returns in text, as they are, so a parser
reads them back as
+ * spaces and line feeds.
+ *
+ * <p>A variant fails with an {@link IllegalArgumentException} if it has no
XML representation:
+ *
+ * <ul>
+ * <li>it isn't an object with a single field for the root element, or the
root is an array,
+ * <li>a key isn't an XML name that {@code PARSE_XML} accepts,
+ * <li>a text or an attribute value has a character that XML 1.0 doesn't
allow, e.g. U+0000,
+ * <li>an array is nested in an array,
+ * <li>an {@code @} field holds an object, an array, or a {@code NULL}, or a
{@code $} field holds
+ * an object,
+ * <li>a {@code NULL} text isn't the only content of its element, or an
element has an {@code
+ * @xsi:nil} field that is true and content,
+ * <li>a {@code #} field isn't an object, doesn't have one integer position
for each item of a
+ * key, or has a position more than once,
+ * <li>an element would need two {@code xsi:type} or two {@code xsi:nil}
attributes, e.g. an
+ * {@code @xsi:type} field and a typed text, or an {@code @xsi:nil}
field and a {@code NULL}
+ * text.
+ * </ul>
+ */
+@Internal
+public final class VariantToXmlWriter {
Review Comment:
We use recursion here. How much effort is it to build this without recursion
and avoid potential stack overflow problems?
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/functions/VariantToXmlWriterTest.java:
##########
@@ -0,0 +1,377 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.types.variant.BinaryVariantInternalBuilder;
+import org.apache.flink.types.variant.Variant;
+import org.apache.flink.types.variant.VariantBuilder;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.util.Arrays;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+/** Tests for {@link VariantToXmlWriter}. */
+class VariantToXmlWriterTest {
+
+ private static final String XSI =
"xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"";;
+ private static final String XSI_AND_XS = XSI + "
xmlns:xs=\"http://www.w3.org/2001/XMLSchema\"";;
+
+ private static final VariantBuilder BUILDER = Variant.newBuilder();
+
+ private final XmlToVariantParser parser = new XmlToVariantParser();
+
+ //
--------------------------------------------------------------------------------------------
+ // Variants from PARSE_XML
+ //
--------------------------------------------------------------------------------------------
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("parsedDocuments")
+ void testWriteParsedDocument(String xml, String expectedXml) {
+ assertThat(VariantToXmlWriter.write(parser.parse(xml,
false))).isEqualTo(expectedXml);
+ }
+
+ static Stream<Arguments> parsedDocuments() {
+ return Stream.of(
+ // The result is the normalized document, with the content in
the order of the #
+ // field and the attributes in the order of their keys.
+ arguments(
+ "<?xml version=\"1.0\"?><!-- c --><book pages=\"320\"
id=\"1\">\n"
+ + " <title>Dune</title>\n"
+ + " This is a description\n"
+ + " <!-- c --><price>12.50</price><?pi x?>\n"
+ + "</book>",
+ "<book id=\"1\" pages=\"320\"><title>Dune</title>This
is a description"
+ + "<price>12.50</price></book>"),
+ // An empty string is an empty element.
+ arguments("<title/>", "<title></title>"),
+ // Namespace prefixes and declarations are kept.
+ arguments(
+ "<ns:a xmlns:ns=\"urn:ns\"
ns:id=\"1\"><ns:b>x</ns:b></ns:a>",
+ "<ns:a ns:id=\"1\"
xmlns:ns=\"urn:ns\"><ns:b>x</ns:b></ns:a>"),
+ // A NULL, and the null text of an element with attributes,
are nil.
+ arguments(
+ "<r><a xsi:nil=\"true\"/><b xsi:nil=\"true\"
id=\"1\">x</b></r>",
+ "<r " + XSI + "><a xsi:nil=\"true\"/><b id=\"1\"
xsi:nil=\"true\"/></r>"),
+ arguments("<a xsi:nil=\"true\"/>", "<a " + XSI + "
xsi:nil=\"true\"/>"),
+ // Typed values are written in the lexical form of their type.
+ arguments(
+ "<r>"
+ + "<a xsi:type=\"boolean\">1</a>"
+ + "<b xsi:type=\"xs:int\">+5</b>"
+ + "<c xsi:type=\"integer\">007</c>"
+ + "<d xsi:type=\"double\">1.5E3</d>"
+ + "<e xsi:type=\"time\">12:30:00.500</e>"
+ + "<f
xsi:type=\"dateTime\">2026-09-23T12:30:45.5+02:00</f>"
+ + "<g
xsi:type=\"dateTime\">2026-09-23T12:30:45</g>"
+ + "<h xsi:type=\"base64Binary\">AQ I D</h>"
+ + "</r>",
+ "<r "
+ + XSI_AND_XS
+ + "><a xsi:type=\"xs:boolean\">true</a>"
+ + "<b xsi:type=\"xs:int\">5</b>"
+ + "<c xsi:type=\"xs:decimal\">7</c>"
+ + "<d xsi:type=\"xs:double\">1500.0</d>"
+ + "<e xsi:type=\"xs:time\">12:30:00.5</e>"
+ + "<f
xsi:type=\"xs:dateTime\">2026-09-23T10:30:45.5Z</f>"
+ + "<g
xsi:type=\"xs:dateTime\">2026-09-23T12:30:45</g>"
+ + "<h xsi:type=\"xs:base64Binary\">AQID</h>"
+ + "</r>"),
+ arguments(
+ "<price xsi:type=\"decimal\"
currency=\"EUR\">13.50</price>",
+ "<price "
+ + XSI_AND_XS
+ + " currency=\"EUR\"
xsi:type=\"xs:decimal\">13.5</price>"),
+ // An xsi:type that is kept as an attribute is written as it
is.
+ arguments(
+ "<a xsi:type=\"int\">five</a>", "<a " + XSI + "
xsi:type=\"int\">five</a>"),
+ // The declarations with the namespaces of XML Schema are
dropped and declared
+ // again where needed, and other declarations are kept.
+ arguments(
+ "<r
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\";><a>x</a></r>",
+ "<r><a>x</a></r>"),
+ arguments(
+ "<xs:schema
xmlns:xs=\"http://www.w3.org/2001/XMLSchema\";>"
+ + "<xs:element name=\"Book\"/></xs:schema>",
+ "<xs:schema
xmlns:xs=\"http://www.w3.org/2001/XMLSchema\";>"
+ + "<xs:element
name=\"Book\"></xs:element></xs:schema>"),
+ arguments(
+ "<r xmlns:xs=\"urn:custom\"><a
xsi:type=\"int\">5</a></r>",
+ "<r " + XSI + " xmlns:xs=\"urn:custom\"><a
xsi:type=\"xs:int\">5</a></r>"),
+ // A declaration on another element doesn't apply to the root
element.
+ arguments(
+ "<r><a xmlns:xsi=\"urn:custom\"
xsi:nil=\"true\"/></r>",
+ "<r " + XSI + "><a xmlns:xsi=\"urn:custom\"
xsi:nil=\"true\"/></r>"));
+ }
+
+ /** {@code PARSE_XML(XML_STRING(v))} is {@code v} for variants from {@code
PARSE_XML}. */
+ @ParameterizedTest(name = "{0}")
+ @ValueSource(
+ strings = {
+ "<book pages=\"320\"><title>Dune</title>This is a description"
+ + "<price>12.50</price>Text in between<price
currency=\"EUR\">13.50"
+ + "</price></book>",
+ "<a><b>1</b><c>2</c><b>3</b><c>4</c>text<b>5</b></a>",
+ "<ns:a xmlns:ns=\"urn:ns\" xmlns=\"urn:default\"
ns:id=\"1\"><ns:b>x</ns:b></ns:a>",
+ "<a b=\""<&> x
\"><&>\"	x y</a>",
+ "<book><a xsi:nil=\"true\" id=\"1\"/><a>x</a><a
xsi:nil=\"true\"/></book>",
+ "<a xsi:type=\"int\">five</a>",
+ "<xs:schema xmlns:xs=\"http://www.w3.org/2001/XMLSchema\";>"
+ + "<xs:element name=\"Book\"
type=\"xs:string\"/></xs:schema>",
+ "<r xmlns:xs=\"urn:custom\"><a xsi:type=\"int\">5</a></r>",
+ "<r><a xmlns:xsi=\"urn:custom\" xsi:nil=\"true\"/></r>",
+ "<r>"
+ + "<a xsi:type=\"boolean\">true</a>"
+ + "<b xsi:type=\"byte\">-1</b>"
+ + "<c xsi:type=\"short\">300</c>"
+ + "<d xsi:type=\"int\">70000</d>"
+ + "<e xsi:type=\"long\">5000000000</e>"
+ + "<f
xsi:type=\"integer\">123456789012345678901234567890</f>"
+ + "<g xsi:type=\"decimal\">-12.50</g>"
+ + "<h xsi:type=\"float\">1.1</h>"
+ + "<i xsi:type=\"double\">1.0E-300</i>"
+ + "<j xsi:type=\"date\">2026-09-23</j>"
+ + "<k xsi:type=\"time\">12:30:45.123456</k>"
+ + "<l
xsi:type=\"dateTime\">2026-09-23T12:30:45.123456</l>"
+ + "<m
xsi:type=\"dateTime\">2026-09-23T12:30:45.123456789</m>"
+ + "<n
xsi:type=\"dateTime\">2026-09-23T12:30:45.123456+02:00</n>"
+ + "<o
xsi:type=\"dateTime\">2026-09-23T12:30:45.123456789Z</o>"
+ + "<p xsi:type=\"string\">5</p>"
+ + "<q xsi:type=\"base64Binary\">AQID</q>"
+ + "</r>",
+ // The limits of the types.
+ "<r>"
+ + "<a xsi:type=\"date\">9999-12-31</a>"
+ + "<b xsi:type=\"date\">-5877641-06-23</b>"
+ + "<c
xsi:type=\"dateTime\">9999-12-31T23:59:59.999999Z</c>"
+ + "<d
xsi:type=\"dateTime\">2262-04-11T23:47:16.854775807</d>"
+ + "<e xsi:type=\"time\">23:59:59.999999</e>"
+ + "<f xsi:type=\"double\">4.9E-324</f>"
+ + "<g xsi:type=\"float\">3.4028235E38</g>"
+ + "</r>"
+ })
+ void testRoundTrip(String xml) {
+ for (boolean forceArray : new boolean[] {false, true}) {
+ final Variant variant = parser.parse(xml, forceArray);
+ final String written = VariantToXmlWriter.write(variant);
+ final Variant reparsed = parser.parse(written, forceArray);
+
assertThat(describe(reparsed)).as(written).isEqualTo(describe(variant));
+ // The written document is normalized, so writing it again gives
the same document.
+ assertThat(VariantToXmlWriter.write(reparsed)).isEqualTo(written);
+ }
+ }
+
+ /**
+ * Tabs and line breaks in attribute values, and carriage returns in text,
are written as they
+ * are, so a parser reads them back as spaces and line feeds.
+ */
+ @Test
+ void testWhitespaceThatParsersNormalize() {
+ final Variant variant = parser.parse("<a
b=\"x	y z w\">x y</a>", false);
+ assertThat(parser.parse(VariantToXmlWriter.write(variant),
false).toJson())
+ .isEqualTo("{\"a\":{\"$\":\"x\\ny\",\"@b\":\"x y z w\"}}");
+ }
+
+ //
--------------------------------------------------------------------------------------------
+ // Scalars
+ //
--------------------------------------------------------------------------------------------
+
+ @ParameterizedTest(name = "{1}")
+ @MethodSource({"typedScalars", "untypedScalars"})
+ void testWriteScalar(Variant scalar, String expectedXml) {
+ assertThat(VariantToXmlWriter.write(BUILDER.object().add("a",
scalar).build()))
+ .isEqualTo(expectedXml);
+ }
+
+ static Stream<Arguments> typedScalars() {
+ return Stream.of(
+ arguments(BUILDER.of(true), typed("boolean", "true")),
+ arguments(BUILDER.of((byte) -1), typed("byte", "-1")),
+ arguments(BUILDER.of((short) 300), typed("short", "300")),
+ arguments(BUILDER.of(70000), typed("int", "70000")),
+ arguments(BUILDER.of(5000000000L), typed("long",
"5000000000")),
+ arguments(BUILDER.of(new BigDecimal("1E+3")), typed("decimal",
"1000")),
+ arguments(BUILDER.of(1.1f), typed("float", "1.1")),
+ arguments(BUILDER.of(1.0E-300), typed("double", "1.0E-300")),
+ // XML Schema writes infinity as INF.
+ arguments(BUILDER.of(Float.NaN), typed("float", "NaN")),
+ arguments(BUILDER.of(Double.NEGATIVE_INFINITY),
typed("double", "-INF")),
+ // Years after 9999 have no + sign.
+ arguments(BUILDER.of(LocalDate.of(10000, 1, 1)), typed("date",
"10000-01-01")),
+ arguments(BUILDER.of(LocalDate.of(-1, 1, 1)), typed("date",
"-0001-01-01")),
+ // The seconds are always written.
+ arguments(BUILDER.of(LocalTime.of(12, 30)), typed("time",
"12:30:00")),
+ arguments(
+ BUILDER.of(LocalDateTime.of(2026, 9, 23, 12, 30, 45,
123_456_789)),
+ typed("dateTime", "2026-09-23T12:30:45.123456789")),
+ // A TIMESTAMP_LTZ is written in UTC.
+ arguments(
+ BUILDER.of(Instant.parse("2026-09-23T10:30:00Z")),
+ typed("dateTime", "2026-09-23T10:30:00Z")),
+ arguments(BUILDER.of(new byte[] {1, 2, 3}),
typed("base64Binary", "AQID")));
+ }
+
+ static Stream<Arguments> untypedScalars() {
+ return Stream.of(
+ arguments(BUILDER.of("x"), "<a>x</a>"),
+ // XML Schema has no type for a UUID.
+ arguments(
+
BUILDER.of(UUID.fromString("123e4567-e89b-12d3-a456-426614174000")),
+ "<a>123e4567-e89b-12d3-a456-426614174000</a>"));
+ }
+
+ private static String typed(String xsiType, String text) {
+ return String.format("<a %s xsi:type=\"xs:%s\">%s</a>", XSI_AND_XS,
xsiType, text);
+ }
+
+ //
--------------------------------------------------------------------------------------------
+ // Variants without an XML representation
+ //
--------------------------------------------------------------------------------------------
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("variantsWithoutXmlRepresentation")
+ void testNoXmlRepresentation(String json, String expectedMessage) throws
IOException {
+ final Variant variant = parseJson(json);
+ assertThatThrownBy(() -> VariantToXmlWriter.write(variant))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage(expectedMessage);
+ }
+
+ static Stream<Arguments> variantsWithoutXmlRepresentation() {
+ return Stream.of(
+ // The root element
+ arguments(
+ "1",
+ "The variant must be an object with a single field for
the root element, "
+ + "but it is of type TINYINT."),
+ arguments(
+ "{\"a\":1,\"b\":2}",
+ "The variant must be an object with a single field for
the root element, "
+ + "but it has 2 fields."),
+ arguments(
+ "{\"a\":[\"x\",\"y\"]}",
+ "The root element 'a' can't be an array, since a
document has only one "
+ + "root element."),
+ arguments(
+ "{\"r\":{\"a\":[[\"x\"]]}}",
+ "The field 'a' of the element 'r' has an array nested
in an array, which "
+ + "XML can't represent."),
+ // Names, so that a key can't add markup
+ arguments(
+ "{\"r\":{\"a><evil/><b\":\"x\"}}",
+ "'a><evil/><b' is not a valid XML element name."),
+ arguments(
+ "{\"a\":{\"@x=\\\"\\\" evil=\\\"1\\\" y\":\"v\"}}",
+ "'x=\"\" evil=\"1\" y' is not a valid XML attribute
name."),
+ // Only the names that PARSE_XML reads.
+ arguments("{\"\\u2070\":\"x\"}", "'\u2070' is not a valid XML
element name."),
+ // Attributes and text
+ arguments(
+ "{\"a\":{\"@b\":{\"c\":\"x\"}}}",
+ "The attribute 'b' of the element 'a' can't be of type
OBJECT."),
+ arguments(
+ "{\"a\":{\"$\":{\"c\":\"x\"}}}",
+ "The text of the element 'a' can't be of type
OBJECT."),
+ // Characters that XML 1.0 can't represent
+ arguments(
+ "{\"a\":\"x\\u0000y\"}",
+ "The text of the element 'a' has the character U+0000,
which XML 1.0 "
+ + "doesn't allow."),
+ arguments(
+ "{\"a\":{\"@b\":\"x\\uFFFEy\"}}",
+ "The attribute 'b' of the element 'a' has the
character U+FFFE, which XML "
+ + "1.0 doesn't allow."),
+ arguments(
+ "{\"a\":{\"$\":null,\"b\":\"x\"}}",
+ "The element 'a' has a null text and other content,
but an element with "
+ + "xsi:nil can't have content."),
+ arguments(
+ "{\"a\":{\"@xsi:nil\":\" 1 \",\"b\":\"x\"}}",
+ "The element 'a' has an '@xsi:nil' field that is true
and content, but an "
+ + "element with xsi:nil can't have content."),
+ // An element can't have an attribute twice.
Review Comment:
The check for an `@xsi:nil` field plus a null text has no test
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantToXmlWriter.java:
##########
@@ -0,0 +1,784 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.runtime.functions.XmlVariantNames.SchemaPrefix;
+import org.apache.flink.types.variant.Variant;
+
+import org.w3c.dom.DOMException;
+import org.w3c.dom.Document;
+
+import javax.annotation.Nullable;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+
+import java.io.StringWriter;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ATTRIBUTE_PREFIX;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.ORDER_KEY;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.TEXT_KEY;
+import static org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_NIL;
+import static
org.apache.flink.table.runtime.functions.XmlVariantNames.XSI_TYPE;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_FORMATTER;
+import static
org.apache.flink.types.variant.BinaryVariantUtil.TIMESTAMP_LTZ_FORMATTER;
+import static org.apache.flink.types.variant.BinaryVariantUtil.TIME_FORMATTER;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Writes a {@link Variant} as an XML string for {@code XML_STRING}. It
reverses the mapping of
+ * {@link XmlToVariantParser}, so that parsing the XML string of a variant
from {@code PARSE_XML}
+ * gives the same variant. The mapping is described in the documentation of
{@code XML_STRING}.
+ *
+ * <p>The content of an element is written in the order of its {@code #}
field. Content without a
+ * position in the {@code #} field follows in the order of its keys. A variant
{@code NULL}, or a
+ * {@code NULL} text, is written as {@code xsi:nil="true"}, and a scalar other
than a string or a
+ * UUID gets an {@code xsi:type}, e.g. {@code xsi:type="xs:int"}. If the
document uses the xsi or xs
+ * prefix, the root element declares it with the namespace of XML Schema,
unless the variant
+ * declares it on the root element itself.
+ *
+ * <p>The XML writer of the JDK writes the tags and escapes the text. It
writes tabs and line breaks
+ * in attribute values, and carriage returns in text, as they are, so a parser
reads them back as
+ * spaces and line feeds.
+ *
+ * <p>A variant fails with an {@link IllegalArgumentException} if it has no
XML representation:
+ *
+ * <ul>
+ * <li>it isn't an object with a single field for the root element, or the
root is an array,
+ * <li>a key isn't an XML name that {@code PARSE_XML} accepts,
+ * <li>a text or an attribute value has a character that XML 1.0 doesn't
allow, e.g. U+0000,
+ * <li>an array is nested in an array,
+ * <li>an {@code @} field holds an object, an array, or a {@code NULL}, or a
{@code $} field holds
+ * an object,
+ * <li>a {@code NULL} text isn't the only content of its element, or an
element has an {@code
+ * @xsi:nil} field that is true and content,
+ * <li>a {@code #} field isn't an object, doesn't have one integer position
for each item of a
+ * key, or has a position more than once,
+ * <li>an element would need two {@code xsi:type} or two {@code xsi:nil}
attributes, e.g. an
+ * {@code @xsi:type} field and a typed text, or an {@code @xsi:nil}
field and a {@code NULL}
+ * text.
+ * </ul>
+ */
+@Internal
+public final class VariantToXmlWriter {
+
+ private static final Set<Variant.Type> INTEGER_TYPES =
+ EnumSet.of(
+ Variant.Type.TINYINT,
+ Variant.Type.SMALLINT,
+ Variant.Type.INT,
+ Variant.Type.BIGINT);
+
+ // The XML writer of the JDK, regardless of other StAX implementations on
the classpath. Its
+ // factory creates a new writer for each call and keeps no state of it, so
it can be shared.
+ private static final XMLOutputFactory OUTPUT_FACTORY =
XMLOutputFactory.newDefaultFactory();
+
+ // Any character outside of the Char production of XML 1.0, see
+ // https://www.w3.org/TR/xml/#charsets.
+ private static final Pattern NON_XML_CHAR =
+
Pattern.compile("[^\t\n\r\u0020-\uD7FF\uE000-\uFFFD\\x{10000}-\\x{10FFFF}]");
+
+ // The DOM document that names are checked with. It isn't thread-safe, so
each thread has its
+ // own.
+ private static final ThreadLocal<Document> NAME_DOCUMENT =
+ ThreadLocal.withInitial(VariantToXmlWriter::newDocument);
+
+ // The document that is written, and the schema prefixes that it uses.
+ private final StringWriter output = new StringWriter();
+ private final XMLStreamWriter xml;
+ private final Set<SchemaPrefix> usedSchemaPrefixes =
EnumSet.noneOf(SchemaPrefix.class);
+
+ private VariantToXmlWriter() {
+ try {
+ xml = OUTPUT_FACTORY.createXMLStreamWriter(output);
+ } catch (XMLStreamException e) {
+ // Only the message, since the location of the exception isn't
serializable.
+ throw new IllegalStateException(e.getMessage());
+ }
+ }
+
+ public static String write(Variant variant) {
+ final String rootName = rootName(variant);
+ final Variant root = variant.getField(rootName);
+ checkRootIsNotArray(rootName, root);
+ return new VariantToXmlWriter().writeDocument(rootName, root);
+ }
+
+ /** Returns the name of the root element, the single field of the variant.
*/
+ private static String rootName(Variant variant) {
+ checkArgument(
+ variant.isObject(),
+ "The variant must be an object with a single field for the
root element, but it "
+ + "is of type %s.",
+ variant.getType());
+ final List<String> fieldNames = variant.getFieldNames();
+ checkArgument(
+ fieldNames.size() == 1,
+ "The variant must be an object with a single field for the
root element, but it "
+ + "has %s fields.",
+ fieldNames.size());
+ return fieldNames.get(0);
+ }
+
+ private static void checkRootIsNotArray(String rootName, Variant root) {
Review Comment:
Should we revert this can call it `throwIfRootIsArray`. It is very twisted
not in not conditions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
