Moritz Manner created FLINK-41439:
-------------------------------------
Summary: XML_STRING can run out of memory for a variant within the
size limit
Key: FLINK-41439
URL: https://issues.apache.org/jira/browse/FLINK-41439
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Reporter: Moritz Manner
The output of {{XML_STRING}} isn’t bounded by the size of the input variant. A
variant stores each key once, but {{XML_STRING}} writes the key name for every
array item. For an array of nulls it also adds an {{xsi:nil}} attribute to each
item. So a small variant can turn into a very large string.
{{XmlStringFunction.eval}} rethrows an {{{}OutOfMemoryError{}}}. A single row
can fail the TaskManager instead of the query failing with a
{{{}TableRuntimeException{}}}.
The variant and {{JSON_STRING}} write the key once per array, {{XML_STRING}}
writes it once per item. The table shows the structure with a small example,
and the measured sizes for a key of 1,000 characters and an array of nulls:
||Nulls||Variant||{{JSON_STRING}}||{{XML_STRING}}||
|2, key {{kkk}}
(example)|{{{"r":\{"kkk":[null,null]}}}}|{{{"r":\{"kkk":[null,null]}}}}|{{<r
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>}}
{{ <kkk xsi:nil="true"/>}}
{{ <kkk xsi:nil="true"/>}}
{{</r>}}|
|10,000, key of 1,000 chars|31 KB|51,012 chars|10,180,061 chars (328x)|
|3,000,000, key of 1,000 chars|12 MB|–|{{OutOfMemoryError}}|
The last variant is below the variant size limit of 16 MiB. Depending on the
heap, it fails with {{Java heap space}} or because the output exceeds the
maximum length of a Java string:
*How to reproduce:*
SELECT XML_STRING(PARSE_JSON(CONCAT(
'\{"r":{"', REPEAT('k', 1000), '":[null', REPEAT(',null', 2999999), ']}}')));
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
