[
https://issues.apache.org/jira/browse/FLINK-41439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Moritz Manner updated FLINK-41439:
----------------------------------
Description:
The output of {{XML_STRING}} isn’t bounded by the size of the input variant. A
variant stores each key once, but {{XML_STRING}} writes the key name for every
array item. For an array of nulls it also adds an {{xsi:nil}} attribute to each
item. So a small variant can turn into a very large string.
{{XmlStringFunction.eval}} rethrows an {{{}OutOfMemoryError{}}}. A single row
can fail the TaskManager instead of the query failing with a
{{{}TableRuntimeException{}}}.
The variant and {{JSON_STRING}} write the key once per array, {{XML_STRING}}
writes it once per item. The table shows the structure with a small example,
and the measured sizes for a key of 1,000 characters and an array of nulls:
||Nulls||Variant||{{JSON_STRING}}||{{XML_STRING}}||
|2, key {{{{kkk}}}} (example)|{{{"r":\{"kkk":[null,null]}}}}
|{{{"r":\{"kkk":[null,null]}}}}|{{<r
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>}}
{{ <kkk xsi:nil="true"/>}}
{{ <kkk xsi:nil="true"/>}}
{{</r>}}|
|10,000, key of 1,000 chars|31 KB|51,012 chars|10,180,061 chars (328x)|
|3,000,000, key of 1,000 chars|12 MB|–|{{OutOfMemoryError}}|
The last variant is below the variant size limit of 16 MiB. Depending on the
heap, it fails with {{Java heap space}} or because the output exceeds the
maximum length of a Java string:
*How to reproduce:*
{code:java}
SELECT XML_STRING(PARSE_JSON(CONCAT('{"r":{"', REPEAT('k', 1000), '":[null',
REPEAT(',null', 2999999), ']}}')));{code}
was:
The output of {{XML_STRING}} isn’t bounded by the size of the input variant. A
variant stores each key once, but {{XML_STRING}} writes the key name for every
array item. For an array of nulls it also adds an {{xsi:nil}} attribute to each
item. So a small variant can turn into a very large string.
{{XmlStringFunction.eval}} rethrows an {{{}OutOfMemoryError{}}}. A single row
can fail the TaskManager instead of the query failing with a
{{{}TableRuntimeException{}}}.
The variant and {{JSON_STRING}} write the key once per array, {{XML_STRING}}
writes it once per item. The table shows the structure with a small example,
and the measured sizes for a key of 1,000 characters and an array of nulls:
||Nulls||Variant||{{JSON_STRING}}||{{XML_STRING}}||
|2, key {{kkk}}
(example)|{{{"r":\{"kkk":[null,null]}}}}|{{{"r":\{"kkk":[null,null]}}}}|{{<r
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>}}
{{ <kkk xsi:nil="true"/>}}
{{ <kkk xsi:nil="true"/>}}
{{</r>}}|
|10,000, key of 1,000 chars|31 KB|51,012 chars|10,180,061 chars (328x)|
|3,000,000, key of 1,000 chars|12 MB|–|{{OutOfMemoryError}}|
The last variant is below the variant size limit of 16 MiB. Depending on the
heap, it fails with {{Java heap space}} or because the output exceeds the
maximum length of a Java string:
*How to reproduce:*
SELECT XML_STRING(PARSE_JSON(CONCAT(
'\{"r":{"', REPEAT('k', 1000), '":[null', REPEAT(',null', 2999999), ']}}')));
> XML_STRING can run out of memory for a variant within the size limit
> --------------------------------------------------------------------
>
> Key: FLINK-41439
> URL: https://issues.apache.org/jira/browse/FLINK-41439
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Reporter: Moritz Manner
> Priority: Major
>
> The output of {{XML_STRING}} isn’t bounded by the size of the input variant.
> A variant stores each key once, but {{XML_STRING}} writes the key name for
> every array item. For an array of nulls it also adds an {{xsi:nil}} attribute
> to each item. So a small variant can turn into a very large string.
> {{XmlStringFunction.eval}} rethrows an {{{}OutOfMemoryError{}}}. A single row
> can fail the TaskManager instead of the query failing with a
> {{{}TableRuntimeException{}}}.
> The variant and {{JSON_STRING}} write the key once per array, {{XML_STRING}}
> writes it once per item. The table shows the structure with a small example,
> and the measured sizes for a key of 1,000 characters and an array of nulls:
> ||Nulls||Variant||{{JSON_STRING}}||{{XML_STRING}}||
> |2, key {{{{kkk}}}} (example)|{{{"r":\{"kkk":[null,null]}}}}
> |{{{"r":\{"kkk":[null,null]}}}}|{{<r
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>}}
> {{ <kkk xsi:nil="true"/>}}
> {{ <kkk xsi:nil="true"/>}}
> {{</r>}}|
> |10,000, key of 1,000 chars|31 KB|51,012 chars|10,180,061 chars (328x)|
> |3,000,000, key of 1,000 chars|12 MB|–|{{OutOfMemoryError}}|
> The last variant is below the variant size limit of 16 MiB. Depending on the
> heap, it fails with {{Java heap space}} or because the output exceeds the
> maximum length of a Java string:
>
>
> *How to reproduce:*
>
> {code:java}
> SELECT XML_STRING(PARSE_JSON(CONCAT('{"r":{"', REPEAT('k', 1000), '":[null',
> REPEAT(',null', 2999999), ']}}')));{code}
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
