[
https://issues.apache.org/jira/browse/FLINK-41062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib reassigned FLINK-41062:
------------------------------------
Assignee: Moritz Manner
> PARSE_XML and TRY_PARSE_XML can run out of memory for large documents
> ---------------------------------------------------------------------
>
> Key: FLINK-41062
> URL: https://issues.apache.org/jira/browse/FLINK-41062
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Reporter: Moritz Manner
> Assignee: Moritz Manner
> Priority: Major
>
> {{PARSE_XML}} reads the whole document into an intermediate tree before it
> writes the {{{}VARIANT{}}}. The tree is a lot larger than the input: a 40 MB
> document with 10 million empty elements needs between 2 and 3 GB of heap.
> The parser limits the nesting depth, entity expansion, name length and number
> of attributes, but not the size of the input or the number of elements. The
> 16 MB limit of a {{VARIANT}} is only checked when the variant is built, after
> the tree is complete.
> An {{OutOfMemoryError}} isn’t caught, so {{TRY_PARSE_XML}} doesn’t return
> {{NULL}} for such a document, and the job fails. With enough heap, the same
> document returns {{{}NULL{}}}, because the variant would be larger than 16
> MB. The generated code also keeps the last read document until the next
> record.
> Query to reproduce:
> {noformat}
> SELECT TRY_PARSE_XML(CONCAT('<a>', REPEAT('<b/>', 10000000),
> '</a>'));{noformat}
>
> With 2 GB of heap, this fails with {{{}java.lang.OutOfMemoryError: Java heap
> space{}}}. With 3 GB, it returns {{{}NULL{}}}.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
