[
https://issues.apache.org/jira/browse/NIFI-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakub Augustyniak updated NIFI-16467:
-------------------------------------
Description:
When a record reader infers the schema of JSON whose nested records have the
same shape at many levels but differ at the bottom, inference takes
exponentially longer with depth. The time roughly doubles with each nesting
level. Rich text documents with deeply nested lists produce exactly this
structure, for example:
{code:json}
{"type":"doc","content":[
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
\{"type":"text","text":"a"},
\{"type":"inlineCard","attrs":{"url":"https://example.com"}}
]}]}]}
]}]}
]}]},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
\{"type":"text","text":"a","marks":[{"type":"link","attrs":{"href":"https://example.com"}}]}
]}]}]}
]}]}
]}]}
]}
{code}
The document holds two sibling lists with the same structure at every level.
They differ only in the innermost paragraph, where neither is wider than the
other: one has an inlineCard with attrs, the other a text with marks. The
example nests each list 2 levels deep. Each further level wraps the innermost
listItem in another
\{"type":"listItem","content":[{"type":"paragraph"},\{"type":"bulletList","content":[...]}]}.
Each list level is two record levels (bulletList and listItem), so time grows
about 4x per list level: 0.8 s at 8 levels, 11 s at 10, and no result after 20
s at 12 or more. A single nested list, or two lists where one innermost
paragraph has a superset of the other's fields, infers in milliseconds.
Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
records, two of which contain a 38-level nested document:
* Inference did not finish within 5 minutes. In a real deployment it ran for
hours without completing.
* Either of the two deep documents alone is enough: it still didn't finish
after 60 s.
* With those documents removed, inference on the same input takes 26 ms.
* The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
was:
When a record reader infers the schema of JSON whose nested records differ in
shape at many levels, inference takes exponentially longer with depth. The time
roughly doubles with each nesting level. Rich text documents with deeply nested
lists produce exactly this structure, for example:
{code:java}
{"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
{"type":"paragraph","content":[{"type":"text","text":"a"}]},
{"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated N
times ... ]}]}
]}]}]}{code}
At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children have
a {{content}} field. Neither record type is wider than the other.
Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
records, two of which contain a 38-level nested document:
- Inference did not finish within 5 minutes. In a real deployment it ran for
hours without completing.
- Either of the two deep documents alone is enough: it still didn't finish
after 60 s.
- With those documents removed, inference on the same input takes 26 ms.
- The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
> Schema inference hangs on deeply nested JSON records
> ----------------------------------------------------
>
> Key: NIFI-16467
> URL: https://issues.apache.org/jira/browse/NIFI-16467
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Jakub Augustyniak
> Assignee: Jakub Augustyniak
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When a record reader infers the schema of JSON whose nested records have the
> same shape at many levels but differ at the bottom, inference takes
> exponentially longer with depth. The time roughly doubles with each nesting
> level. Rich text documents with deeply nested lists produce exactly this
> structure, for example:
> {code:json}
> {"type":"doc","content":[
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> {"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
> \{"type":"text","text":"a"},
> \{"type":"inlineCard","attrs":{"url":"https://example.com"}}
> ]}]}]}
> ]}]}
> ]}]},
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> {"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
>
> \{"type":"text","text":"a","marks":[{"type":"link","attrs":{"href":"https://example.com"}}]}
> ]}]}]}
> ]}]}
> ]}]}
> ]}
> {code}
> The document holds two sibling lists with the same structure at every level.
> They differ only in the innermost paragraph, where neither is wider than the
> other: one has an inlineCard with attrs, the other a text with marks. The
> example nests each list 2 levels deep. Each further level wraps the innermost
> listItem in another
> \{"type":"listItem","content":[{"type":"paragraph"},\{"type":"bulletList","content":[...]}]}.
> Each list level is two record levels (bulletList and listItem), so time grows
> about 4x per list level: 0.8 s at 8 levels, 11 s at 10, and no result after
> 20 s at 12 or more. A single nested list, or two lists where one innermost
> paragraph has a superset of the other's fields, infers in milliseconds.
> Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
> records, two of which contain a 38-level nested document:
> * Inference did not finish within 5 minutes. In a real deployment it ran for
> hours without completing.
> * Either of the two deep documents alone is enough: it still didn't finish
> after 60 s.
> * With those documents removed, inference on the same input takes 26 ms.
> * The processor thread can't be stopped, because inference doesn't respond to
> interruption, and the same FlowFile is picked up again after every restart.
> That blocks the queue indefinitely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
