[
https://issues.apache.org/jira/browse/NIFI-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakub Augustyniak updated NIFI-16467:
-------------------------------------
Description:
When a record reader infers the schema of JSON whose nested records have the
same shape at many levels but differ at the bottom, inference takes
exponentially longer with depth. The time roughly doubles with each nesting
level. Rich text documents with deeply nested lists produce exactly this
structure, for example:
```json
{"type":"doc","content":[
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
\{"type":"text","text":"a"},
\{"type":"inlineCard","attrs":{"url":"https://example.com"}}
]}]}]}
]}]}
]}]},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
\{"type":"text","text":"a","marks":[{"type":"link","attrs":{"href":"https://example.com"}}]}
]}]}]}
]}]}
]}]}
]}
```
The document holds two sibling lists with the same structure at every level.
They differ only in the innermost paragraph, where neither is wider than the
other: one has an `inlineCard` with `attrs`, the other a `text` with `marks`.
The example nests each list 2 levels deep. Each further level wraps the
innermost `listItem` in another
`\{"type":"listItem","content":[{"type":"paragraph"},\{"type":"bulletList","content":[...]}]}`.
Each list level is two record levels (`bulletList` and `listItem`), so time
grows about 4x per list level: 0.8 s at 8 levels, 11 s at 10, and no result
after 20 s at 12 or more. A single nested list, or two lists where one
innermost paragraph has a superset of the other's fields, infers in
milliseconds.
Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
records, two of which contain a 38-level nested document:
- Inference did not finish within 5 minutes. In a real deployment it ran for
hours without completing.
- Either of the two deep documents alone is enough: it still didn't finish
after 60 s.
- With those documents removed, inference on the same input takes 26 ms.
- The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
was:
When a record reader infers the schema of JSON whose nested records have the
same shape at many levels but differ at the bottom, inference takes
exponentially longer with depth. The time roughly doubles with each nesting
level. Rich text documents with deeply nested lists produce exactly this
structure, for example:
{code:json}
{"type":"doc","content":[
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
\{"type":"text","text":"a"},
\{"type":"inlineCard","attrs":{"url":"https://example.com"}}
]}]}]}
]}]}
]}]},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
\{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
\{"type":"text","text":"a","marks":[{"type":"link","attrs":{"href":"https://example.com"}}]}
]}]}]}
]}]}
]}]}
]}
{code}
The document holds two sibling lists with the same structure at every level.
They differ only in the innermost paragraph, where neither is wider than the
other: one has an inlineCard with attrs, the other a text with marks. The
example nests each list 2 levels deep. Each further level wraps the innermost
listItem in another
\{"type":"listItem","content":[{"type":"paragraph"},\{"type":"bulletList","content":[...]}]}.
Each list level is two record levels (bulletList and listItem), so time grows
about 4x per list level: 0.8 s at 8 levels, 11 s at 10, and no result after 20
s at 12 or more. A single nested list, or two lists where one innermost
paragraph has a superset of the other's fields, infers in milliseconds.
Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
records, two of which contain a 38-level nested document:
* Inference did not finish within 5 minutes. In a real deployment it ran for
hours without completing.
* Either of the two deep documents alone is enough: it still didn't finish
after 60 s.
* With those documents removed, inference on the same input takes 26 ms.
* The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
> Schema inference hangs on deeply nested JSON records
> ----------------------------------------------------
>
> Key: NIFI-16467
> URL: https://issues.apache.org/jira/browse/NIFI-16467
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Jakub Augustyniak
> Assignee: Jakub Augustyniak
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When a record reader infers the schema of JSON whose nested records have the
> same shape at many levels but differ at the bottom, inference takes
> exponentially longer with depth. The time roughly doubles with each nesting
> level. Rich text documents with deeply nested lists produce exactly this
> structure, for example:
> ```json
> {"type":"doc","content":[
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> {"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
> \{"type":"text","text":"a"},
> \{"type":"inlineCard","attrs":{"url":"https://example.com"}}
> ]}]}]}
> ]}]}
> ]}]},
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> \{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph"},
>
> {"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[
>
> \{"type":"text","text":"a","marks":[{"type":"link","attrs":{"href":"https://example.com"}}]}
> ]}]}]}
> ]}]}
> ]}]}
> ]}
> ```
> The document holds two sibling lists with the same structure at every level.
> They differ only in the innermost paragraph, where neither is wider than the
> other: one has an `inlineCard` with `attrs`, the other a `text` with `marks`.
> The example nests each list 2 levels deep. Each further level wraps the
> innermost `listItem` in another
> `\{"type":"listItem","content":[{"type":"paragraph"},\{"type":"bulletList","content":[...]}]}`.
> Each list level is two record levels (`bulletList` and `listItem`), so time
> grows about 4x per list level: 0.8 s at 8 levels, 11 s at 10, and no result
> after 20 s at 12 or more. A single nested list, or two lists where one
> innermost paragraph has a superset of the other's fields, infers in
> milliseconds.
> Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
> records, two of which contain a 38-level nested document:
> - Inference did not finish within 5 minutes. In a real deployment it ran for
> hours without completing.
> - Either of the two deep documents alone is enough: it still didn't finish
> after 60 s.
> - With those documents removed, inference on the same input takes 26 ms.
> - The processor thread can't be stopped, because inference doesn't respond to
> interruption, and the same FlowFile is picked up again after every restart.
> That blocks the queue indefinitely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
