[
https://issues.apache.org/jira/browse/HDDS-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-15826:
---------------------------------
Assignee: Siyao Meng
Description:
h2. Problem
Recon serves the disk-usage ({{/du}}), file-size-distribution, and
directory-count views by walking the NSSummary directory tree. Three code paths
do this walk recursively:
* {{EntityHandler.getTotalFileSizeDist}}
* {{EntityHandler.getTotalDirCount}}
* {{ReconUtils.gatherSubPaths}}
Each recurses over {{NSSummary.getChildDir()}} with no protection against a
malformed tree. If the persisted NSSummary data is corrupted so that a
directory lists itself, or one of its ancestors, as a child, the child set
forms a cycle. The recursion then never terminates and Recon dies with a
{{StackOverflowError}}. On the read path this surfaces as an HTTP 500 from
{{/du}} and can take the Recon process down. Because the corruption lives in
persisted state, the failure repeats on every request until the data is
rebuilt. The recursion also had no bound for a non-cyclic but pathologically
deep tree.
h2. Fix
Convert all three walks from recursion to iterative traversal backed by an
explicit stack and a {{visited}} set, so each object is visited (and counted)
at most once. For a well-formed tree the results are unchanged.
Two smaller improvements ride along:
* {{gatherSubPaths}} previously fetched each node's NSSummary twice; it now
fetches once (on pop), halving the {{getNSSummary}} lookups on large trees.
* De-duplicating silently would hide a real data problem, so each walk logs a
single WARN per request (with path/object context) the first time it detects a
repeated reference.
h2. Scope
Read-side hardening only. This does not prevent the corruption at write time,
and does not add response paging or a size limit to {{/du}} (a separate
follow-up for very large but well-formed buckets).
was:Recon disk-usage tree walks recurse over NSSummary child directories with
no cycle guard; a corrupted tree where a directory references itself or an
ancestor causes infinite recursion and a StackOverflowError (HTTP 500 on /du).
> Recon /du endpoint crashes with StackOverflowError on a corrupted NSSummary
> tree
> --------------------------------------------------------------------------------
>
> Key: HDDS-15826
> URL: https://issues.apache.org/jira/browse/HDDS-15826
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> Recon serves the disk-usage ({{/du}}), file-size-distribution, and
> directory-count views by walking the NSSummary directory tree. Three code
> paths do this walk recursively:
> * {{EntityHandler.getTotalFileSizeDist}}
> * {{EntityHandler.getTotalDirCount}}
> * {{ReconUtils.gatherSubPaths}}
> Each recurses over {{NSSummary.getChildDir()}} with no protection against a
> malformed tree. If the persisted NSSummary data is corrupted so that a
> directory lists itself, or one of its ancestors, as a child, the child set
> forms a cycle. The recursion then never terminates and Recon dies with a
> {{StackOverflowError}}. On the read path this surfaces as an HTTP 500 from
> {{/du}} and can take the Recon process down. Because the corruption lives in
> persisted state, the failure repeats on every request until the data is
> rebuilt. The recursion also had no bound for a non-cyclic but pathologically
> deep tree.
> h2. Fix
> Convert all three walks from recursion to iterative traversal backed by an
> explicit stack and a {{visited}} set, so each object is visited (and counted)
> at most once. For a well-formed tree the results are unchanged.
> Two smaller improvements ride along:
> * {{gatherSubPaths}} previously fetched each node's NSSummary twice; it now
> fetches once (on pop), halving the {{getNSSummary}} lookups on large trees.
> * De-duplicating silently would hide a real data problem, so each walk logs a
> single WARN per request (with path/object context) the first time it detects
> a repeated reference.
> h2. Scope
> Read-side hardening only. This does not prevent the corruption at write time,
> and does not add response paging or a size limit to {{/du}} (a separate
> follow-up for very large but well-formed buckets).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
