fgerlits opened a new pull request, #2297: URL: https://github.com/apache/nifi-minifi-cpp/pull/2297
ThreadSanitizer has found a mutex lock order issue between FlowController::mutex_ (M1) and C2MetricsPublisher::metrics_mutex_ (M2): during a flow update, we lock M1 first then M2, and during heartbeats, we lock M2 first and then some metrics nodes call executeOnAllComponents(), which locks M1. This is unlikely to happen in practice, because executeOnAllComponents checks if an update is in progress before trying to acquire the M1 lock -- but the flow update could start in the tiny window between these two lines. Also, if executeOnAllComponents finds that an update is in progress, it returns early without doing its stuff, so some metrics could have been missing from the heartbeat. Claude also found a problem with metrics node types FlowInformation, QueueMetrics and ProcessorMetrics: these store raw pointers to processors or connections inside the current flow, which can become dangling after a flow update. This PR fixes both issues by removing executeOnAllComponents() and replacing it with a component state cache, which has its own leaf mutex. The cache is cleared before the flow update and resnapped afterwards, which happens under this mutex. The new MetricsDuringReloadTests unit test exercises these code paths by running several metrics serializations and flow updates in parallel. Before this change, running under ThreadSanitizer, it fails with both kinds of errors (lock order inversion, invalid read); after the change, it passes. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
