Xinyu Wang created NIFI-16584:
---------------------------------

             Summary: One blocked property validator (e.g. 
DirectoryExistsValidator on an unresponsive NFS mount) stalls the periodic 
validation pass indefinitely
                 Key: NIFI-16584
                 URL: https://issues.apache.org/jira/browse/NIFI-16584
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
    Affects Versions: 2.5.0
            Reporter: Xinyu Wang


{code}
h3. Summary

The periodic validation pass ({{TriggerValidationTask}}) runs all component 
validations serially on a single "Validate Components" thread, and each 
validation calls property validators synchronously with no timeout. If one 
validator performs filesystem I/O on a path that hangs (for example, a 
directory on a {{hard}}-mounted NFS share whose server session is broken), that 
thread blocks forever and the pass never completes. Because the task is 
scheduled with {{scheduleWithFixedDelay}}, no further pass ever starts.

h3. Observed behavior

On a 2.5.0 node, a thread dump showed:

{noformat}
"Validate Components Thread-1" RUNNABLE
  at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
  at java.io.File.exists(File.java:836)
  at 
org.apache.nifi.processor.util.StandardValidators$DirectoryExistsValidator.validate(StandardValidators.java:953)
  at 
org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:180)
  at 
org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:916)
  at 
org.apache.nifi.controller.StandardProcessorNode.computeValidationErrors(StandardProcessorNode.java:1058)
  at 
org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:851)
  at 
org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52)
  at 
org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:42)
{noformat}

At the kernel level the thread was in uninterruptible sleep ({{D}} state) with 
wchan {{rpc_wait_bit_killable}}, and its CPU time did not change over 18 
minutes. The NFSv4.1 lease for the affected mount had expired about 45 hours 
earlier ({{lease_expired}} in {{/proc/<pid>/mountstats}}). The same mount also 
had 8 {{Timer-Driven Process Thread}} threads stuck in {{stat}}/{{access}} 
calls. Nothing is logged, because a {{hard}} mount never returns an error.

The other four threads of the validation pool were idle (parked on the 
executor's {{DelayedWorkQueue}}).

h3. Why this is a problem

- {{FlowController}} creates a 5-thread pool ({{FlowEngine(5, "Validate 
Components", true)}}) but schedules only one {{TriggerValidationTask}} on it 
({{scheduleWithFixedDelay(..., 5, 5, SECONDS)}}).
- {{TriggerValidationTask.run()}} iterates controller services, reporting 
tasks, flow analysis rules, parameter providers, all processors, registry 
clients and connectors sequentially, calling 
{{validationTrigger.trigger(node)}}, which runs {{performValidation()}} 
synchronously on the calling thread.
- A single hung validator therefore blocks the whole periodic validation pass. 
Components whose validity depends on the environment (the case NIFI-98 
describes for {{FileExistsValidator}}) are never re-validated again, and 
nothing is logged or surfaced to the user.
- Components that are newly created or modified go through 
{{resetValidationState()}} and {{triggerAsync()}} and can still be validated by 
the idle pool threads, so the impact is limited to the periodic validation 
pass, but it is silent and permanent until restart.

h3. Expected behavior

A hung or very slow validation of one component should not prevent the 
validation of other components, and should be visible to operators.

h3. Possible approaches

Submit each component's validation to the validation thread pool (as 
{{triggerAsync}} already does) instead of running it inline in the periodic 
validation pass, so one blocked component occupies only one pool thread.

Add a per-component timeout for validation and report the component as INVALID 
(with an explanation such as "validation timed out") so the pass can continue. 
NIFI-15121 took a similar approach for {{ExecuteGroovyScript}} using a virtual 
thread and a timeout.

Skip components whose previous validation is still in progress, so the pass 
does not pile up on the same component.

Log a warning when a validation exceeds a threshold, so a hung validator is 
diagnosable without a thread dump.

h3. Steps to reproduce (not yet verified by the reporter)

Mount an NFS export with {{hard}} on the NiFi host.

Add a {{ListFile}} (or any processor using {{DirectoryExistsValidator}}) with 
{{Input Directory}} pointing at that mount.

Make the NFS server unreachable, for example by dropping its traffic with a 
firewall rule.

Wait for the next periodic validation pass to reach that processor, then take a 
thread dump and check the "Validate Components" threads.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to