Simon Theodosius created NIFI-16583:
---------------------------------------

             Summary: Add ConsumeOPCUA processor for OPC UA data-change 
subscriptions
                 Key: NIFI-16583
                 URL: https://issues.apache.org/jira/browse/NIFI-16583
             Project: Apache NiFi
          Issue Type: New Feature
          Components: Extensions
            Reporter: Simon Theodosius


h3. Motivation

OPC UA servers expose industrial measurements and their quality and timestamps. 
A NiFi source processor that subscribes to value changes would allow these 
measurements to enter a dataflow directly, with configurable sampling and 
batching.

I propose adding ConsumeOPCUA using Eclipse Milo and adapting an existing 
standalone implementation to the current Apache NiFi codebase.

h3. Proposed scope

* Subscribe to OPC UA Value data changes over opc.tcp and emit initial values 
and subsequent changes as batched FlowFiles.
* Configure tags using namespace-index or namespace-URI NodeIds, with aliases, 
sampling intervals, server queue sizes and deadbands.
* Include tag identity, value, status and timestamps in the output.
* Support anonymous and username/password authentication, explicit security 
policy and message security mode selection, and client application 
certificates. Full server certificate validation should be the default.
* Support connection verification, reconnection and subscription recreation, 
with primary-node execution in clustered NiFi.

OPC UA Events/alarms, historical reads, writes and durable recovery are outside 
the proposed initial scope.

h3. Delivery and resource limits

Bound retained notifications by count and serialized bytes. Keep batches 
reserved and counted against both limits until NiFi reports the commit outcome, 
and report buffer overflow and server-reported notification loss. These limits 
do not bound all decoded protocol objects or total JVM heap.

Shutdown should stop accepting notifications and make a time-limited attempt to 
drain buffered data. This does not impose a deadline on NiFi repository 
operations or the overall processor stop.

Uncommitted notifications remain in memory and are not replicated. Outages, 
overflow, JVM termination and primary-node failover can cause gaps or 
duplicates. The processor would not provide end-to-end exactly-once or lossless 
outage recovery.

h3. Existing implementation and design questions

The standalone implementation is available at 
https://github.com/paf91/nifi-opcua and is currently published under the 
processor name SubscribeOpcUa. ConsumeOPCUA is the proposed upstream name. 
Integration into the Apache NiFi source tree and dependency/licensing review 
remain to be completed.

I would appreciate maintainer guidance on two initial design choices:

* Start with JSON/NDJSON output, or expose a RecordSetWriterFactory in the 
first contribution?
* Keep OPC UA identity/trust configuration on the processor, or introduce an 
OPC UA-specific Controller Service?

h3. Proposed verification

* Exercise supported authentication/security combinations and certificate 
failures.
* Cover namespace resolution, value encoding, reconnect/resubscribe, buffer 
saturation, downstream backpressure, commit failures and Stop/Start races.
* Measure memory and CPU use under representative load and repeated connection 
failures.
* Verify interoperability with an independent OPC UA server such as open62541 
and load the resulting NAR in NiFi.
* Complete the current NiFi contribution checks, dependency/licensing review 
and processor documentation.

h3. Related work

* MINIFICPP-819 - OPC Unified Architecture Support.
* MINIFICPP-893 - OPC UA Client Implementation.
* MINIFICPP-2908 - Add processor to fetch OPC events.

These concern MiNiFi C++. This proposal targets the Java NiFi runtime and Value 
data-change subscriptions.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to