Simon Theodosius created NIFI-16583:
---------------------------------------
Summary: Add ConsumeOPCUA processor for OPC UA data-change
subscriptions
Key: NIFI-16583
URL: https://issues.apache.org/jira/browse/NIFI-16583
Project: Apache NiFi
Issue Type: New Feature
Components: Extensions
Reporter: Simon Theodosius
h3. Motivation
OPC UA servers expose industrial measurements and their quality and timestamps.
A NiFi source processor that subscribes to value changes would allow these
measurements to enter a dataflow directly, with configurable sampling and
batching.
I propose adding ConsumeOPCUA using Eclipse Milo and adapting an existing
standalone implementation to the current Apache NiFi codebase.
h3. Proposed scope
* Subscribe to OPC UA Value data changes over opc.tcp and emit initial values
and subsequent changes as batched FlowFiles.
* Configure tags using namespace-index or namespace-URI NodeIds, with aliases,
sampling intervals, server queue sizes and deadbands.
* Include tag identity, value, status and timestamps in the output.
* Support anonymous and username/password authentication, explicit security
policy and message security mode selection, and client application
certificates. Full server certificate validation should be the default.
* Support connection verification, reconnection and subscription recreation,
with primary-node execution in clustered NiFi.
OPC UA Events/alarms, historical reads, writes and durable recovery are outside
the proposed initial scope.
h3. Delivery and resource limits
Bound retained notifications by count and serialized bytes. Keep batches
reserved and counted against both limits until NiFi reports the commit outcome,
and report buffer overflow and server-reported notification loss. These limits
do not bound all decoded protocol objects or total JVM heap.
Shutdown should stop accepting notifications and make a time-limited attempt to
drain buffered data. This does not impose a deadline on NiFi repository
operations or the overall processor stop.
Uncommitted notifications remain in memory and are not replicated. Outages,
overflow, JVM termination and primary-node failover can cause gaps or
duplicates. The processor would not provide end-to-end exactly-once or lossless
outage recovery.
h3. Existing implementation and design questions
The standalone implementation is available at
https://github.com/paf91/nifi-opcua and is currently published under the
processor name SubscribeOpcUa. ConsumeOPCUA is the proposed upstream name.
Integration into the Apache NiFi source tree and dependency/licensing review
remain to be completed.
I would appreciate maintainer guidance on two initial design choices:
* Start with JSON/NDJSON output, or expose a RecordSetWriterFactory in the
first contribution?
* Keep OPC UA identity/trust configuration on the processor, or introduce an
OPC UA-specific Controller Service?
h3. Proposed verification
* Exercise supported authentication/security combinations and certificate
failures.
* Cover namespace resolution, value encoding, reconnect/resubscribe, buffer
saturation, downstream backpressure, commit failures and Stop/Start races.
* Measure memory and CPU use under representative load and repeated connection
failures.
* Verify interoperability with an independent OPC UA server such as open62541
and load the resulting NAR in NiFi.
* Complete the current NiFi contribution checks, dependency/licensing review
and processor documentation.
h3. Related work
* MINIFICPP-819 - OPC Unified Architecture Support.
* MINIFICPP-893 - OPC UA Client Implementation.
* MINIFICPP-2908 - Add processor to fetch OPC events.
These concern MiNiFi C++. This proposal targets the Java NiFi runtime and Value
data-change subscriptions.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)