Martin Šimek created NIFI-6998:
----------------------------------
Summary: PutAzureEventHub to utilize Batch, Properties &
Partitioning key
Key: NIFI-6998
URL: https://issues.apache.org/jira/browse/NIFI-6998
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.8.0
Reporter: Martin Šimek
Fix For: 1.11.0
Current implementation of _PutAzureEventHub_ is too simplistic. It sends one
flow file at a time synchronously which poses a significant bottlenec. It
doesn't lift attributes into the message's _User defined properties_ section.
And most important problem is that you can't specify the _partitioning key_
which plays critical role in a delivery consistency across multiple partitions
in case there are more partitions. Without partitioning key multiple use-cases
can't choose NiFi as an option.
Aim of this improvement is to consume up to specified amount of flow files,
lift all _attributes_ of each flow file into _User defined properties_ section
of each message and let processor to use one of those attributes as a
partitioning key.
Each flow file in a batch is sent asynchronously which is way more faster then
processor wait for all confirmations (or errors) and based on the result
messages are transfered to success or failure output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
