joewitt commented on a change in pull request #3676: NIFI-6597 Azure Event Hub
Version Update
URL: https://github.com/apache/nifi/pull/3676#discussion_r335112998
##########
File path:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java
##########
@@ -312,11 +317,13 @@ public void onScheduled(final ProcessContext context)
throws ProcessException, U
receiverFetchTimeout = null;
}
- final String connectionString = new ConnectionStringBuilder(new
URI("amqps://"+namespace+serviceBusEndpoint), eventHubName, policyName,
policyKey).toString();
- setupReceiver(connectionString);
+ final int numThreads = context.getMaxConcurrentTasks();
+ executor = Executors.newScheduledThreadPool(numThreads);
Review comment:
@mysunnytime it is fine to use the onScheduled to create the necessary
connections to Azure event hubs. For pushing to event hubs it is probably
better to use lazy instantiation but in the consume case eager makes more sense
and the lifecycle hook of onScheduled is fine.
As far as NiFi and thread pooling yes NiFi has a thread pool. Basically the
whole flow controller is a thread pool. It isn't generally desired for us to
create threads via a new thread pool and give it to some library such as the
event hubs sdk. Rather we'd like to bring a thread to the library to do some
meaningful work (fetch data) then process it and repeat. That allows us to
have control over the threads, share them as necessary for splitting work,
etc.. while allowing the user to control overall thread count in the system.
Are you certain the only API for leveraging the event hubs SDK is one where
we have to hand it over a pool of threads it gets to own then we offer event
handlers as data arrives? How would back-pressure work in that case? We dont
want to keep getting data when we're not ready to process it... We much prefer
a model where we can use our thread to go grab data available and ready and
then go do other work and come back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
