Paul-Adrien Cordonnier created NIFI-13544:
---------------------------------------------
Summary: Add configuration to set the size of the threadpool of
pubsub publish
Key: NIFI-13544
URL: https://issues.apache.org/jira/browse/NIFI-13544
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.27.0
Reporter: Paul-Adrien Cordonnier
The GCP java-pubsub library has a default behaviour of creating a threadpool of
5 threads per processor on the machine. On a typical production server it can
be 200 threads per processor, mutliplied by the number of GCP publish processor
it can very well overwhelm a JVM.
[Link to the gcp library
snippet|https://github.com/googleapis/java-pubsub/blob/4e17c34880d0498d2bd891ea7caa8fe095054016/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Publisher.java#L712-L716]
Similar to whats its done with
[putkudu|https://github.com/apache/nifi/blob/support/nifi-1.x/nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java#L148].
I think we should be able to set the size of the threadpool as a property of
the processor. Default behaviour being the one in pubsub-library so that there
are no impact on performance after upgrading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
