Hi Chris, There’s a YouTube video [1] that I created a few months ago that discusses a lot of the considerations to take into account when determining the number of threads to allocate to the overall thread pool and how many to allocate to individual processors. Unfortunately, it’s pretty non-trivial. One thing to note, though, is that the number of nodes shouldn’t really determine the size of the thread pool, as the thread pool is per-node.
Thanks -Mark [1] https://www.youtube.com/watch?v=pZq0EbfDBy4 On Jan 25, 2021, at 12:05 PM, Chris Nava <[email protected]<mailto:[email protected]>> wrote: Good morning from Denver! I'm having a hard time trying to find the answer within the documentation online. I am fairly new to NiFi and was trying to research the possibility of adjusting the maximum thread count of a process based on the environment and the node size. We have the following code and I FEEL like doing some kind of request to see how many nodes are on the given process and then replacing the thread_count with some equation based on the return is the way to go. Am I on the right track or way off-base here?: def update_thread(): """ Updates the overall thread count for NiFi """ if (os.getenv('NIFI_THREAD_OVERRIDE') == 'true'): thread_count = int(os.getenv('NIFI_THREAD_COUNT')) else: thread_count = int(round(multiprocessing.cpu_count()*0.8*4)) api = nipyapi.nifi.apis.controller_api.ControllerApi() controller = api.get_controller_config() controller.component = nipyapi.nifi.models.controller_configuration_dto.ControllerConfigurationDTO(thread_count) api.update_controller_config(controller)
