Poor Performance with Increasing Public Thread Pool Size

[Adrian Corman]

Hi Igniters,

We are using the .net API for Ignite to distribute computing tasks (closures) 
to a cluster. We are having an issue involving the public thread pool size - 
when the tasks are sent to a machine with the default public thread pool size 
(48 processors, 48  threads in the case of our test machine), the jobs run 10x 
slower than if we set the public thread pool to a smaller number such as 6 or 
12. This slowdown doesn't seem to be related to any cache operations or IO - 
logging shows that even pure computational methods are slower overall.

We also discovered that when we run two nodes on the test machine, each with 
half the thread pool size, the jobs run faster than with one node with the full 
pool size (two of 24 run faster than one of 48).

Does anyone have any suggestions on how we can trace the source of this 
slowdown, or otherwise improve performance? I've included scaffolding for the 
types of tests we are running, to illustrate our issue.

Thank you!

public class PerfTest_Ignite
    {
        private IIgnite ignite;

        public void Test_PublicThreadPool_12()
        {
            ignite = Ignition.Start(new IgniteConfiguration
            {
                PublicThreadPoolSize = 12
            });
            string[] tasks = new string[100000];

            var result = ignite.GetCompute().ApplyAsync(new 
LongRunning_DotNet_Task(), tasks);
            // This method performs as expected
        }

        public void Test_PublicThreadPool_48()
        {
            ignite = Ignition.Start(new IgniteConfiguration
            {
                //PublicThreadPoolSize = 48. Do not set Public Thread Pool 
size. (48 is the number of cores on this machine. )
            });
            string[] tasks = new string[100000];

            var result = ignite.GetCompute().ApplyAsync(new 
LongRunning_DotNet_Task(), tasks);

            // This method is 10 TIMES SLOWER?
        }


    }


    public class LongRunning_DotNet_Task : IComputeFunc<string[], bool>
    {
        public bool Invoke(string[] arg)
        {
            // do work that takes up to 5 minutes with 12 threads, but takes 50 
minutes with 48 threads!
            return true;
       }
    }