Hi I am new to Spark. I am confused between correlation in threads and physical cores.
As per my understanding, according to number of partitions in data set, number of tasks is created. For example I have a machine which has 10 physical cores and I have data set which has 100 partitions then in Executor JVM 100 tasks (one per each partitioner will be created) Query 1) But how will it be decided how many threads in Executor are created to execute these 100 tasks and who creates these threads. Query 2) Does parameter "total-executor-cores" define how many threads will be launched in executor JVM to process tasks. If not than what is meaning of "total-executor-cores" in context of both threads inside Executor JVM and physical cores. Thanks and Regards Aniruddh
