[
https://issues.apache.org/jira/browse/SYSTEMML-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Schüler updated SYSTEMML-1127:
------------------------------------
Description:
When executing a Spark parfor, the SparkParforWorker throws an exception which
states that the localtmpdir could not be created. This is due to the fact that
multiple executors are running multithreaded on the same worker.
The createDistributedUniqueID() method in the IDHander.java creates unique IDs
only per pid and host, not per thread. This could potentially be solved by
adding the threadID to the unique ID. The question is if every thread should
have its own cache or if the logic should be changed so that the first creation
will be successful and then the threads share one cache.
was:
When executing a Spark parfor, the SparkParforWorker throws an exception which
states that the localtmpdir could not be created. This is due to the fact that
multiple executors are running multithreaded on the same worker.
The createDistributedUniqueID() method in the IDHander.java creates unique IDs
only per pid and host, not per thread. This could potentially be solved by
adding the threadID to the unique ID.
> Distributed unique IDs are not unique
> -------------------------------------
>
> Key: SYSTEMML-1127
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1127
> Project: SystemML
> Issue Type: Bug
> Components: ParFor
> Reporter: Felix Schüler
>
> When executing a Spark parfor, the SparkParforWorker throws an exception
> which states that the localtmpdir could not be created. This is due to the
> fact that multiple executors are running multithreaded on the same worker.
> The createDistributedUniqueID() method in the IDHander.java creates unique
> IDs only per pid and host, not per thread. This could potentially be solved
> by adding the threadID to the unique ID. The question is if every thread
> should have its own cache or if the logic should be changed so that the first
> creation will be successful and then the threads share one cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
