[
https://issues.apache.org/jira/browse/SYSTEMML-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693344#comment-15693344
]
Matthias Boehm commented on SYSTEMML-1127:
------------------------------------------
just to keep track of the open issues: the parfor remote_spark also misses a
proper cleanup (as workers cannot cleanup before all workers of one executor
finished). I can take care of that once I'm back.
> Distributed unique IDs are not unique
> -------------------------------------
>
> Key: SYSTEMML-1127
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1127
> Project: SystemML
> Issue Type: Bug
> Components: ParFor
> Reporter: Felix Schüler
>
> When executing a Spark parfor, the SparkParforWorker throws an exception
> which states that the localtmpdir could not be created. This is due to the
> fact that multiple executors are running multithreaded on the same worker.
> The createDistributedUniqueID() method in the IDHander.java creates unique
> IDs only per pid and host, not per thread. This could potentially be solved
> by adding the threadID to the unique ID. The question is if every thread
> should have its own cache or if the logic should be changed so that the first
> creation will be successful and then the threads share one cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
