Re: [R] mclapply enters into an infinite loop....
Dear Ivan,
I was on RHEL 8 with aws Ec2 z1d instance.
I infer by your words that this "bug" happens only sometimes..But I
disagree...I run the same code multiple times within a span of 5 minutes, and
the problem persistedAnyway I have removed the bug in the function and I
hope that mclapply runs properly now( if it doesn't you are my only saviour!)
Many thanks for your geekish explanation.
THanking you,
Yours sincerely,
AKSHAY M KULKARNI
From: Ivan Krylov
Sent: Saturday, May 20, 2023 6:00 PM
To: akshay kulkarni
Cc: r-help@r-project.org
Subject: Re: [R] mclapply enters into an infinite loop
� Sat, 20 May 2023 10:59:18 +
akshay kulkarni �:
> By "holding a lock", you mean a bug in the process right
Well... one person's bug ("your threaded program breaks if I fork() the
process") is another person's documented behaviour ("of course it
does, it says 'please do not fork()' right in the manual"). Depending
on how you're running R (which operating system? are you using a
front-end? a multi-threaded BLAS?), this may be applicable, hence the
question about sessionInfo().
A GUI program may need multiple threads in order to stay responsive
(without looking hung and forgetting to repaint its window or
something) while doing something significant in the background. A
multi-threaded linear algebra library may create a thread per CPU core
in order to take your matrix products faster. Inside a process, some
resources have to be shared between the threads, like the bathroom in a
flat rented by multiple people together. "Holding a lock" is like
taking the key to the bathroom with the intent to occupy it for a
while: a completely normal action that prevents embarrassing accidents.
When you fork() a process, though, it ends up creating an exact copy of
the flat (process) just for you (the current thread), with none of the
flat-mates (other threads). If the bathroom was locked by someone else,
it stays locked, and there's no key anywhere.
There do exist mechanisms like pthread_atfork() that could be used to
prevent problems like this, but for that to work, *every* piece of code
that may be creating threads and taking locks inside your process
should be using them right. (The flat-mates need to know to give you
any keys they might be holding before the flat is fork()ed. It's enough
for one of them to forget one key to cause a problem.) The pragmatic
solution may be to avoid fork() and threads being used together.
--
Best regards,
Ivan
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] mclapply enters into an infinite loop....
В Sat, 20 May 2023 10:59:18 +
akshay kulkarni пишет:
> By "holding a lock", you mean a bug in the process right
Well... one person's bug ("your threaded program breaks if I fork() the
process") is another person's documented behaviour ("of course it
does, it says 'please do not fork()' right in the manual"). Depending
on how you're running R (which operating system? are you using a
front-end? a multi-threaded BLAS?), this may be applicable, hence the
question about sessionInfo().
A GUI program may need multiple threads in order to stay responsive
(without looking hung and forgetting to repaint its window or
something) while doing something significant in the background. A
multi-threaded linear algebra library may create a thread per CPU core
in order to take your matrix products faster. Inside a process, some
resources have to be shared between the threads, like the bathroom in a
flat rented by multiple people together. "Holding a lock" is like
taking the key to the bathroom with the intent to occupy it for a
while: a completely normal action that prevents embarrassing accidents.
When you fork() a process, though, it ends up creating an exact copy of
the flat (process) just for you (the current thread), with none of the
flat-mates (other threads). If the bathroom was locked by someone else,
it stays locked, and there's no key anywhere.
There do exist mechanisms like pthread_atfork() that could be used to
prevent problems like this, but for that to work, *every* piece of code
that may be creating threads and taking locks inside your process
should be using them right. (The flat-mates need to know to give you
any keys they might be holding before the flat is fork()ed. It's enough
for one of them to forget one key to cause a problem.) The pragmatic
solution may be to avoid fork() and threads being used together.
--
Best regards,
Ivan
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] mclapply enters into an infinite loop....
Dear Ivan, REgrets to reply this late... By "holding a lock", you mean a bug in the process right (I am not a computer science guy, excuse my naivete)? THanking you, Yours sincerely, AKSHAY M KULKARNI From: Ivan Krylov Sent: Thursday, May 18, 2023 1:08 PM To: akshay kulkarni Cc: r-help@r-project.org Subject: Re: [R] mclapply enters into an infinite loop On Wed, 17 May 2023 13:55:59 + akshay kulkarni wrote: > So that means mclapply should run properly, i.e output a try class > object and exit. But it didn't. Can you shed some light on why this > happened? What's your sessionInfo()? Are you using a GUI frontend? mclapply() relies on the fork() system call, which is tricky to get right in a process where other threads may exist at the same time: when a process calls fork(), the child process ends up with the same contents of virtual memory but no other threads at all. If a thread was holding a lock when the process was forked, the lock will never be released in the child process, leaving it stuck. Depending on how you're running R, it may be threading issues or something else. More information may be obtained by looking at traceback() after you interrupt mclapply() (easy, but low information: did you interrupt selectChildren() or some other function?) and attaching a C debugger to the child process when you become sure it's stuck (hard, requires additional preparation such as installing debugging symbols). -- Best regards, Ivan [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] mclapply enters into an infinite loop....
On Wed, 17 May 2023 13:55:59 + akshay kulkarni wrote: > So that means mclapply should run properly, i.e output a try class > object and exit. But it didn't. Can you shed some light on why this > happened? What's your sessionInfo()? Are you using a GUI frontend? mclapply() relies on the fork() system call, which is tricky to get right in a process where other threads may exist at the same time: when a process calls fork(), the child process ends up with the same contents of virtual memory but no other threads at all. If a thread was holding a lock when the process was forked, the lock will never be released in the child process, leaving it stuck. Depending on how you're running R, it may be threading issues or something else. More information may be obtained by looking at traceback() after you interrupt mclapply() (easy, but low information: did you interrupt selectChildren() or some other function?) and attaching a C debugger to the child process when you become sure it's stuck (hard, requires additional preparation such as installing debugging symbols). -- Best regards, Ivan __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] mclapply enters into an infinite loop....
Dear Jeff, There was a problem in LYGH and lapply threw an error, but mclapply got stuck in an infinite loop. The doc for mclapply says that mclapply runs under try() with silent = TRUE. So that means mclapply should run properly, i.e output a try class object and exit. But it didn't. Can you shed some light on why this happened? THanking you, Yours sincerely, AKSHAY M KULKARNI From: R-help on behalf of Jeff Newmiller Sent: Wednesday, May 17, 2023 5:24 AM To: r-help@r-project.org Subject: Re: [R] mclapply enters into an infinite loop It does not look to me like you are providing the necessary arguments to arfima. Try making this work with lapply first... then try mclapply. On May 16, 2023 3:10:45 PM PDT, akshay kulkarni wrote: >Dear members, > I am using arfima in an mclapply construction (from > the parallel package): > >Browse[2]> LYG <- mclapply(LYGH, FUN = arfima, mc.cores = detectCores()) >^C >Browse[2]> LYG <- mclapply(LYGH[1:10], FUN = arfima, mc.cores = detectCores()) >^C >Browse[2]> LYG <- mclapply(LYGH[1:2], FUN = arfima, mc.cores = detectCores()) >^C > >You can see that I am aborting the execution of mclapply. It doesn't finish >even if I reduce the elements to be iterated over to 2. Why is it entering an >infinite loop? > >Please exhort me if this is to be posted in HPC list. > >THanking you, >Yours sincerely, >ALSHAY M KULKARNI > > [[alternative HTML version deleted]] > >__ >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] mclapply enters into an infinite loop....
It does not look to me like you are providing the necessary arguments to arfima. Try making this work with lapply first... then try mclapply. On May 16, 2023 3:10:45 PM PDT, akshay kulkarni wrote: >Dear members, > I am using arfima in an mclapply construction (from > the parallel package): > >Browse[2]> LYG <- mclapply(LYGH, FUN = arfima, mc.cores = detectCores()) >^C >Browse[2]> LYG <- mclapply(LYGH[1:10], FUN = arfima, mc.cores = detectCores()) >^C >Browse[2]> LYG <- mclapply(LYGH[1:2], FUN = arfima, mc.cores = detectCores()) >^C > >You can see that I am aborting the execution of mclapply. It doesn't finish >even if I reduce the elements to be iterated over to 2. Why is it entering an >infinite loop? > >Please exhort me if this is to be posted in HPC list. > >THanking you, >Yours sincerely, >ALSHAY M KULKARNI > > [[alternative HTML version deleted]] > >__ >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] mclapply enters into an infinite loop....
Dear members, I am using arfima in an mclapply construction (from the parallel package): Browse[2]> LYG <- mclapply(LYGH, FUN = arfima, mc.cores = detectCores()) ^C Browse[2]> LYG <- mclapply(LYGH[1:10], FUN = arfima, mc.cores = detectCores()) ^C Browse[2]> LYG <- mclapply(LYGH[1:2], FUN = arfima, mc.cores = detectCores()) ^C You can see that I am aborting the execution of mclapply. It doesn't finish even if I reduce the elements to be iterated over to 2. Why is it entering an infinite loop? Please exhort me if this is to be posted in HPC list. THanking you, Yours sincerely, ALSHAY M KULKARNI [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
