Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
As for other calls to system. I avoid calling system. In the past I had some (to get memory stats from OS), but they were failing with exactly the same issue. So yes, if I would add call to system before calling quit, I believe it would fail with the same error. At the same time I think (although I am not sure) that new allocations made in R are working fine. So R seems to reserve some memory and can continue to operate, while external call like system will fail. Maybe it is like this by design, don't know. Aside from this problem that is easy to report due to the warning message, I think that gc() is choking at the same time. I tried to make reproducible example for that, multiple times but couldn't, let me try one more time. It happens to manifest when there is 4e8+ unique characters/factors in an R session. I am able to reproduce it using data.table and dplyr (0.8.4 because 1.0.0+ fails even sooner), but using base R is not easy because of the size. I described briefly problem in: https://github.com/h2oai/db-benchmark/issues/110 It would help if gcinfo() could take FALSE/TRUE/2L where 2L will print even more information about gc, like how much time the each gc() process took, how many objects it has to check on each level. Best regards, Jan On Tue, Nov 24, 2020 at 1:05 PM Tomas Kalibera wrote: > > On 11/24/20 11:27 AM, Jan Gorecki wrote: > > Thanks Bill for checking that. > > It was my impression that warnings are raised from some internal > > system calls made when quitting R. At that point I don't have much > > control over checking the return status of those. > > Your suggestion looks good to me. > > > > Tomas, do you think this could help? could this be implemented? > > I think this is a good suggestion. Deleting files on Unix was changed > from system("rm") to doing that in C, and deleting the session directory > should follow. > > It might also help diagnosing your problem, but I don't think it would > solve it. If the diagnostics in R works fine and the OS was so > hopelessly out of memory that it couldn't run any more external > processes, then really this is not a problem of R, but of having > exhausted the resources. And it would be a coincidence that just this > particular call to "system" at the end of the session did not work. > Anything else could break as well close to the end of the script. This > seems the most likely explanation to me. > > Do you get this warning repeatedly, reproducibly at least in slightly > different scripts at the very end, with this warning always from quit()? > So that the "call" part of the warning message has .Internal(quit) like > in the case you posted? Would adding another call to "system" before the > call to "q()" work - with checking the return value? If it is always > only the last call to "system" in "q()", then it is suspicious, perhaps > an indication that some diagnostics in R is not correct. In that case, a > reproducible example would be the key - so either if you could diagnose > on your end what is the problem, or create a reproducible example that > someone else can use to reproduce and debug. > > Best > Tomas > > > > > On Mon, Nov 23, 2020 at 7:10 PM Bill Dunlap > > wrote: > >> The call to system() probably is an internal call used to delete the > >> session's tempdir(). This sort of failure means that a potentially large > >> amount of disk space is not being recovered when R is done. Perhaps > >> R_CleanTempDir() could call R_unlink() instead of having a subprocess call > >> 'rm -rf ...'. Then it could also issue a specific warning if it was > >> impossible to delete all of tempdir(). (That should be very rare.) > >> > >>> q("no") > >> Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf > >> /tmp/RtmppoKPXb") at sysutils.c:311 > >> 311 { > >> (gdb) where > >> #0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf > >> /tmp/RtmppoKPXb") at sysutils.c:311 > >> #1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178 > >> #2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0, > >> runLast=) at sys-std.c:1243 > >> #3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE, > >> status=status@entry=0, runLast=) at system.c:87 > >> #4 0x556cc85e in do_quit (call=, op= >> out>, args=0x57813f90, rho=) at main.c:1393 > >> > >> -Bill > >> > >> On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera > >> wrote: > >>> On 11/21/20 6:51 PM, Jan Gorecki wrote: > Dear R-developers, > > Some of the more fat scripts (50+ GB mem used by R) that I am running, > when they finish they do quit with q("no", status=0) > Quite often it happens that there is an extra stderr output produced > at the very end which looks like this: > > Warning message: > In .Internal(quit(save, status, runLast)) : > system call failed: Cannot allocate memory > > Is there any way to avoid this kind of warnings? I am using stderr > output for
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
On 11/24/20 11:27 AM, Jan Gorecki wrote: Thanks Bill for checking that. It was my impression that warnings are raised from some internal system calls made when quitting R. At that point I don't have much control over checking the return status of those. Your suggestion looks good to me. Tomas, do you think this could help? could this be implemented? I think this is a good suggestion. Deleting files on Unix was changed from system("rm") to doing that in C, and deleting the session directory should follow. It might also help diagnosing your problem, but I don't think it would solve it. If the diagnostics in R works fine and the OS was so hopelessly out of memory that it couldn't run any more external processes, then really this is not a problem of R, but of having exhausted the resources. And it would be a coincidence that just this particular call to "system" at the end of the session did not work. Anything else could break as well close to the end of the script. This seems the most likely explanation to me. Do you get this warning repeatedly, reproducibly at least in slightly different scripts at the very end, with this warning always from quit()? So that the "call" part of the warning message has .Internal(quit) like in the case you posted? Would adding another call to "system" before the call to "q()" work - with checking the return value? If it is always only the last call to "system" in "q()", then it is suspicious, perhaps an indication that some diagnostics in R is not correct. In that case, a reproducible example would be the key - so either if you could diagnose on your end what is the problem, or create a reproducible example that someone else can use to reproduce and debug. Best Tomas On Mon, Nov 23, 2020 at 7:10 PM Bill Dunlap wrote: The call to system() probably is an internal call used to delete the session's tempdir(). This sort of failure means that a potentially large amount of disk space is not being recovered when R is done. Perhaps R_CleanTempDir() could call R_unlink() instead of having a subprocess call 'rm -rf ...'. Then it could also issue a specific warning if it was impossible to delete all of tempdir(). (That should be very rare.) q("no") Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb") at sysutils.c:311 311 { (gdb) where #0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb") at sysutils.c:311 #1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178 #2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0, runLast=) at sys-std.c:1243 #3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE, status=status@entry=0, runLast=) at system.c:87 #4 0x556cc85e in do_quit (call=, op=, args=0x57813f90, rho=) at main.c:1393 -Bill On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera wrote: On 11/21/20 6:51 PM, Jan Gorecki wrote: Dear R-developers, Some of the more fat scripts (50+ GB mem used by R) that I am running, when they finish they do quit with q("no", status=0) Quite often it happens that there is an extra stderr output produced at the very end which looks like this: Warning message: In .Internal(quit(save, status, runLast)) : system call failed: Cannot allocate memory Is there any way to avoid this kind of warnings? I am using stderr output for detecting failures in scripts and this warning is a false positive of a failure. Maybe quit function could wait little bit longer trying to allocate before it raises this warning? If you see this warning, some call to system() or system2() or similar, which executes an external program, failed to even run a shell to run that external program, because there was not enough memory. You should be able to find out where it happens by checking the exit status of system(). Tomas Best regards, Jan Gorecki __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
Thanks Bill for checking that. It was my impression that warnings are raised from some internal system calls made when quitting R. At that point I don't have much control over checking the return status of those. Your suggestion looks good to me. Tomas, do you think this could help? could this be implemented? On Mon, Nov 23, 2020 at 7:10 PM Bill Dunlap wrote: > > The call to system() probably is an internal call used to delete the > session's tempdir(). This sort of failure means that a potentially large > amount of disk space is not being recovered when R is done. Perhaps > R_CleanTempDir() could call R_unlink() instead of having a subprocess call > 'rm -rf ...'. Then it could also issue a specific warning if it was > impossible to delete all of tempdir(). (That should be very rare.) > > > q("no") > Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf > /tmp/RtmppoKPXb") at sysutils.c:311 > 311 { > (gdb) where > #0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb") > at sysutils.c:311 > #1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178 > #2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0, > runLast=) at sys-std.c:1243 > #3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE, > status=status@entry=0, runLast=) at system.c:87 > #4 0x556cc85e in do_quit (call=, op=, > args=0x57813f90, rho=) at main.c:1393 > > -Bill > > On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera > wrote: >> >> On 11/21/20 6:51 PM, Jan Gorecki wrote: >> > Dear R-developers, >> > >> > Some of the more fat scripts (50+ GB mem used by R) that I am running, >> > when they finish they do quit with q("no", status=0) >> > Quite often it happens that there is an extra stderr output produced >> > at the very end which looks like this: >> > >> > Warning message: >> > In .Internal(quit(save, status, runLast)) : >> >system call failed: Cannot allocate memory >> > >> > Is there any way to avoid this kind of warnings? I am using stderr >> > output for detecting failures in scripts and this warning is a false >> > positive of a failure. >> > >> > Maybe quit function could wait little bit longer trying to allocate >> > before it raises this warning? >> >> If you see this warning, some call to system() or system2() or similar, >> which executes an external program, failed to even run a shell to run >> that external program, because there was not enough memory. You should >> be able to find out where it happens by checking the exit status of >> system(). >> >> Tomas >> >> >> > >> > Best regards, >> > Jan Gorecki >> > >> > __ >> > R-devel@r-project.org mailing list >> > https://stat.ethz.ch/mailman/listinfo/r-devel >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
The call to system() probably is an internal call used to delete the session's tempdir(). This sort of failure means that a potentially large amount of disk space is not being recovered when R is done. Perhaps R_CleanTempDir() could call R_unlink() instead of having a subprocess call 'rm -rf ...'. Then it could also issue a specific warning if it was impossible to delete all of tempdir(). (That should be very rare.) > q("no") Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb") at sysutils.c:311 311 { (gdb) where #0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb") at sysutils.c:311 #1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178 #2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0, runLast=) at sys-std.c:1243 #3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE, status=status@entry=0, runLast=) at system.c:87 #4 0x556cc85e in do_quit (call=, op=, args=0x57813f90, rho=) at main.c:1393 -Bill On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera wrote: > On 11/21/20 6:51 PM, Jan Gorecki wrote: > > Dear R-developers, > > > > Some of the more fat scripts (50+ GB mem used by R) that I am running, > > when they finish they do quit with q("no", status=0) > > Quite often it happens that there is an extra stderr output produced > > at the very end which looks like this: > > > > Warning message: > > In .Internal(quit(save, status, runLast)) : > >system call failed: Cannot allocate memory > > > > Is there any way to avoid this kind of warnings? I am using stderr > > output for detecting failures in scripts and this warning is a false > > positive of a failure. > > > > Maybe quit function could wait little bit longer trying to allocate > > before it raises this warning? > > If you see this warning, some call to system() or system2() or similar, > which executes an external program, failed to even run a shell to run > that external program, because there was not enough memory. You should > be able to find out where it happens by checking the exit status of > system(). > > Tomas > > > > > > Best regards, > > Jan Gorecki > > > > __ > > R-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
Try explicitly deleting large data objects by calling `rm`, then `gc`. On Mon, Nov 23, 2020 at 6:15 AM Tomas Kalibera wrote: > On 11/21/20 6:51 PM, Jan Gorecki wrote: > > Dear R-developers, > > > > Some of the more fat scripts (50+ GB mem used by R) that I am running, > > when they finish they do quit with q("no", status=0) > > Quite often it happens that there is an extra stderr output produced > > at the very end which looks like this: > > > > Warning message: > > In .Internal(quit(save, status, runLast)) : > >system call failed: Cannot allocate memory > > > > Is there any way to avoid this kind of warnings? I am using stderr > > output for detecting failures in scripts and this warning is a false > > positive of a failure. > > > > Maybe quit function could wait little bit longer trying to allocate > > before it raises this warning? > > If you see this warning, some call to system() or system2() or similar, > which executes an external program, failed to even run a shell to run > that external program, because there was not enough memory. You should > be able to find out where it happens by checking the exit status of > system(). > > Tomas > > > > > > Best regards, > > Jan Gorecki > > > > __ > > R-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > -- "Whereas true religion and good morals are the only solid foundations of public liberty and happiness . . . it is hereby earnestly recommended to the several States to take the most effectual measures for the encouragement thereof." Continental Congress, 1778 [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
On 11/21/20 6:51 PM, Jan Gorecki wrote: Dear R-developers, Some of the more fat scripts (50+ GB mem used by R) that I am running, when they finish they do quit with q("no", status=0) Quite often it happens that there is an extra stderr output produced at the very end which looks like this: Warning message: In .Internal(quit(save, status, runLast)) : system call failed: Cannot allocate memory Is there any way to avoid this kind of warnings? I am using stderr output for detecting failures in scripts and this warning is a false positive of a failure. Maybe quit function could wait little bit longer trying to allocate before it raises this warning? If you see this warning, some call to system() or system2() or similar, which executes an external program, failed to even run a shell to run that external program, because there was not enough memory. You should be able to find out where it happens by checking the exit status of system(). Tomas Best regards, Jan Gorecki __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
On 21/11/2020 12:51 p.m., Jan Gorecki wrote: Dear R-developers, Some of the more fat scripts (50+ GB mem used by R) that I am running, when they finish they do quit with q("no", status=0) Quite often it happens that there is an extra stderr output produced at the very end which looks like this: Warning message: In .Internal(quit(save, status, runLast)) : system call failed: Cannot allocate memory Is there any way to avoid this kind of warnings? I am using stderr output for detecting failures in scripts and this warning is a false positive of a failure. Maybe quit function could wait little bit longer trying to allocate before it raises this warning? I don't know what waiting would accomplish. Generally speaking the allocation functions in R will try garbage collection before failing, so it looks like you are in a situation where there really is no memory available. (I think code can prevent gc; maybe your code is doing that and not re-enabling it?) Having a reproducible example would help, but I imagine it's not easy to put one together. Duncan Murdoch __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
Dear R-developers, Some of the more fat scripts (50+ GB mem used by R) that I am running, when they finish they do quit with q("no", status=0) Quite often it happens that there is an extra stderr output produced at the very end which looks like this: Warning message: In .Internal(quit(save, status, runLast)) : system call failed: Cannot allocate memory Is there any way to avoid this kind of warnings? I am using stderr output for detecting failures in scripts and this warning is a false positive of a failure. Maybe quit function could wait little bit longer trying to allocate before it raises this warning? Best regards, Jan Gorecki __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel