Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Thanks again Hugo,
2011/1/19 Hugo Mildenberger :
> Dear Mauricio,
>
> what I do not understand at all is the message:
>
>> Error: cannot allocate vector of size 476.2 Mb
>
> Have you tried to allocate a big matrix in between, whith
> the R - statement not being shown in the output?
If I understood correctly your question, in between the lines I showed
you before:
print(gc())
print(gcinfo(TRUE))
system("/dataMZB/2011/memorytest.out")
I didn't try to allocate a matrix, and the R output was a message
printed just before those three lines.
> Probably not. If not, your local R version is buggy for sure. The test shows
> that the OS imposes no artificial constraint on memory consumption,
> at least not up to ~ 1GB. But because you said that you're running a
> 32 - bit version of R (in a virtual machine?),
I'm running a 32 bits OS in a 64 bits hardware, by using a kernel with
PAE extension, in order to see the 12 Gb of ram that I have:
$ uname -a
Linux mymachine 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010
i686 i686 i386 GNU/Linux
> the question is now wether gcc also compiles the test program as a 32 - bit
> program. I'm
> unsure because a 64 - bit system can be configured to compile and
> run programs using either environment. Regrettably I previously
> did not include a test for that.
>
> --- snip ---
> #include
> int main() {
> printf("The size of a pointer is %lu bytes\n", (unsigned
> long)sizeof(void *));
> return 0;
> }
> --- snip ---
The output of that program was:
"The size of a pointer is 4 bytes"
so, I confirmed that I'm running a 32 bits OS.
>
> If the program says "The size of a pointer is 4 bytes" then you're
> running the test program in a 32 bit environment. Then the
> allocation problem could possibly be solved by recompiling/updating
> your R version.
I think is the right way of solving this.
At the other hand, yesterday a friend suggested me to install R in a
live CD with a 64 GNU/Linux OS, to run my script and see if I get the
same error. I'll also try this.
> The system is called Redhat "Enterprise" Linux after all, so calling in
> Redhat support at that point and ask them to do
> something for their money should be an option.
What is sadly funny, is the fact that when I asked the IT guys of my
work if it is possible to ask support for "software"
compilation/installation in RHLE, they told me that is NOT possible,
because the money they are paying in licences is for network
maintenanceand other corporative stuff.
So far, RHLE only have brought problems to me (this is one of "many"),
problems that I never had while using Ubuntu, but unfortunately at
work RHEL was the only possible choice.
I'll let you know if I'm able to overcome this issue.
Thanks a lot for your help,
Cheers,
Mauricio
--
===
Linux user #454569 -- Ubuntu user #17469
===
>
> Best
>
> Hugo
>
>
>
> On Wednesday 19 January 2011 12:54:38 Mauricio Zambrano wrote:
>> Dear Hugo,
>>
>> I tried your memory test program (without further modifications) just
>> after the gc() command:
>>
>> print(gc())
>> print(gcinfo(TRUE))
>> system("/mypath/memorytest.out")
>>
>> and the result that I got was:
>>
>> Number of simulations read from ' Particles.txt ' : 9000
>> ---
>> used (Mb) gc trigger (Mb) max used (Mb)
>> Ncells 480114 12.9 1816453 48.6 5543382 148.1
>> Vcells 5256970 40.2 119171871 909.3 322607910 2461.4
>> [1] TRUE
>> Trying to allocate 10 bytes ... success. Now trying to commit
>> memory ... succeeded.
>> Garbage collection 2667 = 2286+194+187 (level 2) ...
>> 12.9 Mbytes of cons cells used (26%)
>> 278.4 Mbytes of vectors used (29%)
>> Error: cannot allocate vector of size 476.2 Mb
>>
>> What do you understand of this result ?
>>
>> Thanks in advance,
>>
>> Mauricio
>>
>> >> I got the following warning:
>> >> memorytest.c: In function ‘main’:
>> >> memorytest.c:5: warning: return type of ‘main’ is not ‘int’
>> >> Is this important ?
>> >
>> > Hello Mauricio,
>> >
>> > No, your gcc version is unduly puristic here. The traditional return
>> > type of the main function in "C" should be "int", and if that is given,
>> > main has to return an integer number explicitly. By convention,
>> > a main function returning 0 indicates success, and anything other
>> > is considered as an error or warning condition. Specifying "void main()"
>> > normally implies returning zero. But if you are still worried simply
>> > change "void main" into "int main()" and add the statement " return 0; "
>> > just before the last curly bracket (without quotation marks, of course):
>> >
>> > int main() {
>> > [...]
>> > return 0;
>> > }
>> >
>> > Best
>> >
>> > Hugo
>> >
>>
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Dear Mauricio,
what I do not understand at all is the message:
> Error: cannot allocate vector of size 476.2 Mb
Have you tried to allocate a big matrix in between, whith
the R - statement not being shown in the output? Probably
not. If not, your local R version is buggy for sure. The test shows
that the OS imposes no artificial constraint on memory consumption,
at least not up to ~ 1GB. But because you said that you're running a
32 - bit version of R (in a virtual machine?), the question is now
wether gcc also compiles the test program as a 32 - bit program. I'm
unsure because a 64 - bit system can be configured to compile and
run programs using either environment. Regrettably I previously
did not include a test for that.
--- snip ---
#include
int main() {
printf("The size of a pointer is %lu bytes\n", (unsigned
long)sizeof(void *));
return 0;
}
--- snip ---
If the program says "The size of a pointer is 4 bytes" then you're
running the test program in a 32 bit environment. Then the
allocation problem could possibly be solved by recompiling/updating
your R version. The system is called Redhat "Enterprise" Linux after
all, so calling in Redhat support at that point and ask them to do
something for their money should be an option.
Best
Hugo
On Wednesday 19 January 2011 12:54:38 Mauricio Zambrano wrote:
> Dear Hugo,
>
> I tried your memory test program (without further modifications) just
> after the gc() command:
>
> print(gc())
> print(gcinfo(TRUE))
> system("/mypath/memorytest.out")
>
> and the result that I got was:
>
> Number of simulations read from ' Particles.txt ' : 9000
> ---
> used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 480114 12.91816453 48.6 5543382 148.1
> Vcells 5256970 40.2 119171871 909.3 322607910 2461.4
> [1] TRUE
> Trying to allocate 10 bytes ... success. Now trying to commit
> memory ... succeeded.
> Garbage collection 2667 = 2286+194+187 (level 2) ...
> 12.9 Mbytes of cons cells used (26%)
> 278.4 Mbytes of vectors used (29%)
> Error: cannot allocate vector of size 476.2 Mb
>
> What do you understand of this result ?
>
> Thanks in advance,
>
> Mauricio
>
> >> I got the following warning:
> >> memorytest.c: In function ‘main’:
> >> memorytest.c:5: warning: return type of ‘main’ is not ‘int’
> >> Is this important ?
> >
> > Hello Mauricio,
> >
> > No, your gcc version is unduly puristic here. The traditional return
> > type of the main function in "C" should be "int", and if that is given,
> > main has to return an integer number explicitly. By convention,
> > a main function returning 0 indicates success, and anything other
> > is considered as an error or warning condition. Specifying "void main()"
> > normally implies returning zero. But if you are still worried simply
> > change "void main" into "int main()" and add the statement " return 0; "
> > just before the last curly bracket (without quotation marks, of course):
> >
> > int main() {
> > [...]
> > return 0;
> > }
> >
> > Best
> >
> > Hugo
> >
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Dear Hugo,
I tried your memory test program (without further modifications) just
after the gc() command:
print(gc())
print(gcinfo(TRUE))
system("/mypath/memorytest.out")
and the result that I got was:
Number of simulations read from ' Particles.txt ' : 9000
---
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 480114 12.91816453 48.6 5543382 148.1
Vcells 5256970 40.2 119171871 909.3 322607910 2461.4
[1] TRUE
Trying to allocate 10 bytes ... success. Now trying to commit
memory ... succeeded.
Garbage collection 2667 = 2286+194+187 (level 2) ...
12.9 Mbytes of cons cells used (26%)
278.4 Mbytes of vectors used (29%)
Error: cannot allocate vector of size 476.2 Mb
What do you understand of this result ?
Thanks in advance,
Mauricio
--
===
Linux user #454569 -- Ubuntu user #17469
===
2011/1/19 Hugo Mildenberger :
>> I got the following warning:
>> memorytest.c: In function ‘main’:
>> memorytest.c:5: warning: return type of ‘main’ is not ‘int’
>> Is this important ?
>
> Hello Mauricio,
>
> No, your gcc version is unduly puristic here. The traditional return
> type of the main function in "C" should be "int", and if that is given,
> main has to return an integer number explicitly. By convention,
> a main function returning 0 indicates success, and anything other
> is considered as an error or warning condition. Specifying "void main()"
> normally implies returning zero. But if you are still worried simply
> change "void main" into "int main()" and add the statement " return 0; "
> just before the last curly bracket (without quotation marks, of course):
>
> int main() {
> [...]
> return 0;
> }
>
> Best
>
> Hugo
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
> I got the following warning:
> memorytest.c: In function ‘main’:
> memorytest.c:5: warning: return type of ‘main’ is not ‘int’
> Is this important ?
Hello Mauricio,
No, your gcc version is unduly puristic here. The traditional return
type of the main function in "C" should be "int", and if that is given,
main has to return an integer number explicitly. By convention,
a main function returning 0 indicates success, and anything other
is considered as an error or warning condition. Specifying "void main()"
normally implies returning zero. But if you are still worried simply
change "void main" into "int main()" and add the statement " return 0; "
just before the last curly bracket (without quotation marks, of course):
int main() {
[...]
return 0;
}
Best
Hugo
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Thank you very much Hugo for your answer.
Yesterday I was out of my office and I couldn't test the advise you
gave me. Today I'll do it.
I have never used C, so I have to ask.
For creating an executable file with your code, I copied and pasted
the text into a text file, and then tried from the command line:
$ gcc memorytest.c -o memorytes.out
and I got the following warning:
memorytest.c: In function ‘main’:
memorytest.c:5: warning: return type of ‘main’ is not ‘int’
Is this important ?
After your answer, I will execute this program just before the line
that rises the error message, and I'll let you know today.
Thank you very much Hugo,
Cheers,
--
===
Linux user #454569 -- Ubuntu user #17469
===
2011/1/18 Hugo Mildenberger :
> Hello Mauricio,
>
> Today I lost several messages from r-help and also forgot to
> include you in the reply-to address list previously. I'm curious what the test
> program I already proposed in my answer to r-help reveals. I modified it
> a bit to force the operating system to actually provide the memory. On
> an old 32 bit notebook with 512 MB of physical RAM having X running
> in parallel, this test program takes about a minute or so, but eventually
> succeeds.
>
> --- snip ---
>
> #include
> #include
> #include
>
> void main() {
> const size_t size = 10LU;
> fprintf(stderr,"Trying to allocate %lu bytes ... ",(unsigned long)size);
> fflush(stderr);
> void *p = malloc(size);
> if ( p ) {
> fprintf(stderr,"success. Now trying to commit memory ... ");
> fflush(stderr);
> memset(p,0,size);
> fprintf(stderr,"succeeded.\n");
> }else {
> fprintf(stderr,"allocation of %lu bytes failed:%m\n",(unsigned long)
> size);
> }
> }
>
> --- snip ---
>
> put this into a file named, say, "tmalloc.c" and compile it using
>
> gcc tmalloc.c -o tmalloc
>
>
> Best regards
>
> Hugo Mildenberger
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Mauricio,
I tried your matrix allocation on Gentoo-hardened 32 and
64 bit systems. Both work ok, using R-2.11.1 and R-2.12.2 respectively,
and both use a recent 2.6.36 kernel revision.
This is from the 32 bit system with 512 MB physical memory:
>system("free")
total used free sharedbuffers
cached
Mem:469356 61884 407472 0 1368 21592
-/+ buffers/cache: 38924 430432
Swap: 1927796 360961891700
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 120116 3.3 35 9.4 35 9.4
Vcells 78413 0.6 786432 6.0 391299 3.0
> bs <- matrix(NA, nrow=6940,ncol=9000)
> gc()
used (Mb)gc trigger (Mb) max used (Mb)
Ncells 120123 3.3 35 9.4 359.4
Vcells 31308414 238.9 34854943 266.0 31308428 238.9
> system("free")
total used free sharedbuffers cached
Mem:469356 307528 161828 0 1404 22508
-/+ buffers/cache: 283616 185740
Swap: 1927796 360841891712
MZ> I tried to increase the memory limit available for R by using:
MZ> $ R --min-vsize=10M --max-vsize=5000M --min-nsize=500k --max-nsize=5000M
Hmm, I wonder if specifying 5000M is a good idea within a 32-bit environment.
Depending on R's internal implementation, maybe that value could overflow an
tacitly wrap around on a 32 bit integer. (5000M > 2^32 - 1) You may try to
specify
1000M instead. But I think it's more probable that the system or VM
configuration
had setup a memory usage limit per user or per process. How to view/change this
on redhat I don't know. But you may try to compile a small C programm using
malloc() and see what happens if you request say 1Gigabyte:
#include
#include
void main() {
const size_t size = 10LU;
void* p = malloc(size);
if ( p ) {
fprintf(stderr,"successfully allocated %lu bytes\n",size);
}else {
fprintf(stderr,"allocation of %lu bytes failed:%m\n",size);
}
}
put this into a file named, say, "tmalloc.c" and compile it using
gcc tmalloc.c -o tmalloc
Hugo
On Monday 17 January 2011 16:42:43 Mauricio Zambrano wrote:
> Following the advice a colleague, I put the gc() and gcinfo(TRUE)
> commands just before the line I got the problem, and their output
> were:
>
> used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 471485 12.61704095 45.6 7920371 211.5
> Vcells 6408885 48.9 113919753 869.2 347651599 2652.4
>
> Garbage collection 538 = 323+101+114 (level 2) ...
> 13.0 Mbytes of cons cells used (29%)
> 49.0 Mbytes of vectors used (7%)
>
> Error: cannot allocate vector of size 238.1 Mb
>
>
> If I understood correctly, I should have enough memory for allocating
> the new matrix (Q.obs <- matrix(NA, nrow=6940, MZ> ncol=9000) ))
>
> Thanks in advance for any help,
>
> Mauricio
>
>
> >> "MZ" == Mauricio Zambrano
> >> on Mon, 17 Jan 2011 11:46:44 +0100 writes:
> >
> >MZ> Dear R community,
> >MZ> I'm running R 32 bits in a 64-bits machine (with 16Gb of Ram) using a
> >MZ> PAE kernel, as you can see here:
> >
> >MZ> $ uname -a
> >MZ> Linux mymachine 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010
> >MZ> i686 i686 i386 GNU/Linux
> >
> >
> >MZ> When I try to create a large matrix ( Q.obs <- matrix(NA, nrow=6940,
> >MZ> ncol=9000) ), I got the following error:
> >
> >
> >>> Error: cannot allocate vector of size 238.3 Mb
> >
> >
> >MZ> However, the amount of free memory in my machine seems to be much
> >MZ> larger than this:
> >
> >MZ> system("free")
> >MZ> \ total used free sharedbuffers
> > cached
> >MZ> Mem: 1246623663541166112120 0 67596
> > 2107556
> >MZ> -/+ buffers/cache:41789648287272
> >MZ> Swap: 12582904 0 12582904
> >
> >
> >MZ> I tried to increase the memory limit available for R by using:
> >
> >MZ> $ R --min-vsize=10M --max-vsize=5000M --min-nsize=500k
> > --max-nsize=5000M
> >
> >
> >MZ> but it didn't work.
> >
> >
> >MZ> Any hint about how can I get R using all the memory available in the
> > machine ?
> >
> > Install a 64-bit version of Linux, i.e., ubuntu in your case
> > and work from there.
> > I don't think there's a way around that.
> >
> > Martin
> >
>
> __
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Following the advice a colleague, I put the gc() and gcinfo(TRUE)
commands just before the line I got the problem, and their output
were:
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 471485 12.61704095 45.6 7920371 211.5
Vcells 6408885 48.9 113919753 869.2 347651599 2652.4
Garbage collection 538 = 323+101+114 (level 2) ...
13.0 Mbytes of cons cells used (29%)
49.0 Mbytes of vectors used (7%)
Error: cannot allocate vector of size 238.1 Mb
If I understood correctly, I should have enough memory for allocating
the new matrix (Q.obs <- matrix(NA, nrow=6940, MZ> ncol=9000) ))
Thanks in advance for any help,
Mauricio
--
===
Linux user #454569 -- Ubuntu user #17469
===
2011/1/17 Martin Maechler :
>> "MZ" == Mauricio Zambrano
>> on Mon, 17 Jan 2011 11:46:44 +0100 writes:
>
> MZ> Dear R community,
> MZ> I'm running R 32 bits in a 64-bits machine (with 16Gb of Ram) using a
> MZ> PAE kernel, as you can see here:
>
> MZ> $ uname -a
> MZ> Linux mymachine 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010
> MZ> i686 i686 i386 GNU/Linux
>
>
> MZ> When I try to create a large matrix ( Q.obs <- matrix(NA, nrow=6940,
> MZ> ncol=9000) ), I got the following error:
>
>
> >> Error: cannot allocate vector of size 238.3 Mb
>
>
> MZ> However, the amount of free memory in my machine seems to be much
> MZ> larger than this:
>
> MZ> system("free")
> MZ> \ total used free shared buffers
> cached
> MZ> Mem: 12466236 6354116 6112120 0 67596
> 2107556
> MZ> -/+ buffers/cache: 4178964 8287272
> MZ> Swap: 12582904 0 12582904
>
>
> MZ> I tried to increase the memory limit available for R by using:
>
> MZ> $ R --min-vsize=10M --max-vsize=5000M --min-nsize=500k
> --max-nsize=5000M
>
>
> MZ> but it didn't work.
>
>
> MZ> Any hint about how can I get R using all the memory available in the
> machine ?
>
> Install a 64-bit version of Linux, i.e., ubuntu in your case
> and work from there.
> I don't think there's a way around that.
>
> Martin
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
Thanks for your answer Martin, but -unfortunately- the decision about
installing a 32 bits OS in the 64 bits machine, was taken by the IT
guys of my work and not by me.
By the way, due to strong limitations about software installation in
my work place, this problem didn't happen in Ubuntu, but in Red Hat
Enterprise 5. At home I have Ubuntu 10.10 32 bits, but I can not run
the code I need in that machine.
Cheers,
Mauricio
--
===
Linux user #454569 -- Ubuntu user #17469
===
2011/1/17 Martin Maechler :
>> "MZ" == Mauricio Zambrano
>> on Mon, 17 Jan 2011 11:46:44 +0100 writes:
>
> MZ> Dear R community,
> MZ> I'm running R 32 bits in a 64-bits machine (with 16Gb of Ram) using a
> MZ> PAE kernel, as you can see here:
>
> MZ> $ uname -a
> MZ> Linux mymachine 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010
> MZ> i686 i686 i386 GNU/Linux
>
>
> MZ> When I try to create a large matrix ( Q.obs <- matrix(NA, nrow=6940,
> MZ> ncol=9000) ), I got the following error:
>
>
> >> Error: cannot allocate vector of size 238.3 Mb
>
>
> MZ> However, the amount of free memory in my machine seems to be much
> MZ> larger than this:
>
> MZ> system("free")
> MZ> \ total used free shared buffers
> cached
> MZ> Mem: 12466236 6354116 6112120 0 67596
> 2107556
> MZ> -/+ buffers/cache: 4178964 8287272
> MZ> Swap: 12582904 0 12582904
>
>
> MZ> I tried to increase the memory limit available for R by using:
>
> MZ> $ R --min-vsize=10M --max-vsize=5000M --min-nsize=500k
> --max-nsize=5000M
>
>
> MZ> but it didn't work.
>
>
> MZ> Any hint about how can I get R using all the memory available in the
> machine ?
>
> Install a 64-bit version of Linux, i.e., ubuntu in your case
> and work from there.
> I don't think there's a way around that.
>
> Martin
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] "cannot allocate vector of size ..." in RHLE5 PAE kernel
> "MZ" == Mauricio Zambrano
> on Mon, 17 Jan 2011 11:46:44 +0100 writes:
MZ> Dear R community,
MZ> I'm running R 32 bits in a 64-bits machine (with 16Gb of Ram) using a
MZ> PAE kernel, as you can see here:
MZ> $ uname -a
MZ> Linux mymachine 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010
MZ> i686 i686 i386 GNU/Linux
MZ> When I try to create a large matrix ( Q.obs <- matrix(NA, nrow=6940,
MZ> ncol=9000) ), I got the following error:
>> Error: cannot allocate vector of size 238.3 Mb
MZ> However, the amount of free memory in my machine seems to be much
MZ> larger than this:
MZ> system("free")
MZ> \ total used free sharedbuffers
cached
MZ> Mem: 1246623663541166112120 0 67596
2107556
MZ> -/+ buffers/cache:41789648287272
MZ> Swap: 12582904 0 12582904
MZ> I tried to increase the memory limit available for R by using:
MZ> $ R --min-vsize=10M --max-vsize=5000M --min-nsize=500k --max-nsize=5000M
MZ> but it didn't work.
MZ> Any hint about how can I get R using all the memory available in the
machine ?
Install a 64-bit version of Linux, i.e., ubuntu in your case
and work from there.
I don't think there's a way around that.
Martin
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
