Re: [Rd] Crash after (wrongly) applying product operator on object from LIMMA package

2017-04-24 Thread Tomas Kalibera 


Yes, the mentioned pull request for rJava is to workaround a JVM problem 
that is Linux-only.

I am not aware of any related problem on OSX.

In general, you can currently get two kinds of messages with infinite 
recursion:


Error: evaluation nested too deeply: infinite recursion / 
options(expressions=)?

Error: C stack usage XXX is too close to the limit

but in some situations you could get a segfault, an obvious one is 
infinite recursion in native code of a package.


If you're getting a segfault it is probably best to check with a 
debugger - infinite recursion would be easily identified in the stacktrace.


Best
Tomas


On 04/24/2017 11:58 AM, Hilmar Berger wrote:

Hi January,

I believe the root of the xlsx issue has been identified and a fix
suggested by Tomas Kalibera (see https://github.com/s-u/rJava/pull/102).
In a nutshell, Oracle Java on Linux modifies the stack in a way that
makes it smaller and and the same time makes it impossible for R to
detect this change, leading to segfaults. It is not clear to me that the
same problem would occur on Mac, since the behavior of Oracle seems to
be Linux specific. Possibly even Linux users on OpenJDK might not
encounter any problems (not tested).

So possibly the next release of rJava should also fix the xlsx problems
with other packages.

Best regards,
Hilmar


On 24/04/17 11:46, January W. wrote:

Hi Hilmar,

weird. The memory problem seems be due to recursion (my R, version
3.3.3, says: Error: evaluation nested too deeply: infinite recursion /
options(expressions=)?, just write traceback() to see how it happens),
but why does it segfault with xlsx? Nb xlsx is the culprit: neither
rJava nor xlsxjars cause the problem.

On the other hand, quick googling for r+xlsx+segfault returns tons of
reports of how xlsx crashes in dozens of situations. See for example
http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html.
Also, the problem might be platform-specific. It would be interesting
to see whether anyone with a Mac can reproduce it.

kind regards,

j.





On 19 April 2017 at 10:01, Hilmar Berger > wrote:

 Hi,

 following up on my own question, I found smaller example that does
 not require LIMMA:

 setClass("FOOCLASS",
  representation("list")
 )
 ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))

 > ma * ma$M
 Error: C stack usage  7970512 is too close to the limit

 > library(xlsx)
 Loading required package: rJava
 Loading required package: xlsxjars
 > ma * ma$M
 ---> Crash

 xlsx seems to act like a catalyst here, with the product operator
 running in a deep nested iteration, exhausting the stack. Valgrind
 shows thousands of invalid stack accesses when loading xslx, which
 might contribute to the problem. Package xlsx has not been updated
 since 2014, so it might fail with more current versions of R or
 Java (I'm using Oracle Java 8).

 Still, even if xlsx was the package to be blamed for the crash, I
 fail to understand what exactly the product operator is trying to
 do in the multiplication of the matrix with the object.

 Best regards,
 Hilmar


 On 18/04/17 18:57, Hilmar Berger wrote:

 Hi,

 this is a problem that occurs in the presence of two libraries
 (limma, xlsx) and leads to a crash of R. The problematic code
 is the wrong application of sweep or the product ("*")
 function on an LIMMA MAList object. To my knowledge, limma
 does not define a "*" method for MAList objects.

 If only LIMMA is loaded but not package xlsx, the code does
 not crash but rather produces an error ("Error: C stack usage
 7970512 is too close to the limit"). Loading only package
 rJava instead of xlsx does also not produce the crash but the
 error message instead. Note that xlsx functions are not
 explicitly used.

 It could be reproduced on two different Linux machines running
 R-3.2.5, R-3.3.0 and R-3.3.2.

 Code to reproduce the problem:
 -
 library(limma)
 library(xlsx)

 # a MAList
 ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
 M=matrix(rnorm(300), 30,10)))

 # This should actually be sweep(ma$M, ...) for functional
 code, but I omitted the $M...
 #sweep(ma, 2, c(1:10), "*")
 # sweep will crash when doing the final operation of applying
 the function over the input matrix, which in this case is
 function "*"

 f = match.fun("*")
 # This is not exactly the same as in sweep but it also tries
 to multiply the MAList object with a matrix of same size and
 leads to the crash
 f(ma, ma$M)
 # ma * ma$M has the same effect
 -

Re: [Rd] Crash after (wrongly) applying product operator on object from LIMMA package

2017-04-24 Thread Hilmar Berger 
Hi January,

I believe the root of the xlsx issue has been identified and a fix 
suggested by Tomas Kalibera (see https://github.com/s-u/rJava/pull/102). 
In a nutshell, Oracle Java on Linux modifies the stack in a way that 
makes it smaller and and the same time makes it impossible for R to 
detect this change, leading to segfaults. It is not clear to me that the 
same problem would occur on Mac, since the behavior of Oracle seems to 
be Linux specific. Possibly even Linux users on OpenJDK might not 
encounter any problems (not tested).

So possibly the next release of rJava should also fix the xlsx problems 
with other packages.

Best regards,
Hilmar


On 24/04/17 11:46, January W. wrote:
> Hi Hilmar,
>
> weird. The memory problem seems be due to recursion (my R, version 
> 3.3.3, says: Error: evaluation nested too deeply: infinite recursion / 
> options(expressions=)?, just write traceback() to see how it happens), 
> but why does it segfault with xlsx? Nb xlsx is the culprit: neither 
> rJava nor xlsxjars cause the problem.
>
> On the other hand, quick googling for r+xlsx+segfault returns tons of 
> reports of how xlsx crashes in dozens of situations. See for example 
> http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html. 
> Also, the problem might be platform-specific. It would be interesting 
> to see whether anyone with a Mac can reproduce it.
>
> kind regards,
>
> j.
>
>
>
>
>
> On 19 April 2017 at 10:01, Hilmar Berger  > wrote:
>
> Hi,
>
> following up on my own question, I found smaller example that does
> not require LIMMA:
>
> setClass("FOOCLASS",
>  representation("list")
> )
> ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>
> > ma * ma$M
> Error: C stack usage  7970512 is too close to the limit
>
> > library(xlsx)
> Loading required package: rJava
> Loading required package: xlsxjars
> > ma * ma$M
> ---> Crash
>
> xlsx seems to act like a catalyst here, with the product operator
> running in a deep nested iteration, exhausting the stack. Valgrind
> shows thousands of invalid stack accesses when loading xslx, which
> might contribute to the problem. Package xlsx has not been updated
> since 2014, so it might fail with more current versions of R or
> Java (I'm using Oracle Java 8).
>
> Still, even if xlsx was the package to be blamed for the crash, I
> fail to understand what exactly the product operator is trying to
> do in the multiplication of the matrix with the object.
>
> Best regards,
> Hilmar
>
>
> On 18/04/17 18:57, Hilmar Berger wrote:
>
> Hi,
>
> this is a problem that occurs in the presence of two libraries
> (limma, xlsx) and leads to a crash of R. The problematic code
> is the wrong application of sweep or the product ("*")
> function on an LIMMA MAList object. To my knowledge, limma
> does not define a "*" method for MAList objects.
>
> If only LIMMA is loaded but not package xlsx, the code does
> not crash but rather produces an error ("Error: C stack usage 
> 7970512 is too close to the limit"). Loading only package
> rJava instead of xlsx does also not produce the crash but the
> error message instead. Note that xlsx functions are not
> explicitly used.
>
> It could be reproduced on two different Linux machines running
> R-3.2.5, R-3.3.0 and R-3.3.2.
>
> Code to reproduce the problem:
> -
> library(limma)
> library(xlsx)
>
> # a MAList
> ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
> M=matrix(rnorm(300), 30,10)))
>
> # This should actually be sweep(ma$M, ...) for functional
> code, but I omitted the $M...
> #sweep(ma, 2, c(1:10), "*")
> # sweep will crash when doing the final operation of applying
> the function over the input matrix, which in this case is
> function "*"
>
> f = match.fun("*")
> # This is not exactly the same as in sweep but it also tries
> to multiply the MAList object with a matrix of same size and
> leads to the crash
> f(ma, ma$M)
> # ma * ma$M has the same effect
> -
>
> My output:
>
> R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
> Copyright (C) 2016 The R Foundation for Statistical Computing
> Platform: x86_64-pc-linux-gnu (64-bit)
>
> R is free software and comes with ABSOLUTELY NO WARRANTY.
> You are welcome to redistribute it under certain conditions.
> Type 'license()' or 'licence()' for distribution details.
>
>   Natural language support but running in an English locale
>
> R is a collaborative project with many

Re: [Rd] Crash after (wrongly) applying product operator on object from LIMMA package

2017-04-24 Thread January W. 
Hi Hilmar,

weird. The memory problem seems be due to recursion (my R, version 3.3.3,
says: Error: evaluation nested too deeply: infinite recursion /
options(expressions=)?, just write traceback() to see how it happens), but
why does it segfault with xlsx? Nb xlsx is the culprit: neither rJava nor
xlsxjars cause the problem.

On the other hand, quick googling for r+xlsx+segfault returns tons of
reports of how xlsx crashes in dozens of situations. See for example
http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html.
Also, the problem might be platform-specific. It would be interesting to
see whether anyone with a Mac can reproduce it.

kind regards,

j.





On 19 April 2017 at 10:01, Hilmar Berger  wrote:

> Hi,
>
> following up on my own question, I found smaller example that does not
> require LIMMA:
>
> setClass("FOOCLASS",
>  representation("list")
> )
> ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>
> > ma * ma$M
> Error: C stack usage  7970512 is too close to the limit
>
> > library(xlsx)
> Loading required package: rJava
> Loading required package: xlsxjars
> > ma * ma$M
> ---> Crash
>
> xlsx seems to act like a catalyst here, with the product operator running
> in a deep nested iteration, exhausting the stack. Valgrind shows thousands
> of invalid stack accesses when loading xslx, which might contribute to the
> problem. Package xlsx has not been updated since 2014, so it might fail
> with more current versions of R or Java (I'm using Oracle Java 8).
>
> Still, even if xlsx was the package to be blamed for the crash, I fail to
> understand what exactly the product operator is trying to do in the
> multiplication of the matrix with the object.
>
> Best regards,
> Hilmar
>
>
> On 18/04/17 18:57, Hilmar Berger wrote:
>
>> Hi,
>>
>> this is a problem that occurs in the presence of two libraries (limma,
>> xlsx) and leads to a crash of R. The problematic code is the wrong
>> application of sweep or the product ("*") function on an LIMMA MAList
>> object. To my knowledge, limma does not define a "*" method for MAList
>> objects.
>>
>> If only LIMMA is loaded but not package xlsx, the code does not crash but
>> rather produces an error ("Error: C stack usage  7970512 is too close to
>> the limit"). Loading only package rJava instead of xlsx does also not
>> produce the crash but the error message instead. Note that xlsx functions
>> are not explicitly used.
>>
>> It could be reproduced on two different Linux machines running R-3.2.5,
>> R-3.3.0 and R-3.3.2.
>>
>> Code to reproduce the problem:
>> -
>> library(limma)
>> library(xlsx)
>>
>> # a MAList
>> ma = new("MAList", list(A=matrix(rnorm(300), 30,10), M=matrix(rnorm(300),
>> 30,10)))
>>
>> # This should actually be sweep(ma$M, ...) for functional code, but I
>> omitted the $M...
>> #sweep(ma, 2, c(1:10), "*")
>> # sweep will crash when doing the final operation of applying the
>> function over the input matrix, which in this case is function "*"
>>
>> f = match.fun("*")
>> # This is not exactly the same as in sweep but it also tries to multiply
>> the MAList object with a matrix of same size and leads to the crash
>> f(ma, ma$M)
>> # ma * ma$M has the same effect
>> -
>>
>> My output:
>>
>> R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
>> Copyright (C) 2016 The R Foundation for Statistical Computing
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>> You are welcome to redistribute it under certain conditions.
>> Type 'license()' or 'licence()' for distribution details.
>>
>>   Natural language support but running in an English locale
>>
>> R is a collaborative project with many contributors.
>> Type 'contributors()' for more information and
>> 'citation()' on how to cite R or R packages in publications.
>>
>> Type 'demo()' for some demos, 'help()' for on-line help, or
>> 'help.start()' for an HTML browser interface to help.
>> Type 'q()' to quit R.
>>
>> > library(limma)
>> > library(xlsx)
>> Loading required package: rJava
>> Loading required package: xlsxjars
>> >
>> > sessionInfo()
>> R version 3.3.0 (2016-05-03)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>> Running under: Ubuntu 14.04.5 LTS
>>
>> locale:
>>  [1] LC_CTYPE=en_US.UTF-8  LC_NUMERIC=C LC_TIME=en_US.UTF-8
>>  [4] LC_COLLATE=en_US.UTF-8LC_MONETARY=en_US.UTF-8
>> LC_MESSAGES=en_US.UTF-8
>>  [7] LC_PAPER=en_US.UTF-8  LC_NAME=en_US.UTF-8
>> LC_ADDRESS=en_US.UTF-8
>> [10] LC_TELEPHONE=en_US.UTF-8  LC_MEASUREMENT=en_US.UTF-8
>> LC_IDENTIFICATION=en_US.UTF-8
>>
>> attached base packages:
>> [1] stats graphics  grDevices utils datasets  methods base
>>
>> other attached packages:
>> [1] xlsx_0.5.7 xlsxjars_0.6.1 rJava_0.9-8limma_3.30.7
>>
>> loaded via a namespace (and not attached):
>> [1] tools_3.3.0
>> >
>> > ma = new("MAList",

Re: [Rd] Crash after (wrongly) applying product operator on object from LIMMA package

2017-04-19 Thread Hilmar Berger 

Hi,

following up on my own question, I found smaller example that does not 
require LIMMA:


setClass("FOOCLASS",
 representation("list")
)
ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))

> ma * ma$M
Error: C stack usage  7970512 is too close to the limit

> library(xlsx)
Loading required package: rJava
Loading required package: xlsxjars
> ma * ma$M
---> Crash

xlsx seems to act like a catalyst here, with the product operator 
running in a deep nested iteration, exhausting the stack. Valgrind shows 
thousands of invalid stack accesses when loading xslx, which might 
contribute to the problem. Package xlsx has not been updated since 2014, 
so it might fail with more current versions of R or Java (I'm using 
Oracle Java 8).


Still, even if xlsx was the package to be blamed for the crash, I fail 
to understand what exactly the product operator is trying to do in the 
multiplication of the matrix with the object.


Best regards,
Hilmar

On 18/04/17 18:57, Hilmar Berger wrote:

Hi,

this is a problem that occurs in the presence of two libraries (limma, 
xlsx) and leads to a crash of R. The problematic code is the wrong 
application of sweep or the product ("*") function on an LIMMA MAList 
object. To my knowledge, limma does not define a "*" method for MAList 
objects.


If only LIMMA is loaded but not package xlsx, the code does not crash 
but rather produces an error ("Error: C stack usage  7970512 is too 
close to the limit"). Loading only package rJava instead of xlsx does 
also not produce the crash but the error message instead. Note that 
xlsx functions are not explicitly used.


It could be reproduced on two different Linux machines running 
R-3.2.5, R-3.3.0 and R-3.3.2.


Code to reproduce the problem:
-
library(limma)
library(xlsx)

# a MAList
ma = new("MAList", list(A=matrix(rnorm(300), 30,10), 
M=matrix(rnorm(300), 30,10)))


# This should actually be sweep(ma$M, ...) for functional code, but I 
omitted the $M...

#sweep(ma, 2, c(1:10), "*")
# sweep will crash when doing the final operation of applying the 
function over the input matrix, which in this case is function "*"


f = match.fun("*")
# This is not exactly the same as in sweep but it also tries to 
multiply the MAList object with a matrix of same size and leads to the 
crash

f(ma, ma$M)
# ma * ma$M has the same effect
-

My output:

R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(limma)
> library(xlsx)
Loading required package: rJava
Loading required package: xlsxjars
>
> sessionInfo()
R version 3.3.0 (2016-05-03)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.5 LTS

locale:
 [1] LC_CTYPE=en_US.UTF-8  LC_NUMERIC=C LC_TIME=en_US.UTF-8
 [4] LC_COLLATE=en_US.UTF-8LC_MONETARY=en_US.UTF-8 
LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8  LC_NAME=en_US.UTF-8 
LC_ADDRESS=en_US.UTF-8
[10] LC_TELEPHONE=en_US.UTF-8  LC_MEASUREMENT=en_US.UTF-8 
LC_IDENTIFICATION=en_US.UTF-8


attached base packages:
[1] stats graphics  grDevices utils datasets  methods base

other attached packages:
[1] xlsx_0.5.7 xlsxjars_0.6.1 rJava_0.9-8limma_3.30.7

loaded via a namespace (and not attached):
[1] tools_3.3.0
>
> ma = new("MAList", list(A=matrix(rnorm(300), 30,10), 
M=matrix(rnorm(300), 30,10)))

> #sweep(ma, 2, c(1:10), "*")
>
> f = match.fun("*")
> f
function (e1, e2)  .Primitive("*")

> f(ma, ma$M)

> crash to command line with segfault.

Best regards,
Hilmar



--
Dr. Hilmar Berger, MD
Max Planck Institute for Infection Biology
Charitéplatz 1
D-10117 Berlin
GERMANY

Phone:  + 49 30 28460 430
Fax:+ 49 30 28460 401
 
E-Mail: ber...@mpiib-berlin.mpg.de

Web   : www.mpiib-berlin.mpg.de

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Crash after (wrongly) applying product operator on object from LIMMA package

2017-04-18 Thread Hilmar Berger 

Hi,

this is a problem that occurs in the presence of two libraries (limma, 
xlsx) and leads to a crash of R. The problematic code is the wrong 
application of sweep or the product ("*") function on an LIMMA MAList 
object. To my knowledge, limma does not define a "*" method for MAList 
objects.


If only LIMMA is loaded but not package xlsx, the code does not crash 
but rather produces an error ("Error: C stack usage  7970512 is too 
close to the limit"). Loading only package rJava instead of xlsx does 
also not produce the crash but the error message instead. Note that xlsx 
functions are not explicitly used.


It could be reproduced on two different Linux machines running R-3.2.5, 
R-3.3.0 and R-3.3.2.


Code to reproduce the problem:
-
library(limma)
library(xlsx)

# a MAList
ma = new("MAList", list(A=matrix(rnorm(300), 30,10), 
M=matrix(rnorm(300), 30,10)))


# This should actually be sweep(ma$M, ...) for functional code, but I 
omitted the $M...

#sweep(ma, 2, c(1:10), "*")
# sweep will crash when doing the final operation of applying the 
function over the input matrix, which in this case is function "*"


f = match.fun("*")
# This is not exactly the same as in sweep but it also tries to multiply 
the MAList object with a matrix of same size and leads to the crash

f(ma, ma$M)
# ma * ma$M has the same effect
-

My output:

R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(limma)
> library(xlsx)
Loading required package: rJava
Loading required package: xlsxjars
>
> sessionInfo()
R version 3.3.0 (2016-05-03)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.5 LTS

locale:
 [1] LC_CTYPE=en_US.UTF-8  LC_NUMERIC=C LC_TIME=en_US.UTF-8
 [4] LC_COLLATE=en_US.UTF-8LC_MONETARY=en_US.UTF-8 
LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8  LC_NAME=en_US.UTF-8 
LC_ADDRESS=en_US.UTF-8
[10] LC_TELEPHONE=en_US.UTF-8  LC_MEASUREMENT=en_US.UTF-8 
LC_IDENTIFICATION=en_US.UTF-8


attached base packages:
[1] stats graphics  grDevices utils datasets  methods base

other attached packages:
[1] xlsx_0.5.7 xlsxjars_0.6.1 rJava_0.9-8limma_3.30.7

loaded via a namespace (and not attached):
[1] tools_3.3.0
>
> ma = new("MAList", list(A=matrix(rnorm(300), 30,10), 
M=matrix(rnorm(300), 30,10)))

> #sweep(ma, 2, c(1:10), "*")
>
> f = match.fun("*")
> f
function (e1, e2)  .Primitive("*")

> f(ma, ma$M)

> crash to command line with segfault.

Best regards,
Hilmar

--
Dr. Hilmar Berger, MD
Max Planck Institute for Infection Biology
Charitéplatz 1
D-10117 Berlin
GERMANY

Phone:  + 49 30 28460 430
Fax:+ 49 30 28460 401
 
E-Mail: ber...@mpiib-berlin.mpg.de

Web   : www.mpiib-berlin.mpg.de

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel