Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple platform build/check report for BioC 3.12
If you have successfully built the vignette, under correct Bioconductor version
and valid installation, then I suggest pushing your changes to the Bioconductor
git repository. The NIGHTLY builds will occur, and with luck they will be
successful and visible in the next build report (either tomorrow afternoon or,
if you miss the nightly build tonight, then on Thursday afternoon).
I personally would not use conda, but would configure my system to build my
package ‘natively’. If the vignette used features that I was not able to
install, such as the mathptmx LaTeX library (google might be your friend
here…), then I would not use those features and would instead use a simpler
vignette (e.g., without unnecessary LaTeX packages).
I personally would not use RStudio, or devtools, to build and check my package,
because the build system does not use these add-ons, and because they set
options that are convenient but not necessary and might be confusing as I try
to make my package build under standard conditions. Instead I would use R from
the command line
R CMD build epihet
R CMD check epihet_...
where ‘…’ corresponds to the compressed tar file produced by R CMD build.
Martin
From: Xiaowen Chen
Date: Tuesday, November 24, 2020 at 4:51 PM
To: "Kern, Lori" , "mtmorgan.b...@gmail.com"
Cc: "bioc-devel@r-project.org"
Subject: Re: [EXTERNAL]epihet problems reported in the Multiple platform
build/check report for BioC 3.12
Hi Bioconductor,
Yes, I have updated epihet because the issue
“ 'x' is NULL so the result will be NULL
Quitting from lines 483-495 (/Users/ma38727/b/git/epihet/vignettes/epihet.Rnw)
Error in ans[ypos] <- rep(yes, length.out = len)[ypos] :”
Now before to push bioconductor, I would like to build and check,
Now,
I uninstrall tinytex, and used the conda to install texlive “conda install -c
conda-forge texlive-core”
And build again, now it complained no epihet.log file.
So I built epihet without vigenttes, then using Rstudio on my MAC, NOT hpc,
installed epihet, and compile epihet.Rnw, it can successfully transform Rnw to
PDF.
Any suggestion?
I will appreciate it.
Thanks.
Best,
Xiaowen
From: "Kern, Lori"
Date: Monday, November 23, 2020 at 4:05 PM
To: Xiaowen Chen
Cc: "bioc-devel@r-project.org"
Subject: Re: [EXTERNAL]epihet problems reported in the Multiple platform
build/check report for BioC 3.12
And For What Its Worth - I can reproduce the ERROR locally using a git clone
from
g...@git.bioconductor.org:packages/epihet.git
(base) shepherd@jbcj433:~/BioconductorPackages/Software$ R CMD build epihet
Bioconductor version 3.13 (BiocManager 1.30.10), ?BiocManager::install for help
* checking for file 'epihet/DESCRIPTION' ... OK
* preparing 'epihet':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘epihet.Rnw’ using knitr
Loading required package: org.Hs.eg.db
Loading required package: AnnotationDbi
Loading required package: Biobase
Welcome to Bioconductor
Vignettes contain introductory material; view with
'browseVignettes()'. To cite Bioconductor, see
'citation("Biobase")', and for packages 'citation("pkgname")'.
Attaching package: 'AnnotationDbi'
The following object is masked from 'package:clusterProfiler':
select
'select()' returned 1:1 mapping between keys and columns
Warning in rep(yes, length.out = len) :
'x' is NULL so the result will be NULL
Quitting from lines 484-496 (epihet.Rnw)
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
replacement has length zero
--- failed re-building ‘epihet.Rnw’
SUMMARY: processing the following file failed:
‘epihet.Rnw’
Error: Vignette re-building failed.
Execution halted
Lori Shepherd
Bioconductor Core Team
Roswell Park Comprehensive Cancer Center
Department of Biostatistics & Bioinformatics
Elm & Carlton Streets
Buffalo, New York 14263
From: Bioc-devel on behalf of Kern, Lori
Sent: Monday, November 23, 2020 4:01 PM
To: Xiaowen Chen
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple
platform build/check report for BioC 3.12
Please make sure your github repositories are up-to-date with the
git.bioconductor.org repositories. On quick glance, I can see you have not
pulled the changes on the bioconductor server since the version numbers are not
up-to-date for relesae and devel. The version we have at git.bioconductor.org
is the version of the package we are building and checking. not a github
repository. Perhaps you forgot to push changes to the git.bioconductor.org
repository?
I suggest you check out the git help pages especially for
Syncing with Bioconductor:
Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple platform build/check report for BioC 3.12
Hi Bioconductor,
Yes, I have updated epihet because the issue
“ 'x' is NULL so the result will be NULL
Quitting from lines 483-495 (/Users/ma38727/b/git/epihet/vignettes/epihet.Rnw)
Error in ans[ypos] <- rep(yes, length.out = len)[ypos] :”
Now before to push bioconductor, I would like to build and check,
Now,
I uninstrall tinytex, and used the conda to install texlive “conda install -c
conda-forge texlive-core”
And build again, now it complained no epihet.log file.
So I built epihet without vigenttes, then using Rstudio on my MAC, NOT hpc,
installed epihet, and compile epihet.Rnw, it can successfully transform Rnw to
PDF.
Any suggestion?
I will appreciate it.
Thanks.
Best,
Xiaowen
From: "Kern, Lori"
Date: Monday, November 23, 2020 at 4:05 PM
To: Xiaowen Chen
Cc: "bioc-devel@r-project.org"
Subject: Re: [EXTERNAL]epihet problems reported in the Multiple platform
build/check report for BioC 3.12
And For What Its Worth - I can reproduce the ERROR locally using a git clone
from
g...@git.bioconductor.org:packages/epihet.git
(base) shepherd@jbcj433:~/BioconductorPackages/Software$ R CMD build epihet
Bioconductor version 3.13 (BiocManager 1.30.10), ?BiocManager::install for help
* checking for file 'epihet/DESCRIPTION' ... OK
* preparing 'epihet':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘epihet.Rnw’ using knitr
Loading required package: org.Hs.eg.db
Loading required package: AnnotationDbi
Loading required package: Biobase
Welcome to Bioconductor
Vignettes contain introductory material; view with
'browseVignettes()'. To cite Bioconductor, see
'citation("Biobase")', and for packages 'citation("pkgname")'.
Attaching package: 'AnnotationDbi'
The following object is masked from 'package:clusterProfiler':
select
'select()' returned 1:1 mapping between keys and columns
Warning in rep(yes, length.out = len) :
'x' is NULL so the result will be NULL
Quitting from lines 484-496 (epihet.Rnw)
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
replacement has length zero
--- failed re-building ‘epihet.Rnw’
SUMMARY: processing the following file failed:
‘epihet.Rnw’
Error: Vignette re-building failed.
Execution halted
Lori Shepherd
Bioconductor Core Team
Roswell Park Comprehensive Cancer Center
Department of Biostatistics & Bioinformatics
Elm & Carlton Streets
Buffalo, New York 14263
From: Bioc-devel on behalf of Kern, Lori
Sent: Monday, November 23, 2020 4:01 PM
To: Xiaowen Chen
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple
platform build/check report for BioC 3.12
Please make sure your github repositories are up-to-date with the
git.bioconductor.org repositories. On quick glance, I can see you have not
pulled the changes on the bioconductor server since the version numbers are not
up-to-date for relesae and devel. The version we have at git.bioconductor.org
is the version of the package we are building and checking. not a github
repository. Perhaps you forgot to push changes to the git.bioconductor.org
repository?
I suggest you check out the git help pages especially for
Syncing with Bioconductor:
http://secure-web.cisco.com/1_ed4MO3wzGEv9vtahMGt3EjH_fRdxuy-Ggwfv072LHMg33gJDq6NJ-O4j8ElxPP6icz-SAKtYbg7YAb7NdXVsKWMH8rsoXdimZDOKZDL7d1c-Mb6cggFxH5YKCjkmj08UrH5at6XPatjaa6_LPY4cOVXz-hkOZCdt7p7octVbhxBusXEUQExxebo1-rsrzBqvFppOI8qnlYv5hosKQk6izKY5rjKG_JhkfAIfazaOWJBSZXxxE9Ml4V8YhYZOEZXt470d1XSlY8F-lsYsPTTiaLRveK0RbPuwODtpKAm609JGxKZmcpOUNeXoX8TTLMg/http%3A%2F%2Fbioconductor.org%2Fdevelopers%2Fhow-to%2Fgit%2Fsync-existing-repositories%2F
Pushing to Bioconductor release and devel:
http://secure-web.cisco.com/1Xg3LpZwFnSWhmcDJ_Cyy7XvU1VQn3SiYLwUSqjNAoFWlNnuYj0VmoRb53jBGHdlG73IPrxMQlsLZyP0hBmOi9kVhS2yTbZMs5OWxmyWWW-6f_i_lklGJ5LmzPDeW7bYBmZ4NqRPdQenUdQ460VyVO9wcXxyHHv0wpPEs8YAV4BlRDIXRA7Y6QeuDZbInXsw0YnL0ba7Swify49bnp-OyHs4gvC8p11paPMo_w07Zk0dbcexLbs9reMtR-lWRvGV9NGtSycUHQNNZlkYay03mW_eCIAthyAYaaxxFYg8GBD1srNlOD8qKFcL954Spxjp_/http%3A%2F%2Fbioconductor.org%2Fdevelopers%2Fhow-to%2Fgit%2Fbug-fix-in-release-and-devel%2F
Other things to consider:
Make sure you are using the latest version of R and that all package
dependencies are up to date.
For Release you should be using R 4.0.3 with Bioconductor 3.12. Make sure all
packages are up-to-date by running
BiocManager::install(version="3.12")
BiocManager::valid() and/or BiocManager::install()
For Devel you should be using R-devel (what will be R 4.1.0) and
Bioconductor 3.13. Again make sure packages are updated for this version
BiocManager::install(version="devel")
BiocManager::valid() and/or BiocManager::install()
Lori Shepherd
Bioconductor Core Team
Roswell Park Comprehensive Cancer Center
Department of Biostatistics & Bioinformatics
Elm & Carlton Streets
Buffalo, New York 14263
Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple platform build/check report for BioC 3.12
not sure that this is a good answer or not, but on scanning your vignette I
didn't see any use of LaTeX math expressions, so perhaps mathptmx is not needed
at all in and can be removed from the vignette?
At any rate when I remove it and try
epihet/vignettes master$ R CMD Sweave --pdf epihet.Rnw
using R with the correct version (BiocManager::version() returns 3.13) and
valid package installation (BiocManager::valid() returns TRUE) I get
'select()' returned 1:1 mapping between keys and columns
Warning in rep(yes, length.out = len) :
'x' is NULL so the result will be NULL
Quitting from lines 483-495 (/Users/ma38727/b/git/epihet/vignettes/epihet.Rnw)
Error in ans[ypos] <- rep(yes, length.out = len)[ypos] :
replacement has length zero
Calls: ... expand_limits_discrete_trans ->
expand_limits_continuous_trans -> ifelse
Execution halted
which is the same as the build system, and occurs before the LaTeX processing
step and so does not require that mathptmx be installed to troubleshoot this
error...
Martin
On 11/24/20, 3:39 PM, "Bioc-devel on behalf of Xiaowen Chen"
wrote:
Hi Kern,
I followed your suggestion and updated my Bioconductor version and epihet,
Then when I build epihet, it complained that
The following object is masked from 'package:clusterProfiler':
select
'select()' returned 1:1 mapping between keys and columns
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
pdflatex is not available
--- failed re-building ‘epihet.Rnw’
SUMMARY: processing the following file failed:
‘epihet.Rnw’
Error: Vignette re-building failed.
I check my HPC has not pdflatex, so I installed tinytex from the website
https://yihui.org/tinytex/ using following command lines:
remotes::install_github('yihui/tinytex')
tinytex::install_tinytex()
but another error occurred :
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
unable to run pdflatex on 'epihet.tex'
LaTeX errors:
! LaTeX Error: File `mathptmx.sty' not found.
I tried it :
tlmgr install mathptmx.sty
tlmgr: package repository
http://ctan.mirrors.hoobly.com/systems/texlive/tlnet (verified)
tlmgr install: package mathptmx.sty not present in repository.
tlmgr: action install returned an error; continuing.
tlmgr: An error has occurred. See above messages. Exiting.
Could you give me some suggestions?
Thank you!
Best,
Xiaowen
From: "Kern, Lori"
Date: Monday, November 23, 2020 at 4:05 PM
To: Xiaowen Chen
Cc: "bioc-devel@r-project.org"
Subject: Re: [EXTERNAL]epihet problems reported in the Multiple platform
build/check report for BioC 3.12
And For What Its Worth - I can reproduce the ERROR locally using a git
clone from
g...@git.bioconductor.org:packages/epihet.git
(base) shepherd@jbcj433:~/BioconductorPackages/Software$ R CMD build epihet
Bioconductor version 3.13 (BiocManager 1.30.10), ?BiocManager::install for
help
* checking for file 'epihet/DESCRIPTION' ... OK
* preparing 'epihet':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘epihet.Rnw’ using knitr
Loading required package: org.Hs.eg.db
Loading required package: AnnotationDbi
Loading required package: Biobase
Welcome to Bioconductor
Vignettes contain introductory material; view with
'browseVignettes()'. To cite Bioconductor, see
'citation("Biobase")', and for packages 'citation("pkgname")'.
Attaching package: 'AnnotationDbi'
The following object is masked from 'package:clusterProfiler':
select
'select()' returned 1:1 mapping between keys and columns
Warning in rep(yes, length.out = len) :
'x' is NULL so the result will be NULL
Quitting from lines 484-496 (epihet.Rnw)
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
replacement has length zero
--- failed re-building ‘epihet.Rnw’
SUMMARY: processing the following file failed:
‘epihet.Rnw’
Error: Vignette re-building failed.
Execution halted
Lori Shepherd
Bioconductor Core Team
Roswell Park Comprehensive Cancer Center
Department of Biostatistics & Bioinformatics
Elm & Carlton Streets
Buffalo, New York 14263
From: Bioc-devel on behalf of Kern, Lori
Sent: Monday, November 23, 2020 4:01 PM
To: Xiaowen Chen
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the
Multiple platform build/check report for BioC 3.12
Please make sure your github repositories are up-to-date with the
git.bioconductor.org repositories. On quick glance, I can see you have not
pulled the changes on the bioconductor server since the version numbers are not
up-to-date for relesae and devel.
Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple platform build/check report for BioC 3.12
Hi Kern,
I followed your suggestion and updated my Bioconductor version and epihet,
Then when I build epihet, it complained that
The following object is masked from 'package:clusterProfiler':
select
'select()' returned 1:1 mapping between keys and columns
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
pdflatex is not available
--- failed re-building ‘epihet.Rnw’
SUMMARY: processing the following file failed:
‘epihet.Rnw’
Error: Vignette re-building failed.
I check my HPC has not pdflatex, so I installed tinytex from the website
https://yihui.org/tinytex/ using following command lines:
remotes::install_github('yihui/tinytex')
tinytex::install_tinytex()
but another error occurred :
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
unable to run pdflatex on 'epihet.tex'
LaTeX errors:
! LaTeX Error: File `mathptmx.sty' not found.
I tried it :
tlmgr install mathptmx.sty
tlmgr: package repository http://ctan.mirrors.hoobly.com/systems/texlive/tlnet
(verified)
tlmgr install: package mathptmx.sty not present in repository.
tlmgr: action install returned an error; continuing.
tlmgr: An error has occurred. See above messages. Exiting.
Could you give me some suggestions?
Thank you!
Best,
Xiaowen
From: "Kern, Lori"
Date: Monday, November 23, 2020 at 4:05 PM
To: Xiaowen Chen
Cc: "bioc-devel@r-project.org"
Subject: Re: [EXTERNAL]epihet problems reported in the Multiple platform
build/check report for BioC 3.12
And For What Its Worth - I can reproduce the ERROR locally using a git clone
from
g...@git.bioconductor.org:packages/epihet.git
(base) shepherd@jbcj433:~/BioconductorPackages/Software$ R CMD build epihet
Bioconductor version 3.13 (BiocManager 1.30.10), ?BiocManager::install for help
* checking for file 'epihet/DESCRIPTION' ... OK
* preparing 'epihet':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘epihet.Rnw’ using knitr
Loading required package: org.Hs.eg.db
Loading required package: AnnotationDbi
Loading required package: Biobase
Welcome to Bioconductor
Vignettes contain introductory material; view with
'browseVignettes()'. To cite Bioconductor, see
'citation("Biobase")', and for packages 'citation("pkgname")'.
Attaching package: 'AnnotationDbi'
The following object is masked from 'package:clusterProfiler':
select
'select()' returned 1:1 mapping between keys and columns
Warning in rep(yes, length.out = len) :
'x' is NULL so the result will be NULL
Quitting from lines 484-496 (epihet.Rnw)
Error: processing vignette 'epihet.Rnw' failed with diagnostics:
replacement has length zero
--- failed re-building ‘epihet.Rnw’
SUMMARY: processing the following file failed:
‘epihet.Rnw’
Error: Vignette re-building failed.
Execution halted
Lori Shepherd
Bioconductor Core Team
Roswell Park Comprehensive Cancer Center
Department of Biostatistics & Bioinformatics
Elm & Carlton Streets
Buffalo, New York 14263
From: Bioc-devel on behalf of Kern, Lori
Sent: Monday, November 23, 2020 4:01 PM
To: Xiaowen Chen
Cc: bioc-devel@r-project.org
Subject: Re: [Bioc-devel] [EXTERNAL]epihet problems reported in the Multiple
platform build/check report for BioC 3.12
Please make sure your github repositories are up-to-date with the
git.bioconductor.org repositories. On quick glance, I can see you have not
pulled the changes on the bioconductor server since the version numbers are not
up-to-date for relesae and devel. The version we have at git.bioconductor.org
is the version of the package we are building and checking. not a github
repository. Perhaps you forgot to push changes to the git.bioconductor.org
repository?
I suggest you check out the git help pages especially for
Syncing with Bioconductor:
http://secure-web.cisco.com/1_ed4MO3wzGEv9vtahMGt3EjH_fRdxuy-Ggwfv072LHMg33gJDq6NJ-O4j8ElxPP6icz-SAKtYbg7YAb7NdXVsKWMH8rsoXdimZDOKZDL7d1c-Mb6cggFxH5YKCjkmj08UrH5at6XPatjaa6_LPY4cOVXz-hkOZCdt7p7octVbhxBusXEUQExxebo1-rsrzBqvFppOI8qnlYv5hosKQk6izKY5rjKG_JhkfAIfazaOWJBSZXxxE9Ml4V8YhYZOEZXt470d1XSlY8F-lsYsPTTiaLRveK0RbPuwODtpKAm609JGxKZmcpOUNeXoX8TTLMg/http%3A%2F%2Fbioconductor.org%2Fdevelopers%2Fhow-to%2Fgit%2Fsync-existing-repositories%2F
Pushing to Bioconductor release and devel:
http://secure-web.cisco.com/1Xg3LpZwFnSWhmcDJ_Cyy7XvU1VQn3SiYLwUSqjNAoFWlNnuYj0VmoRb53jBGHdlG73IPrxMQlsLZyP0hBmOi9kVhS2yTbZMs5OWxmyWWW-6f_i_lklGJ5LmzPDeW7bYBmZ4NqRPdQenUdQ460VyVO9wcXxyHHv0wpPEs8YAV4BlRDIXRA7Y6QeuDZbInXsw0YnL0ba7Swify49bnp-OyHs4gvC8p11paPMo_w07Zk0dbcexLbs9reMtR-lWRvGV9NGtSycUHQNNZlkYay03mW_eCIAthyAYaaxxFYg8GBD1srNlOD8qKFcL954Spxjp_/http%3A%2F%2Fbioconductor.org%2Fdevelopers%2Fhow-to%2Fgit%2Fbug-fix-in-release-and-devel%2F
Other things to consider:
Make sure you are using the latest version of R and that all package
dependencies are up to date.
For Release you should be using R 4.0.3 with Bioconductor 3.12. Make
Re: [Bioc-devel] Running vignettes one at a time works but build_vignettes causes buffer overflow
same result with BiocManager::version() 3.13
From: Shraddha Pai
Date: Tuesday, November 24, 2020 at 8:13 AM
To: Martin Morgan
Cc: Bioc-devel
Subject: Re: [Bioc-devel] Running vignettes one at a time works but
build_vignettes causes buffer overflow
Hi Martin,
The issue is on the bioc-devel version, not the release version (3.12).
Shraddha
On Tue, Nov 24, 2020 at 4:06 AM Martin Morgan
mailto:mtmorgan.b...@gmail.com>> wrote:
Please respond on the list. After creating a new clone of your package
git clone g...@git.bioconductor.org:packages/netDx
and validating my installation
BiocManager::version() # 3.12
BiocManager::valid()# TRUE
I do not see a problem
> devtools::build_vignettes()
Building netDx vignettes
…
Building vignette index
>
so can’t really help further…?
From: Shraddha Pai mailto:shraddha@utoronto.ca>>
Date: Monday, November 23, 2020 at 12:14 PM
To: Martin Morgan mailto:mtmorgan.b...@gmail.com>>
Subject: Re: [Bioc-devel] Running vignettes one at a time works but
build_vignettes causes buffer overflow
Hi Martin,
Thanks for responding. Sorry, left out the package name(!).
This is netDx, v 1.3.1, currently the version in bioc-devel.
I have tried opening a single session, and then running
> rmarkdown::render("vignette1.rmd")
> rmarkdown::render("vignette2.rmd")
> rmarkdown::render("vignette3.rmd")
This works. But if I call
> devtools::build_vignettes()
I get the buffer overflow error.
I expect that if I use gdb and then run one vignette at a time, I may not get
an error message.
But let me try running gdb and then calling devtools::build_vignettes(), see if
that turns up something.
If it helps, this only started after I switched using rJava to make java calls,
instead of using system2(). I have tried looking to see if I need to empty some
kind of buffer at the end of all java operations for a given vignette (maybe
they build up over time?) but a quick Google search hasn't turned up anything.
Aside from the gdb tool, I basically don't know how to debug the problem. Any
other thoughts would be great.
Thanks,Shraddha
On Mon, Nov 23, 2020 at 11:59 AM Martin Morgan
mailto:mtmorgan.b...@gmail.com>> wrote:
It would really help (maybe this has been mentioned before, but I'm not very
good at remembering things) to indicate what package this is!
When R installs a package, it (probably this is not the right thing to do)
builds all vignettes in the same session. Maybe devtools does this too? You
could work toward debugging by creating the source code of each vignette using
tools::Stangle() or knitr::purl(), then starting a new, clean, R session and
source()ing the first, second... vignettes. Basically the goal is to simplify
the problem so that it can be reproduced in a single R session, and perhaps
even further...
Buffer overflow is a C-level problem, e.g., copying too many characters into a
buffer. If I were trying to debug this I'd brush up on my gdb / lldb skills and
try to reproduce this with something like
$ R -d gdb
gdb> run
> source("vig1.R")
> source("vig2.R")
...
If this triggers the error, you'd find yourself at the gdb prompt, and you
could use `up` to navigate the stack until the location of the buffer overflow
(in your code) is found...
If you provide more information, especially an easily reproducible example, I
can try to provide more specific advice.
Martin
On 11/23/20, 11:39 AM, "Bioc-devel on behalf of Shraddha Pai"
mailto:bioc-devel-boun...@r-project.org> on
behalf of shraddha@utoronto.ca> wrote:
Hello BioC community,
Still haven't figured out why I get a buffer_overflow_detected error when I
build vignettes for our package netDx.
I find that running "devtools::build_vignetttes()" causes a "*** buffer
overflow detected ***" error after the first vignette has been built and
the second is just named.
But running the vignettes one at a time leads to them all running smoothing.
Any thoughts as to what could be causing the problem would be appreciated.
Thanks,Shraddha
--
Shraddha Pai, PhD
http://shraddhapai.com ; @spaiglass on Twitter
Post-doctoral Fellow, http://baderlab.org
The Donnelly Centre for Cellular and Biomolecular Research
University of Toronto
[[alternative HTML version deleted]]
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
[[alternative HTML version deleted]]
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
As for other calls to system. I avoid calling system. In the past I
had some (to get memory stats from OS), but they were failing with
exactly the same issue. So yes, if I would add call to system before
calling quit, I believe it would fail with the same error.
At the same time I think (although I am not sure) that new allocations
made in R are working fine. So R seems to reserve some memory and can
continue to operate, while external call like system will fail. Maybe
it is like this by design, don't know.
Aside from this problem that is easy to report due to the warning
message, I think that gc() is choking at the same time. I tried to
make reproducible example for that, multiple times but couldn't, let
me try one more time.
It happens to manifest when there is 4e8+ unique characters/factors in
an R session. I am able to reproduce it using data.table and dplyr
(0.8.4 because 1.0.0+ fails even sooner), but using base R is not easy
because of the size. I described briefly problem in:
https://github.com/h2oai/db-benchmark/issues/110
It would help if gcinfo() could take FALSE/TRUE/2L where 2L will print
even more information about gc, like how much time the each gc()
process took, how many objects it has to check on each level.
Best regards,
Jan
On Tue, Nov 24, 2020 at 1:05 PM Tomas Kalibera wrote:
>
> On 11/24/20 11:27 AM, Jan Gorecki wrote:
> > Thanks Bill for checking that.
> > It was my impression that warnings are raised from some internal
> > system calls made when quitting R. At that point I don't have much
> > control over checking the return status of those.
> > Your suggestion looks good to me.
> >
> > Tomas, do you think this could help? could this be implemented?
>
> I think this is a good suggestion. Deleting files on Unix was changed
> from system("rm") to doing that in C, and deleting the session directory
> should follow.
>
> It might also help diagnosing your problem, but I don't think it would
> solve it. If the diagnostics in R works fine and the OS was so
> hopelessly out of memory that it couldn't run any more external
> processes, then really this is not a problem of R, but of having
> exhausted the resources. And it would be a coincidence that just this
> particular call to "system" at the end of the session did not work.
> Anything else could break as well close to the end of the script. This
> seems the most likely explanation to me.
>
> Do you get this warning repeatedly, reproducibly at least in slightly
> different scripts at the very end, with this warning always from quit()?
> So that the "call" part of the warning message has .Internal(quit) like
> in the case you posted? Would adding another call to "system" before the
> call to "q()" work - with checking the return value? If it is always
> only the last call to "system" in "q()", then it is suspicious, perhaps
> an indication that some diagnostics in R is not correct. In that case, a
> reproducible example would be the key - so either if you could diagnose
> on your end what is the problem, or create a reproducible example that
> someone else can use to reproduce and debug.
>
> Best
> Tomas
>
> >
> > On Mon, Nov 23, 2020 at 7:10 PM Bill Dunlap
> > wrote:
> >> The call to system() probably is an internal call used to delete the
> >> session's tempdir(). This sort of failure means that a potentially large
> >> amount of disk space is not being recovered when R is done. Perhaps
> >> R_CleanTempDir() could call R_unlink() instead of having a subprocess call
> >> 'rm -rf ...'. Then it could also issue a specific warning if it was
> >> impossible to delete all of tempdir(). (That should be very rare.)
> >>
> >>> q("no")
> >> Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf
> >> /tmp/RtmppoKPXb") at sysutils.c:311
> >> 311 {
> >> (gdb) where
> >> #0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf
> >> /tmp/RtmppoKPXb") at sysutils.c:311
> >> #1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178
> >> #2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0,
> >> runLast=) at sys-std.c:1243
> >> #3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE,
> >> status=status@entry=0, runLast=) at system.c:87
> >> #4 0x556cc85e in do_quit (call=, op= >> out>, args=0x57813f90, rho=) at main.c:1393
> >>
> >> -Bill
> >>
> >> On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera
> >> wrote:
> >>> On 11/21/20 6:51 PM, Jan Gorecki wrote:
> Dear R-developers,
>
> Some of the more fat scripts (50+ GB mem used by R) that I am running,
> when they finish they do quit with q("no", status=0)
> Quite often it happens that there is an extra stderr output produced
> at the very end which looks like this:
>
> Warning message:
> In .Internal(quit(save, status, runLast)) :
> system call failed: Cannot allocate memory
>
> Is there any way to avoid this kind of warnings? I am using stderr
> output for
Re: [Bioc-devel] Running vignettes one at a time works but build_vignettes causes buffer overflow
Hi Martin,
The issue is on the bioc-devel version, not the release version (3.12).
Shraddha
On Tue, Nov 24, 2020 at 4:06 AM Martin Morgan
wrote:
> Please respond on the list. After creating a new clone of your package
>
>
>
> git clone g...@git.bioconductor.org:packages/netDx
>
>
>
> and validating my installation
>
>
>
> BiocManager::version() # 3.12
>
> BiocManager::valid()# TRUE
>
>
>
> I do not see a problem
>
>
>
> > devtools::build_vignettes()
>
> Building netDx vignettes
>
> …
>
> Building vignette index
>
> >
>
>
>
> so can’t really help further…?
>
>
>
> *From: *Shraddha Pai
> *Date: *Monday, November 23, 2020 at 12:14 PM
> *To: *Martin Morgan
> *Subject: *Re: [Bioc-devel] Running vignettes one at a time works but
> build_vignettes causes buffer overflow
>
>
>
> Hi Martin,
>
> Thanks for responding. Sorry, left out the package name(!).
>
> This is netDx, v 1.3.1, currently the version in bioc-devel.
>
>
>
> I have tried opening a single session, and then running
>
> > rmarkdown::render("vignette1.rmd")
>
> > rmarkdown::render("vignette2.rmd")
>
> > rmarkdown::render("vignette3.rmd")
>
>
>
> This works. But if I call
>
> > devtools::build_vignettes()
>
> I get the buffer overflow error.
>
>
>
> I expect that if I use gdb and then run one vignette at a time, I may not
> get an error message.
>
> But let me try running gdb and then calling devtools::build_vignettes(),
> see if that turns up something.
>
>
>
> If it helps, this only started after I switched using rJava to make java
> calls, instead of using system2(). I have tried looking to see if I need to
> empty some kind of buffer at the end of all java operations for a given
> vignette (maybe they build up over time?) but a quick Google search hasn't
> turned up anything.
>
>
>
> Aside from the gdb tool, I basically don't know how to debug the problem.
> Any other thoughts would be great.
>
>
>
> Thanks,Shraddha
>
>
>
>
>
> On Mon, Nov 23, 2020 at 11:59 AM Martin Morgan
> wrote:
>
> It would really help (maybe this has been mentioned before, but I'm not
> very good at remembering things) to indicate what package this is!
>
> When R installs a package, it (probably this is not the right thing to do)
> builds all vignettes in the same session. Maybe devtools does this too? You
> could work toward debugging by creating the source code of each vignette
> using tools::Stangle() or knitr::purl(), then starting a new, clean, R
> session and source()ing the first, second... vignettes. Basically the goal
> is to simplify the problem so that it can be reproduced in a single R
> session, and perhaps even further...
>
> Buffer overflow is a C-level problem, e.g., copying too many characters
> into a buffer. If I were trying to debug this I'd brush up on my gdb / lldb
> skills and try to reproduce this with something like
>
> $ R -d gdb
> gdb> run
> > source("vig1.R")
> > source("vig2.R")
> ...
>
> If this triggers the error, you'd find yourself at the gdb prompt, and you
> could use `up` to navigate the stack until the location of the buffer
> overflow (in your code) is found...
>
> If you provide more information, especially an easily reproducible
> example, I can try to provide more specific advice.
>
> Martin
>
> On 11/23/20, 11:39 AM, "Bioc-devel on behalf of Shraddha Pai" <
> bioc-devel-boun...@r-project.org on behalf of shraddha@utoronto.ca>
> wrote:
>
> Hello BioC community,
> Still haven't figured out why I get a buffer_overflow_detected error
> when I
> build vignettes for our package netDx.
>
> I find that running "devtools::build_vignetttes()" causes a "*** buffer
> overflow detected ***" error after the first vignette has been built
> and
> the second is just named.
> But running the vignettes one at a time leads to them all running
> smoothing.
>
> Any thoughts as to what could be causing the problem would be
> appreciated.
>
> Thanks,Shraddha
>
> --
> Shraddha Pai, PhD
> http://shraddhapai.com ; @spaiglass on Twitter
> Post-doctoral Fellow, http://baderlab.org
> The Donnelly Centre for Cellular and Biomolecular Research
> University of Toronto
>
> [[alternative HTML version deleted]]
>
> ___
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
>
[[alternative HTML version deleted]]
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
On 11/24/20 11:27 AM, Jan Gorecki wrote:
Thanks Bill for checking that.
It was my impression that warnings are raised from some internal
system calls made when quitting R. At that point I don't have much
control over checking the return status of those.
Your suggestion looks good to me.
Tomas, do you think this could help? could this be implemented?
I think this is a good suggestion. Deleting files on Unix was changed
from system("rm") to doing that in C, and deleting the session directory
should follow.
It might also help diagnosing your problem, but I don't think it would
solve it. If the diagnostics in R works fine and the OS was so
hopelessly out of memory that it couldn't run any more external
processes, then really this is not a problem of R, but of having
exhausted the resources. And it would be a coincidence that just this
particular call to "system" at the end of the session did not work.
Anything else could break as well close to the end of the script. This
seems the most likely explanation to me.
Do you get this warning repeatedly, reproducibly at least in slightly
different scripts at the very end, with this warning always from quit()?
So that the "call" part of the warning message has .Internal(quit) like
in the case you posted? Would adding another call to "system" before the
call to "q()" work - with checking the return value? If it is always
only the last call to "system" in "q()", then it is suspicious, perhaps
an indication that some diagnostics in R is not correct. In that case, a
reproducible example would be the key - so either if you could diagnose
on your end what is the problem, or create a reproducible example that
someone else can use to reproduce and debug.
Best
Tomas
On Mon, Nov 23, 2020 at 7:10 PM Bill Dunlap wrote:
The call to system() probably is an internal call used to delete the session's
tempdir(). This sort of failure means that a potentially large amount of disk
space is not being recovered when R is done. Perhaps R_CleanTempDir() could
call R_unlink() instead of having a subprocess call 'rm -rf ...'. Then it
could also issue a specific warning if it was impossible to delete all of
tempdir(). (That should be very rare.)
q("no")
Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf
/tmp/RtmppoKPXb") at sysutils.c:311
311 {
(gdb) where
#0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb") at
sysutils.c:311
#1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178
#2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0,
runLast=) at sys-std.c:1243
#3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE,
status=status@entry=0, runLast=) at system.c:87
#4 0x556cc85e in do_quit (call=, op=,
args=0x57813f90, rho=) at main.c:1393
-Bill
On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera wrote:
On 11/21/20 6:51 PM, Jan Gorecki wrote:
Dear R-developers,
Some of the more fat scripts (50+ GB mem used by R) that I am running,
when they finish they do quit with q("no", status=0)
Quite often it happens that there is an extra stderr output produced
at the very end which looks like this:
Warning message:
In .Internal(quit(save, status, runLast)) :
system call failed: Cannot allocate memory
Is there any way to avoid this kind of warnings? I am using stderr
output for detecting failures in scripts and this warning is a false
positive of a failure.
Maybe quit function could wait little bit longer trying to allocate
before it raises this warning?
If you see this warning, some call to system() or system2() or similar,
which executes an external program, failed to even run a shell to run
that external program, because there was not enough memory. You should
be able to find out where it happens by checking the exit status of
system().
Tomas
Best regards,
Jan Gorecki
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] .Internal(quit(...)): system call failed: Cannot allocate memory
Thanks Bill for checking that.
It was my impression that warnings are raised from some internal
system calls made when quitting R. At that point I don't have much
control over checking the return status of those.
Your suggestion looks good to me.
Tomas, do you think this could help? could this be implemented?
On Mon, Nov 23, 2020 at 7:10 PM Bill Dunlap wrote:
>
> The call to system() probably is an internal call used to delete the
> session's tempdir(). This sort of failure means that a potentially large
> amount of disk space is not being recovered when R is done. Perhaps
> R_CleanTempDir() could call R_unlink() instead of having a subprocess call
> 'rm -rf ...'. Then it could also issue a specific warning if it was
> impossible to delete all of tempdir(). (That should be very rare.)
>
> > q("no")
> Breakpoint 1, R_system (command=command@entry=0x7fffa1e0 "rm -Rf
> /tmp/RtmppoKPXb") at sysutils.c:311
> 311 {
> (gdb) where
> #0 R_system (command=command@entry=0x7fffa1e0 "rm -Rf /tmp/RtmppoKPXb")
> at sysutils.c:311
> #1 0x557c30ec in R_CleanTempDir () at sys-std.c:1178
> #2 0x557c31d7 in Rstd_CleanUp (saveact=, status=0,
> runLast=) at sys-std.c:1243
> #3 0x557c593d in R_CleanUp (saveact=saveact@entry=SA_NOSAVE,
> status=status@entry=0, runLast=) at system.c:87
> #4 0x556cc85e in do_quit (call=, op=,
> args=0x57813f90, rho=) at main.c:1393
>
> -Bill
>
> On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera
> wrote:
>>
>> On 11/21/20 6:51 PM, Jan Gorecki wrote:
>> > Dear R-developers,
>> >
>> > Some of the more fat scripts (50+ GB mem used by R) that I am running,
>> > when they finish they do quit with q("no", status=0)
>> > Quite often it happens that there is an extra stderr output produced
>> > at the very end which looks like this:
>> >
>> > Warning message:
>> > In .Internal(quit(save, status, runLast)) :
>> >system call failed: Cannot allocate memory
>> >
>> > Is there any way to avoid this kind of warnings? I am using stderr
>> > output for detecting failures in scripts and this warning is a false
>> > positive of a failure.
>> >
>> > Maybe quit function could wait little bit longer trying to allocate
>> > before it raises this warning?
>>
>> If you see this warning, some call to system() or system2() or similar,
>> which executes an external program, failed to even run a shell to run
>> that external program, because there was not enough memory. You should
>> be able to find out where it happens by checking the exit status of
>> system().
>>
>> Tomas
>>
>>
>> >
>> > Best regards,
>> > Jan Gorecki
>> >
>> > __
>> > R-devel@r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
