On OS X 10.12.6 (I don't think 10.12.16 exists), I get
$ ulimit -Sn
7168
Interestingly, this is because I use iTerm2 for my command line prompt.
If I do the same command in Terminal I get 256. If I start R inside of
Emacs I get 256 as well. I don't know anything about ulimit and how it
is set, but that is a pretty start difference.
Best,
Kasper
On Fri, Oct 6, 2017 at 3:12 AM, Wolfgang Huber <wolfgang.hu...@embl.de
<mailto:wolfgang.hu...@embl.de>> wrote:
On Mac OSX 10.12.16:
$ ulimit -Sn
256
so the maximum value of R_MAX_NUM_DLLS is 153 ...
Wolfgang
5.10.17 23:02, Henrik Bengtsson scripsit:
About the DLL limit:
Just wanna make sure you're aware of "new" environment variable
R_MAX_NUM_DLLS available in R (>= 3.4.0). It allows you to push the
current default limit of 100 open DLLs a bit higher. It can be
set in
.Renviron or before, e.g.
$ R_MAX_NUM_DLLS=500 R
This, of course, assumes that you can set it, which you might not be
able to do on build servers. Also, there is an upper limit
min(0.6*fd_limit,1000) that depends on the number of files you can
have open at the same time (fd_limit), e.g. on my Ubuntu 16.04 I've
got:
$ ulimit -Sn
1024
so R_MAX_NUM_DLLS=614 is the maximum for me.
/Henrik
On Thu, Oct 5, 2017 at 11:22 AM, Wolfgang Huber
<wolfgang.hu...@embl.de <mailto:wolfgang.hu...@embl.de>> wrote:
Breaking up long workflows into several smaller "modules"
each with a
clearly defined input and output is a good idea, certainly
for didactic &
maintenance reasons.
It doesn't "solve" the DLL issue though, it only avoids it
(for now)...
I believe you can use a Makefile for your vignettes
(https://cran.r-project.org/doc/manuals/R-exts.html#Writing-package-vignettes
<https://cran.r-project.org/doc/manuals/R-exts.html#Writing-package-vignettes>),
and this might be a good way of managing which depends on
which. For passing
along output/input, perhaps local .RData files are good
enough, perhaps some
wheel-reinventing can also be avoided by using
https://bioconductor.org/packages/release/bioc/html/BiocFileCache.html
<https://bioconductor.org/packages/release/bioc/html/BiocFileCache.html>
(haven't actually used it yet, though).
Wolfgang
5.10.17 20:02, Aaron Lun scripsit:
This may relate to what I was thinking with respect to
solving the DLL
problem, by breaking up large workflows into modules
that can be executed in
separate R sessions. The same approach would also make
it easier to
associate package dependencies with specific parts of
the workflow.
In my particular situation, it is easy to break up the
workflow into
sections that can be executed completely independently.
However, I can also
imagine situations where dependencies on previous
objects, etc. make it
difficult to break up the workflow. If multiple files
are present in
vignettes/, can they be directed to execute in a
specific order, and would
output files from one vignette persist during the
execution of another?
-Aaron
------------------------------------------------------------------------
*From:* Wolfgang Huber <wolfgang.hu...@embl.de
<mailto:wolfgang.hu...@embl.de>>
*Sent:* Thursday, 5 October 2017 6:23:47 PM
*To:* Laurent Gatto; Aaron Lun
*Cc:* bioc-devel@r-project.org
<mailto:bioc-devel@r-project.org>
*Subject:* Re: [Bioc-devel] library() calls removed in
simpleSingleCell
workflow
I agree it is nice to be able to only load the packages
needed for a
certain section of a vignette and not the whole thing.
And that too many
`::` can make code look unwieldy (though some may
actually increase
readability).
But relying on manually sprinkled in `library` calls
seems like a hack
prone to error. And there are always bound to be
dependencies that are
non-local, e.g. on general infrastructure like
SummarizedExperiment,
ggplot2, dplyr.
So: do we need a way to computationally determine the
dependencies of a
vignette section, including highlighting/eliminating
potential name
clashes (b/c the warnings about masking emitted at
package loading are
easily ignored)? This seems like a straightforward
engineering task.
Eventually with such code analysis we could get rid of
explicit
`library` calls altogether :)
Wolfgang
5.10.17 08:53, Laurent Gatto scripsit:
On 5 October 2017 00:11, Aaron Lun wrote:
Here's another two cents from me:
The explicit library() calls allow for easy
copy-pasting if people
only want to use/adapt a section of the
workflow. In such cases,
calling "library(simpleSingleCell)" could drag
in a lot of unnecessary
packages (e.g., which could hit the DLL limit).
Reading through the
text to figure out the requirements for each
code chunk seems like a
pain, and lots of "::" are unwieldy.
More generally, the removal of individual
library() calls seems to
encourage the use of a single
"library(simpleSingleCell)" call at the
top of any user-developed custom analysis
scripts based on the
workflow. This seems conceptually odd to me -
the simpleSingleCell
package is simply a vehicle for the compiled
workflow, it shouldn't be
involved in analyses of other data.
I can confirm that this is a possibility.
Before workflows became available, I created the
RforProteomics package
that essentially provided one relatively large
vignette to demonstrate a
variety of applications of R/Bioconductor for mass
spectrometry and
proteomics. I think this has been a useful way to
disseminate R and
Bioconductor in these respective communities, but
also lead to the
confusion that it was that package that "did all the
stuff", i.e. people
saying that they were using RforProteomics to do a
task that was
described in the vignette. The RforProteomics
vignette does explicitly
call library at the beginning of each section and
explained that the
package was only a collection of analyses stemming
from other packages,
but that wasn't enough apparently.
Laurent
-Aaron
________________________________
From: Bioc-devel
<bioc-devel-boun...@r-project.org
<mailto:bioc-devel-boun...@r-project.org>> on
behalf of
Wolfgang Huber <wolfgang.hu...@embl.de
<mailto:wolfgang.hu...@embl.de>>
Sent: Thursday, 5 October 2017 8:26 AM
To: bioc-devel@r-project.org
<mailto:bioc-devel@r-project.org>
Subject: Re: [Bioc-devel] library() calls
removed in simpleSingleCell
workflow
I find `eval=FALSE` chunks not a good idea, since
- they confuse users who only see the rendered
HTML/PDF (where this flag
is not shown)
- they are not tested, so more prone to code rot.
I'd also like to object to the idea that
proximity of a `library` call
to code that uses a package is somehow didactic.
It's actually a bad
habit: the R interpreter does not care. The
relevant package
- can be mentioned in the narrative,
- stated in the code with the pkgname:: prefix.
The latter is good didactics to get people used
to the idea of
namespaces, especially since there is an
increasing frequency of name
clashes in CRAN, tidyverse, BioC (e.g. consider
the various functions
named 'filter' and the obscure malbehaviors that
can result from these).
Best wishes
Wolfgang
On 04/10/2017 22:20, Turaga, Nitesh wrote:
Hi Aaron,
A work around solution maybe to, put all
libraries in a “eval=FALSE”
block in the r code chunk
```{r, eval=FALSE}
library(scran)
library(scater)
```
etc.
This way the users can see the library()
calls in the vignette.
Best,
Nitesh
On Oct 4, 2017, at 4:14 PM, Obenchain,
Valerie
<valerie.obench...@roswellpark.org> wrote:
Hi guys,
A little background on this vignette ->
package conversion. The
workflows were converted to package form
because we want to integrate them
into the nightly build system instead of
supporting separate machines as
we're now doing.
As part of this conversion, packages
loaded in workflow vignettes were
moved to Depends in DESCRIPTION. This
enables the user to load a single
package instead of many. Packages were
moved to Depends instead of Suggests
(as is usually done with software
packages) because these vignette is the
only thing these workflow
packages have going - no defined classes or methods.
This seemed a more
tidy approach and the dependencies are listed in Depends
for the user to
see. This was my (maybe bad?) idea and Nitesh was the
messenger. If you feel
the individual loading of packages in the vignette is a
key part of the
instruction/learning we can leave them as is and list
the packages in
Suggests.
I should also mention that incorporating
the workflows into the build
system won't happen until after the
release. At that time we'll move the
repositories from svn to git and it's
likely we'll have to ask maintainers
to abide by some time/space guidelines.
At that point the build machines
will be building software,
experimental data and workflows and resources aren't
unlimited. When that
time comes we'll update the workflow guidelines and
contact maintainers.
Thanks.
Valerie
On 10/04/2017 12:27 PM, Kasper Daniel
Hansen wrote:
yeah, that is super super useful to
people. In my vignettes (granted,
not
workflows) I have a separate
"Dependencies" section which is basically
a
series of library() calls.
On Wed, Oct 4, 2017 at 3:18 PM, Aaron Lun
<a...@wehi.edu.au
<mailto:a...@wehi.edu.au>><mailto:a...@wehi.edu.au
<mailto:a...@wehi.edu.au>> wrote:
Dear Nitesh, list;
The library() calls in the
simpleSingleCell workflow have been
removed.
Why is this? I find explicit library()
calls to be quite useful for
readers
of the compiled vignette, because it
makes it easier for them to
determine
the packages that are required to adapt
parts of the workflow for
their own
analyses. If it doesn't hurt the build
system, I would prefer to have
these
library() calls in the vignette.
Cheers,
Aaron
[[alternative HTML version
deleted]]
_______________________________________________
Bioc-devel@r-project.org
<mailto:Bioc-devel@r-project.org><mailto:Bioc-devel@r-project.org
<mailto:Bioc-devel@r-project.org>>
mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
[[alternative HTML version
deleted]]
_______________________________________________
Bioc-devel@r-project.org
<mailto:Bioc-devel@r-project.org><mailto:Bioc-devel@r-project.org
<mailto:Bioc-devel@r-project.org>>
mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
This email message may contain legally
privileged and/or confidential
information. If you are not the
intended recipient(s), or the employee or
agent responsible for the delivery of
this message to the intended
recipient(s), you are hereby notified
that any disclosure, copying,
distribution, or use of this email
message is
prohibited. If you have received this message in error,
please notify the
sender immediately by e-mail and delete this email
message from your
computer. Thank you.
[[alternative HTML version
deleted]]
_______________________________________________
Bioc-devel@r-project.org
<mailto:Bioc-devel@r-project.org>
mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
Bioc-devel Info Page - ETH
Zurich<https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>>
stat.ethz.ch <http://stat.ethz.ch>
Your email address: Your name (optional): You
may enter a privacy
password below. This provides only mild
security, but should prevent others
from messing with ...
This email message may contain legally
privileged and/or confidential
information. If you are not the intended
recipient(s), or the employee or
agent responsible for the delivery of this
message to the intended
recipient(s), you are hereby notified that
any disclosure, copying,
distribution, or use of this email message is
prohibited. If you have received this message in error,
please notify the
sender immediately by e-mail and delete this email
message from your
computer. Thank you.
_______________________________________________
Bioc-devel@r-project.org
<mailto:Bioc-devel@r-project.org> mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
Bioc-devel Info Page - ETH
Zurich<https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>>
stat.ethz.ch <http://stat.ethz.ch>
Your email address: Your name (optional): You
may enter a privacy
password below. This provides only mild
security, but should prevent others
from messing with ...
--
With thanks in advance-
Wolfgang
-------
Wolfgang Huber
Principal Investigator, EMBL Senior Scientist
European Molecular Biology Laboratory (EMBL)
Heidelberg, Germany
wolfgang.hu...@embl.de <mailto:wolfgang.hu...@embl.de>
http://www.huber.embl.de
--
With thanks in advance-
Wolfgang
-------
Wolfgang Huber
Principal Investigator, EMBL Senior Scientist
European Molecular Biology Laboratory (EMBL)
Heidelberg, Germany
wolfgang.hu...@embl.de <mailto:wolfgang.hu...@embl.de>
http://www.huber.embl.de
_______________________________________________
Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org>
mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
--
With thanks in advance-
Wolfgang
-------
Wolfgang Huber
Principal Investigator, EMBL Senior Scientist
European Molecular Biology Laboratory (EMBL)
Heidelberg, Germany
wolfgang.hu...@embl.de <mailto:wolfgang.hu...@embl.de>
http://www.huber.embl.de
_______________________________________________
Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org> mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
