On 02/08/2018 04:04 PM, Armen Abnousi wrote:
Hi Martin,
Thanks for your answer. What do you mean by "might go to the trouble of
installing it", is there a way for us to install software on the build
machines?
I meant that individual users might install MPI, even if it were not
installed on the build machines, so they would be able to evaluate your
code in parallel, even though the build machines evaluated it in serial.
This would be an alternative to marking your package as 'not supported
on Windows', which would mean that the user would need to go to special
measures (installing your package from source, rather than the normal
windows .zip archive).
As for the other suggestion, I'm not sure whether using
BiocFileCache::BiocFileCache() will be more efficient than using
parallel apply for example returning all the results to one processor
and then broadcasting them out to all. If every processor is to write
its intermediate results to a file and those files are to be read by
every other processor then there will be "n" file reads, where each file
is possibly on a different location on the disk. Do you happen to have
any performance evaluation results for this method. But in any case I
did not know about BiocFileCache, thanks for sharing it.
I do not know of benchmarks. MPI would use shared memory, perhaps backed
by the file system. Serializing results to the master and back to the
worker would involve R memory management, which is not optimized for
this type of work flow. Saving to disk would provide whatever
efficiencies and guarantees offered by the OS; it seems like these would
be 'more' than R (a trivial example being serialization of very large
vectors, which will cause R to fail) and less the MPI.
Martin
Thanks,
Armen
On Thu, Feb 8, 2018 at 7:58 AM, Martin Morgan
<martin.mor...@roswellpark.org <mailto:martin.mor...@roswellpark.org>>
wrote:
On 02/07/2018 05:01 PM, Armen Abnousi wrote:
Hi Valerie,
Thanks for your reply. I can make it work using snow I believe.
But the
problem is that all functions in these packages that you have
suggested
(including snow; as far as I have seen) return the answer to one
node and
then if I want all nodes to have the same answer I will need to
broadcast
it to all nodes or repeat the same operation. This is very
prohibitive,
requiring a lot of communication. On the other hand if I could
use MPI
functionality, allGather would cut the operation to half (if not
more,
given the size of the communication that will be needed. I
haven't been
able to find a package with a functionality similar to allGather
that will
work without MPI. With larger dataset sizes, this problem
becomes more
threatening and that is just the opposite of what drives us to use
parallelism. I don't know if there is any other option, maybe we
can't do
better, but I thought I'll let you know what my problem is and
maybe you'll
suggest a better way to deal with it. Thank you again.
as a sub-optimal solution one could use the file system for data
transfer, sending tempfile() or app-specific
BiocFileCache::BiocFileCache() file paths between nodes; access
could be synchronized using BiocParallel::ipc* mutex.
Although the build system does not support MPI, sufficiently
motivated users might go to the trouble of installing it; you could
make your parallel code conditional on presence of MPI, defaulting
to serial (maybe with an encouraging message()) otherwise. MPI
depedencies would be moved to the Enhances: field, I think...
Martin
Thanks,
Armen
On Wed, Jan 31, 2018 at 2:37 PM, Obenchain, Valerie <
valerie.obench...@roswellpark.org
<mailto:valerie.obench...@roswellpark.org>> wrote:
Hi Armen,
MPI is not installed on the Windows builders so pbdMPI is
not supported.
Was there a specific reason you chose pbdMPI? There are
several other
options for parallel evaluation which are supported across
all 3 platforms,
e.g., BiocParallel, doParallel and batchtools.
I took a brief look at what naddaR is importing from pdbMPI
and didn't see
anything unusual - I believe you can find all functionality
in one of the
alternatives. Please have a look and let us know if one of
these will work
for you.
Thanks.
Valerie
On 01/27/2018 12:35 PM, Armen Abnousi wrote:
Hi,
I'm submitting a package for build, but for windows it goes
to tokay2 and
fails during the build with the following message:
----------------------------------------------------------------------------------
* checking for file 'naddaR/DESCRIPTION' ... OK
* preparing 'naddaR':
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
-----------------------------------
* installing *source* package 'naddaR' ...
** R
** inst
** preparing package for lazy loading
Error : .onLoad failed in loadNamespace() for 'pbdMPI', details:
call: inDL(x, as.logical(local), as.logical(now), ...)
error: unable to load shared object
'C:/Users/pkgbuild/packagebuilder/workers/jobs/626/R-libs/pbdMPI/libs/x64/pbdMPI.dll':
LoadLibrary failure: The specified module could not be
found.
ERROR: lazy loading failed for package 'naddaR'
* removing
'C:/Users/pkgbuild/AppData/Local/Temp/Rtmp4kcgED/Rinst1ffc2775216d/naddaR'
In R CMD INSTALL
-----------------------------------
ERROR: package installation failed
------------------------------------------------------------
-----------------------
How can I change this? It builds for Linux and Mac on
Merida2 and Malbec2.
Thanks,
Armen
[[alternative HTML version deleted]]
_______________________________________________bioc-de...@r-project.org
<mailto:Bioc-devel@r-project.org> mailing
listhttps://stat.ethz.ch/mailman/listinfo/bioc-devel
<http://stat.ethz.ch/mailman/listinfo/bioc-devel>
This email message may contain legally privileged and/or
confidential
information. If you are not the intended recipient(s), or
the employee or
agent responsible for the delivery of this message to the
intended
recipient(s), you are hereby notified that any disclosure,
copying,
distribution, or use of this email message is prohibited. If
you have
received this message in error, please notify the sender
immediately by
e-mail and delete this email message from your computer.
Thank you.
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org <mailto:Bioc-devel@r-project.org>
mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
<https://stat.ethz.ch/mailman/listinfo/bioc-devel>
This email message may contain legally privileged and/or
confidential information. If you are not the intended recipient(s),
or the employee or agent responsible for the delivery of this
message to the intended recipient(s), you are hereby notified that
any disclosure, copying, distribution, or use of this email message
is prohibited. If you have received this message in error, please
notify the sender immediately by e-mail and delete this email
message from your computer. Thank you.
This email message may contain legally privileged and/or...{{dropped:2}}
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
