Re: [go-nuts] Is there a Go native approch to MPI (Message passing interface) ?

[Samuel Lampa]

Nice, thanks a lot for sharing!

Best
Samuel

On Monday, September 14, 2020 at 1:07:25 PM UTC+2 rcore...@gmail.com wrote:

> I just wrote a wrapper around open mpi in Go: https://github.com/emer/empi
>
> Also, here's a set of random go bindings I found:
> • https://github.com/yoo/go-mpi
> • https://github.com/marcusthierfelder/mpi
> • https://github.com/JohannWeging/go-mpi
> Even a from scratch implementation:
>
> • https://github.com/btracey/mpi
> Also a few libraries for using infiniband directly:
>
> • https://github.com/Mellanox/rdmamap
> • https://github.com/jsgilmore/ib
> • https://github.com/Mellanox/libvma
> Some discussion: 
> https://groups.google.com/forum/#!topic/golang-nuts/t7Vjpfu0sjQ
>
> - Randy
>
> > On Sep 14, 2020, at 3:25 AM, Samuel Lampa  wrote:
> > 
> > On Thursday, June 11, 2015 at 2:53:36 PM UTC+2 Egon wrote:
> > 1. In which cases a cluster of say 4 (or 10 or 100 for instance) 
> Raspberry Pi mini computers can be more cost-effective than a single 
> computer with the same amount of cores (does the cost of communicating the 
> data between the computers via the network not outweigh the fact that they 
> car run tasks simultaneously) ?
> > 
> > The general answer is Amdahl's Law (
> http://en.wikipedia.org/wiki/Amdahl%27s_law), of course it's not always 
> applicable (
> http://www.futurechips.org/thoughts-for-researchers/parallel-programming-gene-amdahl-said.html).
>  
> When moving things to multiple-computers you'll get a larger overhead in 
> communication when compared to a single-computer, at the same time you may 
> reduce resource-contention for disk, RAM (or other resources). So depending 
> where your bottlenecks are, it could go either way...
> > 
> > Yes, and also note that super-computers often use special network 
> protocols/technologies which support so called "Remote direct memory 
> access" (RDMA) [1], such as Infiniband [2], to get acceptable performance 
> for high-performance multi-core computations across compute nodes. 
> Infiniband cards are pretty expensive as far as I know, so will probably 
> outweigh the benefits of buying a lot of RPis.
> > 
> > I'd still be interested to hear if anybody knows about new developments 
> on MPI for Go (for HPC use cases if nothing else)? :)
> > 
> > [1] https://en.wikipedia.org/wiki/Remote_direct_memory_access
> > [2] https://en.wikipedia.org/wiki/InfiniBand
> > 
> > Best
> > Samuel
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/d1d39602-e48e-4c2e-909b-a85d0d7e81ban%40googlegroups.com
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6d58b655-a017-4d8b-8446-f8687eb1412an%40googlegroups.com.

[go-nuts] Re: Is there a Go native approch to MPI (Message passing interface) ?

[Samuel Lampa]

On Thursday, June 11, 2015 at 2:53:36 PM UTC+2 Egon wrote:

> 1. In which cases a cluster of say 4 (or 10 or 100 for instance) Raspberry 
>> Pi mini computers can be more cost-effective than a single computer with 
>> the same amount of cores (does the cost of communicating the data between 
>> the computers via the network not outweigh the fact that they car run tasks 
>> simultaneously) ?
>>
>
> The general answer is Amdahl's Law (
> http://en.wikipedia.org/wiki/Amdahl%27s_law), of course it's not always 
> applicable (
> http://www.futurechips.org/thoughts-for-researchers/parallel-programming-gene-amdahl-said.html).
>  
> When moving things to multiple-computers you'll get a larger overhead in 
> communication when compared to a single-computer, at the same time you may 
> reduce resource-contention for disk, RAM (or other resources). So depending 
> where your bottlenecks are, it could go either way...
>

Yes, and also note that super-computers often use special network 
protocols/technologies which support so called "Remote direct memory 
access" (RDMA) [1], such as Infiniband [2], to get acceptable performance 
for high-performance multi-core computations across compute nodes. 
Infiniband cards are pretty expensive as far as I know, so will probably 
outweigh the benefits of buying a lot of RPis.

I'd still be interested to hear if anybody knows about new developments on 
MPI for Go (for HPC use cases if nothing else)? :)

[1] https://en.wikipedia.org/wiki/Remote_direct_memory_access
[2] https://en.wikipedia.org/wiki/InfiniBand

Best
Samuel

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d1d39602-e48e-4c2e-909b-a85d0d7e81ban%40googlegroups.com.

[go-nuts] Re: Is there a workflow engine written in GO

[Samuel Lampa]

You might want to check out http://scipipe.org

We developed it and are currently using it to run cheminformatics pipelines 
(data mangling, machine learning etc) on a HPC cluster, at http://pharmb.io  
at Uppsala University.

Cheers
// Samuel 

On Friday, May 8, 2015 at 2:30:20 PM UTC+2, Sriram Jagadeesan wrote:
>
> I am trying to automated tasks( mainly server task with command line 
> process).
> I am looking for workflow engine if anything written in GOlang, which 
> could be used ?.
> There were some BPM workflow engines, but not suited for command line 
> server operations.
>
> Any input is high appreciated
>
> /Sriram.J 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Any way to somehow include readable source code into compiled binaries?

[Samuel Lampa]

Hi Seb! :)

On Thursday, October 19, 2017 at 9:00:46 PM UTC+2, Sebastien Binet wrote:
>
> On Thu, Oct 19, 2017 at 8:51 PM, Alex Buchanan  > wrote:
>
>> But, maybe a better solution is to faithfully record the current git (or 
>> other VCS) details you'd need to find the code which resulted in the 
>> binary? That way you have access to the source in its natural state.
>>
>
> yes.
> using go-bindata is a nice, quick'n'dirty solution for the problem at 
> hand, but I fear it's just duplication of work somehow.
> why not just using "dep" that captures the state of the code (ie: git sha) 
> *and* its dependencies.
>
> this sounds more reliable. at least to me.
> and it encourages people to put their code into a repository which, for 
> reproducibility of scientific results, is step-0.
>

Absolutely, and this is what we do for our own use.

What I'm thinking about is how the ability to compile a workflow to a 
binary will make it easy to ship "ready made" workflows for less tech-savvy 
users, who might not be comfortable using git and the go tools, or might do 
unwarranted, undocumented changes if we'd just give them the Go source and 
the Go tools.

Best
// Samuel

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Any way to somehow include readable source code into compiled binaries?

[Samuel Lampa]

We use go-bindata to bundle text files (html, js, css, yaml) into our 
binary. I bet you could use it to bundle Go source.


https://github.com/jteeuwen/go-bindata 





Ah, this is great, Thanks!

On 2017-10-19 20:51, Alex Buchanan wrote:
But, maybe a better solution is to faithfully record the current git 
(or other VCS) details you'd need to find the code which resulted in 
the binary? That way you have access to the source in its natural state.


Yes, this is what we do not, but it turns out, in reality it is often 
too easy that these get disconnected for whatever reason, like that very 
tempting last minute fix, file open in multiple editors at the same time 
(we're on a shared cluster), or similar. Having a "locked" link between 
what is executed and its definition, will make us sleep better at night, 
I think :)


Thanks, and Cheers
// Samuel

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Any way to somehow include readable source code into compiled binaries?

[Samuel Lampa]

Hi Gophers,

*Question:* Is there any way to include a readable verison of the source 
code of my program into the compiled Go binary, either in plain text, or 
the ability extract it somehow, with some go tooling or 3rd party tools?

*Background:* The reason for asking is that we are using Go to write 
scientific workflows with scipipe . This means we can 
compile our scientific workflows into static binaries for easier deployment 
and protecting against undocumented code changes that code make us loose 
track of the provenance ("complete track record") of the steps ... and 
potentially lead to flawed science ("shudder").

But with compiled binaries, we instead run into another problem of 
provenance: Allowing ourselves and others to inspect that a particular 
workflow binary actually does what it says.

Thus, having a statically compiled binary with the source code included in 
some form, would be the best of two worlds.

Cheers
// Samuel
PhD Student @ pharmb.io

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.