Please read this as well, thanks Disclaimer: it's my article.
https://medium.com/@ravishankar.nair/online-and-batch-based-ml-execution-from-same-python-code-preserving-pre-and-post-transformation-ea7ebc27f50f?sk=c33bcf1d6c28b562b7bd36fa39809294 Best, Ravion On Mon, Sep 7, 2020, 8:29 AM Enrico Minack <m...@enrico.minack.dev> wrote: > You could use Horovod to distribute your ML algorithm on a cluster, while > Horovod also supports Spark clusters. > > Enrico > > > Am 06.09.20 um 15:30 schrieb Ankur Das: > > > Good Evening Sir/Madam, > Hope you are doing well, I am experimenting on some ML techniques where I > need to test it on a distributed environment. > For example a particular algorithm I want to run it on different nodes at > the same time and collect the results at the end in one single node or the > parent node. > > So, I would like to know if it is possible or a good choice to use spark > for this. > > Hope to hear from you soon, Stay safe and healthy > Thanking you in advance. > -- > Regards, > Ankur J Das > Research Scholar @ Tezpur University > Tezpur, Assam > > >
