Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop
+1 thanks Ricardo ! > On Apr 7, 2018, at 11:29 AM, Evans Ye wrote: > > Got it. we welcome any kind of contribution. If you find some packaging, > testing, or doc need to be improved, feel free to contribute :) > > Riccardo Murri 於 2018年4月7日 週六,下午4:10寫道: >> Hello! >> >> (Evans Ye, Fri, Apr 06, 2018 at 04:17:27PM +:) >> > Cool. I’m really happy to see people leveraging community works to build up >> > its software stack. It seems that you are using a bigtop packages only. >> > Any reason you didn’t choose provisioner and puppet? Your feedback can help >> > us polish the feature further. >> >> Well, ElastiCluster was born to create HPC/batch-queuing clusters, and >> for that we had already an existing infrastructure built around Ansible >> and Python. I found it simpler to just re-use the packages and the >> tweak the example configs, than to provide another layer of glue to run >> Puppet and the Bigtop Provisioner. (Somebody more versed in Ruby and >> Puppet might have made a different choice.) >> >> Again, thanks for the excellent packaging, the testing, and the repos! >> Already that part alone, even without the provisioner, is a terrific >> help to set up a working Hadoop/Spark environment. >> >> >> > BTW, although we welcome any bigtop related stuffs to be posted here. Using >> > the announce tag is not appropriate. People May misunderstand that the >> > message is sent by Bigtop PMC. >> >> Sorry fot that! I was looking for a way to tag the message as "news" >> but I agree that the choice of the "[announce]" tag was misguided. My >> apologies. >> >> Kind regards, >> Riccardo >> >> -- >> Riccardo Murri >> >> S3IT: Services and Support for Science IT >> University of Zurich
Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop
Got it. we welcome any kind of contribution. If you find some packaging, testing, or doc need to be improved, feel free to contribute :) Riccardo Murri 於 2018年4月7日 週六,下午4:10寫道: > Hello! > > (Evans Ye, Fri, Apr 06, 2018 at 04:17:27PM +:) > > Cool. I’m really happy to see people leveraging community works to build > up > > its software stack. It seems that you are using a bigtop packages only. > > Any reason you didn’t choose provisioner and puppet? Your feedback can > help > > us polish the feature further. > > Well, ElastiCluster was born to create HPC/batch-queuing clusters, and > for that we had already an existing infrastructure built around Ansible > and Python. I found it simpler to just re-use the packages and the > tweak the example configs, than to provide another layer of glue to run > Puppet and the Bigtop Provisioner. (Somebody more versed in Ruby and > Puppet might have made a different choice.) > > Again, thanks for the excellent packaging, the testing, and the repos! > Already that part alone, even without the provisioner, is a terrific > help to set up a working Hadoop/Spark environment. > > > > BTW, although we welcome any bigtop related stuffs to be posted here. > Using > > the announce tag is not appropriate. People May misunderstand that the > > message is sent by Bigtop PMC. > > Sorry fot that! I was looking for a way to tag the message as "news" > but I agree that the choice of the "[announce]" tag was misguided. My > apologies. > > Kind regards, > Riccardo > > -- > Riccardo Murri > > S3IT: Services and Support for Science IT > University of Zurich >
Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop
Hello! (Evans Ye, Fri, Apr 06, 2018 at 04:17:27PM +:) > Cool. I’m really happy to see people leveraging community works to build up > its software stack. It seems that you are using a bigtop packages only. > Any reason you didn’t choose provisioner and puppet? Your feedback can help > us polish the feature further. Well, ElastiCluster was born to create HPC/batch-queuing clusters, and for that we had already an existing infrastructure built around Ansible and Python. I found it simpler to just re-use the packages and the tweak the example configs, than to provide another layer of glue to run Puppet and the Bigtop Provisioner. (Somebody more versed in Ruby and Puppet might have made a different choice.) Again, thanks for the excellent packaging, the testing, and the repos! Already that part alone, even without the provisioner, is a terrific help to set up a working Hadoop/Spark environment. > BTW, although we welcome any bigtop related stuffs to be posted here. Using > the announce tag is not appropriate. People May misunderstand that the > message is sent by Bigtop PMC. Sorry fot that! I was looking for a way to tag the message as "news" but I agree that the choice of the "[announce]" tag was misguided. My apologies. Kind regards, Riccardo -- Riccardo Murri S3IT: Services and Support for Science IT University of Zurich
Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop
Wow! Really nice! Thanks for sharing with us! Thanks, Bruno On 04/04/2018 11:59 AM, Riccardo Murri wrote: Hello! I would like to bring to your attention ElastiCluster [1] [2], a tool for deploy verious kinds of compute clusters on IaaS clouds. Thanks to BigTop (and to the developers behind it!), ElastiCluster can also deploy functional Hadoop+Spark clusters [3]. ElastiCluster does not use the BigTop provisioner, instead opts for its own Ansible-based deployment playbooks: the provisioned software is currently limited to Hadoop + Spark + Thriftserver (from BigTop 1.2.1), but they can be integrated with other non-BigTop software (e.g., JupyterHub). AFAIK, the main use for Hadoop+Spark on ElastiCluster so far has been setting up small clusters for teaching purposes; I'd be glad for any feedback, and especially if anyone is willing to try it for more "serious" use cases, as well as discussing more general topics (here or on the ElastiCluster mailing-list). [1]: http://elasticluster.readthedocs.io/en/latest/ [2]: http://elasticluster.readthedocs.io/en/latest/install.html#quickstart [3]: http://elasticluster.readthedocs.io/en/latest/playbooks.html#hadoop-spark (I hope this kind of announcements is welcome on the list; I could find no policy on allowed topics on the BigTop web site and the mailing list index.) Kind regards, Riccardo -- Riccardo Murri S3IT: Services and Support for Science IT University of Zurich
Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop
Cool. I’m really happy to see people leveraging community works to build up its software stack. It seems that you are using a bigtop packages only. Any reason you didn’t choose provisioner and puppet? Your feedback can help us polish the feature further. BTW, although we welcome any bigtop related stuffs to be posted here. Using the announce tag is not appropriate. People May misunderstand that the message is sent by Bigtop PMC. Riccardo Murri 於 2018年4月5日 週四,上午2:59寫道: > Hello! > > I would like to bring to your attention ElastiCluster [1] [2], a tool > for deploy verious kinds of compute clusters on IaaS clouds. Thanks to > BigTop (and to the developers behind it!), ElastiCluster can also deploy > functional Hadoop+Spark clusters [3]. > > ElastiCluster does not use the BigTop provisioner, instead opts for its > own Ansible-based deployment playbooks: the provisioned software is > currently limited to Hadoop + Spark + Thriftserver (from BigTop 1.2.1), > but they can be integrated with other non-BigTop software (e.g., > JupyterHub). > > AFAIK, the main use for Hadoop+Spark on ElastiCluster so far has been > setting up small clusters for teaching purposes; I'd be glad for any > feedback, and especially if anyone is willing to try it for more > "serious" use cases, as well as discussing more general topics (here or > on the ElastiCluster mailing-list). > > [1]: http://elasticluster.readthedocs.io/en/latest/ > [2]: http://elasticluster.readthedocs.io/en/latest/install.html#quickstart > [3]: > http://elasticluster.readthedocs.io/en/latest/playbooks.html#hadoop-spark > > (I hope this kind of announcements is welcome on the list; I could find > no policy on allowed topics on the BigTop web site and the mailing list > index.) > > Kind regards, > Riccardo > > -- > Riccardo Murri > > S3IT: Services and Support for Science IT > University of Zurich >
[Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop
Hello! I would like to bring to your attention ElastiCluster [1] [2], a tool for deploy verious kinds of compute clusters on IaaS clouds. Thanks to BigTop (and to the developers behind it!), ElastiCluster can also deploy functional Hadoop+Spark clusters [3]. ElastiCluster does not use the BigTop provisioner, instead opts for its own Ansible-based deployment playbooks: the provisioned software is currently limited to Hadoop + Spark + Thriftserver (from BigTop 1.2.1), but they can be integrated with other non-BigTop software (e.g., JupyterHub). AFAIK, the main use for Hadoop+Spark on ElastiCluster so far has been setting up small clusters for teaching purposes; I'd be glad for any feedback, and especially if anyone is willing to try it for more "serious" use cases, as well as discussing more general topics (here or on the ElastiCluster mailing-list). [1]: http://elasticluster.readthedocs.io/en/latest/ [2]: http://elasticluster.readthedocs.io/en/latest/install.html#quickstart [3]: http://elasticluster.readthedocs.io/en/latest/playbooks.html#hadoop-spark (I hope this kind of announcements is welcome on the list; I could find no policy on allowed topics on the BigTop web site and the mailing list index.) Kind regards, Riccardo -- Riccardo Murri S3IT: Services and Support for Science IT University of Zurich