Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop

[Jay Vyas]

+1 thanks Ricardo !


> On Apr 7, 2018, at 11:29 AM, Evans Ye  wrote:
> 
> Got it. we welcome any kind of contribution. If you find some packaging, 
> testing, or doc need to be improved, feel free to contribute :)
> 
> Riccardo Murri 於 2018年4月7日 週六，下午4:10寫道：
>> Hello!
>> 
>> (Evans Ye, Fri, Apr 06, 2018 at 04:17:27PM +:)
>> > Cool. I’m really happy to see people leveraging community works to build up
>> > its software  stack. It seems that you are using a bigtop packages only.
>> > Any reason you didn’t choose provisioner and puppet? Your feedback can help
>> > us polish the feature further.
>> 
>> Well, ElastiCluster was born to create HPC/batch-queuing clusters, and
>> for that we had already an existing infrastructure built around Ansible
>> and Python.  I found it simpler to just re-use the packages and the
>> tweak the example configs, than to provide another layer of glue to run
>> Puppet and the Bigtop Provisioner.  (Somebody more versed in Ruby and
>> Puppet might have made a different choice.)
>> 
>> Again, thanks for the excellent packaging, the testing, and the repos!
>> Already that part alone, even without the provisioner, is a terrific
>> help to set up a working Hadoop/Spark environment.
>> 
>> 
>> > BTW, although we welcome any bigtop related stuffs to be posted here. Using
>> > the announce tag is not appropriate. People May misunderstand that the
>> > message is sent by Bigtop PMC.
>> 
>> Sorry fot that!  I was looking for a way to tag the message as "news"
>> but I agree that the choice of the "[announce]" tag was misguided. My
>> apologies.
>> 
>> Kind regards,
>> Riccardo
>> 
>> --
>> Riccardo Murri
>> 
>> S3IT: Services and Support for Science IT
>> University of Zurich

Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop

[Evans Ye]

Got it. we welcome any kind of contribution. If you find some packaging,
testing, or doc need to be improved, feel free to contribute :)

Riccardo Murri 於 2018年4月7日 週六，下午4:10寫道：

> Hello!
>
> (Evans Ye, Fri, Apr 06, 2018 at 04:17:27PM +:)
> > Cool. I’m really happy to see people leveraging community works to build
> up
> > its software  stack. It seems that you are using a bigtop packages only.
> > Any reason you didn’t choose provisioner and puppet? Your feedback can
> help
> > us polish the feature further.
>
> Well, ElastiCluster was born to create HPC/batch-queuing clusters, and
> for that we had already an existing infrastructure built around Ansible
> and Python.  I found it simpler to just re-use the packages and the
> tweak the example configs, than to provide another layer of glue to run
> Puppet and the Bigtop Provisioner.  (Somebody more versed in Ruby and
> Puppet might have made a different choice.)
>
> Again, thanks for the excellent packaging, the testing, and the repos!
> Already that part alone, even without the provisioner, is a terrific
> help to set up a working Hadoop/Spark environment.
>
>
> > BTW, although we welcome any bigtop related stuffs to be posted here.
> Using
> > the announce tag is not appropriate. People May misunderstand that the
> > message is sent by Bigtop PMC.
>
> Sorry fot that!  I was looking for a way to tag the message as "news"
> but I agree that the choice of the "[announce]" tag was misguided. My
> apologies.
>
> Kind regards,
> Riccardo
>
> --
> Riccardo Murri
>
> S3IT: Services and Support for Science IT
> University of Zurich
>

Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop

[Riccardo Murri]

Hello!

(Evans Ye, Fri, Apr 06, 2018 at 04:17:27PM +:)
> Cool. I’m really happy to see people leveraging community works to build up
> its software  stack. It seems that you are using a bigtop packages only.
> Any reason you didn’t choose provisioner and puppet? Your feedback can help
> us polish the feature further.

Well, ElastiCluster was born to create HPC/batch-queuing clusters, and
for that we had already an existing infrastructure built around Ansible
and Python.  I found it simpler to just re-use the packages and the
tweak the example configs, than to provide another layer of glue to run
Puppet and the Bigtop Provisioner.  (Somebody more versed in Ruby and
Puppet might have made a different choice.)

Again, thanks for the excellent packaging, the testing, and the repos!
Already that part alone, even without the provisioner, is a terrific
help to set up a working Hadoop/Spark environment.


> BTW, although we welcome any bigtop related stuffs to be posted here. Using
> the announce tag is not appropriate. People May misunderstand that the
> message is sent by Bigtop PMC.

Sorry fot that!  I was looking for a way to tag the message as "news"
but I agree that the choice of the "[announce]" tag was misguided. My
apologies.

Kind regards,
Riccardo

--
Riccardo Murri

S3IT: Services and Support for Science IT
University of Zurich

Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop

[Bruno Mahé]

Wow! Really nice!


Thanks for sharing with us!


Thanks,

Bruno


On 04/04/2018 11:59 AM, Riccardo Murri wrote:

Hello!

I would like to bring to your attention ElastiCluster [1] [2], a tool
for deploy verious kinds of compute clusters on IaaS clouds.  Thanks to
BigTop (and to the developers behind it!), ElastiCluster can also deploy
functional Hadoop+Spark clusters [3].

ElastiCluster does not use the BigTop provisioner, instead opts for its
own Ansible-based deployment playbooks: the provisioned software is
currently limited to Hadoop + Spark + Thriftserver (from BigTop 1.2.1),
but they can be integrated with other non-BigTop software (e.g.,
JupyterHub).

AFAIK, the main use for Hadoop+Spark on ElastiCluster so far has been
setting up small clusters for teaching purposes; I'd be glad for any
feedback, and especially if anyone is willing to try it for more
"serious" use cases, as well as discussing more general topics (here or
on the ElastiCluster mailing-list).

[1]: http://elasticluster.readthedocs.io/en/latest/
[2]: http://elasticluster.readthedocs.io/en/latest/install.html#quickstart
[3]: http://elasticluster.readthedocs.io/en/latest/playbooks.html#hadoop-spark

(I hope this kind of announcements is welcome on the list; I could find
no policy on allowed topics on the BigTop web site and the mailing list index.)

Kind regards,
Riccardo

--
Riccardo Murri

S3IT: Services and Support for Science IT
University of Zurich

Re: [Announce] ElastiCluster: a tool to deploy Hadoop/Spark clusters based on BigTop

[Evans Ye]

Cool. I’m really happy to see people leveraging community works to build up
its software  stack. It seems that you are using a bigtop packages only.
Any reason you didn’t choose provisioner and puppet? Your feedback can help
us polish the feature further.

BTW, although we welcome any bigtop related stuffs to be posted here. Using
the announce tag is not appropriate. People May misunderstand that the
message is sent by Bigtop PMC.

Riccardo Murri 於 2018年4月5日 週四，上午2:59寫道：

> Hello!
>
> I would like to bring to your attention ElastiCluster [1] [2], a tool
> for deploy verious kinds of compute clusters on IaaS clouds.  Thanks to
> BigTop (and to the developers behind it!), ElastiCluster can also deploy
> functional Hadoop+Spark clusters [3].
>
> ElastiCluster does not use the BigTop provisioner, instead opts for its
> own Ansible-based deployment playbooks: the provisioned software is
> currently limited to Hadoop + Spark + Thriftserver (from BigTop 1.2.1),
> but they can be integrated with other non-BigTop software (e.g.,
> JupyterHub).
>
> AFAIK, the main use for Hadoop+Spark on ElastiCluster so far has been
> setting up small clusters for teaching purposes; I'd be glad for any
> feedback, and especially if anyone is willing to try it for more
> "serious" use cases, as well as discussing more general topics (here or
> on the ElastiCluster mailing-list).
>
> [1]: http://elasticluster.readthedocs.io/en/latest/
> [2]: http://elasticluster.readthedocs.io/en/latest/install.html#quickstart
> [3]:
> http://elasticluster.readthedocs.io/en/latest/playbooks.html#hadoop-spark
>
> (I hope this kind of announcements is welcome on the list; I could find
> no policy on allowed topics on the BigTop web site and the mailing list
> index.)
>
> Kind regards,
> Riccardo
>
> --
> Riccardo Murri
>
> S3IT: Services and Support for Science IT
> University of Zurich
>