Having used Whirr several times in EC2, it seems like a fine way to spin up
a temporary 'developers' cluster.  Zookeeper is the most likely source of
difficulty on VMs with limited I/O (i.e., it's very chatty and doesn't
tolerate the highly variable latency that smaller AMIs provide).  The HBase
community seems to be very aware of this; there's likely some tips and
tricks to be gleaned from reading their mailing lists.

-Dan

On Wed, Apr 4, 2012 at 11:08 PM, Brian Femiano <bfemi...@gmail.com> wrote:

> I've used it on clusters I started on EC2 launched by Whirr. Simply copy
> the fat
> jar to your client machine and it will distribute normally as a M/R
> dependency.
>
> It works very well.
>
> The only limitation I could potentially find (without much proof) was on
> VMs
> with limited IO the RPC message overhead between workers could be an issue.
> I never tried it on VMs with less than 'High' IO, so take that with a grain
> of salt.
>
> On Thu, Apr 5, 2012 at 12:51 AM, Jakob Homan <jgho...@gmail.com> wrote:
>
> > This is interesting.  Whirr can already spin up Hadoop MR clusters,
> > which can then run the Giraph jobs.  Once Giraph is bootstrapped onto
> > YARN, this will make more sense as a Whirr service.
> >
> > On Wed, Apr 4, 2012 at 9:43 PM, Avery Ching <ach...@apache.org> wrote:
> > > I don't use Whirr...I haven't heard it mentioned on this forum yet.
> >  Anyone?
> > >
> > > Avery
> > >
> > >
> > > On 4/4/12 9:30 PM, Paolo Castagna wrote:
> > >>
> > >> Hi,
> > >> seen this?
> > >>
> > >>   WHIRR-530 - Add Giraph as a service
> > >>   https://issues.apache.org/jira/browse/WHIRR-530
> > >>
> > >> This could be quite useful for users who want to give Giraph a spin on
> > >> cloud
> > >> infrastructure, just for testing or to run a few small experiments.
> > >> My experience with Whirr an small 10-20 nodes clusters has be quite
> > >> positive.
> > >> Less so for larger clusters, but it more a problem/limit with the
> cloud
> > >> provider rather than Whirr itself. I think.
> > >>
> > >> Whirr makes extremely easy and pleasant deploy stuff on-demand.
> > >>
> > >> ... and Whirr already supports YARN:
> > >> https://issues.apache.org/jira/browse/WHIRR-391
> > >>
> > >> Is any Giraph developers/users here also a Whirr user?
> > >>
> > >> Paolo
> > >
> > >
> >
>



-- 
Daniel McClary, Ph.D.
Visiting Scholar
*Amaral Lab and Department of Chemical and Biological Engineering,
Northwestern University*
Bioinformatics Specialist II
*Howard Hughes Medical Institute*
Email: dan.mccl...@northwestern.edu
Phone: (847) 491-1234
Web: http://amaral-lab.org/people/mcclary/
Mailing address:
2145 Sheridan Rd, Room E-136
Northwestern University
Evanston, IL 60208

Reply via email to