Re: Giraph as Whirr service, see WHIRR-530
I've used it on clusters I started on EC2 launched by Whirr. Simply copy the fat jar to your client machine and it will distribute normally as a M/R dependency. It works very well. The only limitation I could potentially find (without much proof) was on VMs with limited IO the RPC message overhead between workers could be an issue. I never tried it on VMs with less than 'High' IO, so take that with a grain of salt. On Thu, Apr 5, 2012 at 12:51 AM, Jakob Homan jgho...@gmail.com wrote: This is interesting. Whirr can already spin up Hadoop MR clusters, which can then run the Giraph jobs. Once Giraph is bootstrapped onto YARN, this will make more sense as a Whirr service. On Wed, Apr 4, 2012 at 9:43 PM, Avery Ching ach...@apache.org wrote: I don't use Whirr...I haven't heard it mentioned on this forum yet. Anyone? Avery On 4/4/12 9:30 PM, Paolo Castagna wrote: Hi, seen this? WHIRR-530 - Add Giraph as a service https://issues.apache.org/jira/browse/WHIRR-530 This could be quite useful for users who want to give Giraph a spin on cloud infrastructure, just for testing or to run a few small experiments. My experience with Whirr an small 10-20 nodes clusters has be quite positive. Less so for larger clusters, but it more a problem/limit with the cloud provider rather than Whirr itself. I think. Whirr makes extremely easy and pleasant deploy stuff on-demand. ... and Whirr already supports YARN: https://issues.apache.org/jira/browse/WHIRR-391 Is any Giraph developers/users here also a Whirr user? Paolo
Re: Giraph as Whirr service, see WHIRR-530
I don't use Whirr...I haven't heard it mentioned on this forum yet. Anyone? Avery On 4/4/12 9:30 PM, Paolo Castagna wrote: Hi, seen this? WHIRR-530 - Add Giraph as a service https://issues.apache.org/jira/browse/WHIRR-530 This could be quite useful for users who want to give Giraph a spin on cloud infrastructure, just for testing or to run a few small experiments. My experience with Whirr an small 10-20 nodes clusters has be quite positive. Less so for larger clusters, but it more a problem/limit with the cloud provider rather than Whirr itself. I think. Whirr makes extremely easy and pleasant deploy stuff on-demand. ... and Whirr already supports YARN: https://issues.apache.org/jira/browse/WHIRR-391 Is any Giraph developers/users here also a Whirr user? Paolo
Re: Giraph as Whirr service, see WHIRR-530
This is interesting. Whirr can already spin up Hadoop MR clusters, which can then run the Giraph jobs. Once Giraph is bootstrapped onto YARN, this will make more sense as a Whirr service. On Wed, Apr 4, 2012 at 9:43 PM, Avery Ching ach...@apache.org wrote: I don't use Whirr...I haven't heard it mentioned on this forum yet. Anyone? Avery On 4/4/12 9:30 PM, Paolo Castagna wrote: Hi, seen this? WHIRR-530 - Add Giraph as a service https://issues.apache.org/jira/browse/WHIRR-530 This could be quite useful for users who want to give Giraph a spin on cloud infrastructure, just for testing or to run a few small experiments. My experience with Whirr an small 10-20 nodes clusters has be quite positive. Less so for larger clusters, but it more a problem/limit with the cloud provider rather than Whirr itself. I think. Whirr makes extremely easy and pleasant deploy stuff on-demand. ... and Whirr already supports YARN: https://issues.apache.org/jira/browse/WHIRR-391 Is any Giraph developers/users here also a Whirr user? Paolo
