Hey Sonali, I believe the point at which YARN became version compatible for 2.* as at 2.1.0-beta. I believe 2.0.5 is not API compatible with later versions of YARN (e.g. 2.2). For this reason, you'll need to upgrade your YARN grid, or use a different one with a higher version.
For its part, Samza should work with YARN grids 2.1.0-beta and beyond, though I haven't tested this. The YARN community has given a commitment to maintaining API compatibility going forward for YARN 2.*, which means that future upgrades should not be required, until YARN 3 comes out. The rest of your understanding is correct. You can run a 1 RM, 2 NM kind of cluster, throw some Kafka brokers on there, and you should be good to go. You can also re-use your existing ZK, if you wish. Cheers, Chris On 2/3/14 10:42 AM, "[email protected]" <[email protected]> wrote: >Thanks Chris/Gary. > >I have an existing Zookeeper and YARN Cluster. However, the YARN version >that I have (that came preinstalled with Pivotal HD) is 2.0.5. So from >what you're saying I cannot reuse it for my Samza deployment. > >So then my option is: >1. Reuse zookeeper. So I'll have to configure Samza to point to the right >cluster >2. Run Samza with its YARN grid and Kafka Installation (I can do this on >multiple servers right? 1 RM, 2 NM kind of situation) > >Thanks, >Sonali > > >-----Original Message----- >From: Chris Riccomini [mailto:[email protected]] >Sent: Friday, January 31, 2014 11:24 AM >To: [email protected] >Subject: Re: Cluster Installation > >Hey Sonali, > >Everything Gary said is correct. > >One other item of note is that if you're interested in running stuff >locally in a dev-mode fashion, you don't need YARN. You can use the >LocalJobFactory instead of the YarnJobFactory factory when configuring >your job's "job.factory.class" setting. > >For "real" deployments, yes you'll need YARN, ZooKeeper, and Kafka. They >can be deployed using any standard way of shipping software around to a >cluster of machines. > >Cheers, >Chris > >On 1/31/14 12:58 AM, "Garry Turkington" <[email protected]> >wrote: > >>Hi Sonali, >> >>This was something that I had some questions about originally as well. >>In terms of required components then yes, for any size of Samza >>deployment you will need all those pieces. >> >>In terms of actual deployment, from what I understand from the LinkedIn >>guys they do run Samza on a dedicated YARN grid that also has a Kafka >>broker collocated on each node. These decisions though appear to be >>more down to convenience than a hard requirement. >> >>In my own setup I have existing ZooKeeper and Kafka clusters that I'm >>pointing Samza at but do need to run a dedicated YARN grid because my >>Hadoop cluster has a pre-2.2 version of YARN running on it. >> >>So if you have existing components you can reuse them, if not then >>repurposing the Hello Samza package is a good starting point to get all >>the things you want on the required hosts. Only caveat would be to not >>drop a ZK node on each host, the ZK quorum should follow the usual >>advice of an odd number of servers and likely no more than 3, 5 or 7 >>depending on your deployment size. >> >>Garry >> >>-----Original Message----- >>From: [email protected] >>[mailto:[email protected]] >>Sent: 30 January 2014 23:38 >>To: [email protected] >>Subject: Cluster Installation >> >>Hi All, >> >>I'm new to working with Samza and have been trying to figure out the >>best cluster configuration. I understand that Samza comes with >>yarn,kafka and zookeeper out of the box. Is that the model just for a >>standalone/local configuration. What if I want a bigger cluster? Do I >>have to install yarn, kafka and zookeeper separately? Any suggestions >>would be great! >> >>Thanks, >>Sonali >> >>Sonali Parthasarathy >>R&D Developer, Data Insights >>Accenture Technology Labs >>703-341-7432 >> >> >>________________________________ >> >>This message is for the designated recipient only and may contain >>privileged, proprietary, or otherwise confidential information. If you >>have received it in error, please notify the sender immediately and >>delete the original. Any other use of the e-mail by you is prohibited. >>Where allowed by local law, electronic communications with Accenture >>and its affiliates, including e-mail and instant messaging (including >>content), may be scanned by our systems for the purposes of information >>security and assessment of internal compliance with Accenture policy. . >>_______________________________________________________________________ >>___ >>____________ >> >>www.accenture.com >> >>----- >>No virus found in this message. >>Checked by AVG - www.avg.com >>Version: 2014.0.4259 / Virus Database: 3684/7046 - Release Date: >>01/30/14 > > > >________________________________ > >This message is for the designated recipient only and may contain >privileged, proprietary, or otherwise confidential information. If you >have received it in error, please notify the sender immediately and >delete the original. Any other use of the e-mail by you is prohibited. >Where allowed by local law, electronic communications with Accenture and >its affiliates, including e-mail and instant messaging (including >content), may be scanned by our systems for the purposes of information >security and assessment of internal compliance with Accenture policy. . >__________________________________________________________________________ >____________ > >www.accenture.com >
