Hi Venkat, >From past three months, Inmobi has been using Oozie in uber mode. Currently we are using falcon-0.7 with oozie-3.2 (along with several patches specific to our use case). Following use cases have been tested and are active in our production environment.
1. NN HA/RM HA 2. Oozie ubermode for launcher job 3. Multi cluster with prism 4. Sanity of falcon features with oozie(in uber mode). After falcon-0.8 release we plan to go forward with Oozie HA feature. Please let me know if you have any more concerns. Regards, Pragya Mittal On Sun, Oct 25, 2015 at 8:15 AM, Srikanth Sundarrajan <[email protected]> wrote: > Am guessing venkat you meant to refer FALCON-1555. Good check list to go > through for each release. > > Specifically on > >> 6. Multi cluster managed with Oozie (with single Falcon instance) > > Since we haven't particularly done anything by isolating the classes and > their dependencies for each cluster with independent class loader, I think > this would work only as long as all the clusters are compatible. If things > were to change say with hadoop-3 for ex, this may no longer work. > > Regards > Srikanth Sundarrajan > > > From: [email protected] > > Date: Sun, 25 Oct 2015 05:29:20 +0530 > > Subject: Re: Oozie uber mode and other modes of running the software > > To: [email protected] > > > > Venkat, > > > > Sounds good, let's discuss it in the next sync up. For the benefit of > > everyone on the mailing list, I will summarise our use cases. Yes, we > > (Inmobi) use Oozie's uber mode in our production environment(IIRC we are > > using it since Apache Falcon 0.7). We do have several thousand jobs > running > > per day(several at minutely frequencies) and a heavily utilised cluster, > so > > this makes sense for us. We do test all Oozie releases, though sometimes > > our Oozie releases are not exactly aligned with Oozie's general release - > > it might contain some extra patches, might be from trunk etc. and our > > testing might be specific to features being used at Inmobi. In fact last > > time Pragya Mittal did the testing for both Apache Falcon 0.7 and Oozie. > > She can add more on this. > > > > List of scenarios looks comprehensive. Just curious, what storage > backends > > are other users using for titan? > > > > > > Cheers > > Ajay Yadava > > > > On Sat, Oct 24, 2015 at 9:20 PM, Venkat Ranganathan < > > [email protected]> wrote: > > > > > FALCON-1551 was interesting for two reasons - that we are still caught > by > > > the FS caching issue (this should have been dealt by most of the > projects > > > one time or another), but the use of Oozie uber mode - I am wondering > if we > > > have done uber mode testing generally - Is this the mode that is used > in > > > your environments – It makes sense when thousands of Oozie jobs are > run – > > > saving one container. I would like to understand more about your > > > experience with uber mode if you have – may be in the next Falcon sync > up. > > > > > > I think we have create a list of modes that we expect the product to > run > > > before we introduce changes generally – Given that there are different > sets > > > of execution modes that we want to test. Please add more on the > scenarios > > > that I am most likely missing. Let us discuss this also if we have > time. > > > > > > 1. Security (Kerberos and Wire encryption) > > > 2. NN HA/RM HA > > > 3. Oozie HA > > > 4. Oozie ubermode for launcher job > > > 5. Multi cluster with prism > > > 6. Multi cluster managed with Oozie (with single Falcon instance) > > > 7. Hadoop/Hive/Oozie versions and the implications (S3 and Azure > blob > > > storage went out of hadoop-common in 2.6.0+ and the implications). > > > > > > > > > Thanks > > > > > > Venkat > > > > > > > > > > -- > > _____________________________________________________________ > > The information contained in this communication is intended solely for > the > > use of the individual or entity to whom it is addressed and others > > authorized to receive it. It may contain confidential or legally > privileged > > information. If you are not the intended recipient you are hereby > notified > > that any disclosure, copying, distribution or taking any action in > reliance > > on the contents of this information is strictly prohibited and may be > > unlawful. If you have received this communication in error, please notify > > us immediately by responding to this email and then delete it from your > > system. The firm is neither liable for the proper and complete > transmission > > of the information contained in this communication nor for any delay in > its > > receipt. > >
