Alejandro (phone typing)
On Sep 30, 2013, at 9:15, Robert Kanter <rkan...@cloudera.com> wrote: >> >> I am not clear as to what the problem here is. > > Suppose you have a script that starts up your cluster (e.g. it starts HDFS, > Oozie, JobTracker, etc). Now the script has to ensure that HDFS has > finished starting up before starting Oozie; previously, the script could > start them in any order and also didn't have to wait for any of them to > finish starting up as there were no dependencies. You'll have HDFS > eventually, but not necessarily when Oozie starts. > > > thanks > - Robert > > > > On Sun, Sep 29, 2013 at 5:42 PM, Rohini Palaniswamy <rohini.adi...@gmail.com >> wrote: > >>> By adding this dependency on HDFS to be running, you can no longer start >> your cluster in any order; HDFS has to come before Oozie. We should have >> it defer the loading of the jars until it either somehow detects that HDFS >> is up or at the first job submission. >> >> I am not clear as to what the problem here is. Even now with 3.3.x and >> 4.0 if HDFS is not up the Oozie's jobs cannot be submitted to the cluster, >> nor can users submit jobs to oozie as parsing workflow.xml will fail. What >> is being made worse with Oozie copying the sharelib jars during startup or >> before submitting the first job? Even if the admin has to run >> oozie-setup.sh before starting Oozie, HDFS needs to be up. >> >> Regards, >> Rohini >> >> >> >> >> On Fri, Sep 27, 2013 at 6:36 PM, Alejandro Abdelnur <t...@cloudera.com >>> wrote: >> >>> IMO, making a required order of services startup is a no go. >>> >>> >>> On Fri, Sep 27, 2013 at 5:10 PM, Robert Kanter <rkan...@cloudera.com> >>> wrote: >>> >>>> By adding this dependency on HDFS to be running, you can no longer >> start >>>> your cluster in any order; HDFS has to come before Oozie. We should >> have >>>> it defer the loading of the jars until it either somehow detects that >>> HDFS >>>> is up or at the first job submission. >>>> >>>> An admin can put oozie in safe mode after it starts and then issues >>> series >>>>> of commands to do the necessary maintenance (install/upgrade/purge) >> of >>>>> sharelibs. >>>> >>>> Also, I agree with Alejandro's point earlier: if the purpose of all >> this >>> is >>>> to make this all automatic so that the admin doesn't have to do >> anything, >>>> then this won't work; it sounds like more steps for the admin to deal >>> with. >>>> >>>> >>>> What if we just enhanced the oozie-setup.sh script to add the logic for >>>> temp and staging sharelib stuff and didn't have Oozie try to do >> anything >>>> itself. e.g. the admin wants to transition to a new set of sharelibs, >>>> they'd run oozie-setup.sh upgrade blah blah. Or perhaps make the admin >>>> command handle this logic and do it only when the command is run; then >> we >>>> don't have to worry about starting up and it would simplify the HA work >>>> because the different Oozies won't be changing the sharelib at startup. >>>> Thoughts? >>>> >>>> - Robert >>>> >>>> >>>> On Fri, Sep 27, 2013 at 11:58 AM, Rohini Palaniswamy < >>>> rohini.adi...@gmail.com> wrote: >>>> >>>>> Basically I am trying to answer the question is what was possible >>> before >>>>> when Oozie was up and hdfs is down? >>>>> - Before oozie could still get job submissions if hdfs was down. >> But >>> we >>>>> are going to try validate the workflow.xml from hdfs and submission >> is >>>>> anyway going to fail. So don't think that is a big issue >>>>> - It can't submit new jobs and input dependency checks will anyway >>>> fail. >>>>> So that is also not a issue >>>>> - Users can continue to query the status of oozie which will now be >>> not >>>>> possible as Oozie will be down. >>>>> >>>>> Is there something else that I am missing? >>>>> >>>>> One thing that I can think of to address this is to defer loading to >>> job >>>>> submission time if hdfs is down during startup time. >>>>> >>>>> -Rohini >>>>> >>>>> On Fri, Sep 27, 2013 at 11:51 AM, Rohini Palaniswamy < >>>>> rohini.adi...@gmail.com> wrote: >>>>> >>>>>>> Unless I'm missing something, is not just a missing hadoop conf >>>> issue, >>>>>> now >>>>>> Oozie won't start if HDFS is not running. >>>>>> >>>>>> Why would this be a issue? If sharelib is in hdfs and that is >> not >>>>>> accessible all the jobs would anyway fail and things are not going >> to >>>>> work. >>>>>> >>>>>> Regards, >>>>>> Rohini >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 27, 2013 at 8:08 AM, Alejandro Abdelnur < >>> t...@cloudera.com >>>>>> wrote: >>>>>> >>>>>>> afaik, the whole point of these changes was to make things >>> adminless, >>>>>>> thus the extra complexity. if we need admin intervention, i'd go >> to >>>> the >>>>>>> previous model. >>>>>>> >>>>>>> thx >>>>>>> >>>>>>> Alejandro >>>>>>> (phone typing) >>>>>>> >>>>>>> On Sep 26, 2013, at 22:46, Virag Kothari <vi...@yahoo-inc.com> >>> wrote: >>>>>>> >>>>>>>> That¹s a good point. An alternative is to do share lib install >>> after >>>>>>> Oozie >>>>>>>> starts. >>>>>>>> An admin can put oozie in safe mode after it starts and then >>> issues >>>>>>> series >>>>>>>> of commands to do the necessary maintenance >>> (install/upgrade/purge) >>>> of >>>>>>>> sharelibs. >>>>>>>> (OOZIE-1519 is already tracking admin upgrade of sharelibs) >>>>>>>> Also all this commands can be optionally made to accept a >>>>> configuration >>>>>>>> parameter so the hadoop conf's are not required in oozie's class >>>> path. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Virag >>>>>>>> >>>>>>>> On 9/26/13 1:54 PM, "Alejandro Abdelnur" <t...@cloudera.com> >>> wrote: >>>>>>>> >>>>>>>>> Unless I'm missing something, is not just a missing hadoop conf >>>>> issue, >>>>>>> now >>>>>>>>> Oozie won't start if HDFS is not running. >>>>>>>>> >>>>>>>>> This does not seem right. >>>>>>>>> >>>>>>>>> If we can sort this out i would prefer the previous manual >>> update >>>> fo >>>>>>> the >>>>>>>>> sharelib via oozie-setup.sh >>>>>>>>> >>>>>>>>> thanks. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Sep 26, 2013 at 1:22 PM, Virag Kothari < >>>> vi...@yahoo-inc.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> There is no need for this setting if you have the hadoop >> configs >>>>> under >>>>>>>>>> conf/hadoop-conf. Having this setting would be more useful if >>> you >>>>> have >>>>>>>>>> oozie configured to talk with multiple hadoops. >>>>>>>>>> This configs are now required at startup as the share lib >>> service >>>>>>> needs >>>>>>>>>> to >>>>>>>>>> connect to the filesystem on which the share lib jars need to >> be >>>>>>> copied >>>>>>>>>> (probably only require the core-site.xml, need to check) >>>>>>>>>> Before also it was recommended to have the hadoop configs on >>>>>>>>>> oozie-server, >>>>>>>>>> so users don't need to define some of this configurations in >>>> their >>>>>>>>>> workflows. As now its mandatory, we should make it clear in >> our >>>>>>>>>> documentation. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Virag >>>>>>>>>> >>>>>>>>>> From: bowen zhang >>>>>>>>>> <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com >>>>>>>>>> Reply-To: bowen zhang <bowenzhang...@yahoo.com<mailto: >>>>>>>>>> bowenzhang...@yahoo.com>> >>>>>>>>>> Date: Thursday, September 26, 2013 12:20 PM >>>>>>>>>> To: "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" < >>>>>>>>>> dev@oozie.apache.org<mailto:dev@oozie.apache.org>>, Virag >>>> Kothari < >>>>>>>>>> vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> >>>>>>>>>> Subject: Re: issue after OOZIE-1461 >>>>>>>>>> >>>>>>>>>> I second on Robert's concern. Right now, even the easiest way >>> for >>>> me >>>>>>> to >>>>>>>>>> get around this is to point " >>>>>>>>>> oozie.service.HadoopAccessorService.hadoop.configurations" to >>>>>>>>>> "*=Absolute >>>>>>>>>> path of my hadoop conf". >>>>>>>>>> Bowen >>>>>>>>>> >>>>>>>>>> ________________________________ >>>>>>>>>> From: Robert Kanter <rkan...@cloudera.com<mailto: >>>>> rkan...@cloudera.com >>>>>>>>> >>>>>>>>>> To: Virag Kothari <vi...@yahoo-inc.com<mailto: >>> vi...@yahoo-inc.com >>>>>> >>>>>>>>>> Cc: "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" < >>>>>>>>>> dev@oozie.apache.org<mailto:dev@oozie.apache.org>>; bowen >>> zhang < >>>>>>>>>> bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>> >>>>>>>>>> Sent: Thursday, September 26, 2013 11:11 AM >>>>>>>>>> Subject: Re: issue after OOZIE-1461 >>>>>>>>>> >>>>>>>>>> Is there any way to make that step not required, or to at >> least >>>>>>>>>> streamline >>>>>>>>>> it somehow? I imagine we'll see many questions from users >>>> wondering >>>>>>> why >>>>>>>>>> their Oozie server doesn't start because of this. Also, Oozie >>>> used >>>>> to >>>>>>>>>> work >>>>>>>>>> out-of-the-box after running a few scripts; now it requires >>>> manually >>>>>>>>>> setting the >>>>> oozie.service.HadoopAccessorService.hadoop.configurations >>>>>>>>>> property. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> thanks >>>>>>>>>> - Robert >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Sep 23, 2013 at 2:57 PM, Virag Kothari < >>>> vi...@yahoo-inc.com >>>>>>>>>> <mailto:vi...@yahoo-inc.com>> wrote: >>>>>>>>>> >>>>>>>>>>> Bowen, >>>>>>>>>>> >>>>>>>>>>> Including hadoop configs on oozie-server is documented at >>> https://oozie.apache.org/docs/3.3.2/AG_HadoopConfiguration.html. >>>>>>>>>>> Documentation for sharelib installation can be updated once >>>>>>> OOZIE-1518 >>>>>>>>>> and >>>>>>>>>>> OOZIE-1519 goes in. We can also update quick start guide >> during >>>>> that >>>>>>>>>> time. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Virag >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 9/23/13 2:31 PM, "bowen zhang" <bowenzhang...@yahoo.com >>>> <mailto: >>>>>>>>>> bowenzhang...@yahoo.com>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Virag, >>>>>>>>>>>> Can you add documentation for this change since the current >>>> build >>>>>>> and >>>>>>>>>>>> setup of oozie doesn't cover this? >>>>>>>>>>>> Bowen >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ________________________________ >>>>>>>>>>>> From: Virag Kothari >>>>>>>>>> <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> >>>>>>>>>>>> To: bowen zhang >>>>>>>>>> <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>; >>>>>>>>>> "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" >>>>>>>>>>>> <dev@oozie.apache.org<mailto:dev@oozie.apache.org>>; Robert >>>>> Kanter >>>>>>> < >>>>>>>>>> rkan...@cloudera.com<mailto:rkan...@cloudera.com>> >>>>>>>>>>>> Sent: Monday, September 16, 2013 2:58 PM >>>>>>>>>>>> Subject: Re: issue after OOZIE-1461 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Robert/Bowen, >>>>>>>>>>>> >>>>>>>>>>>> The hadoop configs need to be there in class path >> (hadoop-conf >>>> dir >>>>>>> or >>>>>>>>>>>> oozie-server/lib). So the HadoopAccessorService can create >> the >>>>>>>>>>>> appropriate filesystem object. This will fix your current >>> issue. >>>>>>>>>>>> But there is one more problem you might face while running a >>> job >>>>>>>>>> where >>>>>>>>>>>> permissions are not recursively applied. This fix is in >>>> OOZIE-1528 >>>>>>>>>> and >>>>>>>>>>>> will be checked in shortly. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Virag >>>>>>>>>>>> >>>>>>>>>>>> From: bowen zhang >>>>>>>>>>>> <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com >>>>> <mailto: >>>>>>>>>> bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>> >>>>>>>>>>>> Reply-To: bowen zhang >>>>>>>>>>>> <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com >>>>> <mailto: >>>>>>>>>> bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>> >>>>>>>>>>>> Date: Monday, September 16, 2013 2:49 PM >>>>>>>>>>>> To: "dev@oozie.apache.org<mailto:dev@oozie.apache.org >>>> <mailto: >>>>>>>>>> dev@oozie.apache.org<mailto:dev@oozie.apache.org>>" >>>>>>>>>>>> <dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: >>>>>>>>>> dev@oozie.apache.org<mailto:dev@oozie.apache.org>>>, Virag >>>> Kothari >>>>>>>>>>>> <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com><mailto: >>>>>>>>>> vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>>> >>>>>>>>>>>> Subject: Re: Fwd: issue after OOZIE-1461 >>>>>>>>>>>> >>>>>>>>>>>> what I found is the variable uri from tmpShareLibPath has >>>>> authority >>>>>>>>>> of >>>>>>>>>>>> "null". >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ________________________________ >>>>>>>>>>>> From: Robert Kanter <rkan...@cloudera.com<mailto: >>>>>>> rkan...@cloudera.com >>>>>>>>>>> <mailto:rkan...@cloudera.com<mailto:rkan...@cloudera.com>>> >>>>>>>>>>>> To: Virag Kothari <vi...@yahoo-inc.com<mailto: >>>> vi...@yahoo-inc.com >>>>>>>>>>> <mailto:vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>>>; >>>>>>>>>>>> "dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: >>>>>>>>>> dev@oozie.apache.org<mailto:dev@oozie.apache.org>>" >>>>>>>>>>>> <dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: >>>>>>>>>> dev@oozie.apache.org<mailto:dev@oozie.apache.org>>> >>>>>>>>>>>> Sent: Monday, September 16, 2013 2:39 PM >>>>>>>>>>>> Subject: Fwd: issue after OOZIE-1461 >>>>>>>>>>>> >>>>>>>>>>>> Hi Virag, >>>>>>>>>>>> >>>>>>>>>>>> After OOZIE-1461, Bowen (and I too) have run into this >>> exception >>>>>>> when >>>>>>>>>>>> starting Oozie, so it fails and won't start. I checked, and >>> for >>>>> me >>>>>>>>>> at >>>>>>>>>>>> least, the share/lib/ dir looks like it has the correct >>>>> permissions. >>>>>>>>>>>> Any thoughts? >>>>>>>>>>>> >>>>>>>>>>>> thanks >>>>>>>>>>>> - Robert >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 16, 2013 at 2:26 PM, Bowen Zhang >>>>>>>>>>>> <bzh...@hortonworks.com<mailto:bzh...@hortonworks.com >>>> <mailto: >>>>>>>>>> bzh...@hortonworks.com<mailto:bzh...@hortonworks.com>>> >> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Robert, >>>>>>>>>>>>> After rebasing to the trunk, I hit this error when trying >> to >>>>> bring >>>>>>>>>> up >>>>>>>>>>>>> oozie. >>>>>>>>>>>>> org.apache.oozie.service.ServiceException: E0100: Could not >>>>>>>>>> initialize >>>>>>>>>>>>> service [org.apache.oozie.service.ShareLibService], Failed >> to >>>> set >>>>>>>>>>>>> permissions of path: >>>>>>>>>> /user/bzhang/share/lib/tmp-20130916135406/oozie >>>>>>>>>> to >>>>>>>>>>>>> 0755 >>>>>>>>>>>>> at >> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:81) >>>>>>>>>>>>> at >>>> org.apache.oozie.service.Services.setServiceInternal(Services.java:368) >>>>>>>>>>>>> at >>> org.apache.oozie.service.Services.setService(Services.java:354) >>>>>>>>>>>>> at >>>> org.apache.oozie.service.Services.loadServices(Services.java:287) >>>>>>>>>>>>> at >>>>> org.apache.oozie.service.Services.init(Services.java:208) >>>>>>>>>>>>> at >>>>>>>>>> >>>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoad >>>>>>>>>>>> er >>>>>>>>>>>>> .java:45) >>>>>>>>>>>>> at >>>>>>>>>>
