Unless I'm missing something, is not just a missing hadoop conf issue, now Oozie won't start if HDFS is not running.
This does not seem right. If we can sort this out i would prefer the previous manual update fo the sharelib via oozie-setup.sh thanks. On Thu, Sep 26, 2013 at 1:22 PM, Virag Kothari <vi...@yahoo-inc.com> wrote: > There is no need for this setting if you have the hadoop configs under > conf/hadoop-conf. Having this setting would be more useful if you have > oozie configured to talk with multiple hadoops. > This configs are now required at startup as the share lib service needs to > connect to the filesystem on which the share lib jars need to be copied > (probably only require the core-site.xml, need to check) > Before also it was recommended to have the hadoop configs on oozie-server, > so users don't need to define some of this configurations in their > workflows. As now its mandatory, we should make it clear in our > documentation. > > Thanks, > Virag > > From: bowen zhang <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com > >> > Reply-To: bowen zhang <bowenzhang...@yahoo.com<mailto: > bowenzhang...@yahoo.com>> > Date: Thursday, September 26, 2013 12:20 PM > To: "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" < > dev@oozie.apache.org<mailto:dev@oozie.apache.org>>, Virag Kothari < > vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> > Subject: Re: issue after OOZIE-1461 > > I second on Robert's concern. Right now, even the easiest way for me to > get around this is to point " > oozie.service.HadoopAccessorService.hadoop.configurations" to "*=Absolute > path of my hadoop conf". > Bowen > > ________________________________ > From: Robert Kanter <rkan...@cloudera.com<mailto:rkan...@cloudera.com>> > To: Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> > Cc: "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" < > dev@oozie.apache.org<mailto:dev@oozie.apache.org>>; bowen zhang < > bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>> > Sent: Thursday, September 26, 2013 11:11 AM > Subject: Re: issue after OOZIE-1461 > > Is there any way to make that step not required, or to at least streamline > it somehow? I imagine we'll see many questions from users wondering why > their Oozie server doesn't start because of this. Also, Oozie used to work > out-of-the-box after running a few scripts; now it requires manually > setting the oozie.service.HadoopAccessorService.hadoop.configurations > property. > > > > thanks > - Robert > > > > On Mon, Sep 23, 2013 at 2:57 PM, Virag Kothari <vi...@yahoo-inc.com > <mailto:vi...@yahoo-inc.com>> wrote: > > > Bowen, > > > > Including hadoop configs on oozie-server is documented at > > https://oozie.apache.org/docs/3.3.2/AG_HadoopConfiguration.html. > > Documentation for sharelib installation can be updated once OOZIE-1518 > and > > OOZIE-1519 goes in. We can also update quick start guide during that > time. > > > > Regards, > > Virag > > > > > > On 9/23/13 2:31 PM, "bowen zhang" <bowenzhang...@yahoo.com<mailto: > bowenzhang...@yahoo.com>> wrote: > > > > >Virag, > > >Can you add documentation for this change since the current build and > > >setup of oozie doesn't cover this? > > >Bowen > > > > > > > > > > > > > > >________________________________ > > > From: Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>> > > >To: bowen zhang <bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>; > "dev@oozie.apache.org<mailto:dev@oozie.apache.org>" > > ><dev@oozie.apache.org<mailto:dev@oozie.apache.org>>; Robert Kanter < > rkan...@cloudera.com<mailto:rkan...@cloudera.com>> > > >Sent: Monday, September 16, 2013 2:58 PM > > >Subject: Re: issue after OOZIE-1461 > > > > > > > > >Hi Robert/Bowen, > > > > > >The hadoop configs need to be there in class path (hadoop-conf dir or > > >oozie-server/lib). So the HadoopAccessorService can create the > > >appropriate filesystem object. This will fix your current issue. > > >But there is one more problem you might face while running a job where > > >permissions are not recursively applied. This fix is in OOZIE-1528 and > > >will be checked in shortly. > > > > > >Thanks, > > >Virag > > > > > >From: bowen zhang > > ><bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com><mailto: > bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>> > > >Reply-To: bowen zhang > > ><bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com><mailto: > bowenzhang...@yahoo.com<mailto:bowenzhang...@yahoo.com>>> > > >Date: Monday, September 16, 2013 2:49 PM > > >To: "dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: > dev@oozie.apache.org<mailto:dev@oozie.apache.org>>" > > ><dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: > dev@oozie.apache.org<mailto:dev@oozie.apache.org>>>, Virag Kothari > > ><vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com><mailto: > vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>>> > > >Subject: Re: Fwd: issue after OOZIE-1461 > > > > > >what I found is the variable uri from tmpShareLibPath has authority of > > >"null". > > > > > > > > >________________________________ > > >From: Robert Kanter <rkan...@cloudera.com<mailto:rkan...@cloudera.com > ><mailto:rkan...@cloudera.com<mailto:rkan...@cloudera.com>>> > > >To: Virag Kothari <vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com > ><mailto:vi...@yahoo-inc.com<mailto:vi...@yahoo-inc.com>>>; > > >"dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: > dev@oozie.apache.org<mailto:dev@oozie.apache.org>>" > > ><dev@oozie.apache.org<mailto:dev@oozie.apache.org><mailto: > dev@oozie.apache.org<mailto:dev@oozie.apache.org>>> > > >Sent: Monday, September 16, 2013 2:39 PM > > >Subject: Fwd: issue after OOZIE-1461 > > > > > >Hi Virag, > > > > > >After OOZIE-1461, Bowen (and I too) have run into this exception when > > >starting Oozie, so it fails and won't start. I checked, and for me at > > >least, the share/lib/ dir looks like it has the correct permissions. > > >Any thoughts? > > > > > >thanks > > >- Robert > > > > > > > > >On Mon, Sep 16, 2013 at 2:26 PM, Bowen Zhang > > ><bzh...@hortonworks.com<mailto:bzh...@hortonworks.com><mailto: > bzh...@hortonworks.com<mailto:bzh...@hortonworks.com>>> wrote: > > > > > >> Hi Robert, > > >> After rebasing to the trunk, I hit this error when trying to bring up > > >> oozie. > > >> org.apache.oozie.service.ServiceException: E0100: Could not initialize > > >> service [org.apache.oozie.service.ShareLibService], Failed to set > > >> permissions of path: /user/bzhang/share/lib/tmp-20130916135406/oozie > to > > >>0755 > > >> at > > >> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:81) > > >> at > > >> > org.apache.oozie.service.Services.setServiceInternal(Services.java:368) > > >> at > > >>org.apache.oozie.service.Services.setService(Services.java:354) > > >> at > > >> org.apache.oozie.service.Services.loadServices(Services.java:287) > > >> at org.apache.oozie.service.Services.init(Services.java:208) > > >> at > > >> > > > >>org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader > > >>.java:45) > > >> at > > >> > > > >>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.ja > > >>va:4206) > > >> at > > >> > > > >>org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) > > >> at > > >> > > > >>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.jav > > >>a:799) > > >> at > > >> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) > > >> at > > >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) > > >> at > > >> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) > > >> at > > >> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) > > >> at > > >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) > > >> at > > >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) > > >> at > > >> > > > >>org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324 > > >>) > > >> at > > >> > > > >>org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSup > > >>port.java:142) > > >> at > > >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) > > >> at > > >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840) > > >> at > > >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) > > >> at > > >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) > > >> at > > >> > org.apache.catalina.core.StandardService.start(StandardService.java:525) > > >> at > > >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754) > > >> at > org.apache.catalina.startup.Catalina.start(Catalina.java:595) > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> at > > >> > > > >>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java > > >>:39) > > >> at > > >> > > > >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > > >>mpl.java:25) > > >> at java.lang.reflect.Method.invoke(Method.java:597) > > >> at > > >>org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > > >> at > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) > > >> Caused by: java.io.IOException: Failed to set permissions of path: > > >> /user/bzhang/share/lib/tmp-20130916135406/oozie to 0755 > > >> at > > >> org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) > > >> at > org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662) > > >> at > > >> > > > >>org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem. > > >>java:509) > > >> at > > >> > > > >>org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java > > >>:286) > > >> at > > >> > > > >>org.apache.oozie.service.ShareLibService.copyJarContainingClasses(ShareLi > > >>bService.java:109) > > >> at > > >> > > > >>org.apache.oozie.service.ShareLibService.copyLauncherJarsToShareLib(Share > > >>LibService.java:90) > > >> at > > >> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:76) > > >> ... 29 more > > >> > > >> > > >> It might have sth to do with the version of hadoop that hadoopAccessor > > >> cannot change file permission. Do you see this problem? > > >> > > >> > > >> CONFIDENTIALITY NOTICE > > >> NOTICE: This message is intended for the use of the individual or > entity > > >> to which it is addressed and may contain information that is > > >>confidential, > > >> privileged and exempt from disclosure under applicable law. If the > > >>reader > > >> of this message is not the intended recipient, you are hereby notified > > >>that > > >> any printing, copying, dissemination, distribution, disclosure or > > >> forwarding of this communication is strictly prohibited. If you have > > >> received this communication in error, please contact the sender > > >>immediately > > >> and delete it from your system. Thank You. > > > > > > > -- Alejandro
