Check the status of coordinator actions. If the number of running actions is not same as coord concurrency, you can check if the enough actions are not materialised. You can look at materialisation configs to tune it
On Sat, Oct 18, 2014 at 4:33 PM, Harshal Vora <[email protected]> wrote: > Dear Swetha, > > We fixed the OOZIE_HTTP_HOSTNAME issue temporarily by hard-coding the > machine name and adding dns entries on every hadoop TT machine. > But the jobs still seem slower than expected. Almost all the time we have > free mapper/reducer slots available on hadoop but enough jobs are not > running. > > Apart from setting the concurrency of the coordinator jobs to a higher > number, is there any other configuration that can be changed so that idle > time is minimum? > I read on a blog regarding maximum number of threads for jobs in each > state and even in the oozie configuration I can see a lot of threadpools > defined with max thread limits. > > Is there any particular config that we need to change? > We have a backlog of jobs for one month and there are a lot of jobs > available that can be submitted on the cluster, so according to me all the > hadoop slots should be completely utilized. > > Regards, > > > On 10/14/2014 11:56 AM, Shwetha GS wrote: > >> This is what oozie server does: >> export OOZIE_HTTP_HOSTNAME=`hostname -f` >> export OOZIE_BASE_URL="http:// >> ${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie" >> catalina_opts="${catalina_opts} -Doozie.base.url=${OOZIE_BASE_URL}"; >> >> Callback url is picked up from oozie.base.url config which is set to >> http://:11000/oozie >> in your case which means oozie failed to get hostname >> >> The jobs run on TT, so the host resolution should work on TTs >> >> -Shwetha >> >> >> On Tue, Oct 14, 2014 at 11:45 AM, Harshal Vora <[email protected]> >> wrote: >> >> Re-sending the shell output >>> >>> ------------------------------------------------------------ >>> ------------------------------ >>> >>> [xxx@server-lp-4 ~]$ hostname -f >>> server-lp-4 >>> [xxx@server-lp-4 ~]$ sudo su yyy >>> [sudo] password for xxx: >>> bash-4.1$ hostname -f >>> server-lp-4 >>> ------------------------------------------------------------ >>> ------------------------------ >>> >>> >>> >>> On 10/14/2014 11:43 AM, Harshal Vora wrote: >>> >>> Dear Shwetha, >>>> >>>> Below is the output of shell on oozie machine for both regular user >>>> (xxx) >>>> as well as oozie user (yyy) which is used to submit the jobs. >>>> For security purpose I have changed the user names to xxx and yyy. >>>> >>>> We use /etc/hosts for DNS mapping as we do not have a central DNS >>>> server. >>>> So I believe we will have to put the oozie machines mapping i.e. >>>> "server-lp-4" on jobtracker machine. >>>> >>>> ------------------------------------------------------------ >>>> ------------------------------ >>>> >>>> [xxx@server-lp-4 ~]$ hostname -f >>>> server-lp-4 >>>> [xxx@packet-server-lp-4 ~]$ sudo su yyy >>>> [sudo] password for xxx: >>>> bash-4.1$ hostname -f >>>> server-lp-4 >>>> ------------------------------------------------------------ >>>> ------------------------------ >>>> >>>> >>>> >>>> >>>> >>>> On 10/14/2014 11:36 AM, Shwetha GS wrote: >>>> >>>> If the command 'hostname -f' was resolving correctly, callback url >>>>> would >>>>> have the host. So, I don't think the command is returning the host. >>>>> >>>>> Once you resolve this, hadoop machine where the jobs runs should >>>>> resolve >>>>> the host. >>>>> >>>>> On Tue, Oct 14, 2014 at 11:14 AM, Harshal Vora <[email protected]> >>>>> wrote: >>>>> >>>>> Dear Swetha, >>>>> >>>>>> On oozie machine, hostname points to "server-lp-4" which is the name >>>>>> we >>>>>> have given to our machine. >>>>>> Does this mean we need a mapping in DNS for this hostname on the >>>>>> hadoop >>>>>> machine? >>>>>> >>>>>> Regards, >>>>>> >>>>>> >>>>>> On 10/14/2014 10:49 AM, Shwetha GS wrote: >>>>>> >>>>>> Oozie uses 'hostname -f' to get the hostname which is used in >>>>>> setting >>>>>> >>>>>>> callback url. Check if its resolving correctly. This is the reason >>>>>>> for >>>>>>> delay between job completion and oozie status update >>>>>>> >>>>>>> -Shwetha >>>>>>> >>>>>>> On Tue, Oct 14, 2014 at 10:17 AM, Harshal Vora <[email protected] >>>>>>> > >>>>>>> wrote: >>>>>>> >>>>>>> Logs on Jobtracker machine: >>>>>>> >>>>>>> ERROR org.apache.hadoop.mapred.JobEndNotifier: Notification failure >>>>>>>> [URL: >>>>>>>> http://:11000/oozie/callback?id=0000443-141012054231695- >>>>>>>> oozie-oozi-W@ >>>>>>>> visitInfoProcessor-cascading&status=SUCCEEDED& remaining retries: 0 >>>>>>>> interval: 30000] >>>>>>>> >>>>>>>> ~/.bashrc on oozie machine. >>>>>>>> export OOZIE_URL=http://localhost:11000/oozie >>>>>>>> >>>>>>>> Is the hostname empty in the callback URL because it is configured >>>>>>>> to >>>>>>>> localhost on oozie machine? >>>>>>>> >>>>>>>> For some reason the transition between oozie actions and job status >>>>>>>> change >>>>>>>> is very very slow. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Harshal Vora >>>>>>>> www.radiolocus.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>> Regards, >>>>>> Harshal Vora >>>>>> www.radiolocus.com >>>>>> >>>>>> >>>>>> >>>>>> >>>> -- >>> Regards, >>> Harshal Vora >>> www.radiolocus.com >>> >>> >>> > > -- > Regards, > Harshal Vora > www.radiolocus.com > > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
