Not sure what could be the reason. I assume you are not using kerberos security.
I can see /user/oozie/share/lib-new. Can you please remove this?Also , for the
time being try to keep only the latest sub-dir /user/oozie/share/ (such as :
/user/oozie/share/lib/lib_20150204063210/*). It could be even better to start
the shared lib part from the beginning after removing all.
On Thursday, February 12, 2015 12:12 AM, Jakub Stransky
<[email protected]> wrote:
Hello,
thanks for replies. I checked the version of share libs and according to
version of pig it seemed to me that version is correct. We installed sharelib
tar which is shipped with oozie 4.1.0. However i attached the lis of sharelibs
listed by command $hadoop fs -lsr /user/oozie/share/
When reading a source code I found that command oozie admin -shareliblist is
calling a REST endpoint
http://bd-prg-en1:11000/oozie/v2/admin/list_sharelib?lib=lib_20150204064003returns
an {} which is empty response.
For completeness I am attaching oozie configuration files as well.
I am running out of ideas what might be wrong.
Thanks for helping me outjakub
On 11 February 2015 at 22:37, Mohammad Islam <[email protected]> wrote:
Can you please send the output snippets of $hadoop fs -lsr /user/oozie/share/
On Wednesday, February 11, 2015 2:06 AM, Jakub Stransky
<[email protected]> wrote:
Hello experienced users,
anybody faced similar issue? I am stack on this for 3 day and still not
found what might be a problem. Since the installation of share libs is
correct as I see a corect folder with a timestamp but cannot figure it out
why this is not added to classpath even when I
specify oozie.use.system.libpath=true in job.properties. When tried
strace oozie
admin -shareliblist command I can see a failing child process but can not
nail it down.
Does anybody has some suggestion what to try or check?
On 6 February 2015 at 10:44, Jakub Stransky <[email protected]> wrote:
> Hello,
>
> I tried a new installation procedure for shreLib installation after
> overcoming issues like: user oozie cannot impersonate user oozie etc. I got
> share/libs sucessfully installed to HDFS under
> /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> result of the command
> oozie admin -shareliblist
>
> returns null
>
> The commad for sharelib creation doesn't print any error to output nor
> oozie servers logs are free of any error. Can somebody help me to nail it
> down? Where oozie keep the information which share lib should be used?
>
> Thanks for helping me out
>
> Jakub
>
>
> On 4 February 2015 at 23:27, Jakub Stransky <[email protected]> wrote:
>
>> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
>> not different from the old one. WIll give a try!
>>
>> Thanks for pointing this out
>> Jakub
>>
>>
>> On 4 February 2015 at 22:55, Mohammad Islam <[email protected]>
>> wrote:
>>
>>> Hi Jakub,Shared library structure was changed in 4.1.0.
>>> It is now in /user/oozie/shared/lib/lib_$timestamp.
>>> Did you follow the new instructions?Some discussion at:How-to: Use the
>>> ShareLib in Apache Oozie (CDH 5)
>>>
>>> | |
>>> | | | | | |
>>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
>>> share/lib/lib_20140403151601/distcp -rw-r--r--
>>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
>>> | |
>>> | View on blog.cloudera.com | Preview by Yahoo |
>>> | |
>>> | |
>>>
>>>
>>> Regards,Mohamamd
>>>
>>>
>>>
>>>
>>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
>>> [email protected]> wrote:
>>>
>>>
>>> Hello experienced oozie users,
>>>
>>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
>>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
>>> using simple authentication etc. When the job gets submitted oozie
>>> launcher
>>> is failing with following:
>>>
>>> 2015-02-04 09:56:33,364 WARN [main]
>>> org.apache.hadoop.mapred.YarnChild: Exception running child :
>>> java.io.IOException: Split class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>>> Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
>>> ... 7 more
>>>
>>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
>>> Runnning cleanup for the task
>>>
>>> Found that this classes are contained in oozie share/lib uploaded to HDFS
>>> to /user/${user.name}/share/lib according to oozie-site.xml
>>> configuration:
>>>
>>> <property>
>>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>>> <value>/user/${user.name}/share/lib</value>
>>> </property>
>>>
>>> where ${user.name} is user under which oozie server is running e.g.
>>> oozie.
>>> For some reason those libs doesn't get propagated to
>>> mapreduce.application.classpath and hence ClassNotFoundException.
>>>
>>> Tried to change following property to true:
>>>
>>> <property>
>>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
>>> <value>true</value>
>>> </property>
>>>
>>> But with no difference. I am not sure what I am missing in installation
>>> instruction here
>>> <
>>> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
>>> >
>>>
>>> Any hint would be appreciated.
>>>
>>> Thx
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jakub Stransky
>> cz.linkedin.com/in/jakubstransky
>>
>>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
--
Jakub Stranskycz.linkedin.com/in/jakubstransky
