Re: ShareLibService Error

2016-06-15 Thread rammohan ganapavarapu 
Ok by adding bellow config it worked.



oozie.service.WorkflowAppService.system.libpath
hdfs:///user/hadoop/share/lib/




oozie.service.HadoopAccessorService.hadoop.configurations
*=/usr/hadoop/etc/hadoop/

Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is
the HOST:PORT of
the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*'
configuration is
used when there is no exact match for an authority. The
HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative
is looked within
the Oozie configuration directory; though the path can be
absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.



On Wed, Jun 15, 2016 at 2:55 PM, rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> But i still have a question, is this sharelib path points to local FS or
> HDFS?
>
> oozie.service.WorkflowAppService.system.libpath <-- what is the correct
> syntax for this property in my case it worked if i point it to local FS
> where i have sharelibs extracted, if i give hdfs:// its not working. any
> idea what is the right way to do it? also when i create sharelib using
> oozie-setup.sh can i give static path?
>
> Ram
>
> On Wed, Jun 15, 2016 at 2:00 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> I have updated the oozie config, when i ran create sharelib its creating
>> under "the destination path for sharelib is:
>> /uap/oozie/share/lib/lib_20160615205749" how can i make to create under
>> just /uap/oozie/share/lib ?
>>
>>
>> 
>> oozie.service.WorkflowAppService.system.libpath
>> /uap/oozie/share/lib/
>> 
>>
>> I think its working now :)
>>
>>
>>
>>  oozie  admin -sharelibupdate  -oozie http://localhost:11000/oozie
>> [ShareLib update status]
>> host = http://localhost:11000/oozie
>> status = Successful
>> sharelibDirOld = /uap/oozie/share/lib
>> sharelibDirNew = /uap/oozie/share/lib
>>
>>
>> [hadoop@localhost bin]$ oozie  admin -shareliblist -oozie
>> http://localhost:11000/oozie
>> [Available ShareLib]
>> oozie
>> hive
>> distcp
>> hcatalog
>> sqoop
>> mapreduce-streaming
>> spark
>> hive2
>> pig
>>
>>
>> Ram
>>
>> On Wed, Jun 15, 2016 at 1:51 PM, Peter Cseh 
>> wrote:
>>
>>> Check your configuration, oozie is looking for the sharelib under
>>> /user/hadoop/share/lib instead of /user/oozie/share/lib
>>>
>>> Where is your sharelib installed on hdfs?
>>>
>>>
>>> On Wed, Jun 15, 2016 at 10:43 PM, rammohan ganapavarapu <
>>> rammohanga...@gmail.com> wrote:
>>>
>>> > Yes, i did that after i send my previous email now i am not seeing
>>> that 500
>>> > error but still dont see any thing in sharelib list.
>>> >
>>> >  oozie  admin -sharelibupdate  -oozie http://localhost:11000/oozie
>>> > [ShareLib update status]
>>> > host = http://localhost:11000/oozie
>>> > status = java.io.FileNotFoundException: File /user/hadoop/share/lib
>>> > does not exist
>>> >
>>> > On Wed, Jun 15, 2016 at 1:40 PM, Peter Cseh 
>>> wrote:
>>> >
>>> > > Based on that e-mail thread:
>>> > >
>>> > > An update of the library from 2.1 to 2.4 into
>>> > > oozie_webapp_folder/WEB-INF/lib solved my problem
>>> > >
>>> > >
>>> > > Try copiing the jar there and restart oozie afterwards
>>> > >
>>> > > gp
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Jun 15, 2016 at 10:35 PM, rammohan ganapavarapu <
>>> > > rammohanga...@gmail.com> wrote:
>>> > >
>>> > > > Peter,
>>> > > >
>>> > > > What is the processes to upgrade common-io-lib? i already have (
>>> > manually
>>> > > > copied) that jar under $OOZIE_HOME/lib dir but still having issues.
>>> > > >
>>> > > > Thanks,
>>> > > > Ram
>>> > > >
>>> > > > On Wed, Jun 15, 2016 at 12:10 PM, Peter Cseh <
>>> gezap...@cloudera.com>
>>> > > > wrote:
>>> > > >
>>> > > > > You could try updating the common-io library to 2.4, that helped
>>> a
>>> > > > similar
>>> > > > > issue before:
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E
>>> > > > >
>>> > > > > gp
>>> > > > >
>>> > > > > On Wed, Jun 15, 2016 at 8:54 PM, rammohan ganapavarapu <
>>> > > > > rammohanga...@gmail.com> wrote:
>>> > > > >
>>> > > > > > I am getting this for update.
>>> > > > > >
>>> > > > > > oozie  admin -sharelibupdate  -oozie
>>> http://localhost:11000/oozie
>>> > > > > > Error: HTTP error code: 500 : Internal Server Error
>>> > > > > >
>>> > > > > > and log from oozie.log
>>> > > > > >
>>> > > > > >
>>> > > > > > 2016-06-15 18:54:04,974  WARN AuthenticationFilter:509 -
>>> > > > > SERVER[localhost]
>>> > > > > > AuthenticationToken ignored:
>>> > > > > > org.apache.hadoop.security.authentication.util.SignerException:
>>> > > Invalid
>>> > > > > > signature
>>> > > > > >
>>> > > > > > On Wed, Jun 15, 2016 at 5:28 AM, Peter Cseh <
>>> gezap...@cloudera.com

Re: ShareLibService Error

2016-06-15 Thread Peter Cseh 
Based on that e-mail thread:

An update of the library from 2.1 to 2.4 into
oozie_webapp_folder/WEB-INF/lib solved my problem


Try copiing the jar there and restart oozie afterwards

gp



On Wed, Jun 15, 2016 at 10:35 PM, rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Peter,
>
> What is the processes to upgrade common-io-lib? i already have ( manually
> copied) that jar under $OOZIE_HOME/lib dir but still having issues.
>
> Thanks,
> Ram
>
> On Wed, Jun 15, 2016 at 12:10 PM, Peter Cseh 
> wrote:
>
> > You could try updating the common-io library to 2.4, that helped a
> similar
> > issue before:
> >
> >
> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E
> >
> > gp
> >
> > On Wed, Jun 15, 2016 at 8:54 PM, rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > I am getting this for update.
> > >
> > > oozie  admin -sharelibupdate  -oozie http://localhost:11000/oozie
> > > Error: HTTP error code: 500 : Internal Server Error
> > >
> > > and log from oozie.log
> > >
> > >
> > > 2016-06-15 18:54:04,974  WARN AuthenticationFilter:509 -
> > SERVER[localhost]
> > > AuthenticationToken ignored:
> > > org.apache.hadoop.security.authentication.util.SignerException: Invalid
> > > signature
> > >
> > > On Wed, Jun 15, 2016 at 5:28 AM, Peter Cseh 
> > wrote:
> > >
> > > > Hi Ram,
> > > >
> > > > Have you told oozie to update the sharelibs with the command:
> > > >
> > > > oozie  admin -sharelibupdate -oozie http://localhost:11000/oozie
> > > >
> > > >
> > > > BRs
> > > >
> > > > gezapeti
> > > >
> > > > On Wed, Jun 15, 2016 at 1:06 AM, rammohan ganapavarapu <
> > > > rammohanga...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am getting this error in oozie-error.log, i have created sharelib
> > > using
> > > > > setup script it got successful but when i do shareliblist i am
> > getting
> > > > > empty output but i can list it from hdfs FS, any help?
> > > > >
> > > > >
> > > > > 1. ./oozie-setup.sh sharelib create -fs hdfs://localhost:8020
> > > > > the destination path for sharelib is:
> > > > > /user/hadoop/share/lib/lib_20160614225830
> > > > >
> > > > >
> > > > >  2. oozie  admin -shareliblist -oozie http://localhost:11000/oozie
> > > > > [Available ShareLib]
> > > > >
> > > > > 3. hadoop fs -ls /uap/oozie/share/lib/lib_20160614164826/
> > > > > Found 10 items
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/distcp
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/hcatalog
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/hive
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/hive2
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/mapreduce-streaming
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/oozie
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/pig
> > > > > -rw-r--r--   3 hadoop hadoop   1361 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/sharelib.properties
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/spark
> > > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > > /uap/oozie/share/lib/lib_20160614164826/sqoop
> > > > > [hadoop@eqp049wo bin]$
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > org.apache.oozie.service.ServiceException: E0104: Could not fully
> > > > > initialize service [org.apache.oozie.service.ShareLibService], Not
> > able
> > > > to
> > > > > cache sharelib. An Admin needs to install the sharelib with
> > > > oozie-setup.sh
> > > > > and issue the 'oozie admin' CLI command to update the sharelib
> > > > > org.apache.oozie.service.ServiceException: E0104: Could not fully
> > > > > initialize service [org.apache.oozie.service.ShareLibService], Not
> > able
> > > > to
> > > > > cache sharelib. An Admin needs to install the sharelib with
> > > > oozie-setup.sh
> > > > > and issue the 'oozie admin' CLI command to update the sharelib
> > > > > at
> > > > >
> > org.apache.oozie.service.ShareLibService.init(ShareLibService.java:139)
> > > > > at
> > > > >
> > org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> > > > > at
> > org.apache.oozie.service.Services.setService(Services.java:372)
> > > > > at
> > > org.apache.oozie.service.Services.loadServices(Services.java:305)
> > > > > at org.apache.oozie.service.Services.init(Services.java:213)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: ShareLibService Error

2016-06-15 Thread rammohan ganapavarapu 
Peter,

What is the processes to upgrade common-io-lib? i already have ( manually
copied) that jar under $OOZIE_HOME/lib dir but still having issues.

Thanks,
Ram

On Wed, Jun 15, 2016 at 12:10 PM, Peter Cseh  wrote:

> You could try updating the common-io library to 2.4, that helped a similar
> issue before:
>
> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E
>
> gp
>
> On Wed, Jun 15, 2016 at 8:54 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > I am getting this for update.
> >
> > oozie  admin -sharelibupdate  -oozie http://localhost:11000/oozie
> > Error: HTTP error code: 500 : Internal Server Error
> >
> > and log from oozie.log
> >
> >
> > 2016-06-15 18:54:04,974  WARN AuthenticationFilter:509 -
> SERVER[localhost]
> > AuthenticationToken ignored:
> > org.apache.hadoop.security.authentication.util.SignerException: Invalid
> > signature
> >
> > On Wed, Jun 15, 2016 at 5:28 AM, Peter Cseh 
> wrote:
> >
> > > Hi Ram,
> > >
> > > Have you told oozie to update the sharelibs with the command:
> > >
> > > oozie  admin -sharelibupdate -oozie http://localhost:11000/oozie
> > >
> > >
> > > BRs
> > >
> > > gezapeti
> > >
> > > On Wed, Jun 15, 2016 at 1:06 AM, rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am getting this error in oozie-error.log, i have created sharelib
> > using
> > > > setup script it got successful but when i do shareliblist i am
> getting
> > > > empty output but i can list it from hdfs FS, any help?
> > > >
> > > >
> > > > 1. ./oozie-setup.sh sharelib create -fs hdfs://localhost:8020
> > > > the destination path for sharelib is:
> > > > /user/hadoop/share/lib/lib_20160614225830
> > > >
> > > >
> > > >  2. oozie  admin -shareliblist -oozie http://localhost:11000/oozie
> > > > [Available ShareLib]
> > > >
> > > > 3. hadoop fs -ls /uap/oozie/share/lib/lib_20160614164826/
> > > > Found 10 items
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/distcp
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/hcatalog
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/hive
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/hive2
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/mapreduce-streaming
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/oozie
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/pig
> > > > -rw-r--r--   3 hadoop hadoop   1361 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/sharelib.properties
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/spark
> > > > drwxr-xr-x   - hadoop hadoop  0 2016-06-14 16:48
> > > > /uap/oozie/share/lib/lib_20160614164826/sqoop
> > > > [hadoop@eqp049wo bin]$
> > > >
> > > >
> > > >
> > > >
> > > > org.apache.oozie.service.ServiceException: E0104: Could not fully
> > > > initialize service [org.apache.oozie.service.ShareLibService], Not
> able
> > > to
> > > > cache sharelib. An Admin needs to install the sharelib with
> > > oozie-setup.sh
> > > > and issue the 'oozie admin' CLI command to update the sharelib
> > > > org.apache.oozie.service.ServiceException: E0104: Could not fully
> > > > initialize service [org.apache.oozie.service.ShareLibService], Not
> able
> > > to
> > > > cache sharelib. An Admin needs to install the sharelib with
> > > oozie-setup.sh
> > > > and issue the 'oozie admin' CLI command to update the sharelib
> > > > at
> > > >
> org.apache.oozie.service.ShareLibService.init(ShareLibService.java:139)
> > > > at
> > > >
> org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> > > > at
> org.apache.oozie.service.Services.setService(Services.java:372)
> > > > at
> > org.apache.oozie.service.Services.loadServices(Services.java:305)
> > > > at org.apache.oozie.service.Services.init(Services.java:213)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
> > > > at
> > > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
> > > > at
> > > >
> > > >
> > >
> >
>

Re: Spark Action NPE

2016-06-15 Thread Amit Kumar 
Unsubscribe

On Tue, Jun 14, 2016 at 9:34 PM, Pierre Villard  wrote:

> Hi,
>
> I am trying to run a Spark action (Oozie 4.2.0) and everything works fine
> if my action is run in local mode. As soon as I switch to cluster mode with
> yarn as master I get a NullPointerException from Oozie:
>
> JOB[724-160602113052690-oozie-oozi-W]
> ACTION[724-160602113052690-oozie-oozi-W@submit-spark-job] Error
> starting action [submit-spark-job]. ErrorType [ERROR], ErrorCode
> [NullPointerException], Message [NullPointerException: null]
> org.apache.oozie.action.ActionExecutorException: NullPointerException: null
>at
>
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:445)
>at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1139)
>at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1293)
>at
>
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
>at
>
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
>at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>at org.apache.oozie.command.XCommand.call(XCommand.java:356)
>at
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:434)
>at
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:76)
>at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>at org.apache.oozie.command.XCommand.call(XCommand.java:356)
>at
>
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
>at
>
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
>at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>at org.apache.oozie.command.XCommand.call(XCommand.java:356)
>at
>
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
>at
>
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
>at org.apache.oozie.command.XCommand.call(XCommand.java:286)
>at
>
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
>at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>at
>
> org.apache.oozie.action.hadoop.SparkActionExecutor.setupActionConf(SparkActionExecutor.java:79)
>at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1042)
>... 20 more
>
> I am wondering if this is somehow related to:
> https://issues.apache.org/jira/browse/SPARK-7706
>
> In any case, what is the cause? is there a workaround?
>
> Thanks for your help,
> Pierre
>