[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370112#comment-16370112
]
Attila Sasvari commented on OOZIE-3170:
---
Failed tests pass locally with {{mvn test
-Dtest=TestPartitionDependencyManagerService,TestRecoveryService,TestPartitionDependencyManagerEhcache,TestJMSAccessorService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} and they should be unrelated
(changes are only in the tools module).
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369998#comment-16369998
]
Attila Sasvari commented on OOZIE-3170:
---
Tested it on a pseudo hadoop cluster:
{code:java}
$ bin/oozie-diag-bundle-collector.sh -oozie http://localhost:11000/oozie
-output /tmp/jobs/
Checking Connection...Done
Using Temporary Directory:
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/1519129669757-0
Getting Sharelib Information...Done
Getting Configuration...Done
Getting OS Environment Variables...Done
Getting Java System Properties...Done
Getting Queue Dump...Done
Getting Thread Dump...Done
Getting Instrumentation...Done
Getting Metrics...Skipping (Metrics are unavailable)
Using Directory: /tmp/jobs/oozie-diag-bundle-1519129669082
Creating Zip File:
/tmp/jobs/oozie-diag-bundle-1519129669082/oozie-diag-bundle.zip ...Done{code}
* Verified that Diag bundle tool logs are placed in the proper directory
{code:java}
$ ls /tmp/jobs/oozie-diag-bundle-1519129669082/ jetty.log
oozie-diag-bundle.zip oozie-instrumentation.log oozie-ops.log
oozie-audit.log oozie-error.log oozie-jpa.log
oozie.log{code}
- Please adjust documentation.
- I am pretty sure test failures are unrelated, but I will run them locally
soon.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.di
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369355#comment-16369355
]
Hadoop QA commented on OOZIE-3170:
--
Testing JIRA OOZIE-3170
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2092
.Tests failed: 3
.Tests errors: 1
.The patch failed the following testcases:
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService)
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)
.Tests failing with errors:
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/407/
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369259#comment-16369259
]
Hadoop QA commented on OOZIE-3170:
--
PreCommit-OOZIE-Build started
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369254#comment-16369254
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks! Sorry for the late response.
- Next time, can you please submit the patch to check whether Oozie tests still
pass?
- If we change the directory structure, we should update the documentation too:
https://github.com/apache/oozie/blob/e26641db469533ac059923c7ed28a5c74f71ad20/docs/src/site/twiki/DG_CommandLineTool.twiki#L1942
- Setting the log directory to the same place as the output zip file sounds
good to me.
I plan to try the patch out with a pseudo hadoop as soon as I have some more
capacity.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
>
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349229#comment-16349229
]
Jason Phelps commented on OOZIE-3170:
-
[~asasvari], I did not notice that there was still an NPE being thrown. That
one was due to the ShareLibService requiring the ActionService class. Like you
said, it was not affecting the outcome of the collection, which ultimately
succeeded.
The main cause for issue in this Jira was because my configuration had the
following set:
{code:java}
oozie.services.ext
org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService,org.apache.oozie.service.MetricsInstrumentationService
{code}
It looks like the PartitionDependencyManagerService required several
additional services, including the JobsConcurrencyService.class I added in the
earlier patch. I don't think the bundle collection would need any custom
classes set, so I modified the code to unset it, as well as remove several
loaded classes that were not necessary (i.e. they didn't fail the collection).
In addition, I think the best way for the logs to be handled would be to dump
them in the same place as the output zip file. I restructured it to output to a
time-stamped directory in the {{-output}} directory (if oozie.log.dir is not
set), and to dump everything in that. I have uploaded a new patch. Please let
me know your thoughts
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.ini
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345077#comment-16345077
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks for reporting this issue and reviewing related parts of
the codebase. I added you as a contributor to the project and assigned this
Jira to you.
I reproduced that the cited NPE exception is thrown. The diag bundle zip is
however generated and contain sharelib info - it is just using the
OozieClient's
[listShareLib()|https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java#L42]:
{code:java}
$ bin/oozie-diag-bundle-collector.sh -oozie http://localhost:11000/oozie
-output /tmp/jobs/
Checking Connection...Done
Using Temporary Directory:
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/1517319232457-0
Getting Sharelib Information...Done
Getting Configuration...Done
Getting OS Environment Variables...Done
Getting Java System Properties...Done
Getting Queue Dump...Done
Getting Thread Dump...Done
Getting Instrumentation...Done
Getting Metrics...Skipping (Metrics are unavailable)
Creating Zip File: /tmp/jobs/oozie-diag-bundle-1517319233190.zip...Done
$ unzip -l /tmp/jobs/oozie-diag-bundle-1517319233190.zip
68029 01-30-18 14:33 /effective-oozie-site.xml
9876 01-30-18 14:33 /instrumentation.txt
38636 01-30-18 14:33 /java-sys-props.txt
3807 01-30-18 14:33 /os-env-vars.txt
279 01-30-18 14:33 /queue-dump.txt
40032 01-30-18 14:33 /sharelib.txt
102084 01-30-18 14:33 /thread-dump.html{code}
* In fact, I am not sure all those Oozie services are really needed here to be
able to collect diagnostic information. If they are not needed they shall not
be loaded at all.
* There is also another problem. By default, logs generated by the tool appear
in the server log if you run the tool from Oozie's home directory. It can make
things very confusing for an admin or anyone who review Oozie server logs.
Setting up logging is the responsibility of the
[XLogService|[https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/core/src/main/java/org/apache/oozie/service/XLogService.java#L145]]
and it is started via Services.init(). It can be controlled by the
{{oozie.log.dir}} system property (e.g. \{{export
JAVA_PROPERTIES="-Doozie.log.dir=/tmp/"}} before running the tool). This is
something we should clarify in the documentation of the tool and/or change the
code/script so that logs are put in the directory where the diag bundle is
generated by default.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver
