[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (106 issues) Subscriber: ooziedaily Key Summary OOZIE-3182 Oozie components fail with checkstyle errors https://issues.apache.org/jira/browse/OOZIE-3182 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3176 Oozie-core fails with checkstyle errors. https://issues.apache.org/jira/browse/OOZIE-3176 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3134 Potential inconsistency between the in-memory SLA map and the Oozie database https://issues.apache.org/jira/browse/OOZIE-3134 OOZIE-3109 Escape log-streaming's HTML-specific characters https://issues.apache.org/jira/browse/OOZIE-3109 OOZIE-3105 testJMXInstrumentation from the org.apache.oozie.util.TestMetricsInstrumentation class is flaky https://issues.apache.org/jira/browse/OOZIE-3105 OOZIE-3094 fix for grammar mistake https://issues.apache.org/jira/browse/OOZIE-3094 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-3061 Kill only those child jobs which are not already killed https://issues.apache.org/jira/browse/OOZIE-3061 OOZIE-3002 address findbugs errors in client lib https://issues.apache.org/jira/browse/OOZIE-3002 OOZIE-2975 code clean up in pig sharelib, replace Exception with more explicit, add try with resources, StringBuilder instead of StringBuffer https://issues.apache.org/jira/browse/OOZIE-2975 OOZIE-2957 Documentation states that starting a coordinator is possible https://issues.apache.org/jira/browse/OOZIE-2957 OOZIE-2956 Fix Findbugs warnings related to reliance on default encoding in oozie-core https://issues.apache.org/jira/browse/OOZIE-2956 OOZIE-2955 Fix Findbugs warnings related to reliance on default encoding in oozie-client https://issues.apache.org/jira/browse/OOZIE-2955 OOZIE-2954 Fix Checkstyle issues in oozie-client https://issues.apache.org/jira/browse/OOZIE-2954 OOZIE-2953 Fix Checkstyle issues in oozie-tools https://issues.apache.org/jira/browse/OOZIE-2953 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2942 Fix Findbugs warnings in oozie-examples https://issues.apache.org/jira/browse/OOZIE-2942 OOZIE-2937 Remove redundant groupId from the child pom's https://issues.apache.org/jira/browse/OOZIE-2937 OOZIE-2934 Fix "Exceptional return value of java.io.File.mkdirs() ignored" Findbugs error in oozie-sharelib-spark https://issues.apache.org/jira/browse/OOZIE-2934 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2914 Consolidate Trim https://issues.apache.org/jira/browse/OOZIE-2914 OOZIE-2883 OOZIE throw the error "Missing [oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups] property" https://issues.apache.org/jira/browse/OOZIE-2883 OOZIE-2877 Oozie Git Action https://issues.apache.org/jira/browse/OOZIE-2877 OOZIE-2867 Timezone handling for Coordinators: emphasize "Continent/City" format https://issues.apache.org/jira/browse/OOZIE-2867 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2829 Improve sharelib upload to accept multiple source folders https://issues.apache.org/jira/browse/OOZIE-2829 OOZIE-2826 Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later https://issues.apache.org/jira/browse/OOZIE-2826 OOZIE-2814 OYA: Update example workflows to newest schemas https://issues.apach
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369355#comment-16369355
]
Hadoop QA commented on OOZIE-3170:
--
Testing JIRA OOZIE-3170
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2092
.Tests failed: 3
.Tests errors: 1
.The patch failed the following testcases:
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService)
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)
.Tests failing with errors:
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/407/
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the
Failed: OOZIE-3170 PreCommit Build #407
Jira: https://issues.apache.org/jira/browse/OOZIE-3170
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/407/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.71 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2253 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3170
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2092
Tests failed: 3
Tests errors: 1
The patch failed the following testcases:
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService)
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)
Tests failing with errors:
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
There is at least one warning, please check
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/407/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13134033/comment/16369355","id":"16369355","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
at hadoop dot apache dot
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerId=hadoopqa&avatarId=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium&ownerId=hadoopqa&avatarId=10393"},"displayName":"Hadoop
QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA
OOZIE-3170\n\nCleaning local git
workspace\n\n\n\n{color:green}+1
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:red}-1
RAW_PATCH_ANALYSIS{color}
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369259#comment-16369259
]
Hadoop QA commented on OOZIE-3170:
--
PreCommit-OOZIE-Build started
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369254#comment-16369254
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks! Sorry for the late response.
- Next time, can you please submit the patch to check whether Oozie tests still
pass?
- If we change the directory structure, we should update the documentation too:
https://github.com/apache/oozie/blob/e26641db469533ac059923c7ed28a5c74f71ad20/docs/src/site/twiki/DG_CommandLineTool.twiki#L1942
- Setting the log directory to the same place as the output zip file sounds
good to me.
I plan to try the patch out with a pseudo hadoop as soon as I have some more
capacity.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
>
[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database
[
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369184#comment-16369184
]
Julia Kinga Marton commented on OOZIE-3134:
---
The test failures are unrelated with the patch.
> Potential inconsistency between the in-memory SLA map and the Oozie database
>
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch,
> OOZIE-3134-003.patch, OOZIE-3134-004.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an
> in-memory concurrent hashmap ("slaMap"), and later [persists
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
> in the Oozie database.
> However, if there is a failure during the database operation, a
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the
> SLA map|
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be
> implemented. It would also make sense to do more sanity/consistency check in
> the Oozie server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database
[
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369177#comment-16369177
]
Hadoop QA commented on OOZIE-3134:
--
Testing JIRA OOZIE-3134
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:red}-1{color} There are [1] new bugs found below threshold in [core]
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
core/findbugs-new.html
. The most important FindBugs errors are:
. At FailingConnectionWrapper.java:[line 175]: This use of
java/sql/Connection.prepareStatement(Ljava/lang/String;II)Ljava/sql/PreparedStatement;
can be vulnerable to SQL injection
. At FailingConnectionWrapper.java:[line 169]
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2096
.Tests failed: 17
.Tests errors: 12
.The patch failed the following testcases:
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testCoordinatorActionSelectors(org.apache.oozie.jms.TestJMSJobEventListener)
testCoordinatorActionSelectorsNegative(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectors(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorJobSuccessEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobSuspendEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectorsAnd(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorJobFailureEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectorsNegative(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectorsOr(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobSuccessEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorActionWaitingEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobFailureEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorActionStartEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobStartedEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService)
.Tests failing with errors:
testOnSLADurationMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAStartMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testSLAJobSelectorsNegative(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAEndMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAStartMissEvent(org.apache.oozie.jms.
Failed: OOZIE-3134 PreCommit Build #406
Jira: https://issues.apache.org/jira/browse/OOZIE-3134 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/406/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.76 MB...] +1 the patch does not seem to introduce new Javadoc warnings WARNING: the current HEAD has 100 Javadoc warning(s) +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings -1 There are [1] new bugs found below threshold in total that must be fixed. +1 There are no new bugs found in [examples]. -1 There are [1] new bugs found below threshold in [core] that must be fixed. You can find the FindBugs diff here (look for the red and orange ones): core/findbugs-new.html The most important FindBugs errors are: At FailingConnectionWrapper.java:[line 175]: This use of java/sql/Connection.prepareStatement(Ljava/lang/String;II)Ljava/sql/PreparedStatement; can be vulnerable to SQL injection At FailingConnectionWrapper.java:[line 169] +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/spark]. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/oozie]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [webapp]. +1 There are no new bugs found in [tools]. +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [server]. +1 There are no new bugs found in [client]. +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files -1 TESTS Tests run: 2096 Tests failed: 17 Tests errors: 12 The patch failed the following testcases: testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService) testCoordinatorActionSelectors(org.apache.oozie.jms.TestJMSJobEventListener) testCoordinatorActionSelectorsNegative(org.apache.oozie.jms.TestJMSJobEventListener) testWorkflowJobSelectors(org.apache.oozie.jms.TestJMSJobEventListener) testOnCoordinatorJobSuccessEvent(org.apache.oozie.jms.TestJMSJobEventListener) testOnWorkflowJobSuspendEvent(org.apache.oozie.jms.TestJMSJobEventListener) testWorkflowJobSelectorsAnd(org.apache.oozie.jms.TestJMSJobEventListener) testOnCoordinatorJobFailureEvent(org.apache.oozie.jms.TestJMSJobEventListener) testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener) testWorkflowJobSelectorsNegative(org.apache.oozie.jms.TestJMSJobEventListener) testWorkflowJobSelectorsOr(org.apache.oozie.jms.TestJMSJobEventListener) testOnWorkflowJobSuccessEvent(org.apache.oozie.jms.TestJMSJobEventListener) testOnCoordinatorActionWaitingEvent(org.apache.oozie.jms.TestJMSJobEventListener) testOnWorkflowJobFailureEvent(org.apache.oozie.jms.TestJMSJobEventListener) testOnCoordinatorActionStartEvent(org.apache.oozie.jms.TestJMSJobEventListener) testOnWorkflowJobStartedEvent(org.apache.oozie.jms.TestJMSJobEventListener) testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerService) Tests failing with errors: testOnSLADurationMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener) testOnSLAStartMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener) testSLAJobSelectorsNegative(org.apache.oozie.jms.TestJMSSLAEventListener) testOnSLAEndMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener) testOnSLAStartMissEvent(org.apache.oozie.jms.TestJMSSLAEventListener) testSLAJobSelectors(org.apache.oozie.jms.TestJMSSLAEventListener) testOnSLAEndMissEvent(org.apache.oozie.jms.TestJMSSLAEventListener) testOnSLADurationMissEvent(org.apache.oozie.jms.TestJMSSLAEventListener) testDropEventTypeMessage(org.apache.oozie.jms.TestHCatMessageHandler) testCacheUpdateByMessage(org.apache.oozie.jms.TestHCatMessageHandler) testThreadLocalSession(org.apache.oozie.jms.TestDefaultConnectionContext) testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService) Tests failed at first run: TestJavaActionExecutor#testCredentialsSkip TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild2 For the complete list of flaky tests, see TEST-SUMMARY-FULL files. +1 DISTRO +1 distro tarball builds with the patch -1 Overall result, please check the reported -1(s) There is at least one warning, please check The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/406/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Spee
Re: Review Request 65481: OOZIE-3134 - Potential inconsistency between the in-memory SLA map and the Oozie database
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65481/
---
(Updated Feb. 19, 2018, 2:14 p.m.)
Review request for oozie, András Piros and Attila Sasvari.
Repository: oozie-git
Description
---
Upon SLACalculatorMemory.addRegistration, Oozie puts an entry into an in-memory
concurrent hashmap ("slaMap"), and later persists it in the Oozie database.
However, if there is a failure during the database operation, a
JPAExecutorException is thrown, and the entry is not removed from the SLA map.
It may introduce inconsistency between the Oozie database and the SLA map.
To prevent this, a rollback mechanism (with proper logging) should be
implemented. It would also make sense to do more sanity/consistency check in
the Oozie server.
Diffs (updated)
-
core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java ef019e73
core/src/main/java/org/apache/oozie/util/db/AlwaysFailingHSQLDriverMapper.java
PRE-CREATION
core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
0e310253
core/src/main/java/org/apache/oozie/util/db/FailingDBHelperForTest.java
PRE-CREATION
core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java
fe9f08b1
core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java
f0e2b181
core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java ee906f45
Diff: https://reviews.apache.org/r/65481/diff/4/
Changes: https://reviews.apache.org/r/65481/diff/3-4/
Testing
---
Thanks,
Kinga Marton
[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database
[
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369103#comment-16369103
]
Hadoop QA commented on OOZIE-3134:
--
PreCommit-OOZIE-Build started
> Potential inconsistency between the in-memory SLA map and the Oozie database
>
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch,
> OOZIE-3134-003.patch, OOZIE-3134-004.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an
> in-memory concurrent hashmap ("slaMap"), and later [persists
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
> in the Oozie database.
> However, if there is a failure during the database operation, a
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the
> SLA map|
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be
> implemented. It would also make sense to do more sanity/consistency check in
> the Oozie server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database
[
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julia Kinga Marton updated OOZIE-3134:
--
Attachment: OOZIE-3134-004.patch
> Potential inconsistency between the in-memory SLA map and the Oozie database
>
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch,
> OOZIE-3134-003.patch, OOZIE-3134-004.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an
> in-memory concurrent hashmap ("slaMap"), and later [persists
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
> in the Oozie database.
> However, if there is a failure during the database operation, a
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the
> SLA map|
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be
> implemented. It would also make sense to do more sanity/consistency check in
> the Oozie server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
