date:20180201

[jira] Subscription: Oozie Patch Available

2018-02-01 Thread jira

Issue Subscription
Filter: Oozie Patch Available (105 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3172  Upgrade non-transitive Jackson dependencies from 
org.codehaus.jackson to com.fasterxml.jackson
https://issues.apache.org/jira/browse/OOZIE-3172
OOZIE-3168  Remove -secure option from DG_QuickStart.twiki and from 
oozie-setup.sh 
https://issues.apache.org/jira/browse/OOZIE-3168
OOZIE-3163  improve documentation rendering: use fluido skin and better config
https://issues.apache.org/jira/browse/OOZIE-3163
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3121  bump all maven plugins to latest versions
https://issues.apache.org/jira/browse/OOZIE-3121
OOZIE-3105  testJMXInstrumentation from the 
org.apache.oozie.util.TestMetricsInstrumentation class is flaky
https://issues.apache.org/jira/browse/OOZIE-3105
OOZIE-3094  fix for grammar mistake
https://issues.apache.org/jira/browse/OOZIE-3094
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2814  OYA: Update example workflows to newest schemas
https://issues.apache.org/jira/browse/OOZIE-2814
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create

[jira] Subscription: Oozie Patch Available

2018-02-01 Thread jira

Issue Subscription
Filter: Oozie Patch Available (111 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3129  Fix test TestConfigurationService.testOozieConfig
https://issues-test.apache.org/jira/browse/OOZIE-3129
OOZIE-3127  Remove redundant check for user
https://issues-test.apache.org/jira/browse/OOZIE-3127
OOZIE-3126  Add option to allow list of users to access system config
https://issues-test.apache.org/jira/browse/OOZIE-3126
OOZIE-3121  bump all maven plugins to latest versions
https://issues-test.apache.org/jira/browse/OOZIE-3121
OOZIE-3118  fix for error: self-closing element not allowed
https://issues-test.apache.org/jira/browse/OOZIE-3118
OOZIE-3117  fix for warning: no @throws for
https://issues-test.apache.org/jira/browse/OOZIE-3117
OOZIE-3116  fix for warning: no description for @throws
https://issues-test.apache.org/jira/browse/OOZIE-3116
OOZIE-3115  fix for javadoc warning: empty  tag
https://issues-test.apache.org/jira/browse/OOZIE-3115
OOZIE-3113  Retry for ZK lock release
https://issues-test.apache.org/jira/browse/OOZIE-3113
OOZIE-3112  SparkConfigrationService overwrites properties provided via 
--properties-file option in SparkAction
https://issues-test.apache.org/jira/browse/OOZIE-3112
OOZIE-3105  testJMXInstrumentation from the 
org.apache.oozie.util.TestMetricsInstrumentation class is flaky
https://issues-test.apache.org/jira/browse/OOZIE-3105
OOZIE-3094  fix for grammar mistake
https://issues-test.apache.org/jira/browse/OOZIE-3094
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues-test.apache.org/jira/browse/OOZIE-3091
OOZIE-3083  Make improved version Info backward compatible
https://issues-test.apache.org/jira/browse/OOZIE-3083
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues-test.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues-test.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues-test.apache.org/jira/browse/OOZIE-3062
OOZIE-3002  address findbugs errors in client lib
https://issues-test.apache.org/jira/browse/OOZIE-3002
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues-test.apache.org/jira/browse/OOZIE-2975
OOZIE-2973  Make sure Oozie works with Hadoop 3 
https://issues-test.apache.org/jira/browse/OOZIE-2973
OOZIE-2969  Drop support for Java 1.7
https://issues-test.apache.org/jira/browse/OOZIE-2969
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues-test.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues-test.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues-test.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues-test.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues-test.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues-test.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues-test.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues-test.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues-test.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues-test.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues-test.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues-test.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues-test.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues-test.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging

Failed: OOZIE-2814 PreCommit Build #366

2018-02-01 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2814
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/366/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.81 MB...]
testExternalChildIds(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testExecutionStats(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testUdfPig(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testSqoopActionWithRedundantArgsAndFreeFormQuery(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testSqoopActionWithArgsAndFreeFormQuery(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testSqoopActionWithBadRedundantArgsAndFreeFormQuery(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testSqoopActionWithBadCommand(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testSqoopEval(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testSqoopActionWithRedundantPrefix(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testSqoopAction(org.apache.oozie.action.hadoop.TestSqoopActionExecutor)
testShellScriptHadoopConfDir(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testShellScriptError(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testSetupMethods(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testShellScript(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testEnvVar(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testShellScriptHadoopConfDirWithNoL4J(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testPerlScript(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testCannotKillActionWhenACLSpecified(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testOutputSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitLauncherConfigurationOverridesLauncherMapperProperties(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCredentialsSkip(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testACLDefaults_noFalseChange(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testJobSubmissionWithoutYarnKill(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testFilesystemScheme(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testExceptionSubmitException(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testKill(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCommaSeparatedFilesAndArchives(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitOKWithLauncherEnvVars(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitOKWithLauncherJavaOpts(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testPrepare(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testAddActionShareLib(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testJobXmlAndNonDefaultNamenode(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testActionLibsPath(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitOKWithVcoresAndMemory(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSetupMethods(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testEmptyArgsWithNullArgsNotAllowed(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitWithLauncherQueue(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testEnvVarsPropagatedFromLauncherConfig(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testJavaOpts(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testExceptionSubmitThrowable(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testAdditionalJarSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testChildKill(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testLibFileArchives(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCredentialsModule(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testIdSwapSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testUpdateConfForTimeLineServiceEnabled(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testAddShareLibSchemeAndAuthority(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testEmptyArgsWithNullArgsAllowed(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testExit0SubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCredentialsWithoutCredTag(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testACLDefaults_launcherACLsSetToDefault(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testActionShareLibWithNonDefaultNamenode(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCredentialsInvalid(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testACLDefaults_explicitLauncherAndActionSettings(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSimpestSleSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

2018-02-01 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349614#comment-16349614
 ] 

Hadoop QA commented on OOZIE-2814:
--


Testing JIRA OOZIE-2814

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2091
.Tests failed: 25
.Tests errors: 98

.The patch failed the following testcases:

testMapReduceWithUberJarDisabled(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
testParserDefaultJobTrackerFail(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testParserGlobalExtensionActions(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testParserDefaultJobTrackerWithGlobal(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testParserGlobalLocalAlreadyExists(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testParserDefaultJobTracker(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testParserGlobalJobXML(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testParserGlobal(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser)
testRerun(org.apache.oozie.action.hadoop.TestRerun)
testSubmitFailsWithInvalidLauncherEnvVars(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitFailsWithNegativeMemory(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitFailsWithNegativeVcores(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testActionStartWithEscapeStrings(org.apache.oozie.command.coord.TestCoordActionStartXCommand)
testWorkflowActionEvent(org.apache.oozie.event.TestEventGeneration)
testWorkflowActionRecoveryUserRetry(org.apache.oozie.service.TestRecoveryService)
testRerunFork(org.apache.oozie.command.wf.TestReRunXCommand)
testSubWorkflowSuspend(org.apache.oozie.action.oozie.TestSubWorkflowActionExecutor)
testSubWorkflowRerun(org.apache.oozie.action.oozie.TestSubWorkflowActionExecutor)
testWfActionKillChildJob(org.apache.oozie.command.wf.TestWorkflowActionKillXCommand)
testWfActionKillSuccess(org.apache.oozie.command.wf.TestWorkflowActionKillXCommand)
testActionReuseWfJobAppPath(org.apache.oozie.command.wf.TestActionStartXCommand)
testActionStart(org.apache.oozie.command.wf.TestActionStartXCommand)
testActionWithEscapedStringAndCDATA(org.apache.oozie.command.wf.TestActionStartXCommand)
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflowRetries)
testParallelFsAndShellWorkflowCompletesSuccessfully(org.apache.oozie.test.TestWorkflow)

.Tests failing with errors:
testActionConfLoadDefaultResources(org.apache.oozie.action.hadoop.TestHiveActionExecutor)

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

2018-02-01 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349472#comment-16349472
 ] 

Hadoop QA commented on OOZIE-2814:
--

PreCommit-OOZIE-Build started


> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas

2018-02-01 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2814:
--
Attachment: OOZIE-2814-01.patch

> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

2018-02-01 Thread Robert Kanter (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349303#comment-16349303
 ] 

Robert Kanter commented on OOZIE-2814:
--

Good thing we're trying to use the new schemas ;)

I'd also say we should change {{jobTracker}} to {{resourceManager}} as well:
{code:xml}
${resourceManager}
{code}


> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class

2018-02-01 Thread Jason Phelps (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Phelps updated OOZIE-3170:

Attachment: OOZIE-3170-2.patch

> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
> 
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>  
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie 
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but 
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[] 
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize 
> service [org.apache.oozie.service.ShareLibService], Not able to cache 
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and 
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize 
> service [org.apache.oozie.service.ShareLibService], Not able to cache 
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and 
> issue the 'oozie admin' CLI command to update the sharelib
>  at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
>  at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
>  at org.apache.oozie.service.Services.setService(Services.java:372)
>  at org.apache.oozie.service.Services.loadServices(Services.java:304)
>  at org.apache.oozie.service.Services.init(Services.java:212)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
>  at 
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
>  at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
>  ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[] 
> PartitionDependencyManagerService initialized. Dependency cache is 
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception 
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
>  at 
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
>  at 
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
>  at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
>  at org.apache.oozie.service.Services.setService(Services.java:372)
>  at org.apache.oozie.service.Services.loadServices(Services.java:304)
>  at org.apache.oozie.service.Services.init(Services.java:212)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load 
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service 
> classes, null
>  at org.apache.oozie.service.Services.loadServices(Services.java:309)
>  at org.apache.oozie.service.Services.init(Services.java:212)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
>  at 
>

[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class

2018-02-01 Thread Jason Phelps (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349229#comment-16349229
 ] 

Jason Phelps commented on OOZIE-3170:
-

[~asasvari], I did not notice that there was still an NPE being thrown. That 
one was due to the ShareLibService requiring the ActionService class. Like you 
said, it was not affecting the outcome of the collection, which ultimately 
succeeded.

The main cause for issue in this Jira  was because my configuration had the 
following set:
{code:java}
 
 oozie.services.ext
 
org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService,org.apache.oozie.service.MetricsInstrumentationService
 {code}
 It looks like the PartitionDependencyManagerService required several 
additional services, including the JobsConcurrencyService.class I added in the 
earlier patch. I don't think the bundle collection would need any custom 
classes set, so I modified the code to unset it, as well as remove several 
loaded classes that were not necessary (i.e. they didn't fail the collection). 

In addition, I think the best way for the logs to be handled would be to dump 
them in the same place as the output zip file. I restructured it to output to a 
time-stamped directory in the {{-output}} directory (if oozie.log.dir is not 
set), and to dump everything in that. I have uploaded a new patch. Please let 
me know your thoughts

> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
> 
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>  
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie 
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but 
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[] 
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize 
> service [org.apache.oozie.service.ShareLibService], Not able to cache 
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and 
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize 
> service [org.apache.oozie.service.ShareLibService], Not able to cache 
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and 
> issue the 'oozie admin' CLI command to update the sharelib
>  at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
>  at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
>  at org.apache.oozie.service.Services.setService(Services.java:372)
>  at org.apache.oozie.service.Services.loadServices(Services.java:304)
>  at org.apache.oozie.service.Services.init(Services.java:212)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
>  at 
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
>  at 
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
>  at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
>  ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[] 
> PartitionDependencyManagerService initialized. Dependency cache is 
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception 
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
>  at 
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
>  at 
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
>  at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
>  at org.apache.oozie.service.Services.setService(Services.java:372)
>  at org.apache.oozie.service.Services.loadServices(Services.java:304)
>  at org.apache.oozie.service.Services.init(Services.java:212)
>  at 
>

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

2018-02-01 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349221#comment-16349221
 ] 

Attila Sasvari commented on OOZIE-2814:
---

[~rkanter] thanks a lot. I changed the java-main example workflow to use 
{{resource-manager}} instead of {{job-tracker}}
{code:java}

    
    
        
            ${jobTracker}{code}
Result of job submission:
{code:java}
Error: E0701 : E0701: XML schema error, No job-tracker defined{code}

Workflow can be validated against the schema just fine, but 
{{LiteWorkflowAppParser}} needs to be adjusted as it only tries to fetch the 
child of the {{job-tracker}} element:

https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java#L424

TestLiteWorkflowAppParser is to be extended too.

> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

2018-02-01 Thread András Piros via Review Board

> On Jan. 11, 2018, 1:53 a.m., Robert Kanter wrote:
> > .gitignore
> > Line 10 (original), 13 (patched)
> > 
> >
> > This can be in a followup JIRA if you want (in which case, please file 
> > it), but we should add an example or two to the oozie-examples of a 
> > workflow using the jobs-api.

There are already several new `jobs-api` examples here in `jobs-client` 
artifact: 
`jobs/jobs-client/src/test/java/org/apache/oozie/jobs/client/examples/Test*Example.java`.
 `TestJavaMainExample.java` rebuilds the `java-main` workflow using `jobs-api`, 
`TestShellExample.java` does the same for `shell`, `TestSparkExample.java` for 
`spark-file-copy`, etc.

Should I move these to a folder under `oozie-examples` artifact?

> On Jan. 11, 2018, 1:53 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java
> > Lines 118 (patched)
> > 
> >
> > This should be a constant.
> > 
> > Also, what happens if this is not set?  You'll end up with a path 
> > /application_ which I don't think we want to do.
> > 
> > Also, I don't think we should do application_.  
> > That might get confusion with Yarn APP IDs, which look like 
> > application__.  Perhaps gen_app_ instead 
> > of application?

Actually, I think this should rather be configurable. Most users are expected 
to use Jobs API workflow submission without putting `job.properties` or any 
other thing to HDFS - we want to make life as easy as possible. So a default 
place where generated `workflow.xml` and `job.properties` are put by Oozie 
deserves configuration.

We have a sane default for the prefix in `oozie-default.xml`: 
`/user/${user.name}/jobs/application/generated`. However, postfix renamed to 
`gen_app_` plus timestamp to avoid any confusions.

> On Jan. 11, 2018, 1:53 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java
> > Lines 117 (patched)
> > 
> >
> > IIRC, APP_PATH, COORDINATOR_APP_PATH, and BUNDLE_APP_PATH can all 
> > actually take an arbitrarily named XML file.  For example, you can set it 
> > to hdfs://host:8020/user/rkanter/my-wf/foo.xml instead of 
> > hdfs://host:8020/user/rkanter/my-wf/ and Oozie will use foo.xml 
> > instead of workflow.xml.  Someone may even just do 
> > hdfs://host:8020/user/rkanter/my-wf/workflow.xml.  We need to be 
> > careful to check for that any not always append 
> > /workflow|coordinator|bundle.xml.

Refactored in a way that if the user submits using `/path/to/something.xml`, we 
just take it (differentiating based on extension only). Otherwise, depending on 
whether the user submits a workflow, coordinator, or bundle job, we put the 
postfix `/workflow.xml`, `/coordinator.xml`, or `/bundle.xml`, respectively.

> On Jan. 11, 2018, 1:53 a.m., Robert Kanter wrote:
> > core/src/main/resources/oozie-default.xml
> > Lines 3312-3322 (patched)
> > 
> >
> > Or would we be better off putting these under a special directory under 
> > each user's home dir?  For example 
> > /user/rkanter/oozie-generated-workflows
> > 
> > With the current way, we're going to store all users' generated 
> > workflows under the oozie user's home dir, which means that all users need 
> > access to this directory.  I think that's probably okay, but it's a little 
> > funny.  If we put these under each user's home dir, there's no problem.

Those are HDFS folders, so maybe have a HDFS folder for each submitting user 
would be overkill. Anyway, isn't `${user.name}` the submitting user's login 
name when going via `OozieCLI`?

> On Jan. 11, 2018, 1:53 a.m., Robert Kanter wrote:
> > jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/GraphVisualization.java
> > Lines 38 (patched)
> > 
> >
> > Why is this here?

Putting class and method level Javadoc. For examples, for better understanding, 
we're generating also PNG images from the graphs.

Removed its unused `main()` method.

- András

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review195200
---

On Jan. 8, 2018, 10:31 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated Jan. 8, 2018, 10:31 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter

Re: Welcoming new Oozie PMC members: Attila Sasvari, Peter Cseh, and Peter Bacsko

2018-02-01 Thread Andras Piros

Congrats Attila, Peter and Geza :) well deserved!

Andras

On Thu, Feb 1, 2018 at 12:30 AM, Artem Ervits  wrote:

> Well deserved, congratulations guys!!
>
> On Jan 31, 2018 6:27 PM, "Robert Kanter"  wrote:
>
> > Hello Oozie community,
> >
> > It's my pleasure to announce that Attila Sasvari, Peter Cseh, and Peter
> > Bascko
> > have become the newest additions to the Oozie PMC.
> >
> > Please join me congratulating them.
> > Congrats!
> >
> >
> > - Robert, on behalf of the Oozie PMC
> >
>

[jira] Subscription: Oozie Patch Available

[jira] Subscription: Oozie Patch Available

Failed: OOZIE-2814 PreCommit Build #366

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

[jira] [Updated] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class

[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

Re: Welcoming new Oozie PMC members: Attila Sasvari, Peter Cseh, and Peter Bacsko

12 matches

Site Navigation

Mail list logo

Footer information