[jira] [Updated] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie
[ https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3235: --- Summary: Upgrade Old ActiveMQ dependency in Oozie (was: Old ActiveMQ dependency in Oozie) > Upgrade Old ActiveMQ dependency in Oozie > > > Key: OOZIE-3235 > URL: https://issues.apache.org/jira/browse/OOZIE-3235 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.1.0 >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Minor > Fix For: 5.1.0 > > > An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3). > It seems that the jars are even included in the oozie-client with "compile" > scope, but we only use them in unit tests. > Oozie-client uses the JMS-API only from these jars, which could be included > separately (javax.jms). > The version could be upgraded to 5.14.2 and the scope changed to "test" > instead of "compile". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3235) Old ActiveMQ dependency in Oozie
Mate Juhasz created OOZIE-3235: -- Summary: Old ActiveMQ dependency in Oozie Key: OOZIE-3235 URL: https://issues.apache.org/jira/browse/OOZIE-3235 Project: Oozie Issue Type: Bug Components: client Affects Versions: 5.1.0 Reporter: Mate Juhasz Assignee: Mate Juhasz Fix For: 5.1.0 An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3). It seems that the jars are even included in the oozie-client with "compile" scope, but we only use them in unit tests. Oozie-client uses the JMS-API only from these jars, which could be included separately (javax.jms). The version could be upgraded to 5.14.2 and the scope changed to "test" instead of "compile". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie
[ https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475417#comment-16475417 ] Mate Juhasz commented on OOZIE-3235: [~andras.piros] [~gezapeti] what is your opinion about it? :) > Upgrade Old ActiveMQ dependency in Oozie > > > Key: OOZIE-3235 > URL: https://issues.apache.org/jira/browse/OOZIE-3235 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.1.0 >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Minor > Fix For: 5.1.0 > > Attachments: OOZIE-3235-001.patch > > > An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3). > It seems that the jars are even included in the oozie-client with "compile" > scope, but we only use them in unit tests. > Oozie-client uses the JMS-API only from these jars, which could be included > separately (javax.jms). > The version could be upgraded to 5.14.2 and the scope changed to "test" > instead of "compile". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie
[ https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475706#comment-16475706 ] Mate Juhasz commented on OOZIE-3235: Thanks for the remark Peter, I uploaded a new patch! :) > Upgrade Old ActiveMQ dependency in Oozie > > > Key: OOZIE-3235 > URL: https://issues.apache.org/jira/browse/OOZIE-3235 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.1.0 >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Minor > Fix For: 5.1.0 > > Attachments: OOZIE-3235-001.patch, OOZIE-3235-002.patch > > > An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3). > It seems that the jars are even included in the oozie-client with "compile" > scope, but we only use them in unit tests. > Oozie-client uses the JMS-API only from these jars, which could be included > separately (javax.jms). > The version could be upgraded to 5.14.2 and the scope changed to "test" > instead of "compile". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475974#comment-16475974 ] Mate Juhasz commented on OOZIE-1624: By sharelib root do you mean the "lib_" directory? For example instead of filtering against the below full paths: * /user/oozie/share/lib/lib123456789/oozie/jackson*.jar * /user/oozie/share/lib/lib123456789/pig/lib/jackson*.jar We could only let jars excluded inside lib123456789: * oozie/jackson*.jar * pig/lib/jackson*.jar > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-1624: -- Assignee: Mate Juhasz (was: Purshotam Shah) > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie
[ https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3235: --- Attachment: OOZIE-3235-002.patch > Upgrade Old ActiveMQ dependency in Oozie > > > Key: OOZIE-3235 > URL: https://issues.apache.org/jira/browse/OOZIE-3235 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.1.0 >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Minor > Fix For: 5.1.0 > > Attachments: OOZIE-3235-001.patch, OOZIE-3235-002.patch > > > An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3). > It seems that the jars are even included in the oozie-client with "compile" > scope, but we only use them in unit tests. > Oozie-client uses the JMS-API only from these jars, which could be included > separately (javax.jms). > The version could be upgraded to 5.14.2 and the scope changed to "test" > instead of "compile". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3282) Prototyping: create Angular 6 workflow list page
[ https://issues.apache.org/jira/browse/OOZIE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3282: -- Assignee: Mate Juhasz > Prototyping: create Angular 6 workflow list page > > > Key: OOZIE-3282 > URL: https://issues.apache.org/jira/browse/OOZIE-3282 > Project: Oozie > Issue Type: Sub-task >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > > Create an [*Angular 6*|https://angular.io/docs] prototype of the workflow > list page. Don't have to handle authentication / authorization for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322352#comment-16322352 ] Mate Juhasz commented on OOZIE-3061: Please add me to the contributors list, thank you! > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Reporter: Satish Subhashrao Saley >Priority: Trivial > Labels: newbie, newbiee > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[
https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323885#comment-16323885
]
Mate Juhasz commented on OOZIE-3061:
What do you think of this solution?
{noformat}
if
(YarnApplicationState.KILLED.equals(yarnClient.getApplicationReport(app).getYarnApplicationState()))
{
System.out.println("Application [" + app + "] already killed.");
} else {
yarnClient.killApplication(app);
System.out.println("Done");
}
{noformat}
> Kill only those child jobs which are not already killed
> ---
>
> Key: OOZIE-3061
> URL: https://issues.apache.org/jira/browse/OOZIE-3061
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Priority: Trivial
> Labels: newbie, newbiee
>
> Here we kill all child jobs.
> https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265
> We should check before killing for already killed application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-002.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: (was: OOZIE-3061-002.patch) > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345137#comment-16345137 ] Mate Juhasz commented on OOZIE-3061: Wouldn't be nicer to check the application state in the YarnClient"s killApplocation method? > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345137#comment-16345137 ] Mate Juhasz edited comment on OOZIE-3061 at 1/30/18 2:48 PM: - Wouldn't be nicer to check the application state in the YarnClient"s killApplication method? was (Author: matijhs): Wouldn't be nicer to check the application state in the YarnClient"s killApplocation method? > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345045#comment-16345045 ] Mate Juhasz commented on OOZIE-3061: [~gezapeti] Sure, I attached the patch containing the changes applied to the JavaActionExecuter. > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-002.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-003.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-001.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Attachments: OOZIE-3061-001.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Fix Version/s: 5.0.0 > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344840#comment-16344840 ] Mate Juhasz commented on OOZIE-3061: Thanks [~gezapeti] for adding me, I am looking forward to start contributing! I submitted the patch and hope I picked the right version numbers. > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-004.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360807#comment-16360807 ] Mate Juhasz commented on OOZIE-3061: That makes sense, thanks for pointing out! I have attached a patch for both classes. > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.0.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3311) [fluent-job] Create an ActionMarshaller
Mate Juhasz created OOZIE-3311: -- Summary: [fluent-job] Create an ActionMarshaller Key: OOZIE-3311 URL: https://issues.apache.org/jira/browse/OOZIE-3311 Project: Oozie Issue Type: Sub-task Reporter: Mate Juhasz Assignee: Mate Juhasz It would be handy to create action xmls in unit tests via the Fluent-Job API, thus an "action to xml" conversion could be implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3273) FsAction should fail on retry if destination path exists
[
https://issues.apache.org/jira/browse/OOZIE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564770#comment-16564770
]
Mate Juhasz commented on OOZIE-3273:
[~gezapeti], I think the file is there, but [~shubham.chhabra] would like to
see an error as the move did not occur, because a file with the same name
already exists.
I tried to reproduce, but I am not familiar with the process when the
'recovery' param changes to true, could you give a short intro about it? :)
> FsAction should fail on retry if destination path exists
>
>
> Key: OOZIE-3273
> URL: https://issues.apache.org/jira/browse/OOZIE-3273
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.2.0
>Reporter: Shubham
>Priority: Major
>
> This FsAction fails with error code FS008 if the source files already exist
> in target folder.
> The expected behavior should be that Oozie will try this action once again
> after 1 minute, and marked the action as failed because the error is still
> there.
> However, Oozie marks the action as success on retry. (we didn't clean up the
> target folder)
> Logs:
> {code}
> 2018-05-15 00:08:05,187 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
> [061-180514024838863-oozie-oozi-W@loading] with user-retry state :
> userRetryCount [0], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:08:05,201 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Error starting action
> [load-staging]. ErrorType [ERROR], ErrorCode [FS008], Message [FS008: move,
> could not move
> [hdfs://nn:8020/user/hive/audit/data/ingestion/USER_ACCOUNT_AF_A/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
> to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]]
> org.apache.oozie.action.ActionExecutorException: FS008: move, could not move
> [hdfs://nn:8020/user/hive/audit/data/ingestion/SAMPLE_WF/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
> to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.move(FsActionExecutor.java:509)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:609)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-05-15 00:08:05,202 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Setting Action Status to
> [DONE]
> 2018-05-15 00:08:05,202 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Preparing retry this
> action [061-180514024838863-oozie-oozi-W@loading], errorCode [FS008],
> userRetryCount [0], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:09:05,254 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
> [061-180514024838863-oozie-oozi-W@loading] with user-retry state :
> userRetryCount [1], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:09:05,276 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading]
>
[jira] [Updated] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3304:
---
Attachment: (was: OOZIE-3304-004.patch)
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
>
[jira] [Updated] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3304:
---
Attachment: OOZIE-3304-004.patch
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch, OOZIE-3304-004.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
>
[jira] [Updated] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3304:
---
Attachment: OOZIE-3304-004.patch
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch, OOZIE-3304-004.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
>
[jira] [Commented] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569949#comment-16569949
]
Mate Juhasz commented on OOZIE-3304:
I did some testing and I suspect that not killing the threads caused the
timeout, so I uploaded a new patch addressing the issue and used
ExecutorService to manage the running threads instead.
Can we kick off a new build to check it? Thanks!
cc [~zsombor], [~dionusos], [~gezapeti], [~andras.piros]
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at
[jira] [Comment Edited] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569949#comment-16569949
]
Mate Juhasz edited comment on OOZIE-3304 at 8/6/18 9:42 AM:
I did some testing and I suspect that not killing the threads caused the
timeout, so I uploaded a new patch addressing the issue and used
ExecutorService to manage the running threads instead.
The NUMBER_OF_FILESTATUSES value have been reduced to 100.000, its still enough
to get NumberFormatException, but significantly faster than with a million :)
Can we kick off a new build to check it? Thanks!
cc [~zsombor], [~dionusos], [~gezapeti], [~andras.piros]
was (Author: matijhs):
I did some testing and I suspect that not killing the threads caused the
timeout, so I uploaded a new patch addressing the issue and used
ExecutorService to manage the running threads instead.
Can we kick off a new build to check it? Thanks!
cc [~zsombor], [~dionusos], [~gezapeti], [~andras.piros]
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
>
[jira] [Updated] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3304:
---
Attachment: OOZIE-3304-003.patch
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
[jira] [Commented] (OOZIE-3304) Parsing sharelib timestamps is not threadsafe
[
https://issues.apache.org/jira/browse/OOZIE-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573099#comment-16573099
]
Mate Juhasz commented on OOZIE-3304:
Seems like the SchedulerService caused some trouble with scheduling a sharelib
purge on the many mocked libs in a separate thread. When its eliminated the
test execution looks good. Could you take a look at the latest patch? I may
have added some unwanted static imports as well... :)
> Parsing sharelib timestamps is not threadsafe
> -
>
> Key: OOZIE-3304
> URL: https://issues.apache.org/jira/browse/OOZIE-3304
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: OOZIE-3304-001.patch, OOZIE-3304-002.patch,
> OOZIE-3304-003.patch, OOZIE-3304-004.patch
>
>
> In rare cases the following Exception can be read in log files when an action
> fails:
> {code:java}
> org.apache.oozie.action.ActionExecutorException: NumberFormatException:
> multiple points
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NumberFormatException: multiple points
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1890)
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> at java.lang.Double.parseDouble(Double.java:538)
> at java.text.DigitList.getDouble(DigitList.java:169)
> at java.text.DecimalFormat.parse(DecimalFormat.java:2056)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2160)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1514)
> at java.text.DateFormat.parse(DateFormat.java:364)
> at
> org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:727)
> at
> org.apache.oozie.service.ShareLibService.getShareLibRootPath(ShareLibService.java:595)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.getSharelibRoot(JavaActionExecutor.java:1297)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.initShareLibExcluder(JavaActionExecutor.java:858)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:866)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1156)
> ... 11 more
> {code}
> or
> {code:java}
> 2018-07-12 04:48:52,649 WARN ForkedActionStartXCommand:523 -
> SERVER[ctr-e138-1518143905142-410551-01-03.hwx.site] USER[user] GROUP[-]
> TOKEN[] APP[demo-wf] JOB[023-180712043119670-oozie-oozi-W]
> ACTION[023-180712043119670-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [NumberFormatException], Message [NumberFormatException: For input string: ""]
> org.apache.oozie.action.ActionExecutorException: NumberFormatException: For
> input string: ""
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1271)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-005.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596085#comment-16596085 ] Mate Juhasz commented on OOZIE-3061: [~andras.piros], I try to add some tests as well, should we set the applications external status to KILLED when we dont actually kill them as they are already finished? > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3273) FsAction should fail on retry if destination path exists
[
https://issues.apache.org/jira/browse/OOZIE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550775#comment-16550775
]
Mate Juhasz commented on OOZIE-3273:
[~shubham.chhabra] if I understand correctly, the functioning can be misleading
as the failed move fs action is marked as succeeded.
[~andras.piros], [~gezapeti] could you please take a look at this issue?
Can this be a bug or 'works as designed'?
> FsAction should fail on retry if destination path exists
>
>
> Key: OOZIE-3273
> URL: https://issues.apache.org/jira/browse/OOZIE-3273
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.2.0
>Reporter: Shubham
>Priority: Major
>
> This FsAction fails with error code FS008 if the source files already exist
> in target folder.
> The expected behavior should be that Oozie will try this action once again
> after 1 minute, and marked the action as failed because the error is still
> there.
> However, Oozie marks the action as success on retry. (we didn't clean up the
> target folder)
> Logs:
> {code}
> 2018-05-15 00:08:05,187 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
> [061-180514024838863-oozie-oozi-W@loading] with user-retry state :
> userRetryCount [0], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:08:05,201 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Error starting action
> [load-staging]. ErrorType [ERROR], ErrorCode [FS008], Message [FS008: move,
> could not move
> [hdfs://nn:8020/user/hive/audit/data/ingestion/USER_ACCOUNT_AF_A/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
> to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]]
> org.apache.oozie.action.ActionExecutorException: FS008: move, could not move
> [hdfs://nn:8020/user/hive/audit/data/ingestion/SAMPLE_WF/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
> to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.move(FsActionExecutor.java:509)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:609)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-05-15 00:08:05,202 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Setting Action Status to
> [DONE]
> 2018-05-15 00:08:05,202 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Preparing retry this
> action [061-180514024838863-oozie-oozi-W@loading], errorCode [FS008],
> userRetryCount [0], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:09:05,254 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
> [061-180514024838863-oozie-oozi-W@loading] with user-retry state :
> userRetryCount [1], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:09:05,276 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading]
> [***061-180514024838863-oozie-oozi-W@loading***]Action status=DONE
> 2018-05-15
[jira] [Updated] (OOZIE-3273) FsAction should fail on retry if destination path exists
[
https://issues.apache.org/jira/browse/OOZIE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3273:
---
Affects Version/s: 4.2.0
> FsAction should fail on retry if destination path exists
>
>
> Key: OOZIE-3273
> URL: https://issues.apache.org/jira/browse/OOZIE-3273
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.2.0
>Reporter: Shubham
>Priority: Major
>
> This FsAction fails with error code FS008 if the source files already exist
> in target folder.
> The expected behavior should be that Oozie will try this action once again
> after 1 minute, and marked the action as failed because the error is still
> there.
> However, Oozie marks the action as success on retry. (we didn't clean up the
> target folder)
> Logs:
> {code}
> 2018-05-15 00:08:05,187 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
> [061-180514024838863-oozie-oozi-W@loading] with user-retry state :
> userRetryCount [0], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:08:05,201 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Error starting action
> [load-staging]. ErrorType [ERROR], ErrorCode [FS008], Message [FS008: move,
> could not move
> [hdfs://nn:8020/user/hive/audit/data/ingestion/USER_ACCOUNT_AF_A/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
> to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]]
> org.apache.oozie.action.ActionExecutorException: FS008: move, could not move
> [hdfs://nn:8020/user/hive/audit/data/ingestion/SAMPLE_WF/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
> to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.move(FsActionExecutor.java:509)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:609)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-05-15 00:08:05,202 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Setting Action Status to
> [DONE]
> 2018-05-15 00:08:05,202 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Preparing retry this
> action [061-180514024838863-oozie-oozi-W@loading], errorCode [FS008],
> userRetryCount [0], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:09:05,254 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
> [061-180514024838863-oozie-oozi-W@loading] with user-retry state :
> userRetryCount [1], userRetryMax [2], userRetryInterval [1]
> 2018-05-15 00:09:05,276 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading]
> [***061-180514024838863-oozie-oozi-W@loading***]Action status=DONE
> 2018-05-15 00:09:05,277 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
> USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
> JOB[061-180514024838863-oozie-oozi-W]
> ACTION[061-180514024838863-oozie-oozi-W@loading]
>
[jira] [Updated] (OOZIE-3273) FsAction should fail on retry if destination path exists
[
https://issues.apache.org/jira/browse/OOZIE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3273:
---
Description:
This FsAction fails with error code FS008 if the source files already exist in
target folder.
The expected behavior should be that Oozie will try this action once again
after 1 minute, and marked the action as failed because the error is still
there.
However, Oozie marks the action as success on retry. (we didn't clean up the
target folder)
Logs:
{code}
2018-05-15 00:08:05,187 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
[061-180514024838863-oozie-oozi-W@loading] with user-retry state :
userRetryCount [0], userRetryMax [2], userRetryInterval [1]
2018-05-15 00:08:05,201 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading] Error starting action
[load-staging]. ErrorType [ERROR], ErrorCode [FS008], Message [FS008: move,
could not move
[hdfs://nn:8020/user/hive/audit/data/ingestion/USER_ACCOUNT_AF_A/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]]
org.apache.oozie.action.ActionExecutorException: FS008: move, could not move
[hdfs://nn:8020/user/hive/audit/data/ingestion/SAMPLE_WF/1522284431816-2018-03-28_1747_11.816-PT2M10.096S-TEST.0-19462_24325-67401946-8fcf-4940-91ec-063016a5da48.avro]
to [hdfs://nn:8020/user/hive/audit/data/staging/USER_ACCOUNT_AF_A]
at
org.apache.oozie.action.hadoop.FsActionExecutor.move(FsActionExecutor.java:509)
at
org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
at
org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:609)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-05-15 00:08:05,202 WARN ActionStartXCommand:523 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading] Setting Action Status to
[DONE]
2018-05-15 00:08:05,202 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading] Preparing retry this
action [061-180514024838863-oozie-oozi-W@loading], errorCode [FS008],
userRetryCount [0], userRetryMax [2], userRetryInterval [1]
2018-05-15 00:09:05,254 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading] Start action
[061-180514024838863-oozie-oozi-W@loading] with user-retry state :
userRetryCount [1], userRetryMax [2], userRetryInterval [1]
2018-05-15 00:09:05,276 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading]
[***061-180514024838863-oozie-oozi-W@loading***]Action status=DONE
2018-05-15 00:09:05,277 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@loading]
[***061-180514024838863-oozie-oozi-W@loading***]Action updated in DB!
2018-05-15 00:09:05,314 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
JOB[061-180514024838863-oozie-oozi-W]
ACTION[061-180514024838863-oozie-oozi-W@end] Start action
[061-180514024838863-oozie-oozi-W@end] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-05-15 00:09:05,314 INFO ActionStartXCommand:520 - SERVER[mn2.sf.priv]
USER[hive] GROUP[-] TOKEN[] APP[sample-wf]
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547557#comment-16547557 ] Mate Juhasz commented on OOZIE-1624: Thanks [~andras.piros], I left the xml builder related issues open only, because I think that could be resolved in a separate Jira. What do you think? > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547578#comment-16547578 ] Mate Juhasz commented on OOZIE-1624: Ignore my previous comment, I just realized how wonderful is the JavaActionBuilder class... :) I create a new patch for the review board, thanks! > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (OOZIE-3156) Retry SSH action check when cannot connect to remote host
[
https://issues.apache.org/jira/browse/OOZIE-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reopened OOZIE-3156:
Reopening the issue for adding the amendment patch.
> Retry SSH action check when cannot connect to remote host
> -
>
> Key: OOZIE-3156
> URL: https://issues.apache.org/jira/browse/OOZIE-3156
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.0.0
>Reporter: TIAN XING
>Assignee: TIAN XING
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3156-v1.patch, OOZIE-3156-v2.patch,
> OOZIE-3156-v3.patch, OOZIE-3156-v4.patch, OOZIE-3156-v5.patch,
> OOZIE-3156-v6.patch, ssh-check-bug.patch
>
>
> When {{check()}} method of {{SshActionExecutor}} gets invoked, oozie will ssh
> connect to the host and check whether the pid of the process that ssh action
> started is still there (by checking the returned value of command "{{ssh
> ps -p }}" ) to determine whether ssh action completes or not.
> However, we found cases where oozie fails to connect to host during action
> status check (e.g., the host is under heavy load, or network is bad etc.).
> In such cases, the return value of command "{{ssh ps -p }}"
> will be 255 (ssh command exits with the exit status of the remote command or
> with 255 if an error occurred.).
> According the current logic of method {{getActionStatus()}} in
> {{SshActionExecutor}}, the action status will be determined as OK which may
> not be correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3156) Retry SSH action check when cannot connect to remote host
[
https://issues.apache.org/jira/browse/OOZIE-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549331#comment-16549331
]
Mate Juhasz commented on OOZIE-3156:
I found a minor typo in oozie-default.xml: "ooozie"
{code:java}
ooozie.action.ssh.check.retries.max{code}
> Retry SSH action check when cannot connect to remote host
> -
>
> Key: OOZIE-3156
> URL: https://issues.apache.org/jira/browse/OOZIE-3156
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.0.0
>Reporter: TIAN XING
>Assignee: TIAN XING
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3156-v1.patch, OOZIE-3156-v2.patch,
> OOZIE-3156-v3.patch, OOZIE-3156-v4.patch, OOZIE-3156-v5.patch,
> OOZIE-3156-v6.patch, ssh-check-bug.patch
>
>
> When {{check()}} method of {{SshActionExecutor}} gets invoked, oozie will ssh
> connect to the host and check whether the pid of the process that ssh action
> started is still there (by checking the returned value of command "{{ssh
> ps -p }}" ) to determine whether ssh action completes or not.
> However, we found cases where oozie fails to connect to host during action
> status check (e.g., the host is under heavy load, or network is bad etc.).
> In such cases, the return value of command "{{ssh ps -p }}"
> will be 255 (ssh command exits with the exit status of the remote command or
> with 255 if an error occurred.).
> According the current logic of method {{getActionStatus()}} in
> {{SshActionExecutor}}, the action status will be determined as OK which may
> not be correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Affects Version/s: 4.3.1 > Exclusion pattern for sharelib. > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381996#comment-16381996 ] Mate Juhasz commented on OOZIE-1624: Hello to Everyone! Although this issue may have been abandoned long ago, I think could be a really useful addition. e.g.: for excluding unwanted jars from the oozie sharelib folder for a specific action. In case you agree, I am happy to upload a new version of the patch. > Exclusion pattern for sharelib. > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[ https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-2792: --- Comment: was deleted (was: Hi "Job _job_xxx has completed successfully"_ does not match the given pattern: "_Job (job_\\_S*) completed successfully". The "has" should be removed I think.) > Hive2 action is not parsing Spark application ID from log file properly when > Hive is on Spark > - > > Key: OOZIE-2792 > URL: https://issues.apache.org/jira/browse/OOZIE-2792 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch, > OOZIE-2792-3.patch, OOZIE-2792-4-amendment.patch, OOZIE-2792-4.patch > > > When Hive2 is on Spark, hive action is not able to parse Spark application ID > from log file as 'externalChildID' like Spark/MR actions. This makes it hard > to tell which job hive launches from Oozie server for a particular workflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3192) OozieDBCLI does not destroy the Services
[
https://issues.apache.org/jira/browse/OOZIE-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428134#comment-16428134
]
Mate Juhasz commented on OOZIE-3192:
[~Prabhu Joseph] indeed!
> OozieDBCLI does not destroy the Services
>
>
> Key: OOZIE-3192
> URL: https://issues.apache.org/jira/browse/OOZIE-3192
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: OOZIE-3192.1.patch
>
>
> Oozie runtime directories are getting accumulated under /tmp. OozieDBCLI does
> not destroy the Services and so the code to delete runtime directories are
> skipped.
>
> {code:java}
> [root@bigdata2 libtools]# ls /tmp/oozie-oozi*
> /tmp/oozie-oozi1000989288737845910.dir:
> /tmp/oozie-oozi1036845504637967814.dir:
> /tmp/oozie-oozi2051016416521814301.dir:
> /tmp/oozie-oozi3194893167383531504.dir:
> /tmp/oozie-oozi3750814910369499964.dir:
> /tmp/oozie-oozi3988391799981903880.dir:
> /tmp/oozie-oozi4308477407910112054.dir:
> /tmp/oozie-oozi4717881377452217148.dir:
> /tmp/oozie-oozi5894080803700698940.dir:
> /tmp/oozie-oozi6367849996578983710.dir:
> /tmp/oozie-oozi682733264186033679.dir:
> /tmp/oozie-oozi7006370502487976692.dir:
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433989#comment-16433989 ] Mate Juhasz commented on OOZIE-1624: [~andras.piros], [~gezapeti] could you please take a look at this? > Exclusion pattern for sharelib. > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2683) Rewrite the Oozie Web UI
[ https://issues.apache.org/jira/browse/OOZIE-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428333#comment-16428333 ] Mate Juhasz commented on OOZIE-2683: +1 for using Angular and/or React.js. [~andras.piros] I would be happy to participate if you can offload some of the work. > Rewrite the Oozie Web UI > > > Key: OOZIE-2683 > URL: https://issues.apache.org/jira/browse/OOZIE-2683 > Project: Oozie > Issue Type: New Feature >Reporter: Robert Kanter >Assignee: Andras Piros >Priority: Major > > We're currently relying on a version of ExtJS that's so old, it's not linked > on their official website, and is often temporarily deleted. Looks like it's > now been deleted for good (OOZIE-2622). It also has a GPL license, so we > can't include it with Oozie, and users are forced to go and download it > themselves. And finally, it's a really outdated UI that's not very good. > We should invest in a new Web UI that's more modern and compatibly licensed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440595#comment-16440595 ] Mate Juhasz commented on OOZIE-1624: [~andras.piros], sure, I created the request! > Exclusion pattern for sharelib. > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440595#comment-16440595 ] Mate Juhasz edited comment on OOZIE-1624 at 4/17/18 8:33 AM: - [~andras.piros], sure, I created the request! https://reviews.apache.org/r/66656/ was (Author: matijhs): [~andras.piros], sure, I created the request! > Exclusion pattern for sharelib. > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib.
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V3.patch > Exclusion pattern for sharelib. > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3355) Regex based search option for searching workflows in the WFM-View of Ambari
[
https://issues.apache.org/jira/browse/OOZIE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639866#comment-16639866
]
Mate Juhasz commented on OOZIE-3355:
As I see, the wfm executes a request like this for getting the jobs in
_org/apache/oozie/ambari/view/OozieDelegate.java#readFromOozie_:
{code:java}
Proxy request for url: [GET]
http://ctr-e138-1518143905142-496852-01-03.hwx.site:11000/oozie/v2/jobs?filter=name=sparkcompare=1=1=wf
{code}
Then Oozie handles it in _org/apache/oozie/servlet/V0JobsServlet.java#getJobs,_
and the actual query is done in
_org/apache/oozie/executor/jpa/WorkflowsJobGetJPAExecutor.java#execute_
WorkflowsJobGetJPAExecutor is a bit heavy, but if we can agree on a solution I
am happy to do it.
> Regex based search option for searching workflows in the WFM-View of Ambari
> ---
>
> Key: OOZIE-3355
> URL: https://issues.apache.org/jira/browse/OOZIE-3355
> Project: Oozie
> Issue Type: New Feature
> Components: bundle, coordinator, core, tools, ui, workflow
>Affects Versions: 4.2.0
>Reporter: Krishnadevan Purushothaman
>Priority: Critical
>
> *Challenge faced :*
> _{color:#d04437}In the WFM view of ambari, there is no Filter option
> available to search for the Workflows. In order to search for the desired
> workflow, one has to type the full name of workflow,coordinator,bundles else
> it does not return anything which becomes a time-consuming job.{color}_
> *Feature description:*
> _{color:#14892c}There is a need for Regex based filter option in order to
> search the workflows without the need of entering the complete name of the
> workflow,coordinator,bundles rather by just typing the first three letters of
> the work post which it should populate suggestions based on the first three
> letters of the workflow,coordinator,bundles through which we can refine and
> optimize the searching mechanism.{color}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3355) Regex based search option for searching workflows in the WFM-View of Ambari
[
https://issues.apache.org/jira/browse/OOZIE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647712#comment-16647712
]
Mate Juhasz commented on OOZIE-3355:
[~andras.piros], these are the available filter parameters from the ambari wfm:
* name
* id (oozie job id)
* user (job submitter user)
* status (SUCCEEDED, RUNNING, FAILED, KILLED, SUSPENDED)
* jobtype (wf, coord, bundles)
* startCreatedTime
* endCreatedTime
The offset and len parameters are used for paginating the results.
* len - always 20
* offset - always (page number - 1) * len
The filter parameter is filled with each of the user defined conditions, but
these seem to be in OR relation instead of AND:
_filter=status=SUCCEEDED;name=mr-wf_
Another observation about time filtering, that the wfm shows the local time,
but the filtering has to be done in GMT as oozie works.
> Regex based search option for searching workflows in the WFM-View of Ambari
> ---
>
> Key: OOZIE-3355
> URL: https://issues.apache.org/jira/browse/OOZIE-3355
> Project: Oozie
> Issue Type: New Feature
> Components: bundle, coordinator, core, tools, ui, workflow
>Affects Versions: 4.2.0
>Reporter: Krishnadevan Purushothaman
>Priority: Critical
>
> *Challenge faced :*
> _{color:#d04437}In the WFM view of ambari, there is no Filter option
> available to search for the Workflows. In order to search for the desired
> workflow, one has to type the full name of workflow,coordinator,bundles else
> it does not return anything which becomes a time-consuming job.{color}_
> *Feature description:*
> _{color:#14892c}There is a need for Regex based filter option in order to
> search the workflows without the need of entering the complete name of the
> workflow,coordinator,bundles rather by just typing the first three letters of
> the work post which it should populate suggestions based on the first three
> letters of the workflow,coordinator,bundles through which we can refine and
> optimize the searching mechanism.{color}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-006.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602159#comment-16602159 ] Mate Juhasz commented on OOZIE-3061: Thanks [~andras.piros], I uploaded a patch according to your comments > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[
https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802711#comment-16802711
]
Mate Juhasz commented on OOZIE-3450:
I have managed to reduce to number of jars on the git sharelib (9) with better
chosen dependency scopes, but unfortunately I could not find an environment to
run the examples code. [~asalamon74] could you please help to check what
happens if we keep the below jars only? I upload a patch as well, but these are
really minor changes in code.
{noformat}
find share/lib/git -name "*.jar" | sort
share/lib/git/JavaEWAH-1.1.6.jar
share/lib/git/commons-lang3-3.3.2.jar
share/lib/git/httpclient-4.3.6.jar
share/lib/git/httpcore-4.3.3.jar
share/lib/git/jsch-0.1.54.jar
share/lib/git/jzlib-1.1.1.jar
share/lib/git/oozie-sharelib-git-5.2.0-SNAPSHOT.jar
share/lib/git/org.eclipse.jgit-5.0.1.201806211838-r.jar
share/lib/git/slf4j-api-1.6.6.jar
{noformat}
> Investigate and clean git sharelib
> --
>
> Key: OOZIE-3450
> URL: https://issues.apache.org/jira/browse/OOZIE-3450
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3540-v1.patch
>
>
> I've checked the number of jars in the Oozie sharelibs and realized that git
> sharelib contains the highest number of jars (203), it's much more than the
> hive (85), pig (67). Not to mention that we have really small sharelibs like
> distcp (3).
> I don't really understand the reason for this, we need to check if we really
> need all the jars here. The huge number of jars make it slower and it's more
> likely that we get strange errors because of jar conflicts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3450) Investigate and clean git sharelib
[ https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3450: --- Attachment: OOZIE-3540-v1.patch > Investigate and clean git sharelib > -- > > Key: OOZIE-3450 > URL: https://issues.apache.org/jira/browse/OOZIE-3450 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Priority: Major > Attachments: OOZIE-3540-v1.patch > > > I've checked the number of jars in the Oozie sharelibs and realized that git > sharelib contains the highest number of jars (203), it's much more than the > hive (85), pig (67). Not to mention that we have really small sharelibs like > distcp (3). > I don't really understand the reason for this, we need to check if we really > need all the jars here. The huge number of jars make it slower and it's more > likely that we get strange errors because of jar conflicts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V4.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch, > OOZIE-3443-V3.patch, OOZIE-3443-V4.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803043#comment-16803043 ] Mate Juhasz commented on OOZIE-3443: I agree [~asalamon74], but in this case I would exclude the transitive joda-time from oozie-core, so we will only package them in the necessary sharelib projects. > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch, > OOZIE-3443-V3.patch, OOZIE-3443-V4.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3443) Migrate from joda time to java.time
[
https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801857#comment-16801857
]
Mate Juhasz commented on OOZIE-3443:
I agree :) I put back the dependency to the root pom.xml to force the versions
to 2.9.9, thanks [~asalamon74]!
Now the distro will look like this:
{noformat}
./embedded-oozie-server/webapp/WEB-INF/lib/joda-time-2.9.9.jar
./share/lib/pig/joda-time-2.9.9.jar
./share/lib/hcatalog/joda-time-2.9.9.jar
./share/lib/hive2/joda-time-2.9.9.jar
./share/lib/hive/joda-time-2.9.9.jar
./share/lib/spark/joda-time-2.9.9.jar
{noformat}
> Migrate from joda time to java.time
> ---
>
> Key: OOZIE-3443
> URL: https://issues.apache.org/jira/browse/OOZIE-3443
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch,
> OOZIE-3443-V3.patch
>
>
> From Java SE 8 onwards, users are asked to migrate to from [joda
> time|https://www.joda.org/joda-time/] to java.time (JSR-310).
> It seems to me, that we only use it directly in one place:
> [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java]
> so it would be not too difficult to migrate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V3.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch, > OOZIE-3443-V3.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V2.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[ https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804899#comment-16804899 ] Mate Juhasz commented on OOZIE-3450: Thanks [~asalamon74] for the testing. Could you please give the new patch a try as well? Interesting that each of the sharelib projects are using the oozie-core as a provided dependency, the only difference is that while most of them just reffering to static constants in the ActionExecutor-s, sharelib-git's GitMain tries to call a method in GitActionExecutor$ActionConfVerifier. This method is the ActionConfVerifier#checkAndGetTrimmed, which only returns an action conf value, not connecting strongly anywhere, so I moved it to the GitMain class. > Investigate and clean git sharelib > -- > > Key: OOZIE-3450 > URL: https://issues.apache.org/jira/browse/OOZIE-3450 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-3450-v2.patch, OOZIE-3540-v1.patch > > > I've checked the number of jars in the Oozie sharelibs and realized that git > sharelib contains the highest number of jars (203), it's much more than the > hive (85), pig (67). Not to mention that we have really small sharelibs like > distcp (3). > I don't really understand the reason for this, we need to check if we really > need all the jars here. The huge number of jars make it slower and it's more > likely that we get strange errors because of jar conflicts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3196) Authorization: restrict world readability by user
[ https://issues.apache.org/jira/browse/OOZIE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3196: -- Assignee: Mate Juhasz (was: Peter Orova) > Authorization: restrict world readability by user > - > > Key: OOZIE-3196 > URL: https://issues.apache.org/jira/browse/OOZIE-3196 > Project: Oozie > Issue Type: New Feature > Components: bundle, coordinator, workflow >Affects Versions: 5.0.0b1, 5.0.0 >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3196.001.patch > > > The [*current authorization > model*|https://issues.apache.org/jira/browse/OOZIE-228] does not fit the > enterprise requirements as everything is readable and writable by everyone by > default. > Write access can be restricted using authorization but restricting read > rights is only possible via Yarn ACLs and HDFS rights which still does not > prevent accessing the workflow, coordinator or bundle job’s configurations > for everyone. > Improve authorization so it’s possible to configure read/write access for > workflows, coordinators, and bundles in a more granular way. Could involve > Sentry during implementation or create and design a new system that fits the > needs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3196) Authorization: restrict world readability by user
[ https://issues.apache.org/jira/browse/OOZIE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812246#comment-16812246 ] Mate Juhasz commented on OOZIE-3196: [~orova] thank you for letting me to take over the remaining tasks. I would like to get introduced with the related code parts first, then I will update the jira, hopefully with a POC. > Authorization: restrict world readability by user > - > > Key: OOZIE-3196 > URL: https://issues.apache.org/jira/browse/OOZIE-3196 > Project: Oozie > Issue Type: New Feature > Components: bundle, coordinator, workflow >Affects Versions: 5.0.0b1, 5.0.0 >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3196.001.patch > > > The [*current authorization > model*|https://issues.apache.org/jira/browse/OOZIE-228] does not fit the > enterprise requirements as everything is readable and writable by everyone by > default. > Write access can be restricted using authorization but restricting read > rights is only possible via Yarn ACLs and HDFS rights which still does not > prevent accessing the workflow, coordinator or bundle job’s configurations > for everyone. > Improve authorization so it’s possible to configure read/write access for > workflows, coordinators, and bundles in a more granular way. Could involve > Sentry during implementation or create and design a new system that fits the > needs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3282) Prototyping: create Angular 6 workflow list page
[ https://issues.apache.org/jira/browse/OOZIE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776773#comment-16776773 ] Mate Juhasz commented on OOZIE-3282: [~kmarton] I didnt have time to work on it and I found the react protoype really great, maybe we should go with that one > Prototyping: create Angular 6 workflow list page > > > Key: OOZIE-3282 > URL: https://issues.apache.org/jira/browse/OOZIE-3282 > Project: Oozie > Issue Type: Sub-task >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > > Create an [*Angular 6*|https://angular.io/docs] prototype of the workflow > list page. Don't have to handle authentication / authorization for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781688#comment-16781688 ] Mate Juhasz commented on OOZIE-1624: Uploaded new patch based on the current master > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V5.patch > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781760#comment-16781760 ] Mate Juhasz commented on OOZIE-1624: Thanks [~asalamon74], uploaded to the review board as well. I agree with you, the ActionMarshaller could go in separately. Waiting for your feedback, thanks! > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786802#comment-16786802 ] Mate Juhasz commented on OOZIE-1624: Thanks [~asalamon74], I forgot to add it. Attached to the review, hope it's clear and understandable. > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786844#comment-16786844 ] Mate Juhasz commented on OOZIE-1624: Fixed the typos and line breaks and also added the warning message for using exclusions :) > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V10.patch > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V10.patch, OOZIE-1624-V2.patch, > OOZIE-1624-V3.patch, OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, > OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V10.patch > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V10.patch, OOZIE-1624-V2.patch, > OOZIE-1624-V3.patch, OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, > OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3443: -- Assignee: Mate Juhasz > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V1.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V1.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: (was: OOZIE-3443-V1.patch) > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3024:
--
Assignee: Mate Juhasz
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3024:
---
Attachment: OOZIE-3024.patch
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885210#comment-16885210
]
Mate Juhasz commented on OOZIE-3024:
This issue has popped up recently, so I picked it up quickly. [~asalamon74]
could you take a look?
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921406#comment-16921406
]
Mate Juhasz commented on OOZIE-3405:
Taking this over [~orova] if you dont mind :)
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3405:
--
Assignee: Mate Juhasz (was: Peter Orova)
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3405:
---
Attachment: OOZIE-3405-V1.patch
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3405-V1.patch
>
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3405:
---
Attachment: OOZIE-3405-V2.patch
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-3405-V1.patch, OOZIE-3405-V2.patch
>
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887091#comment-16887091
]
Mate Juhasz edited comment on OOZIE-3024 at 7/17/19 2:20 PM:
-
[~asalamon74], I added a test for emails with attachments and verified on a
cluster as well that the fix is working.
was (Author: matijhs):
[~asalamon74], I added a test for emails with attachments and verified that the
fix is working.
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024-V2.patch, OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887091#comment-16887091
]
Mate Juhasz commented on OOZIE-3024:
[~asalamon74], I added a test for emails with attachments and verified that the
fix is working.
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024-V2.patch, OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3024:
---
Attachment: OOZIE-3024-V2.patch
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024-V2.patch, OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-1974) SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in [PREP] stage
[
https://issues.apache.org/jira/browse/OOZIE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938413#comment-16938413
]
Mate Juhasz commented on OOZIE-1974:
Not related, the ssh-wrapper.sh code is "designed" in a way that it does not
support compound commands
> SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in
> [PREP] stage
> --
>
> Key: OOZIE-1974
> URL: https://issues.apache.org/jira/browse/OOZIE-1974
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Michalis Kongtongk
>Assignee: Mate Juhasz
>Priority: Major
>
> example WF that will fail:
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Workaround is to execute the compound command in subshell eg: $(cmd1 && cmd2)
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> $(kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
> )
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Stack trace "org.apache.oozie.command.CommandException: E0800: Action it is
> not running its in [PREP] state,"
> {code}
> 2014-08-05 23:29:49,721 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() begins
> 2014-08-05 23:29:49,723 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] Attempting to copy ssh base
> scripts to remote host [m...@192-168-88-213.lunix.lan]
> 2014-08-05 23:29:52,691 INFO org.apache.oozie.servlet.CallbackServlet:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] callback for action
> [008-140805224842389-oozie-oozi-W@Ssh]
> 2014-08-05 23:29:52,714 ERROR
> org.apache.oozie.command.wf.CompletedActionXCommand:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] XException,
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:52,714 WARN
> org.apache.oozie.service.CallableQueueService$CallableWrapper:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] exception callable [callback], E0800: Action it is not running its
> in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:57,262 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() ends
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3543) Upgrade quartz to 2.3.1
[ https://issues.apache.org/jira/browse/OOZIE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3543: -- Assignee: Mate Juhasz > Upgrade quartz to 2.3.1 > --- > > Key: OOZIE-3543 > URL: https://issues.apache.org/jira/browse/OOZIE-3543 > Project: Oozie > Issue Type: Bug >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Blocker > Fix For: 5.2.0 > > > Oozie uses quartz 2.1.7, which is six years old and depends on c3p0:c3p0 > 0.9.1.1 which has LGPL license. We should upgrade to quartz 2.3.1 which > depends on com.mchange:c3p0. This newer version of c3p0 has dual license, EPL > is good for us. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3543) Upgrade quartz to 2.3.1
[ https://issues.apache.org/jira/browse/OOZIE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3543: --- Attachment: OOZIE-3543.patch > Upgrade quartz to 2.3.1 > --- > > Key: OOZIE-3543 > URL: https://issues.apache.org/jira/browse/OOZIE-3543 > Project: Oozie > Issue Type: Bug >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Blocker > Fix For: 5.2.0 > > Attachments: OOZIE-3543.patch > > > Oozie uses quartz 2.1.7, which is six years old and depends on c3p0:c3p0 > 0.9.1.1 which has LGPL license. We should upgrade to quartz 2.3.1 which > depends on com.mchange:c3p0. This newer version of c3p0 has dual license, EPL > is good for us. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-2576) Oozie ssh action Cannot run program "scp"
[
https://issues.apache.org/jira/browse/OOZIE-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936818#comment-16936818
]
Mate Juhasz commented on OOZIE-2576:
Its a bit late to answer, but I think the error is because adding multiple
commands are not supported in case of ssh actions.
It should work if you specify the command like below:
{noformat}
/tmp/test.sh
{noformat}
Or:
{noformat}
sh
/tmp/test.sh
{noformat}
> Oozie ssh action Cannot run program "scp"
> -
>
> Key: OOZIE-2576
> URL: https://issues.apache.org/jira/browse/OOZIE-2576
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.0.0
> Environment: Oozie server build version: 4.0.0-cdh5.2.0
>Reporter: ywheel
>Priority: Blocker
>
> We used oozie ssh action in a production environment, the following xml codes
> is the example:
> {noformat}
>
>
>
>
> root@192.168.1.154
> sh /tmp/test.sh
>
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
> However, we meet the errors below:
> {noformat}
> 2016-06-12 22:30:54,713 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[Master] USER[hdfs] GROUP[-] TOKEN[] APP[TestSsh]
> JOB[201-160113124428061-oozie-oozi-W]
> ACTION[201-160113124428061-oozie-oozi-W@ShellAction] Attempting to copy
> ssh base scripts to remote host [root@192.168.1.154]
> 2016-06-12 22:30:54,869 WARN org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[Master] USER[hdfs] GROUP[-] TOKEN[] APP[TestSsh]
> JOB[201-160113124428061-oozie-oozi-W]
> ACTION[201-160113124428061-oozie-oozi-W@ShellAction] Error while
> executing ssh EXECUTION
> 2016-06-12 22:30:54,870 WARN org.apache.oozie.command.wf.ActionStartXCommand:
> SERVER[Master] USER[hdfs] GROUP[-] TOKEN[] APP[TestSsh]
> JOB[201-160113124428061-oozie-oozi-W]
> ACTION[201-160113124428061-oozie-oozi-W@ShellAction] Error starting
> action [ShellAction]. ErrorType [ERROR], ErrorCode [UNKOWN_ERROR], Message
> [UNKOWN_ERROR: Cannot run program "scp": error=2, No such file or directory]
> org.apache.oozie.action.ActionExecutorException: UNKOWN_ERROR: Cannot run
> program "scp": error=2, No such file or directory
> at
> org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:599)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.start(SshActionExecutor.java:204)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> at org.apache.oozie.command.XCommand.call(XCommand.java:281)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.IOException: Cannot run program "scp": error=2, No such
> file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
> at java.lang.Runtime.exec(Runtime.java:617)
> at java.lang.Runtime.exec(Runtime.java:485)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.executeCommand(SshActionExecutor.java:332)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.setupRemote(SshActionExecutor.java:376)
> at
> org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:206)
> at
> org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:204)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:548)
> ... 10 more
> Caused by: java.io.IOException: error=2, No such file or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:135)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
> ... 17 more
> {noformat}
> I checked the codes in {{org.apache.oozie.action.ssh.SshActionExecutor}}
> class, and I found oozie would execute 'scp' command to copy two
> files(ssh-base.sh,ssh-wrapper.sh) to the workspace folder on remote host. But
> the logs could not show any information about which file is not found.
> The workspace folder on remote host was created so the 'ssh'+ 'mkdir'
[jira] [Assigned] (OOZIE-3465) Migrate from commons-codec
[
https://issues.apache.org/jira/browse/OOZIE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3465:
--
Assignee: Mate Juhasz
> Migrate from commons-codec
> --
>
> Key: OOZIE-3465
> URL: https://issues.apache.org/jira/browse/OOZIE-3465
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
>
> Ooize uses {{commons-codec}} 1.4 which is almost [10
> years|https://commons.apache.org/proper/commons-codec/changes-report.html]
> old. We hardly use this library I've only found a few references to the
> {{Base64}} class in Oozie. Java8 defines a
> [Base64|https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html]
> class which we could use instead.
> We should be careful to put this jar into the sharelibs if and only if some
> other library requires it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (OOZIE-3363) Hadoop's cleanup of local directory in uber mode causing failures for sqoop actions
[
https://issues.apache.org/jira/browse/OOZIE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz resolved OOZIE-3363.
Resolution: Duplicate
> Hadoop's cleanup of local directory in uber mode causing failures for sqoop
> actions
> ---
>
> Key: OOZIE-3363
> URL: https://issues.apache.org/jira/browse/OOZIE-3363
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0, 5.0.0
>Reporter: Clemens Valiente
>Priority: Blocker
>
> See related task OOZIE-2536 and discussion
> https://issues.apache.org/jira/browse/OOZIE-2536?focusedCommentId=16617206=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16617206
> the same issue cleaning up local directories also affects sqoop jobs that
> create the sqoop.xml file inbetween.
> {code:java}
> 2018-09-15 00:31:09,416 [AsyncDispatcher event handler] INFO
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl - Num completed Tasks: 1
> 2018-09-15 00:31:09,417 [AsyncDispatcher event handler] INFO
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl -
> job_1535972033593_259806Job Transitioned from RUNNING to COMMITTING
> 2018-09-15 00:31:09,419 [CommitterEvent Processor #1] INFO
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler - Processing
> the event EventType: JOB_COMMIT
> 2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN
> org.apache.hadoop.mapred.LocalContainerLauncher - Unable to delete
> unexpected local file/dir .action.xml.crc: insufficient permissions?
> 2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN
> org.apache.hadoop.mapred.LocalContainerLauncher - Unable to delete
> unexpected local file/dir .action.xml.crc: insufficient permissions?
> 2018-09-15 00:31:09,459 [CommitterEvent Processor #1] FATAL
> org.apache.hadoop.conf.Configuration - error parsing conf sqoop-site.xml
> java.io.FileNotFoundException:
> /appdata/hdfs/v7/yarn/nm/usercache/SEM/appcache/application_1535972033593_259806/container_e100_1535972033593_259806_01_01/sqoop-site.xml
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileInputStream.(FileInputStream.java:101)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2483)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2554)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2507)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2413)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:984)
> at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1034)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1254)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.touchz(CommitterEventHandler.java:268)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:282)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-2966) OozieCLI and TestOozieCLI test class are in different packages
[ https://issues.apache.org/jira/browse/OOZIE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991422#comment-16991422 ] Mate Juhasz commented on OOZIE-2966: The idea is good and valid I think, but the solution is not that trivial unfortunately. If we move the test classes to the client project we would need to reference for example to core and fluent-job sub-projects and that would mean circular dependencies among core and client. The only solution for resolving the circularity is to restructure a lot of code, but this way I dont think it worth the time. > OozieCLI and TestOozieCLI test class are in different packages > -- > > Key: OOZIE-2966 > URL: https://issues.apache.org/jira/browse/OOZIE-2966 > Project: Oozie > Issue Type: Task > Components: tests >Affects Versions: 4.3.0 >Reporter: Artem Ervits >Assignee: Artem Ervits >Priority: Trivial > > OozieCLI class is in client/org/apache/oozie/cli whereas the associated test > class resides in core/org/apache/oozie/client shouldn't they belong to same > project? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3549:
--
Assignee: Mate Juhasz
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952869#comment-16952869
]
Mate Juhasz commented on OOZIE-3549:
Uploaded a patch, but it is still to be tested on a real deployment
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: OOZIE-3549.patch
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959011#comment-16959011
]
Mate Juhasz commented on OOZIE-3549:
Thanks [~asalamon74] for the summary. However it turned out, that the issue
might be somewhere else... there is this function
SSLServerConnectorFactory#setKeystorePass where we call
sslContextFactory.KeyManagerPassword(keystorePass) instead of
setKeyStorePassword.
The patch is working if we change this part, which I could not understand well,
so I would like to test it out better in a deployment.
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3487) Confusing E0820 error message
[
https://issues.apache.org/jira/browse/OOZIE-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3487:
--
Assignee: Mate Juhasz
> Confusing E0820 error message
> -
>
> Key: OOZIE-3487
> URL: https://issues.apache.org/jira/browse/OOZIE-3487
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3487.patch
>
>
> E0820 error messages are confusing:
> {noformat}
> Action user retry max [3] is over system defined max [3], re-assign to use
> system max.{noformat}
> It's not true that 3 is over 3.
>
> It's a bug in
> [LiteWorkflowStoreService|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java]
> {noformat}
> if (ret > max) {
> ret = max;
> log.warn(ErrorCode.E0820.getTemplate(), ret, max);
> }
> {noformat}
> We overwrite the max value before we print out the error messge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3487) Confusing E0820 error message
[
https://issues.apache.org/jira/browse/OOZIE-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3487:
---
Attachment: OOZIE-3487.patch
> Confusing E0820 error message
> -
>
> Key: OOZIE-3487
> URL: https://issues.apache.org/jira/browse/OOZIE-3487
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3487.patch
>
>
> E0820 error messages are confusing:
> {noformat}
> Action user retry max [3] is over system defined max [3], re-assign to use
> system max.{noformat}
> It's not true that 3 is over 3.
>
> It's a bug in
> [LiteWorkflowStoreService|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java]
> {noformat}
> if (ret > max) {
> ret = max;
> log.warn(ErrorCode.E0820.getTemplate(), ret, max);
> }
> {noformat}
> We overwrite the max value before we print out the error messge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3491:
--
Assignee: Mate Juhasz
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3491:
---
Attachment: OOZIE-3491.patch
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-3491.patch
>
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
