[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279670#comment-16279670
]
Abhishek Bafna commented on OOZIE-3139:
---
[~dionusos] Looks good to me. +1.
Can we also include the below scenario. Thanks.
{code}
kill
{code}
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OOZIE-3080) testOldFormatRead fails with classnotfound
[
https://issues.apache.org/jira/browse/OOZIE-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna resolved OOZIE-3080.
---
Resolution: Not A Problem
> testOldFormatRead fails with classnotfound
> --
>
> Key: OOZIE-3080
> URL: https://issues.apache.org/jira/browse/OOZIE-3080
> Project: Oozie
> Issue Type: Bug
> Components: tests
>Reporter: Yesha Vora
>
> testOldFormatRead UT fails with classnotfound
> {code}
> Error Message
> java.lang.ClassNotFoundException: null
> Stacktrace
> java.io.IOException: java.lang.ClassNotFoundException: null
> at
> org.apache.oozie.workflow.lite.NodeDef.readVersionZero(NodeDef.java:140)
> at org.apache.oozie.workflow.lite.NodeDef.readFields(NodeDef.java:190)
> at
> org.apache.oozie.workflow.lite.LiteWorkflowApp.readFields(LiteWorkflowApp.java:147)
> at
> org.apache.oozie.workflow.lite.TestLiteWorkflowApp.testOldFormatRead(TestLiteWorkflowApp.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: null
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.oozie.workflow.lite.NodeDef.readVersionZero(NodeDef.java:137)
> ... 24 more{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2927) Append new line character for Hive2 query using query tag
[ https://issues.apache.org/jira/browse/OOZIE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049051#comment-16049051 ] Abhishek Bafna commented on OOZIE-2927: --- Thanks [~gezapeti]. I think, I will modify the existing test case to cover the scenario. > Append new line character for Hive2 query using query tag > - > > Key: OOZIE-2927 > URL: https://issues.apache.org/jira/browse/OOZIE-2927 > Project: Oozie > Issue Type: Bug > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2927-00.patch, OOZIE-2927-01.patch > > > Without the new line character in the query, beeline does not do anything. > Though hive2 action completes successfully but query is not executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2927) Append new line character for Hive2 query using query tag
[ https://issues.apache.org/jira/browse/OOZIE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2927: -- Attachment: OOZIE-2927-01.patch > Append new line character for Hive2 query using query tag > - > > Key: OOZIE-2927 > URL: https://issues.apache.org/jira/browse/OOZIE-2927 > Project: Oozie > Issue Type: Bug > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2927-00.patch, OOZIE-2927-01.patch > > > Without the new line character in the query, beeline does not do anything. > Though hive2 action completes successfully but query is not executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2927) Append new line character for Hive2 query using query tag
[ https://issues.apache.org/jira/browse/OOZIE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2927: -- Attachment: OOZIE-2927-00.patch > Append new line character for Hive2 query using query tag > - > > Key: OOZIE-2927 > URL: https://issues.apache.org/jira/browse/OOZIE-2927 > Project: Oozie > Issue Type: Bug > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2927-00.patch > > > Without the new line character in the query, beeline does not do anything. > Though hive2 action completes successfully but query is not executed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OOZIE-2927) Append new line character for Hive2 query using query tag
Abhishek Bafna created OOZIE-2927: - Summary: Append new line character for Hive2 query using query tag Key: OOZIE-2927 URL: https://issues.apache.org/jira/browse/OOZIE-2927 Project: Oozie Issue Type: Bug Components: core Reporter: Abhishek Bafna Assignee: Abhishek Bafna Without the new line character in the query, beeline does not do anything. Though hive2 action completes successfully but query is not executed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2927) Append new line character for Hive2 query using query tag
[ https://issues.apache.org/jira/browse/OOZIE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2927: -- Fix Version/s: 5.0.0 > Append new line character for Hive2 query using query tag > - > > Key: OOZIE-2927 > URL: https://issues.apache.org/jira/browse/OOZIE-2927 > Project: Oozie > Issue Type: Bug > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > > Without the new line character in the query, beeline does not do anything. > Though hive2 action completes successfully but query is not executed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient
[
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001875#comment-16001875
]
Abhishek Bafna commented on OOZIE-2751:
---
Thanks [~rkanter] and others for reviews.
> LocalOozieClient is missing methods from OozieClient
>
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch,
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch,
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch,
> OOZIE-2751-08.patch, OOZIE-2751-09.patch, OOZIE-2751-10.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the
> normal {{OozieClient}} methods instead of making REST calls to an Oozie
> Server. However, it currently doesn't override all methods, which means you
> hit the {{OozieClient}} implementation, which tries to make a REST call and
> will fail (because it's not setup for that). For example, {{getStatus(String
> jobId)}} isn't overridden, and doesn't work when used from
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient
[
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2751:
--
Attachment: OOZIE-2751-10.patch
[~rkanter] Addressing your review comments. Thanks.
> LocalOozieClient is missing methods from OozieClient
>
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch,
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch,
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch,
> OOZIE-2751-08.patch, OOZIE-2751-09.patch, OOZIE-2751-10.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the
> normal {{OozieClient}} methods instead of making REST calls to an Oozie
> Server. However, it currently doesn't override all methods, which means you
> hit the {{OozieClient}} implementation, which tries to make a REST call and
> will fail (because it's not setup for that). For example, {{getStatus(String
> jobId)}} isn't overridden, and doesn't work when used from
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient
[
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2751:
--
Attachment: OOZIE-2751-09.patch
> LocalOozieClient is missing methods from OozieClient
>
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch,
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch,
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch,
> OOZIE-2751-08.patch, OOZIE-2751-09.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the
> normal {{OozieClient}} methods instead of making REST calls to an Oozie
> Server. However, it currently doesn't override all methods, which means you
> hit the {{OozieClient}} implementation, which tries to make a REST call and
> will fail (because it's not setup for that). For example, {{getStatus(String
> jobId)}} isn't overridden, and doesn't work when used from
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient
[
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992879#comment-15992879
]
Abhishek Bafna commented on OOZIE-2751:
---
Ping [~pbacsko], [~gezapeti].
> LocalOozieClient is missing methods from OozieClient
>
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch,
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch,
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch,
> OOZIE-2751-08.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the
> normal {{OozieClient}} methods instead of making REST calls to an Oozie
> Server. However, it currently doesn't override all methods, which means you
> hit the {{OozieClient}} implementation, which tries to make a REST call and
> will fail (because it's not setup for that). For example, {{getStatus(String
> jobId)}} isn't overridden, and doesn't work when used from
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2795) Create lib directory or symlink for Oozie CLI during packaging
[
https://issues.apache.org/jira/browse/OOZIE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981361#comment-15981361
]
Abhishek Bafna commented on OOZIE-2795:
---
Few Points:
1. The {{/lib}} does not have the additional jars from {{/libext}} added by the
users. The {{prepare_jetty}} executes after the jars from WEB-INF are already
got copied.
2. line: 240 Have incorrect path into the {{ls $JETTY_LIB_DIR/$jarPath}}
command. It should be just {{$jarPath}}.
3. We should not exit in case of a jar or library already copied. There could
be other jars in the array which are newly added by the user into {{libext}}.
line: 234, 245
Thanks.
> Create lib directory or symlink for Oozie CLI during packaging
> --
>
> Key: OOZIE-2795
> URL: https://issues.apache.org/jira/browse/OOZIE-2795
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
> Attachments: OOZIE-2795-001.patch, OOZIE-2795-002.patch,
> OOZIE-2795-003.patch
>
>
> After creating an Oozie distro, lib directory is not present in the distro
> directory (e.g. oozie-4.4.0-SNAPSHOT) in the generated tar.gz .
>
> As a consequence, first run of theOozie CLI will fail if one tries to execute
> it prior to server start:
> {noformat}
> $ bin/oozie
> unzip: cannot find or open
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war,
>
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.zip
> or
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.ZIP.
> mv: rename
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib/*.jar
> to
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/*.jar:
> No such file or directory
> rmdir:
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib:
> No such file or directory
> rmdir:
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF:
> No such file or directory
> Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
> {noformat}
> If you start the server first however, a symlink is created (see
> {{bin/oozie-jetty-server.sh}}), and this error will not be visible.
> Earlier {{bin/oozie}} created this library by extracting it from the WAR file
> used for Tomcat. Comment in the script:
> {noformat}
> #Create lib directory from war if lib doesn't exist
> {noformat}
> We could create a symlink to embedded-oozie-server/webapp/WEB-INF/lib/ or
> copy that directory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient
[
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2751:
--
Attachment: OOZIE-2751-08.patch
[~pbacsko] Addressing your review comments. Thanks.
> LocalOozieClient is missing methods from OozieClient
>
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch,
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch,
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch,
> OOZIE-2751-08.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the
> normal {{OozieClient}} methods instead of making REST calls to an Oozie
> Server. However, it currently doesn't override all methods, which means you
> hit the {{OozieClient}} implementation, which tries to make a REST call and
> will fail (because it's not setup for that). For example, {{getStatus(String
> jobId)}} isn't overridden, and doesn't work when used from
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (OOZIE-2795) Create lib directory or symlink for Oozie CLI during packaging
[
https://issues.apache.org/jira/browse/OOZIE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978862#comment-15978862
]
Abhishek Bafna edited comment on OOZIE-2795 at 4/21/17 3:42 PM:
[~gezapeti] If create the {{lib}} directory in {{distro.xml}} then it is
affecting other logic which has been put into different scripts. The logic
based on; if {{lib}} is not present, then create and copy the jars from
{{WEB-INF/lib}}.
Thanks.
was (Author: abhishekbafna):
[~gezapeti] Given that we are creating and copying the jars into lib directory
{{src/main/assemblies/distro.xml}}. I think copying from WEB-INF into lib is
not required.
I tried it, after removing those steps from: {{client/src/main/bin/oozie}},
{{distro/src/main/bin/oozie-setup.sh}} and {{tools/src/main/bin/ooziedb.sh}}.
The sharelib and start of the oozie service worked fine for me. Can you check
that I did not miss anything here.
Thanks.
> Create lib directory or symlink for Oozie CLI during packaging
> --
>
> Key: OOZIE-2795
> URL: https://issues.apache.org/jira/browse/OOZIE-2795
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
> Attachments: OOZIE-2795-001.patch, OOZIE-2795-002.patch,
> OOZIE-2795-003.patch
>
>
> After creating an Oozie distro, lib directory is not present in the distro
> directory (e.g. oozie-4.4.0-SNAPSHOT) in the generated tar.gz .
>
> As a consequence, first run of theOozie CLI will fail if one tries to execute
> it prior to server start:
> {noformat}
> $ bin/oozie
> unzip: cannot find or open
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war,
>
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.zip
> or
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.ZIP.
> mv: rename
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib/*.jar
> to
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/*.jar:
> No such file or directory
> rmdir:
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib:
> No such file or directory
> rmdir:
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF:
> No such file or directory
> Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
> {noformat}
> If you start the server first however, a symlink is created (see
> {{bin/oozie-jetty-server.sh}}), and this error will not be visible.
> Earlier {{bin/oozie}} created this library by extracting it from the WAR file
> used for Tomcat. Comment in the script:
> {noformat}
> #Create lib directory from war if lib doesn't exist
> {noformat}
> We could create a symlink to embedded-oozie-server/webapp/WEB-INF/lib/ or
> copy that directory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2795) Create lib directory or symlink for Oozie CLI during packaging
[
https://issues.apache.org/jira/browse/OOZIE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978862#comment-15978862
]
Abhishek Bafna commented on OOZIE-2795:
---
[~gezapeti] Given that we are creating and copying the jars into lib directory
{{src/main/assemblies/distro.xml}}. I think copying from WEB-INF into lib is
not required.
I tried it, after removing those steps from: {{client/src/main/bin/oozie}},
{{distro/src/main/bin/oozie-setup.sh}} and {{tools/src/main/bin/ooziedb.sh}}.
The sharelib and start of the oozie service worked fine for me. Can you check
that I did not miss anything here.
Thanks.
> Create lib directory or symlink for Oozie CLI during packaging
> --
>
> Key: OOZIE-2795
> URL: https://issues.apache.org/jira/browse/OOZIE-2795
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
> Attachments: OOZIE-2795-001.patch, OOZIE-2795-002.patch,
> OOZIE-2795-003.patch
>
>
> After creating an Oozie distro, lib directory is not present in the distro
> directory (e.g. oozie-4.4.0-SNAPSHOT) in the generated tar.gz .
>
> As a consequence, first run of theOozie CLI will fail if one tries to execute
> it prior to server start:
> {noformat}
> $ bin/oozie
> unzip: cannot find or open
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war,
>
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.zip
> or
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.ZIP.
> mv: rename
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib/*.jar
> to
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/*.jar:
> No such file or directory
> rmdir:
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib:
> No such file or directory
> rmdir:
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF:
> No such file or directory
> Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
> {noformat}
> If you start the server first however, a symlink is created (see
> {{bin/oozie-jetty-server.sh}}), and this error will not be visible.
> Earlier {{bin/oozie}} created this library by extracting it from the WAR file
> used for Tomcat. Comment in the script:
> {noformat}
> #Create lib directory from war if lib doesn't exist
> {noformat}
> We could create a symlink to embedded-oozie-server/webapp/WEB-INF/lib/ or
> copy that directory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2827) More directly view of the coordinator’s history from perspective of workflow action.
[ https://issues.apache.org/jira/browse/OOZIE-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978754#comment-15978754 ] Abhishek Bafna commented on OOZIE-2827: --- [~pbacsko] As part of the commit, one wrong file got committed. Can you please check. Thanks. Commit: https://github.com/apache/oozie/commit/4061831378daca80191e5abe296c3ee75e79c8bd File: core/src/main/java/org/apache/oozie/CoordinatorEngine.java.orig > More directly view of the coordinator’s history from perspective of workflow > action. > > > Key: OOZIE-2827 > URL: https://issues.apache.org/jira/browse/OOZIE-2827 > Project: Oozie > Issue Type: New Feature >Reporter: Alonzo Zhou > Fix For: 5.0.0 > > Attachments: OOZIE-2827-0001.patch, OOZIE-2827-0002.patch, > OOZIE-2827-0003.patch, OOZIE-2827-0004.patch, OOZIE-2827-0005.patch > > > Background: > When we monitor a coordinator job, statistics of historical workflow > instances such as running time can be directly visualized. > But, we lack the ability to monitor historical statistics of coordinator from > perspective of workflow action.For example, running time of a coordinator > action is longer than before, but we can't see the exact change tendency of a > sepcific workflow action in it。 > Functional optimization: > More detailed view of the coordinator’s history can be observed from > perspective of workflow action.Furthermore, the trend of a workflow action > can be shown directly on front-end graph. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958990#comment-15958990 ] Abhishek Bafna commented on OOZIE-2701: --- Thank you [~gezapeti] and others for review. Committed to master. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch, > OOZIE-2701-11.patch, OOZIE-2701-12.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-12.patch > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch, > OOZIE-2701-11.patch, OOZIE-2701-12.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955171#comment-15955171
]
Abhishek Bafna commented on OOZIE-2701:
---
[~gezapeti] The difference is between two terms {{multiple HCatalog servers in
the URI}} v/s {{multiple HCatalog URI}}.
The first is regarding, when user is having single HCat URI which gets parsed
using the HCatURIParser#parseURI.
The second is regarding, when user is having multiple HCat URIs in place.
I have improved the testcase
{{TestHCatURIParser#testParsingMultipleHCatServerURI}} and added some more doc.
Please let me know, If this needs more improvement. :)
PS: The patch will have a doc line which is more than 132 character.
Thanks.
> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch,
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch,
> OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch,
> OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch,
> OOZIE-2701-11.patch, OOZIE-2701-12.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-11.patch [~gezapeti] Added the Doc. Please check. Thanks. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch, > OOZIE-2701-11.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf
[
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953654#comment-15953654
]
Abhishek Bafna commented on OOZIE-2845:
---
+1. Committed to master. Thanks [~pbacsko] for the patch.
> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 5.0.0
>
> Attachments: OOZIE-2845-001.patch, OOZIE-2845-002.patch
>
>
> In different places, we use the following reflection-based code to reset a
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null,
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time.
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2841) Limit FindBugs diff errors in JIRA comments
[
https://issues.apache.org/jira/browse/OOZIE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947204#comment-15947204
]
Abhishek Bafna commented on OOZIE-2841:
---
[~gezapeti] I have pushed the contributor change into release-log, in the this
commit:
https://github.com/apache/oozie/commit/7c404ad0ea4c61e90e8c86015de25ef196168c29
Thanks.
> Limit FindBugs diff errors in JIRA comments
> ---
>
> Key: OOZIE-2841
> URL: https://issues.apache.org/jira/browse/OOZIE-2841
> Project: Oozie
> Issue Type: Improvement
> Components: build
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Attachments: OOZIE-2841.001.patch, OOZIE-2841.002.patch,
> OOZIE-2841.amend.patch, OOZIE-2841.wrong.patch,
> test-patch-reports.after.tar.gz, test-patch-reports.before.tar.gz
>
>
> In OOZIE-1986 the FindBugs diff functionality was introduced. At times,
> however, there are cases when it's painful that we print out each and every
> FindBugs error (the ones considered mandatory to be fixed) [*to the JIRA
> comment*|https://issues.apache.org/jira/browse/OOZIE-2812?focusedCommentId=15937447=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937447].
> We'll limit the FindBugs errors to be displayed within a module (let's say,
> top 5 from a module). This limitation will only be valid for
> {{TEST-SUMMARY.jira}} and {{TEST-SUMMARY.txt}} - for
> {{TEST-SUMMARY-FULL.jira}} and {{TEST-SUMMARY-FULL.txt}} the output remains
> unchanged.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf
[
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947154#comment-15947154
]
Abhishek Bafna commented on OOZIE-2845:
---
[~pbacsko] I think
{{HiveConf.setHiveSiteLocation(null);}}
should be
{{HiveConf.setHiveSiteLocation(HiveConf.class.getClassLoader().getResource("hive-site.xml"));}}
Because, we set the {{hiveSiteURL}} to the available {{hive-site.xml}} in the
classpath.
Did we test this, on actual cluster? If not, we should I guess.
Thanks.
> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch
>
>
> In different places, we use the following reflection-based code to reset a
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null,
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time.
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-10.patch [~gezapeti] Thank You. it was left from local testing. Removed it. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2841) Limit FindBugs diff errors in JIRA comments
[
https://issues.apache.org/jira/browse/OOZIE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945258#comment-15945258
]
Abhishek Bafna commented on OOZIE-2841:
---
[~gezapeti] I think this can be committed directly without any amend patch.
Otherwise, we can take care of this into our next commit. Thanks.
> Limit FindBugs diff errors in JIRA comments
> ---
>
> Key: OOZIE-2841
> URL: https://issues.apache.org/jira/browse/OOZIE-2841
> Project: Oozie
> Issue Type: Improvement
> Components: build
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Attachments: OOZIE-2841.001.patch, OOZIE-2841.002.patch,
> OOZIE-2841.amend.patch, OOZIE-2841.wrong.patch,
> test-patch-reports.after.tar.gz, test-patch-reports.before.tar.gz
>
>
> In OOZIE-1986 the FindBugs diff functionality was introduced. At times,
> however, there are cases when it's painful that we print out each and every
> FindBugs error (the ones considered mandatory to be fixed) [*to the JIRA
> comment*|https://issues.apache.org/jira/browse/OOZIE-2812?focusedCommentId=15937447=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937447].
> We'll limit the FindBugs errors to be displayed within a module (let's say,
> top 5 from a module). This limitation will only be valid for
> {{TEST-SUMMARY.jira}} and {{TEST-SUMMARY.txt}} - for
> {{TEST-SUMMARY-FULL.jira}} and {{TEST-SUMMARY-FULL.txt}} the output remains
> unchanged.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2753) Update Docs for Jetty Tomcat changes
[
https://issues.apache.org/jira/browse/OOZIE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945222#comment-15945222
]
Abhishek Bafna commented on OOZIE-2753:
---
[~asasvari] Sorry missed some of the points in the last review:
We should clean up the {{oozie-env.sh}}. Also it is referred into
{{AG_install}} in the end. Can we update that well.
> Update Docs for Jetty Tomcat changes
>
>
> Key: OOZIE-2753
> URL: https://issues.apache.org/jira/browse/OOZIE-2753
> Project: Oozie
> Issue Type: Sub-task
> Components: docs
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2753-01.patch, OOZIE-2753-02.patch,
> OOZIE-2753-03.patch, OOZIE-2753-04.patch, OOZIE-2753-05.patch
>
>
> Update the docs based on the changes for Jetty and Tomcat (OOZIE-2317 and
> child JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944828#comment-15944828 ] Abhishek Bafna edited comment on OOZIE-2701 at 3/28/17 9:09 AM: [~gezapeti] There was change in regex used for identifying the URLs and moved the test cases also into the core module. Can you take another look, for resolving this. Thanks. was (Author: abhishekbafna): [~gezapeti] There was change in regex used of identifying the URLs and moved the test cases also into the core module. Can you take another look, for resolving this. Thanks. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944828#comment-15944828 ] Abhishek Bafna commented on OOZIE-2701: --- [~gezapeti] There was change in regex used of identifying the URLs and moved the test cases also into the core module. Can you take another look, for resolving this. Thanks. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2841) Limit FindBugs diff errors in JIRA comments
[
https://issues.apache.org/jira/browse/OOZIE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943596#comment-15943596
]
Abhishek Bafna commented on OOZIE-2841:
---
[~gezapeti] You missed to add the username in the release-log.txt entry. Thanks.
Commit Link:
https://github.com/apache/oozie/commit/890f69ead827467381983b185fd4fa9a6a0780cd
{{OOZIE-2841 Limit FindBugs diff errors in JIRA comments}}
> Limit FindBugs diff errors in JIRA comments
> ---
>
> Key: OOZIE-2841
> URL: https://issues.apache.org/jira/browse/OOZIE-2841
> Project: Oozie
> Issue Type: Improvement
> Components: build
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Attachments: OOZIE-2841.001.patch, OOZIE-2841.002.patch,
> OOZIE-2841.wrong.patch, test-patch-reports.after.tar.gz,
> test-patch-reports.before.tar.gz
>
>
> In OOZIE-1986 the FindBugs diff functionality was introduced. At times,
> however, there are cases when it's painful that we print out each and every
> FindBugs error (the ones considered mandatory to be fixed) [*to the JIRA
> comment*|https://issues.apache.org/jira/browse/OOZIE-2812?focusedCommentId=15937447=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937447].
> We'll limit the FindBugs errors to be displayed within a module (let's say,
> top 5 from a module). This limitation will only be valid for
> {{TEST-SUMMARY.jira}} and {{TEST-SUMMARY.txt}} - for
> {{TEST-SUMMARY-FULL.jira}} and {{TEST-SUMMARY-FULL.txt}} the output remains
> unchanged.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2753) Update Docs for Jetty Tomcat changes
[
https://issues.apache.org/jira/browse/OOZIE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943384#comment-15943384
]
Abhishek Bafna commented on OOZIE-2753:
---
Thanks [~asasvari].
1. There is small typo {{trustore}} in the AG_install.twiki, line 737.
2. In the AG_install.twiki, we have "To use a Self-Signed Certificate" section.
For example commands, we are still using "tomcat" as alias, can you update that
as well.
3. In the "DG_CommandLineTool.twiki", can you check and update the output of
the {{-osenv}}, {{-javasysprops}} oozie admin command options. line 1334
on-wards.
4. There is still some left over documentation for {{prepare-war}} in the
AG_install.twiki and DG_QuickStart.twiki.
Can you create a Jira for Windows environment and link it here.
Thanks.
> Update Docs for Jetty Tomcat changes
>
>
> Key: OOZIE-2753
> URL: https://issues.apache.org/jira/browse/OOZIE-2753
> Project: Oozie
> Issue Type: Sub-task
> Components: docs
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2753-01.patch, OOZIE-2753-02.patch,
> OOZIE-2753-03.patch
>
>
> Update the docs based on the changes for Jetty and Tomcat (OOZIE-2317 and
> child JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2681) fix javadoc to compile on JDK8 again
[ https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2681: -- Attachment: OOZIE-2681-04.patch [~pbacsko] Thanks for pointing it out again. I hope, I got them this time. :) > fix javadoc to compile on JDK8 again > > > Key: OOZIE-2681 > URL: https://issues.apache.org/jira/browse/OOZIE-2681 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 4.3.0 >Reporter: Robert Kanter >Assignee: Abhishek Bafna >Priority: Critical > Fix For: 5.0.0 > > Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch, > OOZIE-2681-02.patch, OOZIE-2681-03.patch, OOZIE-2681-04.patch > > > We have a lot of Javadoc build failures, which causes Oozie to fail to build > on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all > of these that existed at that time, but we've since introduced a lot more. > OOZIE-2487 made a workaround to prevent this from causing the build to fail, > but we should fix these (again) and remove the OOZIE-2487 workaround. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2681) fix javadoc to compile on JDK8 again
[ https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942576#comment-15942576 ] Abhishek Bafna commented on OOZIE-2681: --- The testcase passes locally. Thanks. > fix javadoc to compile on JDK8 again > > > Key: OOZIE-2681 > URL: https://issues.apache.org/jira/browse/OOZIE-2681 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 4.3.0 >Reporter: Robert Kanter >Assignee: Abhishek Bafna >Priority: Critical > Fix For: 5.0.0 > > Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch, > OOZIE-2681-02.patch, OOZIE-2681-03.patch > > > We have a lot of Javadoc build failures, which causes Oozie to fail to build > on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all > of these that existed at that time, but we've since introduced a lot more. > OOZIE-2487 made a workaround to prevent this from causing the build to fail, > but we should fix these (again) and remove the OOZIE-2487 workaround. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-09.patch > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, > OOZIE-2701-08.patch, OOZIE-2701-09.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2701:
--
Attachment: OOZIE-2701-08.patch
Thanks for identifying. Moved the test {{TestHCatURIParser}} into core module.
> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch,
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch,
> OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch,
> OOZIE-2701-08.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2681) fix javadoc to compile on JDK8 again
[ https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940248#comment-15940248 ] Abhishek Bafna commented on OOZIE-2681: --- [~pbacsko] I will take a another look and fix them. Thanks. > fix javadoc to compile on JDK8 again > > > Key: OOZIE-2681 > URL: https://issues.apache.org/jira/browse/OOZIE-2681 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 4.3.0 >Reporter: Robert Kanter >Assignee: Abhishek Bafna >Priority: Critical > Fix For: 5.0.0 > > Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch, > OOZIE-2681-02.patch > > > We have a lot of Javadoc build failures, which causes Oozie to fail to build > on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all > of these that existed at that time, but we've since introduced a lot more. > OOZIE-2487 made a workaround to prevent this from causing the build to fail, > but we should fix these (again) and remove the OOZIE-2487 workaround. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2817) Increase test case stability in pre-commit job
[ https://issues.apache.org/jira/browse/OOZIE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940228#comment-15940228 ] Abhishek Bafna commented on OOZIE-2817: --- +1. Committed to master. Thanks [~gezapeti]. Just a thought, going forward we should create a separate issue for this, as we have already done two amends. :) > Increase test case stability in pre-commit job > -- > > Key: OOZIE-2817 > URL: https://issues.apache.org/jira/browse/OOZIE-2817 > Project: Oozie > Issue Type: Bug >Reporter: Peter Cseh >Assignee: Peter Cseh > Fix For: 5.0.0 > > Attachments: OOZIE-2817-001.patch, OOZIE-2817-002.patch, > OOZIE-2817-003.patch, OOZIE-2817-amend2.patch, OOZIE-2817-amend.patch > > > We could collect the failed tests and rerun them in the pre-commit hook. > They will still fail if they are broken, but they'll probably pass if they > are just flaky. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-07.patch [~asasvari] Thanks for helping in improving the regex. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-06.patch > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch, OOZIE-2701-06.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939785#comment-15939785 ] Abhishek Bafna commented on OOZIE-2701: --- Thanks [~asasvari] for feedback. For the regex part, The regex is for splitting individual complete URIs. I guess user will provide the correct hostname. Made the change regarding repetitive initialization of Pattern. I did not understand your point regarding 'not to test scheme's', can you please elaborate little bit. Thanks. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2817) Increase test case stability in pre-commit job
[
https://issues.apache.org/jira/browse/OOZIE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938192#comment-15938192
]
Abhishek Bafna commented on OOZIE-2817:
---
For the Build:
https://builds.apache.org/job/oozie-trunk-precommit-build/3722/consoleFull.
This build had an flask test which got passed in the rerun. But that
information did not got printed in the Jira
[comment|https://issues.apache.org/jira/browse/OOZIE-2701?focusedCommentId=15938155=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15938155].
{code}
flakyTests=$(cat ${TEMPDIR}/${TASKNAME}.flakies.out)
echo "{color:green}+1 ${TASKNAME}{color}" >> $SUMMARYFILE
echo ".Tests run: $testsRun" >> $SUMMARYFILE
if [ -n "${flakyTests}" ]; then
echo ".Tests rerun: ${testsReRun}" >> $SUMMARYFILE
echo ".Tests failed at first run: ${flakyTests}" >> $SUMMARYFILE
{code}
I think issue was:
{code}
failures=$(find . -name '*\.txt' | grep target/surefire-reports | xargs
grep "<<<" | grep -v "Tests run:" | sed 's/.*\.txt\://' | sed 's/ .*//' | grep
-o '(\(.*\))' | tr -d '()' | uniq | awk '{ sub ("$", ""); printf "%s",
$0"," } END { print "" }')
if [ -n "${failures}" ]; then
echo "Rerunning failed tests: ${failures}"
echo $FAILURES > ${TEMPDIR}/${TASKNAME}.flakies.out
{code}
The case for the variable {{FAILURES}} vs {{failures}}.
Total tests run in Jenkins Test report: 1,909 (Link:
https://builds.apache.org/job/oozie-trunk-precommit-build/3722/testReport)
Total tests run in Comment from Report: 1912.
Not sure, why there is difference, could not reach to a conclusion.
[~gezapeti] Can you please take look here?
Thanks.
> Increase test case stability in pre-commit job
> --
>
> Key: OOZIE-2817
> URL: https://issues.apache.org/jira/browse/OOZIE-2817
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Fix For: 5.0.0
>
> Attachments: OOZIE-2817-001.patch, OOZIE-2817-002.patch,
> OOZIE-2817-003.patch, OOZIE-2817-amend.patch
>
>
> We could collect the failed tests and rerun them in the pre-commit hook.
> They will still fail if they are broken, but they'll probably pass if they
> are just flaky.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-05.patch > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, > OOZIE-2701-05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937988#comment-15937988
]
Abhishek Bafna commented on OOZIE-2701:
---
[~asasvari] I think this {{/\w/\w+}} will force that database name can be of
single character.
I will split the regex for our better understanding:
{noformat}
\[a-z\]+= Scheme
\[\w+\.\-\w+\]+ = Hostname or IP address.
\w+ = database name
\w+ = table name
\[/\w+=;\-\]* = partition information
{noformat}
Please let me know If I am missing something. Thanks.
> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch,
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch,
> OOZIE-2701-05.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient
[
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937753#comment-15937753
]
Abhishek Bafna commented on OOZIE-2751:
---
Ping [~gezapeti] and [~pbacsko]. Thanks.
> LocalOozieClient is missing methods from OozieClient
>
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch,
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch,
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the
> normal {{OozieClient}} methods instead of making REST calls to an Oozie
> Server. However, it currently doesn't override all methods, which means you
> hit the {{OozieClient}} implementation, which tries to make a REST call and
> will fail (because it's not setup for that). For example, {{getStatus(String
> jobId)}} isn't overridden, and doesn't work when used from
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2701:
--
Attachment: OOZIE-2701-04.patch
The regex pattern in oozie-default.xml is updated after OOZIE-2393.
{noformat}
oozie.hcat.uri.regex.pattern
([a-z]+://[\w+\.\-\w+]+:\d+[,]*)+/\w+/\w+[/\w+=;\-]*
Regex pattern for HCat URIs. The regex can be modified by
users as per requirement
for parsing/splitting the HCat URIs.
{noformat}
CC: [~gezapeti] and [~andras.piros].
> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch,
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2817) Increase test case stability in pre-commit job
[ https://issues.apache.org/jira/browse/OOZIE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936503#comment-15936503 ] Abhishek Bafna commented on OOZIE-2817: --- +1. > Increase test case stability in pre-commit job > -- > > Key: OOZIE-2817 > URL: https://issues.apache.org/jira/browse/OOZIE-2817 > Project: Oozie > Issue Type: Bug >Reporter: Peter Cseh >Assignee: Peter Cseh > Fix For: 5.0.0 > > Attachments: OOZIE-2817-001.patch, OOZIE-2817-002.patch, > OOZIE-2817-003.patch, OOZIE-2817-amend.patch > > > We could collect the failed tests and rerun them in the pre-commit hook. > They will still fail if they are broken, but they'll probably pass if they > are just flaky. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-03.patch Uploading a rebase patch for Jenkin, before it gets committed. Thanks. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch, OOZIE-2701-03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936459#comment-15936459
]
Abhishek Bafna commented on OOZIE-2041:
---
Sure. Thanks for let me know. I will keep in mind until we fix this. :)
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch,
> OOZIE-2041-08.patch, OOZIE-2041-09.patch, OOZIE-2041-10.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2393) Allow table drop in hcat prepare
[
https://issues.apache.org/jira/browse/OOZIE-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936444#comment-15936444
]
Abhishek Bafna commented on OOZIE-2393:
---
Thanks [~gezapeti] and others for the review. Committed to master.
> Allow table drop in hcat prepare
>
>
> Key: OOZIE-2393
> URL: https://issues.apache.org/jira/browse/OOZIE-2393
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.1.0, 4.2.0
>Reporter: Muhammad Ehsan ul Haque
>Assignee: Abhishek Bafna
>Priority: Minor
> Labels: hcatalog, prepare
> Fix For: 5.0.0
>
> Attachments: OOZIE-2393-00.patch, OOZIE-2393-01.patch
>
>
> The hcat prepare only allows to drop partitions. It would be nice to also
> allow dropping of table, depending on the URL. The current format of the URL
> is
> {code}hcat://[metastore server]:[port]/[database name]/[table
> name]/[partkey1]=[value];[partkey2]=[value]{code}
> where at least one partition must be provided, otherwise the prepare step
> fails with the fololwing exception.
> {code}
> Starting the execution of prepare actions
> Creating HCatClient for user=ehsan.haq (auth:SIMPLE) and
> server=thrift://datavault-prod-app2.internal.machines:9083
> Prepare execution in the Launcher Mapper has failed
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), Error
> trying to drop
> hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
> org.apache.oozie.action.hadoop.LauncherException: Error trying to drop
> hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
> at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:178)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.oozie.action.hadoop.LauncherException: Error trying to
> drop
> hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
> at
> org.apache.oozie.action.hadoop.HCatLauncherURIHandler.delete(HCatLauncherURIHandler.java:64)
> at
> org.apache.oozie.action.hadoop.PrepareActionsDriver.execute(PrepareActionsDriver.java:89)
> at
> org.apache.oozie.action.hadoop.PrepareActionsDriver.doOperations(PrepareActionsDriver.java:67)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.executePrepare(LauncherMapper.java:446)
> at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:174)
> ... 8 more
> Caused by: java.net.URISyntaxException: URI path is not in expected format:
> hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
> at org.apache.oozie.util.HCatURI.parse(HCatURI.java:66)
> at org.apache.oozie.util.HCatURI.(HCatURI.java:52)
> at org.apache.oozie.util.HCatURI.(HCatURI.java:48)
> at
> org.apache.oozie.action.hadoop.HCatLauncherURIHandler.delete(HCatLauncherURIHandler.java:52)
> ... 12 more
> Oozie Launcher failed, finishing Hadoop job gracefully
> {code}
> h3. Suggestion
> In the URL if the partition is not provided then it should delete the entire
> table.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936439#comment-15936439
]
Abhishek Bafna commented on OOZIE-2041:
---
Committed to master. Thanks [~gezapeti] and others for review.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch,
> OOZIE-2041-08.patch, OOZIE-2041-09.patch, OOZIE-2041-10.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936432#comment-15936432
]
Abhishek Bafna commented on OOZIE-2041:
---
Did not find any test failure in the Jenkins job, though here it say one.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch,
> OOZIE-2041-08.patch, OOZIE-2041-09.patch, OOZIE-2041-10.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2041:
--
Attachment: OOZIE-2041-10.patch
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch,
> OOZIE-2041-08.patch, OOZIE-2041-09.patch, OOZIE-2041-10.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2681) fix javadoc to compile on JDK8 again
[ https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935704#comment-15935704 ] Abhishek Bafna commented on OOZIE-2681: --- Thanks [~asasvari] for the review. [~gezapeti] and [~pbacsko] Can you please take a look? Thanks. > fix javadoc to compile on JDK8 again > > > Key: OOZIE-2681 > URL: https://issues.apache.org/jira/browse/OOZIE-2681 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 4.3.0 >Reporter: Robert Kanter >Assignee: Abhishek Bafna >Priority: Critical > Fix For: 5.0.0 > > Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch, > OOZIE-2681-02.patch > > > We have a lot of Javadoc build failures, which causes Oozie to fail to build > on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all > of these that existed at that time, but we've since introduced a lot more. > OOZIE-2487 made a workaround to prevent this from causing the build to fail, > but we should fix these (again) and remove the OOZIE-2487 workaround. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2540) Create a PySpark example
[ https://issues.apache.org/jira/browse/OOZIE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927587#comment-15927587 ] Abhishek Bafna commented on OOZIE-2540: --- Thanks [~rkanter] and others for review and committing it. > Create a PySpark example > > > Key: OOZIE-2540 > URL: https://issues.apache.org/jira/browse/OOZIE-2540 > Project: Oozie > Issue Type: Task > Components: examples >Reporter: Robert Kanter >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2540-00.patch > > > Now that we have PySpark working correctly in the Spark Action, we should > make an example that runs a PySpark job to give users an example of how to do > it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2779) Mask Hive2 action Beeline JDBC password
[
https://issues.apache.org/jira/browse/OOZIE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898409#comment-15898409
]
Abhishek Bafna commented on OOZIE-2779:
---
Testcase passed locally. Thanks.
> Mask Hive2 action Beeline JDBC password
> ---
>
> Key: OOZIE-2779
> URL: https://issues.apache.org/jira/browse/OOZIE-2779
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Labels: security
> Fix For: 5.0.0
>
> Attachments: OOZIE-2779-00.patch, OOZIE-2779-01.patch,
> OOZIE-2779-01.patch
>
>
> Hive2 Oozie launcher job prints the JDBC password into launcher stdout logs.
> {noformat}
> Beeline command arguments :
> -u
> jdbc:hive2://source-1:1/default
> -n
> ambari-qa
> -p
> DUMMY
> -d
> org.apache.hive.jdbc.HiveDriver
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878103#comment-15878103
]
Abhishek Bafna commented on OOZIE-2803:
---
[~pbacsko] Thanks for pointing to the previous discussion. I think we can keep
it same.
For the JavaDoc: {{@param String}} and {{@param Input}} there should be
parameter names, which is missing.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877556#comment-15877556
]
Abhishek Bafna commented on OOZIE-2803:
---
Few points:
There are some unused imports in the {{MapReduceMain}}.
Fix the java doc for {{PasswordMasker:maskPasswordsIfNecessary}} and
{{PasswordMasker:mask}}.
All the field members of {{PasswordMasker}} are declared final and class in
itself does not have any state. it can be converted into Utility Class.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2779) Mask Hive2 action Beeline JDBC password
[
https://issues.apache.org/jira/browse/OOZIE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875887#comment-15875887
]
Abhishek Bafna commented on OOZIE-2779:
---
[~rkanter] can you please take a look?
> Mask Hive2 action Beeline JDBC password
> ---
>
> Key: OOZIE-2779
> URL: https://issues.apache.org/jira/browse/OOZIE-2779
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Labels: security
> Fix For: 5.0.0
>
> Attachments: OOZIE-2779-00.patch
>
>
> Hive2 Oozie launcher job prints the JDBC password into launcher stdout logs.
> {noformat}
> Beeline command arguments :
> -u
> jdbc:hive2://source-1:1/default
> -n
> ambari-qa
> -p
> DUMMY
> -d
> org.apache.hive.jdbc.HiveDriver
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2684) Bad database schema error for WF_ACTIONS table
[
https://issues.apache.org/jira/browse/OOZIE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875885#comment-15875885
]
Abhishek Bafna commented on OOZIE-2684:
---
[~rkanter] Can you please take a look? Thanks.
> Bad database schema error for WF_ACTIONS table
> --
>
> Key: OOZIE-2684
> URL: https://issues.apache.org/jira/browse/OOZIE-2684
> Project: Oozie
> Issue Type: Bug
> Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2684-00.patch
>
>
> In {{SchemaCheckerService}}, Oozie compares expected and found indexed
> columns and generates the below error message which could be confusing to the
> users.
> {noformat}
> 2016-09-16 12:39:26,564 INFO SchemaCheckXCommand:520 -
> SERVER[c6402.ambari.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] About to check database schema
> 2016-09-16 12:39:26,703 ERROR SchemaCheckXCommand:517 -
> SERVER[c6402.ambari.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] Found [1] extra indexes for columns in table [WF_ACTIONS]: [wf_id]
> 2016-09-16 12:39:26,723 ERROR SchemaCheckXCommand:517 -
> SERVER[c6402.ambari.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] Database schema is BAD! Check previous error log messages for
> details
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2805) create derbyoozie
[
https://issues.apache.org/jira/browse/OOZIE-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2805:
--
Description:
I guessed there is a bug because it generated "Exception" instead "Error"
message.
I tried to use build Oozie package.
my Hadoop is installed on the remote server.
when I run
{code}
$oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh create
-run
command I encountered below message:
setting
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Validate DB Connection
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/ReflectionUtils
at
org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
at org.apache.oozie.service.Services.(Services.java:111)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:165)
at
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:864)
at
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:872)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:183)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:127)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:78)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.util.ReflectionUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more
{code}
was:
I guessed there is a bug because it generated "Exception" instead "Error"
message.
I tried to use build Oozie package.
my Hadoop is installed on the remote server.
when I run
oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh create
-run
command I encountered below message:
setting
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Validate DB Connection
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/ReflectionUtils
at
org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
at org.apache.oozie.service.Services.(Services.java:111)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:165)
at
org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:864)
at
org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:872)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:183)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:127)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:78)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.util.ReflectionUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more
> create derbyoozie
> -
>
> Key: OOZIE-2805
> URL: https://issues.apache.org/jira/browse/OOZIE-2805
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
> Environment: OS: CentOS 7
> used database is derbydb in oozie-default.xml.reference file
> user hduser and oozie added in Hadoop Hdfs config files.
>Reporter: Hossein Vatani
> Fix For: 4.3.0
>
>
> I guessed there is a bug because it generated "Exception" instead "Error"
> message.
> I tried to use build Oozie package.
> my Hadoop is installed on the remote server.
> when I run
> {code}
> $oozie-4.3.0/distro/target/oozie-4.3.0-distro/oozie-4.3.0/bin/ooziedb.sh
> create -run
> command I encountered below message:
> setting
> JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el7_3.x86_64
> setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
> Validate DB Connection
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/util/ReflectionUtils
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:377)
> at org.apache.oozie.service.Services.(Services.java:111)
> at
[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles
[
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863653#comment-15863653
]
Abhishek Bafna commented on OOZIE-2718:
---
The output of the bundle dryrun prints {{***coordJob after parsing: *** ...
etc}} where as it outputs parsed bundle.
[Here|https://github.com/apache/oozie/blob/master/client/src/main/java/org/apache/oozie/cli/OozieCLI.java#L1005]
Also, add a section like
https://oozie.apache.org/docs/4.3.0/DG_CommandLineTool.html#Dryrun_of_Workflow_Job
for Bundle.
> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch,
> OOZIE-2718-3.patch, OOZIE-2718-4.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
> states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate
> error message for it. It's also possible to add support for bundle dryruns.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer
[
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860660#comment-15860660
]
Abhishek Bafna commented on OOZIE-2788:
---
Thank You @robert kanter and everyone for the review and committing it.
> Fix jobs API servlet mapping for EmbeddedOozieServer
>
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not
> allow Oozie to submit coordinator and bundle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer
[
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859465#comment-15859465
]
Abhishek Bafna commented on OOZIE-2788:
---
[~rohini], [~puru], [~satishsaley] Can someone take a look. Because this is
breaking the oozie master while running oozie with embedded jetty. It fails
while submitting coordinator or bundle due to missing servlet mapping. Thanks.
> Fix jobs API servlet mapping for EmbeddedOozieServer
>
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not
> allow Oozie to submit coordinator and bundle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail
[
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858946#comment-15858946
]
Abhishek Bafna commented on OOZIE-2787:
---
[~satishsaley] It would be nice to document the same for oozie-spark action.
Some thing like "How to specify application path in Oozie-Spark action". Thanks.
> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch,
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch,
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to
> distributed cache. Since this gets added twice, the job fails. This is
> observed from spark 2.1.0 which introduces a check for same file and fails
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters
[
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858227#comment-15858227
]
Abhishek Bafna commented on OOZIE-2791:
---
I tried the Oozie sharelib installation with {{-concurrency}} option with
different number of parallel thread and It was able to install the oozie
sharelib.
Cluster Information: 3 node cluster, build using virtual box within Mac.
The load on the cluster was not much, a bunch of MR jobs were running.
Values tried for number of threads: 50, 150, 250, 350, 450.
Thanks.
> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 1 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r-- 3 asasvari supergroup 0 2017-02-07 10:59
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong.
> In case of such an exception, we should retry copying or rollback changes. We
> should also consider throttling HDFS requests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[ https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857401#comment-15857401 ] Abhishek Bafna commented on OOZIE-2778: --- +1. Thanks [~asasvari] for working on this. Thanks [~andras.piros] for the review. Committed to master. > Copy only jetty.version related server dependencies to distro > -- > > Key: OOZIE-2778 > URL: https://issues.apache.org/jira/browse/OOZIE-2778 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Fix For: 5.0.0 > > Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, > OOZIE-2778-06.patch, OOZIE-2778-07.patch, OOZIE-2778-2.patch, > OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch > > > To compile JSP, org.eclipse.jdt.core is used (see > http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps). > However, additional currently jasper libraries are copied to > embedded-oozie-server/dependency/ in the distro directory. It can cause > problems (e.g. NoSuchMethodException at runtime) if classes are loaded from > that jars. > Furthermore, transitive dependencies can pull in old jetty libraries (such as > jetty-util 6) that can cause runtime issues (at server startup). > We can make sure with the maven assembly plugin that only jetty.version > related dependencies are copied to distro. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2231: -- Summary: Upgrade curator to latest version 2.12.0 (was: Upgrade curator to latest version 2.11.0) > Upgrade curator to latest version 2.12.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Assignee: Abhishek Bafna >Priority: Blocker > Fix For: 5.0.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.11.0
[ https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856377#comment-15856377 ] Abhishek Bafna commented on OOZIE-2231: --- Sure [~rkanter]. I will update the issue title for the same. Thanks. > Upgrade curator to latest version 2.11.0 > > > Key: OOZIE-2231 > URL: https://issues.apache.org/jira/browse/OOZIE-2231 > Project: Oozie > Issue Type: Bug > Components: HA >Reporter: Purshotam Shah >Assignee: Abhishek Bafna >Priority: Blocker > Fix For: 5.0.0 > > Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, > OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch > > > It have some fix related to InterProcessReadWriteLock, ChildReaper, > LeaderSelector which we use. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856304#comment-15856304
]
Abhishek Bafna edited comment on OOZIE-2778 at 2/7/17 4:59 PM:
---
Hi [~asasvari], one of the exclusion is at wrong place. Can you please do
{{mvn:dependency}} output for the server module, please have a look, with the
recent patch.
The exclusion needs to be in {{hadoop-common}} dependency. Thanks.
was (Author: abhishekbafna):
Hi [~asasvari], I think the exclusion is at wrong place. I am attaching a
mvn:dependency output for the server module, please have a look. This is after
the recent patch.
The exclusion needs to be in {{hadoop-common}} dependency. Thanks.
> Copy only jetty.version related server dependencies to distro
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch,
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch,
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to
> embedded-oozie-server/dependency/ in the distro directory. It can cause
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from
> that jars.
> Furthermore, transitive dependencies can pull in old jetty libraries (such as
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version
> related dependencies are copied to distro.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856304#comment-15856304
]
Abhishek Bafna edited comment on OOZIE-2778 at 2/7/17 4:57 PM:
---
Hi [~asasvari], I think the exclusion is at wrong place. I am attaching a
mvn:dependency output for the server module, please have a look. This is after
the recent patch.
The exclusion needs to be in {{hadoop-common}} dependency. Thanks.
was (Author: abhishekbafna):
+1.
> Copy only jetty.version related server dependencies to distro
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch,
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch,
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to
> embedded-oozie-server/dependency/ in the distro directory. It can cause
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from
> that jars.
> Furthermore, transitive dependencies can pull in old jetty libraries (such as
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version
> related dependencies are copied to distro.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[ https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856304#comment-15856304 ] Abhishek Bafna commented on OOZIE-2778: --- +1. > Copy only jetty.version related server dependencies to distro > -- > > Key: OOZIE-2778 > URL: https://issues.apache.org/jira/browse/OOZIE-2778 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, > OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, > OOZIE-2778-4.patch, OOZIE-2778-5.patch > > > To compile JSP, org.eclipse.jdt.core is used (see > http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps). > However, additional currently jasper libraries are copied to > embedded-oozie-server/dependency/ in the distro directory. It can cause > problems (e.g. NoSuchMethodException at runtime) if classes are loaded from > that jars. > Furthermore, transitive dependencies can pull in old jetty libraries (such as > jetty-util 6) that can cause runtime issues (at server startup). > We can make sure with the maven assembly plugin that only jetty.version > related dependencies are copied to distro. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[ https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855905#comment-15855905 ] Abhishek Bafna commented on OOZIE-2778: --- [~asasvari] I am afraid the recent patch will also fail to apply because it contains the same diff twice like the earlier one. There are no new changes compared to the last one. I think you need to check the last failed UTs locally for the patch [^OOZIE-2778-05.patch]. Thanks. > Copy only jetty.version related server dependencies to distro > -- > > Key: OOZIE-2778 > URL: https://issues.apache.org/jira/browse/OOZIE-2778 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, > OOZIE-2778-2.patch, OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch > > > To compile JSP, org.eclipse.jdt.core is used (see > http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps). > However, additional currently jasper libraries are copied to > embedded-oozie-server/dependency/ in the distro directory. It can cause > problems (e.g. NoSuchMethodException at runtime) if classes are loaded from > that jars. > Furthermore, transitive dependencies can pull in old jetty libraries (such as > jetty-util 6) that can cause runtime issues (at server startup). > We can make sure with the maven assembly plugin that only jetty.version > related dependencies are copied to distro. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2041:
--
Attachment: OOZIE-2041-09.patch
Rebase last (08) patch.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch,
> OOZIE-2041-08.patch, OOZIE-2041-09.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer
[
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852836#comment-15852836
]
Abhishek Bafna commented on OOZIE-2788:
---
Testcases passes locally. Thanks.
[~rkanter] can you please review this?
> Fix jobs API servlet mapping for EmbeddedOozieServer
>
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not
> allow Oozie to submit coordinator and bundle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2789) Maven complains about checkstyle error during build
[ https://issues.apache.org/jira/browse/OOZIE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852669#comment-15852669 ] Abhishek Bafna commented on OOZIE-2789: --- Thanks [~zhengxb2005] for the patch. Committed to master. > Maven complains about checkstyle error during build > --- > > Key: OOZIE-2789 > URL: https://issues.apache.org/jira/browse/OOZIE-2789 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: trunk >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Fix For: 5.0.0 > > Attachments: OOZIE-2789-1.patch > > > While working on this for the 1st time. My maven does not like the style of > some files and I would get 'There are 2 checkstyle errors' and build failed. > My workaround is to use 'mvn xxx -Dcheckstyle.skip'. However I think it is > still nice to fix it in case someone else is also facing this. > $ mvn -v > Apache Maven 3.3.9 > OS name: "mac os x", version: "10.11.1", arch: "x86_64", family: "mac" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2789) Maven complains about checkstyle error during build
[ https://issues.apache.org/jira/browse/OOZIE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852667#comment-15852667 ] Abhishek Bafna commented on OOZIE-2789: --- +1. > Maven complains about checkstyle error during build > --- > > Key: OOZIE-2789 > URL: https://issues.apache.org/jira/browse/OOZIE-2789 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: trunk >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Attachments: OOZIE-2789-1.patch > > > While working on this for the 1st time. My maven does not like the style of > some files and I would get 'There are 2 checkstyle errors' and build failed. > My workaround is to use 'mvn xxx -Dcheckstyle.skip'. However I think it is > still nice to fix it in case someone else is also facing this. > $ mvn -v > Apache Maven 3.3.9 > OS name: "mac os x", version: "10.11.1", arch: "x86_64", family: "mac" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852653#comment-15852653
]
Abhishek Bafna edited comment on OOZIE-2778 at 2/4/17 7:08 AM:
---
Thanks [~asasvari] for working on this.
Though I still faced some issue and adding exclusion of the {{jsp-api}} helped
me. Please verify at your end and update the patch as required.
Pasting a {{git diff}} of the additional changes on top of the current patch
(01).
{noformat}
iff --git a/server/pom.xml b/server/pom.xml
index a336aa8..dae6594 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -89,6 +89,10 @@
org.mortbay.jetty
jetty-util
+
+javax.servlet.jsp
+jsp-api
+
@@ -125,6 +129,10 @@
org.eclipse.jetty.aggregate
jetty-all
+
+javax.servlet.jsp
+jsp-api
+
{noformat}
was (Author: abhishekbafna):
Thanks [~asasvari] for working on this.
Though I still faced some issue and addeingexclusion of the {{jsp-api}} helped
me. Please verify at your end and update the patch as required. Pasting a {{git
diff}} of the additional changes on top of the current patch.
{noformat}
iff --git a/server/pom.xml b/server/pom.xml
index a336aa8..dae6594 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -89,6 +89,10 @@
org.mortbay.jetty
jetty-util
+
+javax.servlet.jsp
+jsp-api
+
@@ -125,6 +129,10 @@
org.eclipse.jetty.aggregate
jetty-all
+
+javax.servlet.jsp
+jsp-api
+
{noformat}
> Copy only jetty.version related server dependencies to distro
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to
> embedded-oozie-server/dependency/ in the distro directory. It can cause
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from
> that jars.
> Furthermore, transitive dependencies can pull in old jetty libraries (such as
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version
> related dependencies are copied to distro.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer
[
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2788:
--
Attachment: OOZIE-2788-00.patch
> Fix jobs API servlet mapping for EmbeddedOozieServer
>
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not
> allow Oozie to submit coordinator and bundle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro
[
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852653#comment-15852653
]
Abhishek Bafna commented on OOZIE-2778:
---
Thanks [~asasvari] for working on this.
Though I still faced some issue and addeingexclusion of the {{jsp-api}} helped
me. Please verify at your end and update the patch as required. Pasting a {{git
diff}} of the additional changes on top of the current patch.
{noformat}
iff --git a/server/pom.xml b/server/pom.xml
index a336aa8..dae6594 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -89,6 +89,10 @@
org.mortbay.jetty
jetty-util
+
+javax.servlet.jsp
+jsp-api
+
@@ -125,6 +129,10 @@
org.eclipse.jetty.aggregate
jetty-all
+
+javax.servlet.jsp
+jsp-api
+
{noformat}
> Copy only jetty.version related server dependencies to distro
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to
> embedded-oozie-server/dependency/ in the distro directory. It can cause
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from
> that jars.
> Furthermore, transitive dependencies can pull in old jetty libraries (such as
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version
> related dependencies are copied to distro.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer
Abhishek Bafna created OOZIE-2788:
-
Summary: Fix jobs API servlet mapping for EmbeddedOozieServer
Key: OOZIE-2788
URL: https://issues.apache.org/jira/browse/OOZIE-2788
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 5.0.0
Reporter: Abhishek Bafna
Assignee: Abhishek Bafna
Priority: Blocker
Fix For: 5.0.0
The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not
allow Oozie to submit coordinator and bundle.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2771) Allow retrieving keystore and truststore passwords from Hadoop Credential Provider
[
https://issues.apache.org/jira/browse/OOZIE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837295#comment-15837295
]
Abhishek Bafna commented on OOZIE-2771:
---
Thanks [~asasvari] for the patch and [~gezapeti] [~andras.piros] for the review.
Committed to master.
> Allow retrieving keystore and truststore passwords from Hadoop Credential
> Provider
> --
>
> Key: OOZIE-2771
> URL: https://issues.apache.org/jira/browse/OOZIE-2771
> Project: Oozie
> Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2771-01.patch
>
>
> Right now passwords for keystore and truststore
> ({{oozie.https.keystore.pass}}, {{oozie.https.truststore.pass}}) are stored
> in {{oozie-site.xml}} as cleartext.
> However, Oozie could take advantage of the Hadoop Credential Provider for
> storing and retrieving that passwords similarly how the JDBC password
> ({{oozie.service.JPAService.jdbc.password}}) is handled today (see
> https://issues.apache.org/jira/browse/OOZIE-2272).
> This way keystore and truststore passwords could be masked in oozie-site.
> Note: {{ConfigurationService.getPassword}} is worth to look at.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2771) Allow retrieving keystore and truststore passwords from Hadoop Credential Provider
[
https://issues.apache.org/jira/browse/OOZIE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837293#comment-15837293
]
Abhishek Bafna commented on OOZIE-2771:
---
+1.
> Allow retrieving keystore and truststore passwords from Hadoop Credential
> Provider
> --
>
> Key: OOZIE-2771
> URL: https://issues.apache.org/jira/browse/OOZIE-2771
> Project: Oozie
> Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2771-01.patch
>
>
> Right now passwords for keystore and truststore
> ({{oozie.https.keystore.pass}}, {{oozie.https.truststore.pass}}) are stored
> in {{oozie-site.xml}} as cleartext.
> However, Oozie could take advantage of the Hadoop Credential Provider for
> storing and retrieving that passwords similarly how the JDBC password
> ({{oozie.service.JPAService.jdbc.password}}) is handled today (see
> https://issues.apache.org/jira/browse/OOZIE-2272).
> This way keystore and truststore passwords could be masked in oozie-site.
> Note: {{ConfigurationService.getPassword}} is worth to look at.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2619) Make Hive action defaults to match hive defaults when running from command line
[ https://issues.apache.org/jira/browse/OOZIE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837284#comment-15837284 ] Abhishek Bafna commented on OOZIE-2619: --- Thanks [~venkatnrangan] for the patch. Committed to master. > Make Hive action defaults to match hive defaults when running from command > line > > > Key: OOZIE-2619 > URL: https://issues.apache.org/jira/browse/OOZIE-2619 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 3.3.0, 4.2.0 >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Fix For: 5.0.0 > > Attachments: OOZIE-2619.patch, OOZIE-2619.patch.2 > > > Over a few patches, we have done a few fixes to make Oozie Hive actions > easier for users. > One of them was OOZIE-2051 which allows default hive and tez site xml configs > to be added to hive actions automatically by introducing action specific > configuration directory under oozie conf/action-conf directory and as a bonus > in an Ambari managed cluster the hive site changes done as part of the Hive > components are automatically reflected into the oozie hive action defaults. > But there is one issue pending for Oozie hive actions. > Oozie Hive jobs launched via hive action are historically restricted to one > reducer by default (and also there are few other in terms of split sizes > etc). Thisvis because of the way Oozie action config management is done and > how Hive was determining the reducers. Hive uses mapreduce.job.reduces to > determine if the reducers have to be dynamically determined (when this > parameter is initialized to an invalid value -1) or explicitly determined by > the users. In HiveConf, this is internally set to -1 if not in > hive-site.xml or in one of the set statements. > Oozie, when it prepares the action configuration, has the > mapreduce.job.reduces set to 1 (from mapred-default). As part of the hive > action, Oozie writes the action configuration prepared (the action.xml) also > as hive-site.xml with the value for mapreduce.job.reduces set to 1. > There are a few ways to overcome this issue, true to Oozie being very > flexible with lots of options :). I may be missing a few other > options here! > # Explicitly set the mapreduce.job.reduces parameter in the configuration > element of the action > Every hive workflow configuration has be changed > # Add the parameter to a job-xml for the action > Once again affects all actions > # Set the parameter to disable loading of the default *-site.xml > files as provided by OOZIE-2205 >We need to make sure that the *-site.xml are otherwise available to the > containers - either have hadoop conf directory (typically /etc/hadoop/conf) > in the mapred framework classpath or explicitly make the files using other > mechanisms available (as files, archives, in sharelib ec). The big issue is > that this affects rolling upgrades once you add explicit config dependency > Unfortunately we can't use the default action config addition introduced in > OOZIE-2051 for adding one more configuration file to the oozie hive action > conf directory with hive MR defaults. > The config files under the action-conf/hive/*.xml or action-conf/hive.xml are > all merged using the method injectDefaults which only updates the target only > if it does not exist in the target configuration map. In our case, > mapreduce.job.reduces already exsits in the action default configuration > (coming from mapred-default.xml) and hence does not get overwritten from the > action-conf/hive configuration files. > The fix (essentially one line of code change) is to use the copy method of > XConfiguration to copy the action-default config instead of using the > injectDefaults method and then provide the action-default/hive.xm with the > required mapred hive parameters with hive expected initial values. > This patch introduces a change that has potential backward compatibility > issues. > * If the action-conf/.xml currently has entries that were no-ops so > far, they can be added to the action configuration. > * Hive will work as expected when run as an Oozie action without users > needing to resort to changes! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2619) Make Hive action defaults to match hive defaults when running from command line
[ https://issues.apache.org/jira/browse/OOZIE-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837281#comment-15837281 ] Abhishek Bafna commented on OOZIE-2619: --- +1. testcase passes locally. > Make Hive action defaults to match hive defaults when running from command > line > > > Key: OOZIE-2619 > URL: https://issues.apache.org/jira/browse/OOZIE-2619 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 3.3.0, 4.2.0 >Reporter: Venkat Ranganathan >Assignee: Venkat Ranganathan > Attachments: OOZIE-2619.patch, OOZIE-2619.patch.2 > > > Over a few patches, we have done a few fixes to make Oozie Hive actions > easier for users. > One of them was OOZIE-2051 which allows default hive and tez site xml configs > to be added to hive actions automatically by introducing action specific > configuration directory under oozie conf/action-conf directory and as a bonus > in an Ambari managed cluster the hive site changes done as part of the Hive > components are automatically reflected into the oozie hive action defaults. > But there is one issue pending for Oozie hive actions. > Oozie Hive jobs launched via hive action are historically restricted to one > reducer by default (and also there are few other in terms of split sizes > etc). Thisvis because of the way Oozie action config management is done and > how Hive was determining the reducers. Hive uses mapreduce.job.reduces to > determine if the reducers have to be dynamically determined (when this > parameter is initialized to an invalid value -1) or explicitly determined by > the users. In HiveConf, this is internally set to -1 if not in > hive-site.xml or in one of the set statements. > Oozie, when it prepares the action configuration, has the > mapreduce.job.reduces set to 1 (from mapred-default). As part of the hive > action, Oozie writes the action configuration prepared (the action.xml) also > as hive-site.xml with the value for mapreduce.job.reduces set to 1. > There are a few ways to overcome this issue, true to Oozie being very > flexible with lots of options :). I may be missing a few other > options here! > # Explicitly set the mapreduce.job.reduces parameter in the configuration > element of the action > Every hive workflow configuration has be changed > # Add the parameter to a job-xml for the action > Once again affects all actions > # Set the parameter to disable loading of the default *-site.xml > files as provided by OOZIE-2205 >We need to make sure that the *-site.xml are otherwise available to the > containers - either have hadoop conf directory (typically /etc/hadoop/conf) > in the mapred framework classpath or explicitly make the files using other > mechanisms available (as files, archives, in sharelib ec). The big issue is > that this affects rolling upgrades once you add explicit config dependency > Unfortunately we can't use the default action config addition introduced in > OOZIE-2051 for adding one more configuration file to the oozie hive action > conf directory with hive MR defaults. > The config files under the action-conf/hive/*.xml or action-conf/hive.xml are > all merged using the method injectDefaults which only updates the target only > if it does not exist in the target configuration map. In our case, > mapreduce.job.reduces already exsits in the action default configuration > (coming from mapred-default.xml) and hence does not get overwritten from the > action-conf/hive configuration files. > The fix (essentially one line of code change) is to use the copy method of > XConfiguration to copy the action-default config instead of using the > injectDefaults method and then provide the action-default/hive.xm with the > required mapred hive parameters with hive expected initial values. > This patch introduces a change that has potential backward compatibility > issues. > * If the action-conf/.xml currently has entries that were no-ops so > far, they can be added to the action configuration. > * Hive will work as expected when run as an Oozie action without users > needing to resort to changes! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837221#comment-15837221
]
Abhishek Bafna commented on OOZIE-2041:
---
[~rohini] Can you please take a look? Thanks.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch,
> OOZIE-2041-08.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834696#comment-15834696
]
Abhishek Bafna edited comment on OOZIE-2041 at 1/23/17 3:43 PM:
Thanks [~asasvari] for looking into the patch.
For the typo, It should be {{"bundle records (bundle=7) older than 7 days."}},
which was correct in earlier patches I think.
For the validation, I will create a ValidationUtil class into oozie-client
module, oozie-core have access to it and then will reuse the same. This class
can be used in future for adding more validation checks.
Thanks.
was (Author: abhishekbafna):
Thanks [~asasvari] for looking into the patch.
For the typo, It should be {{"bundle records (bundle=6) older than 7 days."}},
which was correct in earlier patches I think.
For the validation, I will create a ValidationUtil class into oozie-client
module, oozie-core have access to it and then will reuse the same. This class
can be used in future for adding more validation checks.
Thanks.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834775#comment-15834775
]
Abhishek Bafna commented on OOZIE-2041:
---
[~gezapeti] :) I got your point and typo. Thanks a lot.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834696#comment-15834696
]
Abhishek Bafna commented on OOZIE-2041:
---
Thanks [~asasvari] for looking into the patch.
For the typo, It should be {{"bundle records (bundle=6) older than 7 days."}},
which was correct in earlier patches I think.
For the validation, I will create a ValidationUtil class into oozie-client
module, oozie-core have access to it and then will reuse the same. This class
can be used in future for adding more validation checks.
Thanks.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2779) Mask Hive2 action Beeline JDBC password
[
https://issues.apache.org/jira/browse/OOZIE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2779:
--
Attachment: OOZIE-2779-00.patch
> Mask Hive2 action Beeline JDBC password
> ---
>
> Key: OOZIE-2779
> URL: https://issues.apache.org/jira/browse/OOZIE-2779
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Labels: security
> Fix For: 5.0.0
>
> Attachments: OOZIE-2779-00.patch
>
>
> Hive2 Oozie launcher job prints the JDBC password into launcher stdout logs.
> {noformat}
> Beeline command arguments :
> -u
> jdbc:hive2://source-1:1/default
> -n
> ambari-qa
> -p
> DUMMY
> -d
> org.apache.hive.jdbc.HiveDriver
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OOZIE-2779) Mask Hive2 action Beeline JDBC password
Abhishek Bafna created OOZIE-2779:
-
Summary: Mask Hive2 action Beeline JDBC password
Key: OOZIE-2779
URL: https://issues.apache.org/jira/browse/OOZIE-2779
Project: Oozie
Issue Type: Bug
Components: action
Reporter: Abhishek Bafna
Assignee: Abhishek Bafna
Fix For: 5.0.0
Hive2 Oozie launcher job prints the JDBC password into launcher stdout logs.
{noformat}
Beeline command arguments :
-u
jdbc:hive2://source-1:1/default
-n
ambari-qa
-p
DUMMY
-d
org.apache.hive.jdbc.HiveDriver
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2768) Oozie workflow goes into suspended state with JA009: Output directory already exists
[ https://issues.apache.org/jira/browse/OOZIE-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2768: -- Assignee: (was: Abhishek Bafna) > Oozie workflow goes into suspended state with JA009: Output directory > already exists > - > > Key: OOZIE-2768 > URL: https://issues.apache.org/jira/browse/OOZIE-2768 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.2.0 >Reporter: sandeep samudrala > > At times oozie workflow action fails with following error. > JA009: Output directory > /user/test/oozie-oozi/0397117-160802162641301-oozie-oozi-W/action--java/output > already exists -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2041:
--
Attachment: OOZIE-2041-07.patch
Thanks [~asasvari] for the feedback.
bq. Typo in the documentation: "bundle records (bundle=6) older than 7 days."
I am not able to catch the typo in here. Can you please help by pointing out.
bq. In {{OozieClient}}, you could add some parameter validation like this
The validation is added into the {{BaseAdminServlet}} so that it can work
informally with CLI and REST. I think it will be redundant to add in the
{{OozieClient}}. Do let me know your thoughts.
You mentioned about, playing around with CLI, If there is anything do post it
here. Thanks.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch, OOZIE-2041-07.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821258#comment-15821258 ] Abhishek Bafna commented on OOZIE-2701: --- Thanks [~andras.piros] and [~gezapeti] for the feedback. > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2041) Add an admin command to run the PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Bafna updated OOZIE-2041:
--
Attachment: OOZIE-2041-06.patch
[~gezapeti], [~andras.piros] and [~asasvari] Thanks for the feedback. Please
have a look into the updated patch. Thanks.
[~gezapeti] There is no change into {{setOozieMode}}, just a refactoring as we
got one more 'admin' command in the {{doPut}}.
> Add an admin command to run the PurgeXCommand
> -
>
> Key: OOZIE-2041
> URL: https://issues.apache.org/jira/browse/OOZIE-2041
> Project: Oozie
> Issue Type: New Feature
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2041-00.patch, OOZIE-2041-01.patch,
> OOZIE-2041-02.patch, OOZIE-2041-03.patch, OOZIE-2041-04.patch,
> OOZIE-2041-05.patch, OOZIE-2041-06.patch
>
>
> Some users may find it useful to be able to run the PurgeXCommand on-demand.
> We can add a new admin command to run this. e.g.
> {noformat}
> oozie admin -purge
> {noformat}
> With no args, it can use the "older than" values from oozie-site, but we
> could have it take some arguments to override those for the command.
> Another thing to worry about is making sure that we don't run this if either
> it's already running (i.e. the user sent it twice) or if the scheduled
> PurgeService is already running it. On top of that, we may need extra
> consideration for HA setups where we currently only run it on the leader.
> We should probably also have a to not schedule the PurgeService in ooize-site
> for users who only want to run it manually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs
[ https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2701: -- Attachment: OOZIE-2701-02.patch > Oozie to support Multiple HCatalog URIs > --- > > Key: OOZIE-2701 > URL: https://issues.apache.org/jira/browse/OOZIE-2701 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna > Fix For: 5.0.0 > > Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, > OOZIE-2701-02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2540) Create a PySpark example
[
https://issues.apache.org/jira/browse/OOZIE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818399#comment-15818399
]
Abhishek Bafna commented on OOZIE-2540:
---
Thank you [~andras.piros] and [~gezapeti] for the review.
Patch adds example for pyspark into {{examples}} module. Testcase failure is
not relevant to the patch.
[~satishsaley] Can you please take a look? Thanks.
> Create a PySpark example
>
>
> Key: OOZIE-2540
> URL: https://issues.apache.org/jira/browse/OOZIE-2540
> Project: Oozie
> Issue Type: Task
> Components: examples
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2540-00.patch
>
>
> Now that we have PySpark working correctly in the Spark Action, we should
> make an example that runs a PySpark job to give users an example of how to do
> it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2540) Create a PySpark example
[ https://issues.apache.org/jira/browse/OOZIE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated OOZIE-2540: -- Attachment: OOZIE-2540-00.patch > Create a PySpark example > > > Key: OOZIE-2540 > URL: https://issues.apache.org/jira/browse/OOZIE-2540 > Project: Oozie > Issue Type: Task > Components: examples >Affects Versions: trunk >Reporter: Robert Kanter >Assignee: Abhishek Bafna > Attachments: OOZIE-2540-00.patch > > > Now that we have PySpark working correctly in the Spark Action, we should > make an example that runs a PySpark job to give users an example of how to do > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (OOZIE-2540) Create a PySpark example
[ https://issues.apache.org/jira/browse/OOZIE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna reassigned OOZIE-2540: - Assignee: Abhishek Bafna > Create a PySpark example > > > Key: OOZIE-2540 > URL: https://issues.apache.org/jira/browse/OOZIE-2540 > Project: Oozie > Issue Type: Task > Components: examples >Affects Versions: trunk >Reporter: Robert Kanter >Assignee: Abhishek Bafna > > Now that we have PySpark working correctly in the Spark Action, we should > make an example that runs a PySpark job to give users an example of how to do > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2753) Update Docs for Jetty Tomcat changes
[
https://issues.apache.org/jira/browse/OOZIE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817485#comment-15817485
]
Abhishek Bafna commented on OOZIE-2753:
---
Can you please rebase the patch. Some minor things:
* remove {{-Dtomcat.version= - default 6.0.47}} from QuickStart and
Building twiki.
* update {{@rem Settings for the Embedded Tomcat that runs Oozie}} into
oozie-env.cmd file.
> Update Docs for Jetty Tomcat changes
>
>
> Key: OOZIE-2753
> URL: https://issues.apache.org/jira/browse/OOZIE-2753
> Project: Oozie
> Issue Type: Sub-task
> Components: docs
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2753-01.patch
>
>
> Update the docs based on the changes for Jetty and Tomcat (OOZIE-2317 and
> child JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2732) Remove login server example
[
https://issues.apache.org/jira/browse/OOZIE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817374#comment-15817374
]
Abhishek Bafna commented on OOZIE-2732:
---
+1. Committed to master.
Thanks [~rkanter] for the patch and [~andras.piros] for the review.
> Remove login server example
> ---
>
> Key: OOZIE-2732
> URL: https://issues.apache.org/jira/browse/OOZIE-2732
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 5.0.0
>
> Attachments: OOZIE-2732.001.patch, OOZIE-2732.002.patch
>
>
> 4 years ago, OOZIE-1103 added a "login" module which provided two
> (non-secure) examples of hadoop-auth's {{AltKerberosAuthenticationHandler}}.
> This module is built only when the "login" profile is activated. I think
> it's time to remove this because:
> - it's super old and not maintained
> - it's not widely used AFAIK (if at all)
> - it doesn't really belong in Oozie; it's an example for a hadoop-auth feature
> - it's not secure, so it's not a great example anyway
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2681) fix javadoc to compile on JDK8 again
[ https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817312#comment-15817312 ] Abhishek Bafna commented on OOZIE-2681: --- Thanks [~andras.piros] for the review on RB. [~satishsaley] can you please take look? Thanks. > fix javadoc to compile on JDK8 again > > > Key: OOZIE-2681 > URL: https://issues.apache.org/jira/browse/OOZIE-2681 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 4.3.0 >Reporter: Robert Kanter >Assignee: Abhishek Bafna >Priority: Critical > Fix For: 5.0.0 > > Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch, > OOZIE-2681-02.patch > > > We have a lot of Javadoc build failures, which causes Oozie to fail to build > on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all > of these that existed at that time, but we've since introduced a lot more. > OOZIE-2487 made a workaround to prevent this from causing the build to fail, > but we should fix these (again) and remove the OOZIE-2487 workaround. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
