[jira] [Created] (OOZIE-3451) Example in Git action's documentation refers to incorrect worflow version
Attila Sasvari created OOZIE-3451:
-
Summary: Example in Git action's documentation refers to incorrect
worflow version
Key: OOZIE-3451
URL: https://issues.apache.org/jira/browse/OOZIE-3451
Project: Oozie
Issue Type: Bug
Components: docs
Reporter: Attila Sasvari
https://oozie.apache.org/docs/5.1.0/DG_GitActionExtension.html uses an example
with
{{}}
That is incorrect as per:
https://github.com/apache/oozie/blob/master/client/src/main/resources/git-action-1.0.xsd
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3395) Findbugs is no longer maintained
Attila Sasvari created OOZIE-3395:
-
Summary: Findbugs is no longer maintained
Key: OOZIE-3395
URL: https://issues.apache.org/jira/browse/OOZIE-3395
Project: Oozie
Issue Type: Task
Reporter: Attila Sasvari
https://gleclaire.github.io/findbugs-maven-plugin/
{quote}
Status: Since Findbugs is no longer maintained, please use Spotbugs which has a
Maven plugin.
{quote}
The plugin author recommends to migrate to Spotbugs: https://spotbugs.github.io/
It might worth to investigate this plugin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
Attila Sasvari created OOZIE-3390:
-
Summary: shell action's stderr contains a bogus error message
(since 5.0.0)
Key: OOZIE-3390
URL: https://issues.apache.org/jira/browse/OOZIE-3390
Project: Oozie
Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari
Shell action's stderr contains an error message about path.
I executed shell action example to test Oozie 5.1 rc1, the job finished with
success, but stderr of the job contained "Path echo doesn't appear to exist".
https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
{code}
writeLine(errorStream, "Path " + filename + " doesn't appear to
exist");
{code}
The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3389) NPE after OOZIE-3370
Attila Sasvari created OOZIE-3389:
-
Summary: NPE after OOZIE-3370
Key: OOZIE-3389
URL: https://issues.apache.org/jira/browse/OOZIE-3389
Project: Oozie
Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari
I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
clicked on one of the coordinator jobs on the Oozie Web UI, but the details of
the job were not displayed. I noticed the following NPE in the server log:
{code}
2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] XException,
org.apache.oozie.command.CommandException: E1028: Coord input logic error. null
at
org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
at
org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
at org.apache.oozie.command.XCommand.call(XCommand.java:291)
at
org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
at
org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3386) Misleading error message when workflow application does not exist
Attila Sasvari created OOZIE-3386:
-
Summary: Misleading error message when workflow application does
not exist
Key: OOZIE-3386
URL: https://issues.apache.org/jira/browse/OOZIE-3386
Project: Oozie
Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari
Using 5.1.0 rc1, I tried to run an example workflow. Because of user error
{{oozie.wf.application.path}} in job.properties pointed to a directory in HDFS
that did not exist. Upon submitting the workflow, the following was returned
{code}
bin/oozie job -oozie http://localhost:11000/oozie -config
examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run
-DnameNode=hdfs://localhost:9000
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Error: E0307 : E0307: Runtime error [Configuration entry
oozie.jobs.api.generated.xml not present]
{code}
The server.log contained the following:
{code}
2018-11-19 15:07:13,244 WARN V1JobsServlet:523 -
SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] URL[POST
http://localhost:11000/oozie/v2/jobs?action=start=asasvari] error[E0307],
E0307: Runtime error [Configuration entry oozie.jobs.api.generated.xml not
present]
org.apache.oozie.servlet.XServletException: E0307: Runtime error [Configuration
entry oozie.jobs.api.generated.xml not present]
at
org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
at
org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at
[jira] [Commented] (OOZIE-3354) [core] [SSH action] SSH action gets hung
[
https://issues.apache.org/jira/browse/OOZIE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631560#comment-16631560
]
Attila Sasvari commented on OOZIE-3354:
---
[~andras.piros] thanks for the patch! I have run into an issue when an SSH
action with {{}} got hung after an Oozie upgrade.
Can we add some sleep (say 100ms) when an IllegalThreadStateException exception
is thrown to reduce busy-waiting?
> [core] [SSH action] SSH action gets hung
>
>
> Key: OOZIE-3354
> URL: https://issues.apache.org/jira/browse/OOZIE-3354
> Project: Oozie
> Issue Type: Bug
> Components: action, core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3354.001.patch
>
>
> In OOZIE-3183 {{SshActionExecutor#drainBuffers()}} has changed. Previously,
> it called {{Process#exitCode()}} that would return immediately either with
> the exit code, or would throw an {{IllegalThreadStateException}} if the
> process would still be running.
> In the current implementation introduced by OOZIE-3183, {{Process#waitFor()}}
> is used that would block until the process finishes. Given the fact that
> sometime {{SshActionExecutor#check()}} calls {{ssh ... cat stdout}}, and this
> SSH process can be trapped even after {{cat stdout}} has been finished on the
> target host, it can happen that {{SshActionExecutor#drainBuffers()}} waits
> indefinitely without a chance to gather any {{stdout}} or {{stderr}} logs.
> Hence this particular one is a compatibility breaking change with existing
> SSH action behavior.
> Let's re-introduce the former behavior in
> {{SshActionExecutor#drainBuffers()}} that keeps polling
> {{Process#exitValue()}} and reading the progress on {{stdout}} and {{stderr}}
> till the process finishes, for backwards compatibility.
> [This
> article|https://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html]
> covers the behavioral details of {{Process#waitFor()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3210) revision information is empty
[
https://issues.apache.org/jira/browse/OOZIE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593345#comment-16593345
]
Attila Sasvari commented on OOZIE-3210:
---
[~dbist13] how did you build Oozie? Did you run {{bin/mkdistro.sh}}?
> revision information is empty
> -
>
> Key: OOZIE-3210
> URL: https://issues.apache.org/jira/browse/OOZIE-3210
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297;
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.17.1.el7.x86_64", arch: "amd64",
> family: "unix"{code}
>Reporter: Artem Ervits
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.1.0
>
>
> compiled from RC0
> {code:java}
> STARTUP MSG: Oozie BUILD_VERSION [5.0.0] compiled by [oozie] on
> [2018.04.02-19:55:20GMT]
> STARTUP MSG: revision [unavailable]@[unavailable]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3319) Log SSH action callback error output
[
https://issues.apache.org/jira/browse/OOZIE-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574564#comment-16574564
]
Attila Sasvari commented on OOZIE-3319:
---
[~ericlin] thanks for the patch!
Remarks:
- If we redirect the standard output of the callback command to stdout, it may
interfere with the actual command's standard output; is not desirable. See
https://github.com/apache/oozie/blob/e229e4dbb42b7bc0fdb866ccc530b533a2b2008b/core/src/main/resources/ssh-wrapper.sh#L41
{code}
if $cmnd "$@" >>${stdout} 2>>${stderr}; then
{code}
- I checked the new wrapper: upon a successful execution, the stderr file will
contain
{code}
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
^M 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0^M100 50 0 100 5 0691 --:--:-- --:--:-- --:--:--
714
{code}
- Upon failure:
{code}
Not able to perform callback operation.
Callback command: curl
http://localhost:21000/oozie/callback?id=000-180809004852667-oozie-oozi-W@ssh-6185=OK
--data-binary
@/root/oozie-oozi/000-180809004852667-oozie-oozi-W.bak/ssh-6185--ssh/20907.000-180809004852667-oozie-oozi-W@ssh-6185@0.stdout
--request POST --header content-type:text/plain
{code}
I would suggest to only log the callback command in the failure case, and do
not redirect stdout.
- Did you consider to add a new test case to TestSshActionExecutor with an
invalid callback URL?
> Log SSH action callback error output
>
>
> Key: OOZIE-3319
> URL: https://issues.apache.org/jira/browse/OOZIE-3319
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Minor
> Attachments: OOZIE-3319.patch
>
>
> When SSH callback fails, there is no error message logged to tell us what was
> going on
> {code}
> if $callback; then
> echo "Callback operation successful."
> else
> echo "Not able to perform callback operation." >>${stderr}
> fi
> {code}
> The only message showed in the remote machine's log was "Not able to perform
> callback operation." which is useless.
> Request to log the stderr out of the $callback action to stderr log as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OOZIE-3086) Oozie shell action ignores absolute path
[
https://issues.apache.org/jira/browse/OOZIE-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari resolved OOZIE-3086.
---
Resolution: Duplicate
> Oozie shell action ignores absolute path
>
>
> Key: OOZIE-3086
> URL: https://issues.apache.org/jira/browse/OOZIE-3086
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 3.1.3, 3.2.0, 3.3.0, trunk, 3.3.1, 3.3.2, 4.0.0, 4.1.0,
> 4.0.1, 4.2.0, 4.3.0
>Reporter: Attila Sasvari
>Priority: Major
>
> [ShellActionExecutor's setupActionConf() method|
> https://github.com/apache/oozie/blob/883c7556d2f302261eae5aec6e323a5b1de04f74/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java#L57]
> retrieves only the *basename* for the shell script specified by the user.
> {code}
> Configuration setupActionConf(Configuration actionConf, Context context,
> Element actionXml, Path appPath)
> {
> ...
> String exec = actionXml.getChild("exec", ns).getTextTrim();
> String execName = new Path(exec).getName();
> actionConf.set(ShellMain.CONF_OOZIE_SHELL_EXEC, execName);
> {code}
> As a result, {{ShellMain}} will
> [execute|https://github.com/apache/oozie/blob/883c7556d2f302261eae5aec6e323a5b1de04f74/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java#L356]
> the default command available in $PATH.
> {code}
> protected String getExec(Configuration actionConf) {
> String exec = actionConf.get(CONF_OOZIE_SHELL_EXEC);
> {code}
> Consequences:
> - This behaviour makes it also hard/impossible to use custom executable at a
> given location (that is available on each node where a shell action can run,
> for example a specific python version).
> - Even if a user specifies a wrong path (e.g. a non-existing {{/usr/bn/id}})
> on purpose to force failure of a shell action, the command might succeed with
> no apparent reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3320) Oozie ShellAction should support absolute bash file path
[
https://issues.apache.org/jira/browse/OOZIE-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574357#comment-16574357
]
Attila Sasvari commented on OOZIE-3320:
---
Thanks [~roeyshemtov24]. It duplicates
[OOZIE-3086|https://issues.apache.org/jira/browse/OOZIE-3086].
> Oozie ShellAction should support absolute bash file path
>
>
> Key: OOZIE-3320
> URL: https://issues.apache.org/jira/browse/OOZIE-3320
> Project: Oozie
> Issue Type: Improvement
> Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Roey Shem Tov
>Assignee: Roey Shem Tov
>Priority: Major
> Labels: patch
> Attachments: OOZIE-3086.patch
>
>
> bash files that saved on shared mount, cannot be execute by the ShellAction
> in proper way.
> Example:
> Worker-1,Worker-2,Worker-3 have shared mount /mnt/hadoop
> on /mnt/hadoop there is a file script.sh
> Right now there is two options to submit it using ShellAction:
> # Upload it to hdfs, add it as a file and submit script.sh
> # use bash as exec and file location (/mnt/hadoop/script.sh) as argument
> (e.g bash>/exec>/mnt/hadoop/script.sh
> Best option is that the command will support shared mounted file :
> /mnt/hadoop/script.sh
>
> This code is taking only the file name instead it full path:
> {code:java}
> String exec = actionXml.getChild("exec", ns).getTextTrim();
> String execName = new Path(exec).getName();
> actionConf.set(ShellMain.CONF_OOZIE_SHELL_EXEC, execName);
> {code}
>
> Best option to support shared mount file is to support file:// starting for
> bash files that are local (or shared by mount), e.g:
> {code:java}
> String exec = actionXml.getChild("exec", ns).getTextTrim(); String execName;
> String localFilePrefix = "file://";
> // When exec starts with 'file://' refer it as local file.
> if (exec.startsWith(localFilePrefix))
> execName = exec.substring(localFilePrefix.length());
> else execName = new Path(exec).getName();
> actionConf.set(ShellMain.CONF_OOZIE_SHELL_EXEC, execName);
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[ https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571498#comment-16571498 ] Attila Sasvari commented on OOZIE-3298: --- Can you file a reviewboard request? > OYA: external ID is not filled properly and failing MR job is treated as > SUCCEEDED > -- > > Key: OOZIE-3298 > URL: https://issues.apache.org/jira/browse/OOZIE-3298 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Bacsko >Assignee: Andras Piros >Priority: Blocker > Fix For: 5.1.0 > > Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, > OOZIE-3298.003.patch > > > When a MapReduce action is launched from Oozie (OYA), we don't properly fill > the external ID field. It gets populated with the YARN id of the LauncherAM, > not with the id of the actual MR job. If the MR job is succesfully submitted > then fails, it will be treated as a successfully executed action, which is > very misleading and can potentially confuse Oozie users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders
[
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549315#comment-16549315
]
Attila Sasvari commented on OOZIE-2829:
---
If you look at how Hadoop components are handling/pass multiple files/jars, you
pretty often see a comma separated list of files:
- YARN / Map-Reduce: {{--libjars}}:
https://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options
- Apache Spark: spark-submit
https://spark.apache.org/docs/latest/submitting-applications.html
- Apache Pig - pig.additional.jars.uris
- Apache Oozie - Sharelib mapping file:
https://oozie.apache.org/docs/5.0.0/AG_Install.html#Oozie_Share_Lib
I think it would be better to specify a comma separated list of files for a
given extra sharelib.
> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
> Issue Type: Bug
> Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch,
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch,
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder.
> It would be nice to have the possibility to define additional folders to be
> uploaded into the sharelib, so the users don't have to copy or link the files
> together on their machine.
> The syntax could be something like -additional-lib
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[ https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546559#comment-16546559 ] Attila Sasvari commented on OOZIE-3298: --- [~andras.piros] thanks for the patch! Can you add a new test case that validates the changes? TestMapReduceMain perhaps? > OYA: external ID is not filled properly and failing MR job is treated as > SUCCEEDED > -- > > Key: OOZIE-3298 > URL: https://issues.apache.org/jira/browse/OOZIE-3298 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Bacsko >Assignee: Andras Piros >Priority: Critical > Fix For: 5.1.0 > > Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch > > > When a MapReduce action is launched from Oozie (OYA), we don't properly fill > the external ID field. It gets populated with the YARN id of the LauncherAM, > not with the id of the actual MR job. If the MR job is succesfully submitted > then fails, it will be treated as a successfully executed action, which is > very misleading and can potentially confuse Oozie users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3301) Update NOTICE file
[ https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532431#comment-16532431 ] Attila Sasvari commented on OOZIE-3301: --- Thanks for digging into it, and filing the JIRA about the broken link to the Dependencies Report, Licenses section. Here are some examples from other Apache projects: - Hadoop: https://github.com/apache/hadoop/blob/trunk/NOTICE.txt - HBase: https://github.com/apache/hbase/blob/master/NOTICE.txt - Spark: https://github.com/apache/spark/blob/master/NOTICE-binary - Kafka: https://github.com/apache/kafka/blob/trunk/NOTICE - Flink: https://github.com/apache/flink/blob/master/NOTICE [Oozie license text file|https://github.com/apache/oozie/blob/master/LICENSE.txt] lists many licenses but there are dependencies (such as jetty, netty, gmetric4j, JSch) that are not mentioned / license is missing. Maven has a [license plugin|https://www.mojohaus.org/license-maven-plugin/] that migth be useful, but it needs further investigation. > Update NOTICE file > -- > > Key: OOZIE-3301 > URL: https://issues.apache.org/jira/browse/OOZIE-3301 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Attila Sasvari >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3301-001.patch > > > Oozie NOTICE file is not up to date. Please overview and update it as per > http://www.apache.org/licenses/LICENSE-2.0.html#redistribution > For example: Oozie uses embedded jetty, however Eclipse license > (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is > not listed in > [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3301) Update NOTICE file
[ https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531427#comment-16531427 ] Attila Sasvari commented on OOZIE-3301: --- Hi [~kmarton] thanks for working on it, we should ideally review all the licenses of included Oozie dependencies (extjs etc). Maybe you could use a maven plugin or some tool to automate this. > Update NOTICE file > -- > > Key: OOZIE-3301 > URL: https://issues.apache.org/jira/browse/OOZIE-3301 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Attila Sasvari >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3301-001.patch > > > Oozie NOTICE file is not up to date. Please overview and update it as per > http://www.apache.org/licenses/LICENSE-2.0.html#redistribution > For example: Oozie uses embedded jetty, however Eclipse license > (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is > not listed in > [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2392) Add Kafka support for notifications
[ https://issues.apache.org/jira/browse/OOZIE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531375#comment-16531375 ] Attila Sasvari commented on OOZIE-2392: --- It is an interesting idea that would help to integrate Oozie jobs with external systems. I am wondering if you are still working on it, [~rupalph]? > Add Kafka support for notifications > --- > > Key: OOZIE-2392 > URL: https://issues.apache.org/jira/browse/OOZIE-2392 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Robert Kanter >Assignee: Rupal H >Priority: Major > > Oozie currently lets you send JMS and email notifications. It could be > useful to add support for Kafka as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3301) Update NOTICE file
Attila Sasvari created OOZIE-3301: - Summary: Update NOTICE file Key: OOZIE-3301 URL: https://issues.apache.org/jira/browse/OOZIE-3301 Project: Oozie Issue Type: Bug Affects Versions: 5.0.0 Reporter: Attila Sasvari Oozie NOTICE file is not up to date. Please overview and update it as per http://www.apache.org/licenses/LICENSE-2.0.html#redistribution For example: Oozie uses embedded jetty, however Eclipse license (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is not listed in [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders
[
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516813#comment-16516813
]
Attila Sasvari commented on OOZIE-2829:
---
[~andras.piros], [~pbacsko], [~kmarton] or anyone who reads it: I don't have to
work on this improvement. Feel free to take it over.
> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
> Issue Type: Bug
> Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch,
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder.
> It would be nice to have the possibility to define additional folders to be
> uploaded into the sharelib, so the users don't have to copy or link the files
> together on their machine.
> The syntax could be something like -additional-lib
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3224) Upgrade jetty version to 9.3
Attila Sasvari created OOZIE-3224: - Summary: Upgrade jetty version to 9.3 Key: OOZIE-3224 URL: https://issues.apache.org/jira/browse/OOZIE-3224 Project: Oozie Issue Type: Task Affects Versions: 5.0.0 Reporter: Attila Sasvari Jetty 9.2 has reached end of life as of Jan 2018 http://www.eclipse.org/jetty/documentation/current/what-jetty-version.html#d0e203 Current version used in Oozie 5.0: 9.2.19.v20160908 Please upgrade to a newer Jetty version 9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-2836) Fix ps1 and cmd scripts to make Oozie runnable on Windows
[
https://issues.apache.org/jira/browse/OOZIE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari reassigned OOZIE-2836:
-
Assignee: (was: Attila Sasvari)
> Fix ps1 and cmd scripts to make Oozie runnable on Windows
> -
>
> Key: OOZIE-2836
> URL: https://issues.apache.org/jira/browse/OOZIE-2836
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-2836-01.patch
>
>
> With OOZIE-2666 Oozie was migrated from Tomcat to embedded jetty. Distro
> shell scripts were updated, but Windows scripts were not adjusted in
> {{distro/src/main/bin}}.
> It would be nice if someone with a Windows machine could fix the following
> scripts:
> - oozie-setup.cmd
> - oozie-setup.ps1
> - oozie-sys.cmd
> - oozie.cmd
> - oozied.cmd
> and verify that Oozie works on Windows.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395612#comment-16395612 ] Attila Sasvari commented on OOZIE-2814: --- [~gezapeti], [~andras.piros] can you review the latest patch? Thanks. > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch, OOZIE-2814-06.patch, OOZIE-2814-07.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370112#comment-16370112
]
Attila Sasvari commented on OOZIE-3170:
---
Failed tests pass locally with {{mvn test
-Dtest=TestPartitionDependencyManagerService,TestRecoveryService,TestPartitionDependencyManagerEhcache,TestJMSAccessorService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} and they should be unrelated
(changes are only in the tools module).
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
>
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369998#comment-16369998
]
Attila Sasvari commented on OOZIE-3170:
---
Tested it on a pseudo hadoop cluster:
{code:java}
$ bin/oozie-diag-bundle-collector.sh -oozie http://localhost:11000/oozie
-output /tmp/jobs/
Checking Connection...Done
Using Temporary Directory:
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/1519129669757-0
Getting Sharelib Information...Done
Getting Configuration...Done
Getting OS Environment Variables...Done
Getting Java System Properties...Done
Getting Queue Dump...Done
Getting Thread Dump...Done
Getting Instrumentation...Done
Getting Metrics...Skipping (Metrics are unavailable)
Using Directory: /tmp/jobs/oozie-diag-bundle-1519129669082
Creating Zip File:
/tmp/jobs/oozie-diag-bundle-1519129669082/oozie-diag-bundle.zip ...Done{code}
* Verified that Diag bundle tool logs are placed in the proper directory
{code:java}
$ ls /tmp/jobs/oozie-diag-bundle-1519129669082/ jetty.log
oozie-diag-bundle.zip oozie-instrumentation.log oozie-ops.log
oozie-audit.log oozie-error.log oozie-jpa.log
oozie.log{code}
- Please adjust documentation.
- I am pretty sure test failures are unrelated, but I will run them locally
soon.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
>
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369254#comment-16369254
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks! Sorry for the late response.
- Next time, can you please submit the patch to check whether Oozie tests still
pass?
- If we change the directory structure, we should update the documentation too:
https://github.com/apache/oozie/blob/e26641db469533ac059923c7ed28a5c74f71ad20/docs/src/site/twiki/DG_CommandLineTool.twiki#L1942
- Setting the log directory to the same place as the output zip file sounds
good to me.
I plan to try the patch out with a pseudo hadoop as soon as I have some more
capacity.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
>
[jira] [Comment Edited] (OOZIE-2814) OYA: Update example workflows to newest schemas
[
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365983#comment-16365983
]
Attila Sasvari edited comment on OOZIE-2814 at 2/15/18 5:30 PM:
{{mvn test
-Dtest=TestHCatMessageHandler,TestDefaultConnectionContext,TestJMSSLAEventListener,TestJMSJobEventListener,TestPartitionDependencyManagerEhcache,TestRecoveryService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} passes locally.
was (Author: asasvari):
{{mvn test
-Dtest=TestHCatMessageHandler,TestDefaultConnectionContext,TestJMSSLAEventListener,TestJMSJobEventListener,TestPartitionDependencyManagerEhcache,TestRecoveryService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} pass locally.
> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch,
> OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch,
> OOZIE-2814-05.patch, OOZIE-2814-06.patch, OOZIE-2814-07.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more
> Yarn-centric. Most, if not all, examples currently use pretty old versions
> of the schemas. We should take this opportunity to update the examples to
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows
> still work. We have some examples where we have two workflow XML files that
> do the same thing in different ways.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-07.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch, OOZIE-2814-06.patch, OOZIE-2814-07.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-06.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch, OOZIE-2814-06.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363064#comment-16363064 ] Attila Sasvari commented on OOZIE-2814: --- Can someone review this please? > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-05.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-04.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-03.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360817#comment-16360817
]
Attila Sasvari commented on OOZIE-3172:
---
+1
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch,
> OOZIE-3172.003.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-02.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3175) PurgeService should detect and delete orphaned jobs/actions
[
https://issues.apache.org/jira/browse/OOZIE-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357179#comment-16357179
]
Attila Sasvari commented on OOZIE-3175:
---
An example SQL query to detect orphan workflow actions:
{{SELECT a.id, a.wf_id FROM WF_ACTIONS a WHERE NOT EXISTS (SELECT 1 FROM
WF_JOBS j WHERE j.id=a.wf_id);}}
> PurgeService should detect and delete orphaned jobs/actions
>
>
> Key: OOZIE-3175
> URL: https://issues.apache.org/jira/browse/OOZIE-3175
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Attila Sasvari
>Priority: Major
>
> Improvement idea: the purge service could look for orphaned jobs/actions and
> delete those. It would help the cases where the user went in and only
> partially removed things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3176) Oozie-core fails with checkstyle errors.
[ https://issues.apache.org/jira/browse/OOZIE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reassigned OOZIE-3176: - Assignee: Alisha Prabhu > Oozie-core fails with checkstyle errors. > > > Key: OOZIE-3176 > URL: https://issues.apache.org/jira/browse/OOZIE-3176 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 5.0.0b1 > Environment: # uname -a > Linux in-ibmibm283.persistent.co.in 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Aug > 25 11:21:22 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Alisha Prabhu >Assignee: Alisha Prabhu >Priority: Minor > Attachments: Checkstyle-erros.txt > > > There are multiple checkstyle violations in the Oozie-core module. > Maven command used is mvn clean install -DskipTests > I have attached the Checkstyle-errors.txt file, that has the errors occurred > while building. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3176) Oozie-core fails with checkstyle errors.
[
https://issues.apache.org/jira/browse/OOZIE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355416#comment-16355416
]
Attila Sasvari commented on OOZIE-3176:
---
[~alishap] many thanks for filing this JIRA, I added you to contributors and
assigned the ticket too you.
I can confirm {{mvn clean install -DskipTests
-Dmaven.javadoc.opts="-Xdoclint:none"}} fails on master because of a checkstyle
error, {{core/target/checkstyle-result.xml}} shows:
{code}
http://www.apache.org/licenses/LICENSE-2.0.;
source="com.puppycrawl.tool
s.checkstyle.checks.header.HeaderCheck"/>
{code}
WorkflowJobsBasicInfoFromParentId was introduced by OOZIE-1401.amend.003.patch.
On that day, the Oozie PreCommit build failed to add comment (JIRA was
updated).
> Oozie-core fails with checkstyle errors.
>
>
> Key: OOZIE-3176
> URL: https://issues.apache.org/jira/browse/OOZIE-3176
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1
> Environment: # uname -a
> Linux in-ibmibm283.persistent.co.in 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Aug
> 25 11:21:22 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Minor
> Attachments: Checkstyle-erros.txt
>
>
> There are multiple checkstyle violations in the Oozie-core module.
> Maven command used is mvn clean install -DskipTests
> I have attached the Checkstyle-errors.txt file, that has the errors occurred
> while building.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355233#comment-16355233
]
Attila Sasvari commented on OOZIE-3172:
---
[~andras.piros] thanks for submitting example workflows. Workflows succeeded,
so it looks good to me. [~gezapeti] can you answer last question?
(If it breaks Hive workflows, we can file a separate JIRA. It would be nice to
validate.)
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2847:
--
Affects Version/s: 5.0.0
5.0.0b1
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0, 5.0.0b1, 5.0.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
> at java.security.AccessController.doPrivileged(Native
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353807#comment-16353807
]
Attila Sasvari commented on OOZIE-2847:
---
I decided to only push it to master. [~gezapeti] is the release manager of
4.3.1, that is in progress. He will commit OOZIE-2847-4.3.patch to branch-4.3
later and update fix version.
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>
[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2847:
--
Fix Version/s: (was: 4.3.1)
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
> at java.security.AccessController.doPrivileged(Native Method)
> at
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353797#comment-16353797
]
Attila Sasvari commented on OOZIE-2847:
---
[~dionusos] thanks for your contribution, +1. I will commit the patches to
branch-4.3 and master soon.
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
>
[jira] [Created] (OOZIE-3175) PurgeService should detect and delete orphaned jobs/actions
Attila Sasvari created OOZIE-3175: - Summary: PurgeService should detect and delete orphaned jobs/actions Key: OOZIE-3175 URL: https://issues.apache.org/jira/browse/OOZIE-3175 Project: Oozie Issue Type: Improvement Components: core Affects Versions: trunk Reporter: Attila Sasvari Improvement idea: the purge service could look for orphaned jobs/actions and delete those. It would help the cases where the user went in and only partially removed things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3174) Allow users to retain SQL file generated by OozieDBCLI
Attila Sasvari created OOZIE-3174: - Summary: Allow users to retain SQL file generated by OozieDBCLI Key: OOZIE-3174 URL: https://issues.apache.org/jira/browse/OOZIE-3174 Project: Oozie Issue Type: Improvement Components: tools Affects Versions: trunk Reporter: Attila Sasvari OozieDBCLI writes SQL statements to a temporary file. It is deleted without question at the end of the execution (see OOZIE-2727). We should allow the user to decide if they want to delete it or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353729#comment-16353729 ] Attila Sasvari commented on OOZIE-1717: --- TestHiveActionExecutor locally passes, committed to master > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2847:
--
Fix Version/s: (was: 5.0.0b1)
5.0.0
trunk
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
> at
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: amend-OOZIE-1717-01.patch > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: (was: amend-OOZIE-1717-01.patch) > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: amend-OOZIE-1717-01.patch > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: amend-OOZIE-1717-00.patch > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reopened OOZIE-1717: --- > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Fix Version/s: trunk > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Summary: Add indexes to speed up db queries (was: Index user_name column) > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352389#comment-16352389 ] Attila Sasvari commented on OOZIE-1717: --- TestRecoveryService,TestJMSJobEventListener,TestPartitionDependencyManagerService,TestJMSSLAEventListener,TestHCatMessageHandler,TestDefaultConnectionContext,TestJMSAccessorService pass locally. > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352387#comment-16352387
]
Attila Sasvari commented on OOZIE-2847:
---
OOZIE-1770 is about Oozie on Yarn that migrates the launcher from MapReduce to
Yarn Application Master. It is only included in Oozie 5.0.0-beta1 and the
master branch. So, {{LauncherMapper}} only exists in 4.3.1. Related mechanism,
{{setRecoveryId()}}, on the master branch resides in
[LauncherAM|[https://github.com/apache/oozie/blob/ba665da34c23b1fa86bf1570724147e6f8c85b1d/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java#L481]|https://github.com/apache/oozie/blob/ba665da34c23b1fa86bf1570724147e6f8c85b1d/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java#L481].]
. It might have similar issue in HA mode. Anyway, it would be nice to solve it
on branch-4.3.{{}}
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
>
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352300#comment-16352300
]
Attila Sasvari commented on OOZIE-2847:
---
[~dionusos] can you please rebase the patch? Thanks.
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1519)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-004.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-01.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349221#comment-16349221
]
Attila Sasvari commented on OOZIE-2814:
---
[~rkanter] thanks a lot. I changed the java-main example workflow to use
{{resource-manager}} instead of {{job-tracker}}
{code:java}
${jobTracker}{code}
Result of job submission:
{code:java}
Error: E0701 : E0701: XML schema error, No job-tracker defined{code}
Workflow can be validated against the schema just fine, but
{{LiteWorkflowAppParser}} needs to be adjusted as it only tries to fetch the
child of the {{job-tracker}} element:
https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java#L424
TestLiteWorkflowAppParser is to be extended too.
> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more
> Yarn-centric. Most, if not all, examples currently use pretty old versions
> of the schemas. We should take this opportunity to update the examples to
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows
> still work. We have some examples where we have two workflow XML files that
> do the same thing in different ways.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346507#comment-16346507
]
Attila Sasvari commented on OOZIE-3172:
---
There are transitive dependencies. Spark ShareLib for example:
{code:java}
[INFO] Building Apache Oozie Share Lib Spark 5.0.0-beta1
[INFO]
[INFO]
[INFO] --- maven-dependency-plugin:2.4:tree (default-cli) @
oozie-sharelib-spark ---
[INFO] org.apache.oozie:oozie-sharelib-spark:jar:5.0.0-beta1
[INFO] +- com.google.guava:guava:jar:14.0.1:compile
[INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0:compile
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | +- org.apache.commons:commons-compress:jar1.4.1:compile
[INFO] | | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | +- com.sun.jersey:jersey-client:jar:1.9:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile{code}
I would submit the example workflows to verify it does not cause issues.
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346216#comment-16346216
]
Attila Sasvari commented on OOZIE-3172:
---
{{mvn dependency:tree}} shows that multiple sharelibs are depending on
{{jackson-core-asl}} (for example Hive, Hive 2, Pig, Spark, Sqoop, Streaming).
I am a bit concerned that doing this change may cause runtime failures because
of conflicting versions. Have you submitted example workflows?
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345236#comment-16345236 ] Attila Sasvari commented on OOZIE-2814: --- I am not sure if it makes sense to keep older workflow schemas. Those are there in earlier releases and also under version control. Patch 00 only changes example workflows, but I guess we want to also update documentation and unit tests. > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-00.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reassigned OOZIE-2814: - Assignee: Attila Sasvari > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter >Assignee: Attila Sasvari >Priority: Blocker > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari reassigned OOZIE-3170:
-
Assignee: Jason Phelps
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
>
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345077#comment-16345077
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks for reporting this issue and reviewing related parts of
the codebase. I added you as a contributor to the project and assigned this
Jira to you.
I reproduced that the cited NPE exception is thrown. The diag bundle zip is
however generated and contain sharelib info - it is just using the
OozieClient's
[listShareLib()|https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java#L42]:
{code:java}
$ bin/oozie-diag-bundle-collector.sh -oozie http://localhost:11000/oozie
-output /tmp/jobs/
Checking Connection...Done
Using Temporary Directory:
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/1517319232457-0
Getting Sharelib Information...Done
Getting Configuration...Done
Getting OS Environment Variables...Done
Getting Java System Properties...Done
Getting Queue Dump...Done
Getting Thread Dump...Done
Getting Instrumentation...Done
Getting Metrics...Skipping (Metrics are unavailable)
Creating Zip File: /tmp/jobs/oozie-diag-bundle-1517319233190.zip...Done
$ unzip -l /tmp/jobs/oozie-diag-bundle-1517319233190.zip
68029 01-30-18 14:33 /effective-oozie-site.xml
9876 01-30-18 14:33 /instrumentation.txt
38636 01-30-18 14:33 /java-sys-props.txt
3807 01-30-18 14:33 /os-env-vars.txt
279 01-30-18 14:33 /queue-dump.txt
40032 01-30-18 14:33 /sharelib.txt
102084 01-30-18 14:33 /thread-dump.html{code}
* In fact, I am not sure all those Oozie services are really needed here to be
able to collect diagnostic information. If they are not needed they shall not
be loaded at all.
* There is also another problem. By default, logs generated by the tool appear
in the server log if you run the tool from Oozie's home directory. It can make
things very confusing for an admin or anyone who review Oozie server logs.
Setting up logging is the responsibility of the
[XLogService|[https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/core/src/main/java/org/apache/oozie/service/XLogService.java#L145]]
and it is started via Services.init(). It can be controlled by the
{{oozie.log.dir}} system property (e.g. \{{export
JAVA_PROPERTIES="-Doozie.log.dir=/tmp/"}} before running the tool). This is
something we should clarify in the documentation of the tool and/or change the
code/script so that logs are put in the directory where the diag bundle is
generated by default.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
>
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343484#comment-16343484
]
Attila Sasvari commented on OOZIE-1717:
---
OpenJPA 2.4.2 does not support JPA 2.1, so I had to add the following composite
indices with {{java.sql.Statement's execUpdate()}}:
{code:java}
CREATE INDEX I_WF_JOBS_STATUS_CREATED_TIME ON WF_JOBS (status, created_time)
CREATE INDEX I_COORD_ACTIONS_JOB_ID_STATUS ON COORD_ACTIONS (job_id, status)
CREATE INDEX I_COORD_JOBS_STATUS_CREATED_TIME ON COORD_JOBS (status,
created_time)
CREATE INDEX I_COORD_JOBS_STATUS_LAST_MODIFIED_TIME ON COORD_JOBS (status,
last_modified_time)
CREATE INDEX I_COORD_JOBS_PENDING_DONE_MATERIALIZATION_LAST_MODIFIED_TIME ON
COORD_JOBS (pending, done_materialization, last_modified_time)
"CREATE INDEX I_COORD_JOBS_PENDING_LAST_MODIFIED_TIME ON COORD_JOBS (pending,
last_modified_time)
CREATE INDEX I_BUNLDE_JOBS_STATUS_CREATED_TIME ON BUNDLE_JOBS (status,
created_time)
CREATE INDEX I_BUNLDE_JOBS_STATUS_LAST_MODIFIED_TIME ON BUNDLE_JOBS (status,
last_modified_time)
CREATE INDEX I_BUNLDE_ACTIONS_PENDING_LAST_MODIFIED_TIME ON BUNDLE_ACTIONS
(pending, last_modified_time){code}
After the change, the following indices are present in the oozie database:
{code:java}
> SELECT DISTINCT TABLE_NAME, COLUMN_NAME, INDEX_NAME FROM
> INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = 'oozie';
++--+--+
| TABLE_NAME | COLUMN_NAME | INDEX_NAME
|
++--+--+
| BUNDLE_ACTIONS | bundle_action_id | PRIMARY
|
| BUNDLE_ACTIONS | bundle_id | I_BNDLTNS_BUNDLE_ID
|
| BUNDLE_ACTIONS | pending |
I_BUNLDE_ACTIONS_PENDING_LAST_MODIFIED_TIME |
| BUNDLE_ACTIONS | last_modified_time |
I_BUNLDE_ACTIONS_PENDING_LAST_MODIFIED_TIME |
| BUNDLE_JOBS | id | PRIMARY
|
| BUNDLE_JOBS | created_time | I_BNDLJBS_CREATED_TIME
|
| BUNDLE_JOBS | end_time | I_BNDLJBS_END_TIME
|
| BUNDLE_JOBS | last_modified_time | I_BNDLJBS_LAST_MODIFIED_TIME
|
| BUNDLE_JOBS | status | I_BNDLJBS_STATUS
|
| BUNDLE_JOBS | suspended_time | I_BNDLJBS_SUSPENDED_TIME
|
| BUNDLE_JOBS | user_name | I_BNDLJBS_USER_NAME
|
| BUNDLE_JOBS | status |
I_BUNLDE_JOBS_STATUS_CREATED_TIME |
| BUNDLE_JOBS | created_time |
I_BUNLDE_JOBS_STATUS_CREATED_TIME |
| BUNDLE_JOBS | status |
I_BUNLDE_JOBS_STATUS_LAST_MODIFIED_TIME |
| BUNDLE_JOBS | last_modified_time |
I_BUNLDE_JOBS_STATUS_LAST_MODIFIED_TIME |
| COORD_ACTIONS | id | PRIMARY
|
| COORD_ACTIONS | created_time | I_CRD_TNS_CREATED_TIME
|
| COORD_ACTIONS | external_id | I_CRD_TNS_EXTERNAL_ID
|
| COORD_ACTIONS | job_id | I_CRD_TNS_JOB_ID
|
| COORD_ACTIONS | last_modified_time | I_CRD_TNS_LAST_MODIFIED_TIME
|
| COORD_ACTIONS | nominal_time | I_CRD_TNS_NOMINAL_TIME
|
| COORD_ACTIONS | rerun_time | I_CRD_TNS_RERUN_TIME
|
| COORD_ACTIONS | status | I_CRD_TNS_STATUS
|
| COORD_ACTIONS | job_id | I_COORD_ACTIONS_JOB_ID_STATUS
|
| COORD_ACTIONS | status | I_COORD_ACTIONS_JOB_ID_STATUS
|
| COORD_JOBS | id | PRIMARY
|
| COORD_JOBS | bundle_id | I_CRD_JBS_BUNDLE_ID
|
| COORD_JOBS | created_time | I_CRD_JBS_CREATED_TIME
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-003.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-002.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-01.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340291#comment-16340291
]
Attila Sasvari commented on OOZIE-1717:
---
[~todd.farmer] thanks for the recommendation.
Oozie uses OpenJPA 2.4.2 to talk with databases (Oracle, PostgreSQL, etc. are
also supported). OpenJPA 2.4.2 supports only [JPA
2.0|http://openjpa.apache.org/]. Do you know how we can define a covering
composite index with OpenJPA, I could not find it in the documentation.
I tried the following things in org.apache.oozie.CoordinatorJobBean
{code:java}
@Index(name = "index1", columnNames = {"status, created_time"})
@Index(name = "index2", columnNames = {"status", "created_time"}){code}
but none of them resulted in a composite index. The [Table
annotation|[https://github.com/apache/oozie/blob/acdf291aef48fd0f9a5b0e3270afacb7ee4a1543/core/src/main/java/org/apache/oozie/CoordinatorJobBean.java#L158|https://github.com/apache/oozie/blob/acdf291aef48fd0f9a5b0e3270afacb7ee4a1543/core/src/main/java/org/apache/oozie/CoordinatorJobBean.java#L158]]]
in CoordinatorJobBean did not accept indices either.
I can see in
[OozieDBCLI|https://github.com/apache/oozie/blob/acdf291aef48fd0f9a5b0e3270afacb7ee4a1543/tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java#L977]
that a clustered index is created "manually". We can follow this pattern if
really needed.
It would be good to have a list for more interesting (composite) indexes.
> Index user_name column
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-1717-00.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3093) Test Scenarios for Oozie 5.0
[ https://issues.apache.org/jira/browse/OOZIE-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3093: -- Fix Version/s: (was: 5.0.0b1) 5.0.0 > Test Scenarios for Oozie 5.0 > > > Key: OOZIE-3093 > URL: https://issues.apache.org/jira/browse/OOZIE-3093 > Project: Oozie > Issue Type: Test >Affects Versions: 5.0.0b1 >Reporter: Artem Ervits >Priority: Blocker > Fix For: 5.0.0 > > > This is umbrella jira to track test scenarios on Oozie 5.0. In my discussion > with [~gezapeti] we can have a joint effort to test different test cases. > [~clayb] please provide your input as well. > 1. Oozie HA > 2. Oozie secure cluster > 3. Run all examples > 4. Execute against every Hadoop profile > etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1987) Fix existing findbugs warnings
[ https://issues.apache.org/jira/browse/OOZIE-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1987: -- Fix Version/s: (was: 5.0.0b1) 5.0.0 > Fix existing findbugs warnings > -- > > Key: OOZIE-1987 > URL: https://issues.apache.org/jira/browse/OOZIE-1987 > Project: Oozie > Issue Type: Bug >Reporter: Robert Kanter >Assignee: Jan Hentschel >Priority: Major > Fix For: 5.0.0 > > > Now that we have OOZIE-1793, we should actually fix the warnings that > findbugs finds in the existing codebase. It may be a good idea to split this > into subtasks or multiple patches as there are a lot of warnings: over 500 in > oozie-core alone! > If we think something is ok, but findbugs complains, we should exclude it. > There's a way to create a file that lists exclusions (Hadoop does this). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2600) OYA: Update Documentation
[
https://issues.apache.org/jira/browse/OOZIE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2600:
--
Fix Version/s: (was: 5.0.0b1)
5.0.0
> OYA: Update Documentation
> -
>
> Key: OOZIE-2600
> URL: https://issues.apache.org/jira/browse/OOZIE-2600
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: oya, 5.0.0
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 5.0.0
>
>
> Based on the work done in OOZIE-2896, document also the way how
> {{oozie.launcher.override.\*}} and {{oozie.launcher.prepend.\*}} parameters
> function.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2942) Fix Findbugs warnings in oozie-examples
[
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2942:
--
Fix Version/s: (was: 5.0.0b1)
5.0.0
> Fix Findbugs warnings in oozie-examples
> ---
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
> Issue Type: Sub-task
> Components: build
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close
> stream on exception [org.apache.oozie.example.DateList] At
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in
> org.apache.oozie.example.DateList.main(String[])
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues
> at LocalOozieExample.java:[line 76]Path continues at
> LocalOozieExample.java:[line 77]Path continues at
> LocalOozieExample.java:[line 78]Path continues at
> LocalOozieExample.java:[line 81]Path continues at
> LocalOozieExample.java:[line 88]Path continues at
> LocalOozieExample.java:[line 89]Path continues at
> LocalOozieExample.java:[line 91]Path continues at
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail
> to close stream [org.apache.oozie.example.LocalOozieExample] At
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal
> representation by returning Repeatable.baseline
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose
> internal representation by storing an externally mutable object into
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2975) code clean up in pig sharelib, replace Exception with more explicit, add try with resources, StringBuilder instead of StringBuffer
[ https://issues.apache.org/jira/browse/OOZIE-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2975: -- Fix Version/s: (was: 5.0.0b1) 5.0.0 > code clean up in pig sharelib, replace Exception with more explicit, add try > with resources, StringBuilder instead of StringBuffer > -- > > Key: OOZIE-2975 > URL: https://issues.apache.org/jira/browse/OOZIE-2975 > Project: Oozie > Issue Type: Improvement >Reporter: Artem Ervits >Assignee: Artem Ervits >Priority: Trivial > Labels: newbie > Fix For: 5.0.0 > > Attachments: OOZIE-2975-0.patch > > > 1. Replaced StringBuffer with StringBuilder > 2. Replaced Exception with IOException and ExecException > 3. Replaced old Hashmap notation with diamond notation > 4. Added try-with-resources notation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3169) Create a download page for releases
Attila Sasvari created OOZIE-3169: - Summary: Create a download page for releases Key: OOZIE-3169 URL: https://issues.apache.org/jira/browse/OOZIE-3169 Project: Oozie Issue Type: New Feature Components: site Reporter: Attila Sasvari Today to download a release, you need to click on the Releases tab and select a release. It will take a user to an external page where you can download the release from an Apache mirrors (e.g. [http://www.apache.org/dyn/closer.lua/oozie/5.0.0-beta1).] It would be better to create a page on the Oozie site to download releases and related hash and PGP signature files. This way a visitor could stay on the Oozie site. Examples: [http://hadoop.apache.org/releases.html] [https://flume.apache.org/releases/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2681) fix javadoc to compile on JDK8 again
[
https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337408#comment-16337408
]
Attila Sasvari commented on OOZIE-2681:
---
{{mvn clean deploy -Papache-release -DskipTests fails with the following while
generating Javadoc:}}
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:2.10.4:jar (attach-javadocs) on
project oozie-tools: MavenReportException: Error while generating Javadoc:
[ERROR] Exit code: 1 -
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:44:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:44:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:49:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:49:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:82:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:82:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:88:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:88:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
{code}
> fix javadoc to compile on JDK8 again
>
>
> Key: OOZIE-2681
> URL: https://issues.apache.org/jira/browse/OOZIE-2681
> Project: Oozie
> Issue Type: Sub-task
> Components: build
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Peter Cseh
>Priority: Critical
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch,
> OOZIE-2681-02.patch, OOZIE-2681-03.patch, OOZIE-2681-04.patch,
> OOZIE-2681.06.patch, OOZIE-2681.07.patch, OOZIE-2681.08.patch,
> OOZIE-2681.09.patch, OOZIE-2681.10.patch, OOZIE-2681.11.patch,
> OOZIE-2681.12.patch, OOZIE-2681.13.patch, OOZIE-2681.14.patch,
> OOZIE-2681.15.patch
>
>
> We have a lot of Javadoc build failures, which causes Oozie to fail to build
> on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all
> of these that existed at that time, but we've since introduced a lot more.
> OOZIE-2487 made a workaround to prevent this from causing the build to fail,
> but we should fix these (again) and remove the OOZIE-2487 workaround.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335989#comment-16335989
]
Attila Sasvari commented on OOZIE-1717:
---
I can imagine adding index on the created_time column of COORD_JOBS , could
help too (a lot of coordinator jobs might be created). I saw this in slow
query log when going through the coordinator jobs:
{code:java}
SELECT t0.id, t0.app_name, t0.status, t0.user_name, t0.group_name,
t0.start_time, t0.end_time, t0.app_path, t0.concurrency, t0.frequency,
t0.last_action, t0.next_matd_time, t0.created_time, t0.time_unit, t0.time_zone,
t0.time_out, t0.bundle_id FROM COORD_JOBS t0 WHERE ((t0.status = 'RUNNING' OR
t0.status = 'PREP' OR t0.status = 'SUSPENDED' OR t0.status = 'RUNNINGWITHERROR'
OR t0.status = 'PREPSUSPENDED' OR t0.status = 'SUSPENDEDWITHERROR' OR t0.status
= 'PREPPAUSED' OR t0.status = 'PAUSED' OR t0.status = 'PAUSEDWITHERROR') AND
t0.status IS NOT NULL) ORDER BY t0.created_time DESC LIMIT 100, 100;{code}
> Index user_name column
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-1717-00.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335934#comment-16335934 ] Attila Sasvari commented on OOZIE-1717: --- [~gezapeti] thanks for the review, I will do such changes. I will also set "4" for DB_VERSION_FOR_5_0 following existing pattern. I ran some explain query for selects that appeared in slow query log of MariaDB and after applying the new indices I saw that for example filesort was not used anymore (proper index was used). Anyway, I want to load a really large database and continue investigation (clicking though web UI and watching slow query-s). It would be nice to have a list of the most frequent queries. > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2775) Oozie server does not stop if there is an exception during service initalization at startup
[
https://issues.apache.org/jira/browse/OOZIE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335854#comment-16335854
]
Attila Sasvari commented on OOZIE-2775:
---
[~andras.piros] thanks for the review! Regarding the failure:
junit.framework.AssertionFailedError: YARN App state for app
application_1516711234396_0003 expected: but was: at
org.apache.oozie.action.hadoop.TestHiveActionExecutor.testHiveAction(TestHiveActionExecutor.java:158)
it happened twice (rerun).
Failed test passes locally and I am pretty sure it is unrelated to my changes:
the patch does not touches oozie-core and a seperate embedded jetty server and
miniservers are used for testing in the core module.
> Oozie server does not stop if there is an exception during service
> initalization at startup
> ---
>
> Key: OOZIE-2775
> URL: https://issues.apache.org/jira/browse/OOZIE-2775
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2775-01.patch, OOZIE-2775-02.patch
>
>
> OOZIE-2666 brought in embedded jetty in Oozie. {{com.google.inject.guice}}
> was selected for dependency management. Unfortunately if there is an
> exception at starting Oozie services (e.g. JPAService initialization, called
> via
> https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L194),
> Oozie won't stop if it cannot destroy Oozie services.
> In OOZIE-2666 a shutdown hook
> (https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L184)
> was added to print out startup errors and exceptions, destroy pending oozie
> services, and exit eventually, but it does not work.
> Steps to reproduce a startup failure (for example simulate that Oozie cannot
> find required database driver):
> {code}
> $ cd oozie
> $ mvn clean install assembly:single -DskipTests -DjavaVersion=1.8
> -DtargetVersion=1.7 -Phadoop-2 -Dhadoop.version=2.6.0 -Puber;
> $ cd distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/
> $ export JETTY_OPTS="-Djavax.net.debug=all
> -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
> -Dlog4j.logger.org.eclipse.jetty=TRACE
> -Dlog4j.configuration=./conf/oozie-log4j.properties"
> $ bin/oozied.sh run
> $ mv embedded-oozie-server/webapp/WEB-INF/lib/derby-10.10.1.1.jar ..
>
> {code}
> Exception is printed to stdout
> {code}
>
> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver
> class 'org.apache.derby.jdbc.EmbeddedDriver'
> at
> org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:106)
> at
> org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
> at
> org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1520)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:533)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:458)
> at
> org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:121)
> at
> org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
> at
> org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:642)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:202)
> at
> org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:154)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:226)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:153)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:59)
> at
> org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:514)
> at org.apache.oozie.service.JPAService.init(JPAService.java:215)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:387)
> at org.apache.oozie.service.Services.setService(Services.java:373)
> at
[jira] [Updated] (OOZIE-2775) Oozie server does not stop if there is an exception during service initalization at startup
[
https://issues.apache.org/jira/browse/OOZIE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2775:
--
Attachment: OOZIE-2775-02.patch
> Oozie server does not stop if there is an exception during service
> initalization at startup
> ---
>
> Key: OOZIE-2775
> URL: https://issues.apache.org/jira/browse/OOZIE-2775
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2775-01.patch, OOZIE-2775-02.patch
>
>
> OOZIE-2666 brought in embedded jetty in Oozie. {{com.google.inject.guice}}
> was selected for dependency management. Unfortunately if there is an
> exception at starting Oozie services (e.g. JPAService initialization, called
> via
> https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L194),
> Oozie won't stop if it cannot destroy Oozie services.
> In OOZIE-2666 a shutdown hook
> (https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L184)
> was added to print out startup errors and exceptions, destroy pending oozie
> services, and exit eventually, but it does not work.
> Steps to reproduce a startup failure (for example simulate that Oozie cannot
> find required database driver):
> {code}
> $ cd oozie
> $ mvn clean install assembly:single -DskipTests -DjavaVersion=1.8
> -DtargetVersion=1.7 -Phadoop-2 -Dhadoop.version=2.6.0 -Puber;
> $ cd distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/
> $ export JETTY_OPTS="-Djavax.net.debug=all
> -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
> -Dlog4j.logger.org.eclipse.jetty=TRACE
> -Dlog4j.configuration=./conf/oozie-log4j.properties"
> $ bin/oozied.sh run
> $ mv embedded-oozie-server/webapp/WEB-INF/lib/derby-10.10.1.1.jar ..
>
> {code}
> Exception is printed to stdout
> {code}
>
> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver
> class 'org.apache.derby.jdbc.EmbeddedDriver'
> at
> org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:106)
> at
> org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
> at
> org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1520)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:533)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:458)
> at
> org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:121)
> at
> org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
> at
> org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:642)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:202)
> at
> org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:154)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:226)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:153)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:59)
> at
> org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:514)
> at org.apache.oozie.service.JPAService.init(JPAService.java:215)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:387)
> at org.apache.oozie.service.Services.setService(Services.java:373)
> at org.apache.oozie.service.Services.loadServices(Services.java:305)
> at org.apache.oozie.service.Services.init(Services.java:213)
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:31)
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:25)
> at
> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at
>
[jira] [Commented] (OOZIE-2775) Oozie server does not stop if there is an exception during service initalization at startup
[
https://issues.apache.org/jira/browse/OOZIE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334999#comment-16334999
]
Attila Sasvari commented on OOZIE-2775:
---
{color:#33}[~andras.piros] Thanks for your comments!{color}
* {color:#33}When a {color}ProvisionException is thrown during
{{guiceInjector.getInstance()}} , {{embeddedOozieServer}} will be null. In the
current form, without additional changes: you can't get a reference to the
Jetty server as it does not exist at this point of the execution (it is
actually created after the Oozie services started). The (failed) Oozie services
created by the provider cannot be stopped either. I wonder if it was really
necessary. If so, we can probably handle it in the ServicesProvider. I will try
to address this in a new patch uploaded to *Reviewboard* *first*.
* Regarding a stop() method: can you expand how it should work? Right now,
stopping the Oozie server happens via a SIGTERM signal sent to the process (see
{{bin/oozie-jetty-server.sh}}. This behaviour is pretty similar to the earlier
version (with Tomcat) where this case was not handled by the means of the
[shutdown hook
mechanism|https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/signals006.html]
as far as I remember. I suppose it is OK that we add such a handler, but of
course there might be races (for example what if signal arrives before we
install it). What do you think?
* I believe
[ExceptionInInitializerError|https://docs.oracle.com/javase/7/docs/api/java/lang/ExceptionInInitializerError.html]
is not suitable for handling this sort of service initialization problems,
[ExceptionInInitializerError|https://docs.oracle.com/javase/7/docs/api/java/lang/ExceptionInInitializerError.html]
is a kind of LinkageError. Documentation says "{{ExceptionInInitializerError}}
is thrown to indicate that an exception occurred during evaluation of a static
initializer or the initializer for a static variable.". A more pragmatic answer
is that BoundProviderFactory's {{get()}} by guice only catches Runtime
exceptions.
> Oozie server does not stop if there is an exception during service
> initalization at startup
> ---
>
> Key: OOZIE-2775
> URL: https://issues.apache.org/jira/browse/OOZIE-2775
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2775-01.patch
>
>
> OOZIE-2666 brought in embedded jetty in Oozie. {{com.google.inject.guice}}
> was selected for dependency management. Unfortunately if there is an
> exception at starting Oozie services (e.g. JPAService initialization, called
> via
> https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L194),
> Oozie won't stop if it cannot destroy Oozie services.
> In OOZIE-2666 a shutdown hook
> (https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L184)
> was added to print out startup errors and exceptions, destroy pending oozie
> services, and exit eventually, but it does not work.
> Steps to reproduce a startup failure (for example simulate that Oozie cannot
> find required database driver):
> {code}
> $ cd oozie
> $ mvn clean install assembly:single -DskipTests -DjavaVersion=1.8
> -DtargetVersion=1.7 -Phadoop-2 -Dhadoop.version=2.6.0 -Puber;
> $ cd distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/
> $ export JETTY_OPTS="-Djavax.net.debug=all
> -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
> -Dlog4j.logger.org.eclipse.jetty=TRACE
> -Dlog4j.configuration=./conf/oozie-log4j.properties"
> $ bin/oozied.sh run
> $ mv embedded-oozie-server/webapp/WEB-INF/lib/derby-10.10.1.1.jar ..
>
> {code}
> Exception is printed to stdout
> {code}
>
> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver
> class 'org.apache.derby.jdbc.EmbeddedDriver'
> at
> org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:106)
> at
> org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
> at
> org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1520)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:533)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:458)
> at
> org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:121)
> at
> org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
> at
>
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334430#comment-16334430
]
Attila Sasvari commented on OOZIE-1717:
---
patch 00 adds 6 new indices and changes *db.version* to *3.1*. Here are the
indices from the SQL file generated by the OozieDBCLI tool:
CREATE INDEX I_BNDLTNS_BUNDLE_ID ON BUNDLE_ACTIONS (bundle_id);
CREATE INDEX I_BNDLJBS_CREATED_TIME ON BUNDLE_JOBS (created_time);
{color:#33}*CREATE INDEX I_BNDLJBS_END_TIME ON BUNDLE_JOBS
(end_time);*{color}
CREATE INDEX I_BNDLJBS_LAST_MODIFIED_TIME ON BUNDLE_JOBS (last_modified_time);
CREATE INDEX I_BNDLJBS_STATUS ON BUNDLE_JOBS (status);
CREATE INDEX I_BNDLJBS_SUSPENDED_TIME ON BUNDLE_JOBS (suspended_time);
*CREATE INDEX I_BNDLJBS_USER_NAME ON BUNDLE_JOBS (user_name);*
CREATE INDEX I_CRD_TNS_CREATED_TIME ON COORD_ACTIONS (created_time);
CREATE INDEX I_CRD_TNS_EXTERNAL_ID ON COORD_ACTIONS (external_id);
CREATE INDEX I_CRD_TNS_JOB_ID ON COORD_ACTIONS (job_id);
CREATE INDEX I_CRD_TNS_LAST_MODIFIED_TIME ON COORD_ACTIONS (last_modified_time);
CREATE INDEX I_CRD_TNS_NOMINAL_TIME ON COORD_ACTIONS (nominal_time);
CREATE INDEX I_CRD_TNS_RERUN_TIME ON COORD_ACTIONS (rerun_time);
CREATE INDEX I_CRD_TNS_STATUS ON COORD_ACTIONS (status);
CREATE INDEX I_CRD_JBS_BUNDLE_ID ON COORD_JOBS (bundle_id);
CREATE INDEX I_CRD_JBS_CREATED_TIME ON COORD_JOBS (created_time);
*CREATE INDEX I_CRD_JBS_END_TIME ON COORD_JOBS (end_time);*
CREATE INDEX I_CRD_JBS_LAST_MODIFIED_TIME ON COORD_JOBS (last_modified_time);
CREATE INDEX I_CRD_JBS_NEXT_MATD_TIME ON COORD_JOBS (next_matd_time);
CREATE INDEX I_CRD_JBS_STATUS ON COORD_JOBS (status);
CREATE INDEX I_CRD_JBS_SUSPENDED_TIME ON COORD_JOBS (suspended_time);
*CREATE INDEX I_CRD_JBS_USER_NAME ON COORD_JOBS (user_name);*
CREATE INDEX I_SL_VNTS_DTYPE ON SLA_EVENTS (bean_type);
CREATE INDEX I_SL_RRTN_NOMINAL_TIME ON SLA_REGISTRATION (nominal_time);
CREATE INDEX I_SL_SMRY_APP_NAME ON SLA_SUMMARY (app_name);
CREATE INDEX I_SL_SMRY_EVENT_PROCESSED ON SLA_SUMMARY (event_processed);
CREATE INDEX I_SL_SMRY_LAST_MODIFIED ON SLA_SUMMARY (last_modified);
CREATE INDEX I_SL_SMRY_NOMINAL_TIME ON SLA_SUMMARY (nominal_time);
CREATE INDEX I_SL_SMRY_PARENT_ID ON SLA_SUMMARY (parent_id);
CREATE INDEX I_WF_CTNS_PENDING_AGE ON WF_ACTIONS (pending_age);
CREATE INDEX I_WF_CTNS_STATUS ON WF_ACTIONS (status);
CREATE INDEX I_WF_CTNS_WF_ID ON WF_ACTIONS (wf_id);
*CREATE INDEX I_WF_JOBS_CREATED_TIME ON WF_JOBS (created_time);*
CREATE INDEX I_WF_JOBS_END_TIME ON WF_JOBS (end_time);
CREATE INDEX I_WF_JOBS_EXTERNAL_ID ON WF_JOBS (external_id);
CREATE INDEX I_WF_JOBS_LAST_MODIFIED_TIME ON WF_JOBS (last_modified_time);
CREATE INDEX I_WF_JOBS_PARENT_ID ON WF_JOBS (parent_id);
CREATE INDEX I_WF_JOBS_STATUS ON WF_JOBS (status);
*CREATE INDEX I_WF_JOBS_USER_NAME ON WF_JOBS (user_name);*
Do we want to add anything else?
I am going to run some more tests using Hue / Oozie Web UI (using MariaDB with
slow query logging enabled).
> Index user_name column
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-1717-00.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-00.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reassigned OOZIE-1717: - Assignee: Attila Sasvari > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2775) Oozie server does not stop if there is an exception during service initalization at startup
[
https://issues.apache.org/jira/browse/OOZIE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334196#comment-16334196
]
Attila Sasvari commented on OOZIE-2775:
---
[~andras.piros], [~gezapeti] can you take a look at the first patch? Do you see
any problems with the current fail-fast approach? Would it be enough to call
{{Services.destroy()}} or some extra care is required? If so, could you
recommend a better way to gracefully stop Oozie services that might have been
already initialized and started during startup? Thanks.
> Oozie server does not stop if there is an exception during service
> initalization at startup
> ---
>
> Key: OOZIE-2775
> URL: https://issues.apache.org/jira/browse/OOZIE-2775
> Project: Oozie
> Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2775-01.patch
>
>
> OOZIE-2666 brought in embedded jetty in Oozie. {{com.google.inject.guice}}
> was selected for dependency management. Unfortunately if there is an
> exception at starting Oozie services (e.g. JPAService initialization, called
> via
> https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L194),
> Oozie won't stop if it cannot destroy Oozie services.
> In OOZIE-2666 a shutdown hook
> (https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L184)
> was added to print out startup errors and exceptions, destroy pending oozie
> services, and exit eventually, but it does not work.
> Steps to reproduce a startup failure (for example simulate that Oozie cannot
> find required database driver):
> {code}
> $ cd oozie
> $ mvn clean install assembly:single -DskipTests -DjavaVersion=1.8
> -DtargetVersion=1.7 -Phadoop-2 -Dhadoop.version=2.6.0 -Puber;
> $ cd distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/
> $ export JETTY_OPTS="-Djavax.net.debug=all
> -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
> -Dlog4j.logger.org.eclipse.jetty=TRACE
> -Dlog4j.configuration=./conf/oozie-log4j.properties"
> $ bin/oozied.sh run
> $ mv embedded-oozie-server/webapp/WEB-INF/lib/derby-10.10.1.1.jar ..
>
> {code}
> Exception is printed to stdout
> {code}
>
> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver
> class 'org.apache.derby.jdbc.EmbeddedDriver'
> at
> org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:106)
> at
> org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
> at
> org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1520)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:533)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:458)
> at
> org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:121)
> at
> org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
> at
> org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:642)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:202)
> at
> org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:154)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:226)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:153)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:59)
> at
> org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:514)
> at org.apache.oozie.service.JPAService.init(JPAService.java:215)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:387)
> at org.apache.oozie.service.Services.setService(Services.java:373)
> at org.apache.oozie.service.Services.loadServices(Services.java:305)
> at org.apache.oozie.service.Services.init(Services.java:213)
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:31)
> at
[jira] [Commented] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328591#comment-16328591 ] Attila Sasvari commented on OOZIE-3162: --- [~andras.piros] thanks for the review, committed to master > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Fix For: trunk > > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch, OOZIE-3162-003.patch, OOZIE-3162-004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3159) Spark Action fails because of absence of hadoop mapreduce jar(s)
[
https://issues.apache.org/jira/browse/OOZIE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-3159:
--
Attachment: OOZIE-3159-004.patch
> Spark Action fails because of absence of hadoop mapreduce jar(s)
>
>
> Key: OOZIE-3159
> URL: https://issues.apache.org/jira/browse/OOZIE-3159
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Attila Sasvari
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3159-001.patch, OOZIE-3159-002.patch,
> OOZIE-3159-003.patch, OOZIE-3159-004.patch
>
>
> OOZIE-2869 removed map reduce dependencies from getting added to Spark
> action. Spark action uses
> org.apache.hadoop.filecache.DistributedCache. It is not available anymore in
> Spack action's classpath, causing it to fail.
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:56)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:142)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/filecache/DistributedCache
> at
> org.apache.oozie.action.hadoop.SparkArgsExtractor.extract(SparkArgsExtractor.java:309)
> at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:74)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101)
> at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:60)
> ... 16 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.filecache.DistributedCache
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 20 more
> Failing Oozie Launcher, org/apache/hadoop/filecache/DistributedCache
> java.lang.NoClassDefFoundError: org/apache/hadoop/filecache/DistributedCache
> at
> org.apache.oozie.action.hadoop.SparkArgsExtractor.extract(SparkArgsExtractor.java:309)
> at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:74)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101)
> at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:56)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:142)
> Caused by:
[jira] [Commented] (OOZIE-1401) PurgeCommand should purge the workflow jobs w/o end_time
[
https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328511#comment-16328511
]
Attila Sasvari commented on OOZIE-1401:
---
committed to master
> PurgeCommand should purge the workflow jobs w/o end_time
>
>
> Key: OOZIE-1401
> URL: https://issues.apache.org/jira/browse/OOZIE-1401
> Project: Oozie
> Issue Type: Sub-task
> Components: bundle, coordinator, workflow
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Attila Sasvari
>Priority: Major
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-1401-001.patch, OOZIE-1401.amend.003.patch,
> amend-OOZIE-1401-001.patch, amend-OOZIE-1401-002.patch
>
>
> Currently, {{PurgeXCommand}} logic is not working with those workflow jobs
> with {{end_time=null}}. This command needs to take care of those jobs as
> well. This happens in the case of long stuck jobs after Hadoop restarts or DB
> failures. It could be done by checking {{last_modified_time}} instead, if
> {{end_time}} is not available.
> The current query:
> {code:sql}
> select w from WorkflowJobBean w where w.endTimestamp < :endTime
> {code}
> There is also an issue when:
> * there is a parent workflow that has its {{end_time}} set
> * is otherwise eligible for {{PurgeXCommand}}: {{end_time}} is older than
> configured number of days, and has {{status}} either {{KILLED}}, or
> {{FAILED}}, or {{SUCCEEDED}}
> * has a child workflow that has the {{parent_id}} set to the {{id}} of the
> parent workflow
> * child workflow has its {{end_time = NULL}}
> In this case,
> [*{{PurgeXCommand#fetchTerminatedWorkflow()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L249]
> throws a {{NullPointerException}} like this:
> {noformat}
> 2017-09-29 07:59:46,365 DEBUG org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] Purging workflows of long running coordinators is turned on
> 2017-09-29 07:59:46,371 DEBUG org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] Execute command [purge] key [null]
> 2017-09-29 07:59:46,371 INFO org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] STARTED Purge to purge Workflow Jobs older than [1] days,
> Coordinator Jobs older than [1] days, and Bundlejobs older than [1] days.
> 2017-09-29 07:59:46,375 ERROR org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] Exception,
> java.lang.NullPointerException
> at
> org.apache.oozie.command.PurgeXCommand.fetchTerminatedWorkflow(PurgeXCommand.java:249)
> at
> org.apache.oozie.command.PurgeXCommand.processWorkflowsHelper(PurgeXCommand.java:227)
> at
> org.apache.oozie.command.PurgeXCommand.processWorkflows(PurgeXCommand.java:199)
> at
> org.apache.oozie.command.PurgeXCommand.execute(PurgeXCommand.java:150)
> at org.apache.oozie.command.PurgeXCommand.execute(PurgeXCommand.java:53)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328508#comment-16328508 ] Attila Sasvari commented on OOZIE-3162: --- [~gezapeti] can you take a look at patch 004? > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch, OOZIE-3162-003.patch, OOZIE-3162-004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3162: -- Attachment: OOZIE-3162-004.patch > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch, OOZIE-3162-003.patch, OOZIE-3162-004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328454#comment-16328454 ] Attila Sasvari commented on OOZIE-3162: --- Test comment from curl... > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch, OOZIE-3162-003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3162: -- Attachment: OOZIE-3162-003.patch > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch, OOZIE-3162-003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3162: -- Comment: was deleted (was: Test comment from curl... ) > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3162: -- Comment: was deleted (was: comment.) > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3162: -- Attachment: OOZIE-3162-002.patch > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch, > OOZIE-3162-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327691#comment-16327691 ] Attila Sasvari commented on OOZIE-3162: --- Test comment from curl... > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1401) PurgeCommand should purge the workflow jobs w/o end_time
[
https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327655#comment-16327655
]
Attila Sasvari commented on OOZIE-1401:
---
https://builds.apache.org/job/PreCommit-OOZIE-Build/323/console
{code:java}
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
-1 the patch contains 2 line(s) longer than 132 characters
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2087
Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch {code}
> PurgeCommand should purge the workflow jobs w/o end_time
>
>
> Key: OOZIE-1401
> URL: https://issues.apache.org/jira/browse/OOZIE-1401
> Project: Oozie
> Issue Type: Sub-task
> Components: bundle, coordinator, workflow
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Attila Sasvari
>Priority: Major
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-1401-001.patch, OOZIE-1401.amend.003.patch,
> amend-OOZIE-1401-001.patch, amend-OOZIE-1401-002.patch
>
>
> Currently, {{PurgeXCommand}} logic is not working with those workflow jobs
> with {{end_time=null}}. This command needs to take care of those jobs as
> well. This happens in the case of long stuck jobs after Hadoop restarts or DB
> failures. It could be done by checking {{last_modified_time}} instead, if
> {{end_time}} is not available.
> The current query:
> {code:sql}
> select w from WorkflowJobBean w where w.endTimestamp < :endTime
> {code}
> There is also an issue when:
> * there is a parent workflow that has its {{end_time}} set
> * is otherwise eligible for {{PurgeXCommand}}: {{end_time}} is older than
> configured number of days, and has {{status}} either {{KILLED}}, or
> {{FAILED}}, or {{SUCCEEDED}}
> * has a child workflow that has the {{parent_id}} set to the {{id}} of the
> parent workflow
> * child workflow has its {{end_time = NULL}}
> In this case,
> [*{{PurgeXCommand#fetchTerminatedWorkflow()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L249]
> throws a {{NullPointerException}} like this:
> {noformat}
> 2017-09-29 07:59:46,365 DEBUG org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] Purging workflows of long running coordinators is turned on
> 2017-09-29 07:59:46,371 DEBUG org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] Execute command [purge] key [null]
> 2017-09-29 07:59:46,371 INFO org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] STARTED Purge to purge Workflow Jobs older than [1] days,
> Coordinator Jobs older than [1] days, and Bundlejobs older than [1] days.
> 2017-09-29 07:59:46,375 ERROR org.apache.oozie.command.PurgeXCommand:
> SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] Exception,
> java.lang.NullPointerException
> at
> org.apache.oozie.command.PurgeXCommand.fetchTerminatedWorkflow(PurgeXCommand.java:249)
> at
> org.apache.oozie.command.PurgeXCommand.processWorkflowsHelper(PurgeXCommand.java:227)
> at
>
[jira] [Updated] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3162: -- Attachment: OOZIE-3162-001.patch > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch, OOZIE-3162-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3162) Precommit build fails to add comment to JIRA
[ https://issues.apache.org/jira/browse/OOZIE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327600#comment-16327600 ] Attila Sasvari commented on OOZIE-3162: --- comment. > Precommit build fails to add comment to JIRA > > > Key: OOZIE-3162 > URL: https://issues.apache.org/jira/browse/OOZIE-3162 > Project: Oozie > Issue Type: Bug > Components: build > Environment: Oozie precommit build, > [https://builds.apache.org/job/PreCommit-OOZIE-Build/321/,] failed to add > comment to JIRA > Reason: > Adding comment to JIRA > Unable to log in to server: > [https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2] with user: > hadoopqa. > It is related to the JIRA major update on 1/14. As per INFRA-15843: > "The SOAP RPC methods have been deprecated since 2013 and were completely > removed in Jira 7. You'll have to make use of the REST API instead" > We need to update the Oozie precommit job because it currently relies on > jira-cli-2.6.0. If we upgrade to atlassian-cli-7.5.0 and use jira.sh, it > shall work. >Reporter: Attila Sasvari >Assignee: Attila Sasvari >Priority: Major > Attachments: OOZIE-3162-000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
