[jira] [Created] (OOZIE-3451) Example in Git action's documentation refers to incorrect worflow version
Attila Sasvari created OOZIE-3451:
-
Summary: Example in Git action's documentation refers to incorrect
worflow version
Key: OOZIE-3451
URL: https://issues.apache.org/jira/browse/OOZIE-3451
Project: Oozie
Issue Type: Bug
Components: docs
Reporter: Attila Sasvari
https://oozie.apache.org/docs/5.1.0/DG_GitActionExtension.html uses an example
with
{{}}
That is incorrect as per:
https://github.com/apache/oozie/blob/master/client/src/main/resources/git-action-1.0.xsd
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release Oozie 5.1.0 (candidate 2)
Hi there, +1 (but please see my observations below) Here is what I did: - built Oozie with Hadoop version 2.7.5, installed Oozie sharelib, started the server, - executed example workflows using a single node, pseudo Hadoop cluster, - clicked through the Oozie Web UI to validate basic functionality (checked job statuses, logs, DAG), - created a demo workflow using the Oozie Fluent job API, validated and submitted via Oozie CLI, - verified the issues reported on RC 1 are now fixed. Some observations: - I was hit by OOZIE-3378. As I understand that it will be included in the next minor release. - Some of the error messages contain error code twice. For example: "Error: E0307 : E0307: Runtime error [App directory [hdfs://localhost:9000/user/root/examples/apps/subwf] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]]" - I noticed that when we validate workflows generated via the fluent job API using the Oozie CLI with "-validatejar" and "--verbose" options, temporary XML files are created in a world readable directory (/tmp) on the client side. It is not a good thing, but I am not sure if it qualifies as a blocker. What do you think? If we consider it a blocker, then OOZIE-3378 could be also included in the next RC. - Attila On Wed, Dec 12, 2018 at 5:15 PM Andras Salamon wrote: > Thanks Peter, > > 1) I've opened an upstream jira: > https://issues.apache.org/jira/browse/OOZIE-3401 > > Sala > > On Wed, Dec 12, 2018 at 5:05 PM Andras Piros > wrote: > > > Thanks Peter, > > > > 2) resembles *PIG-5253 <http://PIG-5253>* which is in progress. Oozie's > > Hadoop 3 support efforts are tracked under *OOZIE-2973 > > <https://issues.apache.org/jira/browse/OOZIE-2973>*. > > > > Regards, > > > > Andras > > > > On Wed, Dec 12, 2018 at 3:30 PM Peter Bacsko > > > > > wrote: > > > > > +1 (binding, conditional) > > > > > > - verified hash > > > - verified signature > > > - built distro & ran all tests > > > - created Hadoop pseudo dist cluster, executed sample Java workflow > > > > > > Things I observed: > > > 1. TestPyspark constantly fails locally, but passes in Jenkins. Please > > > investigate that it's a test failure and create a JIRA to track it. > > > 2. Not possible to build distro if -Dhadoop.version is set to 2.9.2. > > Please > > > confirm that it's not Oozie's fault. > > > > > > Peter > > > > > > > > > On Mon, Dec 10, 2018 at 1:50 PM Andras Piros > wrote: > > > > > > > Hi, > > > > > > > > as no votes were cast so far extending the deadline by one week, > until > > > > Friday, December 14, 2018 at 12:00am PDT. > > > > > > > > Please check the new RC and cast your vote accordingly. > > > > > > > > Thanks, > > > > > > > > Andras > > > > > > > > > > > > On Mon, Dec 3, 2018 at 2:29 PM Andras Piros > wrote: > > > > > > > > > Hi there, > > > > > > > > > > I have created a build for Oozie 5.1.0, release candidate 2. > > > > > > > > > > This release introduces following major changes and fixes: > > > > > - Fluent Job API > > > > > - Git action > > > > > - reduced heap waste > > > > > - switched documentation from TWiki to Markdown > > > > > - callable queue improvements > > > > > - Spark action fixes > > > > > - SLA improvements > > > > > - third party dependency revamp > > > > > - fixed CVE-2018-11799 > > > > > > > > > > Keys to verify the signature of the release artifact are available > at > > > > > > > > > > http://www.apache.org/dist/oozie/KEYS > > > > > > > > > > Please download, test, and try it out: > > > > > > > > > > http://people.apache.org/~andras/oozie-5.1.0-rc2/ > > > > > > > > > > The release candidate, sha512 signature, gpg signature, and rat > > report > > > > can > > > > > all be found > > > > > at the above address. > > > > > > > > > > The vote is open until Friday, December 7, 2018 at 12:00am PDT. > > > > > > > > > > Thanks, > > > > > > > > > > Andras Piros > > > > > > > > > > > > > > > -- -- Attila Sasvari Software Engineer <http://www.cloudera.com/>
[jira] [Created] (OOZIE-3395) Findbugs is no longer maintained
Attila Sasvari created OOZIE-3395:
-
Summary: Findbugs is no longer maintained
Key: OOZIE-3395
URL: https://issues.apache.org/jira/browse/OOZIE-3395
Project: Oozie
Issue Type: Task
Reporter: Attila Sasvari
https://gleclaire.github.io/findbugs-maven-plugin/
{quote}
Status: Since Findbugs is no longer maintained, please use Spotbugs which has a
Maven plugin.
{quote}
The plugin author recommends to migrate to Spotbugs: https://spotbugs.github.io/
It might worth to investigate this plugin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release Oozie 5.1.0 (candidate 1)
Hi, Thanks for driving the release Andras. -1 Here is what I did: - Tested with Hadoop 2.7.5, Java 8 - signature, checksum is fine - rat report is ok - distro builds - I submitted example workflows and coordinators and examined jobs via the Oozie Web UI, I have found and filed the following issues: https://issues.apache.org/jira/browse/OOZIE-3386 https://issues.apache.org/jira/browse/OOZIE-3389 https://issues.apache.org/jira/browse/OOZIE-3390 - I ran mkdistro.sh, then executed "mvn site" that failed with the following [ERROR] Failed to execute goal on project oozie-client: Could not resolve dependencies for project org.apache.oozie:oozie-client:jar:5.1.0: Could not find artifact org.apache.oozie:oozie-fluent-job-api:jar:5.1.0 in central ( http://repo1.maven.org/maven2) -> [Help 1] ( Installing dependencies locally solves this problem ) Please fix OOZIE-3386, OOZIE-3389, OOZIE-3390 and create a new RC. Best regards, - Attila On Wed, Nov 14, 2018 at 2:29 PM Andras Piros wrote: > Thanks Gp, > > extending the vote by two weeks (Fri Nov 23, 12pm PDT) to see what others > find. > > Andras > > On Tue, Nov 13, 2018 at 2:01 PM Peter Cseh > wrote: > > > Hi! > > > > Tested with Hadoop2.7.2 > > - signature, checksum is fine > > - rat report is ok > > - distro builds > > - tried some of the examples (MR, Hive, Spark), they are running fine. > > > > +1 > > > > > > On Tue, Nov 13, 2018 at 12:59 PM Andras Piros > > wrote: > > > > > Thanks Kinga for checking out RC1, and good to hear it passed from your > > > perspective. > > > > > > Regards, > > > > > > Andras > > > > > > On Tue, Nov 13, 2018 at 12:36 PM Julia Kinga Marton > > > wrote: > > > > > > > Hi, > > > > > > > > I made some testing on the RC, and it seems OK. > > > > > > > > Here is what I checked: > > > > - sha512 > > > > - gpg signature > > > > - rat report > > > > - built Oozie against Hadoop-2.6.0 > > > > - ran a few examples (wf and coordinator), clicked the Web UI, ran > some > > > CLI > > > > commands > > > > - checked Fluent Job API by running Java and Shell examples > > > > - looked briefly at the documentation > > > > > > > > +1 from my side (non-binding) > > > > > > > > Best regards, > > > > Kinga > > > > > > > > On Mon, Nov 5, 2018 at 8:44 PM Andras Piros > wrote: > > > > > > > > > Hi there, > > > > > > > > > > I have created a build for Oozie 5.1.0, release candidate 1. > > > > > > > > > > This release introduces following major changes and fixes: > > > > > - Fluent Job API > > > > > - Git action > > > > > - reduced heap waste > > > > > - switched documentation from TWiki to Markdown > > > > > - callable queue improvements > > > > > - Spark action fixes > > > > > - SLA improvements > > > > > - third party dependency revamp > > > > > - fixed CVE-2018-11799 > > > > > > > > > > Keys to verify the signature of the release artifact are available > at > > > > > > > > > > http://www.apache.org/dist/oozie/KEYS > > > > > > > > > > Please download, test, and try it out: > > > > > > > > > > http://people.apache.org/~andras/oozie-5.1.0-rc1/ > > > > > > > > > > The release candidate, sha512 signature, gpg signature, and rat > > report > > > > can > > > > > all be found > > > > > at the above address. > > > > > > > > > > The vote is open until Friday, November 9, 2018 at 12:00am PDT. > > > > > > > > > > Thanks, > > > > > > > > > > Andras Piros > > > > > > > > > > > > > > > > > > -- > > *Peter Cseh *| Software Engineer > > cloudera.com <https://www.cloudera.com> > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > -- > > > -- -- Attila Sasvari Software Engineer <http://www.cloudera.com/>
[jira] [Created] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
Attila Sasvari created OOZIE-3390:
-
Summary: shell action's stderr contains a bogus error message
(since 5.0.0)
Key: OOZIE-3390
URL: https://issues.apache.org/jira/browse/OOZIE-3390
Project: Oozie
Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari
Shell action's stderr contains an error message about path.
I executed shell action example to test Oozie 5.1 rc1, the job finished with
success, but stderr of the job contained "Path echo doesn't appear to exist".
https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
{code}
writeLine(errorStream, "Path " + filename + " doesn't appear to
exist");
{code}
The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3389) NPE after OOZIE-3370
Attila Sasvari created OOZIE-3389:
-
Summary: NPE after OOZIE-3370
Key: OOZIE-3389
URL: https://issues.apache.org/jira/browse/OOZIE-3389
Project: Oozie
Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari
I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
clicked on one of the coordinator jobs on the Oozie Web UI, but the details of
the job were not displayed. I noticed the following NPE in the server log:
{code}
2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] XException,
org.apache.oozie.command.CommandException: E1028: Coord input logic error. null
at
org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
at
org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
at org.apache.oozie.command.XCommand.call(XCommand.java:291)
at
org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
at
org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3386) Misleading error message when workflow application does not exist
Attila Sasvari created OOZIE-3386:
-
Summary: Misleading error message when workflow application does
not exist
Key: OOZIE-3386
URL: https://issues.apache.org/jira/browse/OOZIE-3386
Project: Oozie
Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari
Using 5.1.0 rc1, I tried to run an example workflow. Because of user error
{{oozie.wf.application.path}} in job.properties pointed to a directory in HDFS
that did not exist. Upon submitting the workflow, the following was returned
{code}
bin/oozie job -oozie http://localhost:11000/oozie -config
examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run
-DnameNode=hdfs://localhost:9000
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Error: E0307 : E0307: Runtime error [Configuration entry
oozie.jobs.api.generated.xml not present]
{code}
The server.log contained the following:
{code}
2018-11-19 15:07:13,244 WARN V1JobsServlet:523 -
SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] URL[POST
http://localhost:11000/oozie/v2/jobs?action=start=asasvari] error[E0307],
E0307: Runtime error [Configuration entry oozie.jobs.api.generated.xml not
present]
org.apache.oozie.servlet.XServletException: E0307: Runtime error [Configuration
entry oozie.jobs.api.generated.xml not present]
at
org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
at
org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run
[jira] [Commented] (OOZIE-3354) [core] [SSH action] SSH action gets hung
[
https://issues.apache.org/jira/browse/OOZIE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631560#comment-16631560
]
Attila Sasvari commented on OOZIE-3354:
---
[~andras.piros] thanks for the patch! I have run into an issue when an SSH
action with {{}} got hung after an Oozie upgrade.
Can we add some sleep (say 100ms) when an IllegalThreadStateException exception
is thrown to reduce busy-waiting?
> [core] [SSH action] SSH action gets hung
>
>
> Key: OOZIE-3354
> URL: https://issues.apache.org/jira/browse/OOZIE-3354
> Project: Oozie
> Issue Type: Bug
> Components: action, core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3354.001.patch
>
>
> In OOZIE-3183 {{SshActionExecutor#drainBuffers()}} has changed. Previously,
> it called {{Process#exitCode()}} that would return immediately either with
> the exit code, or would throw an {{IllegalThreadStateException}} if the
> process would still be running.
> In the current implementation introduced by OOZIE-3183, {{Process#waitFor()}}
> is used that would block until the process finishes. Given the fact that
> sometime {{SshActionExecutor#check()}} calls {{ssh ... cat stdout}}, and this
> SSH process can be trapped even after {{cat stdout}} has been finished on the
> target host, it can happen that {{SshActionExecutor#drainBuffers()}} waits
> indefinitely without a chance to gather any {{stdout}} or {{stderr}} logs.
> Hence this particular one is a compatibility breaking change with existing
> SSH action behavior.
> Let's re-introduce the former behavior in
> {{SshActionExecutor#drainBuffers()}} that keeps polling
> {{Process#exitValue()}} and reading the progress on {{stdout}} and {{stderr}}
> till the process finishes, for backwards compatibility.
> [This
> article|https://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html]
> covers the behavioral details of {{Process#waitFor()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3210) revision information is empty
[
https://issues.apache.org/jira/browse/OOZIE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593345#comment-16593345
]
Attila Sasvari commented on OOZIE-3210:
---
[~dbist13] how did you build Oozie? Did you run {{bin/mkdistro.sh}}?
> revision information is empty
> -
>
> Key: OOZIE-3210
> URL: https://issues.apache.org/jira/browse/OOZIE-3210
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297;
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.17.1.el7.x86_64", arch: "amd64",
> family: "unix"{code}
>Reporter: Artem Ervits
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.1.0
>
>
> compiled from RC0
> {code:java}
> STARTUP MSG: Oozie BUILD_VERSION [5.0.0] compiled by [oozie] on
> [2018.04.02-19:55:20GMT]
> STARTUP MSG: revision [unavailable]@[unavailable]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3319) Log SSH action callback error output
[
https://issues.apache.org/jira/browse/OOZIE-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574564#comment-16574564
]
Attila Sasvari commented on OOZIE-3319:
---
[~ericlin] thanks for the patch!
Remarks:
- If we redirect the standard output of the callback command to stdout, it may
interfere with the actual command's standard output; is not desirable. See
https://github.com/apache/oozie/blob/e229e4dbb42b7bc0fdb866ccc530b533a2b2008b/core/src/main/resources/ssh-wrapper.sh#L41
{code}
if $cmnd "$@" >>${stdout} 2>>${stderr}; then
{code}
- I checked the new wrapper: upon a successful execution, the stderr file will
contain
{code}
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
^M 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0^M100 50 0 100 5 0691 --:--:-- --:--:-- --:--:--
714
{code}
- Upon failure:
{code}
Not able to perform callback operation.
Callback command: curl
http://localhost:21000/oozie/callback?id=000-180809004852667-oozie-oozi-W@ssh-6185=OK
--data-binary
@/root/oozie-oozi/000-180809004852667-oozie-oozi-W.bak/ssh-6185--ssh/20907.000-180809004852667-oozie-oozi-W@ssh-6185@0.stdout
--request POST --header content-type:text/plain
{code}
I would suggest to only log the callback command in the failure case, and do
not redirect stdout.
- Did you consider to add a new test case to TestSshActionExecutor with an
invalid callback URL?
> Log SSH action callback error output
>
>
> Key: OOZIE-3319
> URL: https://issues.apache.org/jira/browse/OOZIE-3319
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Minor
> Attachments: OOZIE-3319.patch
>
>
> When SSH callback fails, there is no error message logged to tell us what was
> going on
> {code}
> if $callback; then
> echo "Callback operation successful."
> else
> echo "Not able to perform callback operation." >>${stderr}
> fi
> {code}
> The only message showed in the remote machine's log was "Not able to perform
> callback operation." which is useless.
> Request to log the stderr out of the $callback action to stderr log as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OOZIE-3086) Oozie shell action ignores absolute path
[
https://issues.apache.org/jira/browse/OOZIE-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari resolved OOZIE-3086.
---
Resolution: Duplicate
> Oozie shell action ignores absolute path
>
>
> Key: OOZIE-3086
> URL: https://issues.apache.org/jira/browse/OOZIE-3086
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 3.1.3, 3.2.0, 3.3.0, trunk, 3.3.1, 3.3.2, 4.0.0, 4.1.0,
> 4.0.1, 4.2.0, 4.3.0
> Reporter: Attila Sasvari
>Priority: Major
>
> [ShellActionExecutor's setupActionConf() method|
> https://github.com/apache/oozie/blob/883c7556d2f302261eae5aec6e323a5b1de04f74/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java#L57]
> retrieves only the *basename* for the shell script specified by the user.
> {code}
> Configuration setupActionConf(Configuration actionConf, Context context,
> Element actionXml, Path appPath)
> {
> ...
> String exec = actionXml.getChild("exec", ns).getTextTrim();
> String execName = new Path(exec).getName();
> actionConf.set(ShellMain.CONF_OOZIE_SHELL_EXEC, execName);
> {code}
> As a result, {{ShellMain}} will
> [execute|https://github.com/apache/oozie/blob/883c7556d2f302261eae5aec6e323a5b1de04f74/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java#L356]
> the default command available in $PATH.
> {code}
> protected String getExec(Configuration actionConf) {
> String exec = actionConf.get(CONF_OOZIE_SHELL_EXEC);
> {code}
> Consequences:
> - This behaviour makes it also hard/impossible to use custom executable at a
> given location (that is available on each node where a shell action can run,
> for example a specific python version).
> - Even if a user specifies a wrong path (e.g. a non-existing {{/usr/bn/id}})
> on purpose to force failure of a shell action, the command might succeed with
> no apparent reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3320) Oozie ShellAction should support absolute bash file path
[
https://issues.apache.org/jira/browse/OOZIE-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574357#comment-16574357
]
Attila Sasvari commented on OOZIE-3320:
---
Thanks [~roeyshemtov24]. It duplicates
[OOZIE-3086|https://issues.apache.org/jira/browse/OOZIE-3086].
> Oozie ShellAction should support absolute bash file path
>
>
> Key: OOZIE-3320
> URL: https://issues.apache.org/jira/browse/OOZIE-3320
> Project: Oozie
> Issue Type: Improvement
> Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Roey Shem Tov
>Assignee: Roey Shem Tov
>Priority: Major
> Labels: patch
> Attachments: OOZIE-3086.patch
>
>
> bash files that saved on shared mount, cannot be execute by the ShellAction
> in proper way.
> Example:
> Worker-1,Worker-2,Worker-3 have shared mount /mnt/hadoop
> on /mnt/hadoop there is a file script.sh
> Right now there is two options to submit it using ShellAction:
> # Upload it to hdfs, add it as a file and submit script.sh
> # use bash as exec and file location (/mnt/hadoop/script.sh) as argument
> (e.g bash>/exec>/mnt/hadoop/script.sh
> Best option is that the command will support shared mounted file :
> /mnt/hadoop/script.sh
>
> This code is taking only the file name instead it full path:
> {code:java}
> String exec = actionXml.getChild("exec", ns).getTextTrim();
> String execName = new Path(exec).getName();
> actionConf.set(ShellMain.CONF_OOZIE_SHELL_EXEC, execName);
> {code}
>
> Best option to support shared mount file is to support file:// starting for
> bash files that are local (or shared by mount), e.g:
> {code:java}
> String exec = actionXml.getChild("exec", ns).getTextTrim(); String execName;
> String localFilePrefix = "file://";
> // When exec starts with 'file://' refer it as local file.
> if (exec.startsWith(localFilePrefix))
> execName = exec.substring(localFilePrefix.length());
> else execName = new Path(exec).getName();
> actionConf.set(ShellMain.CONF_OOZIE_SHELL_EXEC, execName);
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[ https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571498#comment-16571498 ] Attila Sasvari commented on OOZIE-3298: --- Can you file a reviewboard request? > OYA: external ID is not filled properly and failing MR job is treated as > SUCCEEDED > -- > > Key: OOZIE-3298 > URL: https://issues.apache.org/jira/browse/OOZIE-3298 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Bacsko >Assignee: Andras Piros >Priority: Blocker > Fix For: 5.1.0 > > Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, > OOZIE-3298.003.patch > > > When a MapReduce action is launched from Oozie (OYA), we don't properly fill > the external ID field. It gets populated with the YARN id of the LauncherAM, > not with the id of the actual MR job. If the MR job is succesfully submitted > then fails, it will be treated as a successfully executed action, which is > very misleading and can potentially confuse Oozie users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders
[
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549315#comment-16549315
]
Attila Sasvari commented on OOZIE-2829:
---
If you look at how Hadoop components are handling/pass multiple files/jars, you
pretty often see a comma separated list of files:
- YARN / Map-Reduce: {{--libjars}}:
https://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options
- Apache Spark: spark-submit
https://spark.apache.org/docs/latest/submitting-applications.html
- Apache Pig - pig.additional.jars.uris
- Apache Oozie - Sharelib mapping file:
https://oozie.apache.org/docs/5.0.0/AG_Install.html#Oozie_Share_Lib
I think it would be better to specify a comma separated list of files for a
given extra sharelib.
> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
> Issue Type: Bug
> Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch,
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch,
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder.
> It would be nice to have the possibility to define additional folders to be
> uploaded into the sharelib, so the users don't have to copy or link the files
> together on their machine.
> The syntax could be something like -additional-lib
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[ https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546559#comment-16546559 ] Attila Sasvari commented on OOZIE-3298: --- [~andras.piros] thanks for the patch! Can you add a new test case that validates the changes? TestMapReduceMain perhaps? > OYA: external ID is not filled properly and failing MR job is treated as > SUCCEEDED > -- > > Key: OOZIE-3298 > URL: https://issues.apache.org/jira/browse/OOZIE-3298 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Bacsko >Assignee: Andras Piros >Priority: Critical > Fix For: 5.1.0 > > Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch > > > When a MapReduce action is launched from Oozie (OYA), we don't properly fill > the external ID field. It gets populated with the YARN id of the LauncherAM, > not with the id of the actual MR job. If the MR job is succesfully submitted > then fails, it will be treated as a successfully executed action, which is > very misleading and can potentially confuse Oozie users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3301) Update NOTICE file
[ https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532431#comment-16532431 ] Attila Sasvari commented on OOZIE-3301: --- Thanks for digging into it, and filing the JIRA about the broken link to the Dependencies Report, Licenses section. Here are some examples from other Apache projects: - Hadoop: https://github.com/apache/hadoop/blob/trunk/NOTICE.txt - HBase: https://github.com/apache/hbase/blob/master/NOTICE.txt - Spark: https://github.com/apache/spark/blob/master/NOTICE-binary - Kafka: https://github.com/apache/kafka/blob/trunk/NOTICE - Flink: https://github.com/apache/flink/blob/master/NOTICE [Oozie license text file|https://github.com/apache/oozie/blob/master/LICENSE.txt] lists many licenses but there are dependencies (such as jetty, netty, gmetric4j, JSch) that are not mentioned / license is missing. Maven has a [license plugin|https://www.mojohaus.org/license-maven-plugin/] that migth be useful, but it needs further investigation. > Update NOTICE file > -- > > Key: OOZIE-3301 > URL: https://issues.apache.org/jira/browse/OOZIE-3301 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 > Reporter: Attila Sasvari >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3301-001.patch > > > Oozie NOTICE file is not up to date. Please overview and update it as per > http://www.apache.org/licenses/LICENSE-2.0.html#redistribution > For example: Oozie uses embedded jetty, however Eclipse license > (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is > not listed in > [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3301) Update NOTICE file
[ https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531427#comment-16531427 ] Attila Sasvari commented on OOZIE-3301: --- Hi [~kmarton] thanks for working on it, we should ideally review all the licenses of included Oozie dependencies (extjs etc). Maybe you could use a maven plugin or some tool to automate this. > Update NOTICE file > -- > > Key: OOZIE-3301 > URL: https://issues.apache.org/jira/browse/OOZIE-3301 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 > Reporter: Attila Sasvari >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3301-001.patch > > > Oozie NOTICE file is not up to date. Please overview and update it as per > http://www.apache.org/licenses/LICENSE-2.0.html#redistribution > For example: Oozie uses embedded jetty, however Eclipse license > (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is > not listed in > [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2392) Add Kafka support for notifications
[ https://issues.apache.org/jira/browse/OOZIE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531375#comment-16531375 ] Attila Sasvari commented on OOZIE-2392: --- It is an interesting idea that would help to integrate Oozie jobs with external systems. I am wondering if you are still working on it, [~rupalph]? > Add Kafka support for notifications > --- > > Key: OOZIE-2392 > URL: https://issues.apache.org/jira/browse/OOZIE-2392 > Project: Oozie > Issue Type: New Feature > Components: core >Reporter: Robert Kanter >Assignee: Rupal H >Priority: Major > > Oozie currently lets you send JMS and email notifications. It could be > useful to add support for Kafka as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3301) Update NOTICE file
Attila Sasvari created OOZIE-3301: - Summary: Update NOTICE file Key: OOZIE-3301 URL: https://issues.apache.org/jira/browse/OOZIE-3301 Project: Oozie Issue Type: Bug Affects Versions: 5.0.0 Reporter: Attila Sasvari Oozie NOTICE file is not up to date. Please overview and update it as per http://www.apache.org/licenses/LICENSE-2.0.html#redistribution For example: Oozie uses embedded jetty, however Eclipse license (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is not listed in [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders
[
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516813#comment-16516813
]
Attila Sasvari commented on OOZIE-2829:
---
[~andras.piros], [~pbacsko], [~kmarton] or anyone who reads it: I don't have to
work on this improvement. Feel free to take it over.
> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
> Issue Type: Bug
> Components: tools
>Reporter: Peter Cseh
> Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch,
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder.
> It would be nice to have the possibility to define additional folders to be
> uploaded into the sharelib, so the users don't have to copy or link the files
> together on their machine.
> The syntax could be something like -additional-lib
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3224) Upgrade jetty version to 9.3
Attila Sasvari created OOZIE-3224: - Summary: Upgrade jetty version to 9.3 Key: OOZIE-3224 URL: https://issues.apache.org/jira/browse/OOZIE-3224 Project: Oozie Issue Type: Task Affects Versions: 5.0.0 Reporter: Attila Sasvari Jetty 9.2 has reached end of life as of Jan 2018 http://www.eclipse.org/jetty/documentation/current/what-jetty-version.html#d0e203 Current version used in Oozie 5.0: 9.2.19.v20160908 Please upgrade to a newer Jetty version 9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-2836) Fix ps1 and cmd scripts to make Oozie runnable on Windows
[
https://issues.apache.org/jira/browse/OOZIE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari reassigned OOZIE-2836:
-
Assignee: (was: Attila Sasvari)
> Fix ps1 and cmd scripts to make Oozie runnable on Windows
> -
>
> Key: OOZIE-2836
> URL: https://issues.apache.org/jira/browse/OOZIE-2836
> Project: Oozie
> Issue Type: Bug
> Reporter: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-2836-01.patch
>
>
> With OOZIE-2666 Oozie was migrated from Tomcat to embedded jetty. Distro
> shell scripts were updated, but Windows scripts were not adjusted in
> {{distro/src/main/bin}}.
> It would be nice if someone with a Windows machine could fix the following
> scripts:
> - oozie-setup.cmd
> - oozie-setup.ps1
> - oozie-sys.cmd
> - oozie.cmd
> - oozied.cmd
> and verify that Oozie works on Windows.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release Oozie 5.0.0 (candidate 0)
; Keys to verify the signature of the release artifact are available at > >> > >> http://www.apache.org/dist/oozie/KEYS > >> > >> Please download, test, and try it out: > >> > >> http://people.apache.org/~andras/oozie-5.0.0-rc0/ > >> > >> The release candidate, sha512 signature, gpg signature, and rat report can > >> all be found > >> at the above address. > >> > >> The vote is open until Saturday, March 31, 2018 at 12:00am PDT. > >> > >> Thanks, > >> > >> Andras Piros > >> > > > > -- -- Attila Sasvari Software Engineer
Re: Review Request 66084: Oozie-core fails with checkstyle errors.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66084/#review199384 --- core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java Lines 44 (patched) <https://reviews.apache.org/r/66084/#comment279726> It breaks the precommit build (mkdistro, see https://builds.apache.org/job/PreCommit-OOZIE-Build/416/artifact/test-patch/reports/DISTRO.out) core/src/main/resources/jetty-logging.properties Line 1 (original), 1 (patched) <https://reviews.apache.org/r/66084/#comment279713> Please leave .properties files intact. ``/**/`` is not a valid comment. Format: https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.Reader- core/src/main/resources/localoozie-log4j.properties Line 1 (original), 1 (patched) <https://reviews.apache.org/r/66084/#comment279714> Please leave .properties files intact. ``/**/`` is not a valid comment. Format: https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.Reader- core/src/main/resources/oozie-log4j.properties Line 1 (original), 1 (patched) <https://reviews.apache.org/r/66084/#comment279715> Please leave .properties files intact. ``/**/`` is not a valid comment. Format: https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.Reader- - Attila Sasvari On March 15, 2018, 9:49 a.m., Alisha Prabhu wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66084/ > --- > > (Updated March 15, 2018, 9:49 a.m.) > > > Review request for oozie. > > > Bugs: OOZIE-3176 > https://issues.apache.org/jira/browse/OOZIE-3176 > > > Repository: oozie-git > > > Description > --- > > There are multiple checkstyle violations in the Oozie-core module. > Maven command used is mvn clean install -DskipTests > > > Diffs > - > > core/src/main/java/org/apache/oozie/BundleActionBean.java eeec549 > core/src/main/java/org/apache/oozie/BundleJobBean.java dec0967 > core/src/main/java/org/apache/oozie/CoordinatorActionBean.java ddec1e6 > core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 5999f40 > core/src/main/java/org/apache/oozie/CoordinatorJobInfo.java 04c5bc9 > core/src/main/java/org/apache/oozie/DagELFunctions.java 22ac624 > core/src/main/java/org/apache/oozie/SLAEventBean.java b30867b > core/src/main/java/org/apache/oozie/WorkflowActionBean.java 6b05b89 > core/src/main/java/org/apache/oozie/WorkflowJobBean.java cb06bec > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > bc0f405 > core/src/main/java/org/apache/oozie/client/rest/JsonBean.java c8bd7c2 > > core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java > 669ac5e > > core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java > 179cc45 > > core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java > 16d7036 > core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java > ff7bf8a > core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java > fbc6152 > > core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java > 2b91253 > > core/src/main/java/org/apache/oozie/command/coord/CoordPushDependencyCheckXCommand.java > 1e864ea > core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java > 1998924 > core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java > 42bdc99 > core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java > 1195ef0 > core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java > 86ee1cc > core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java > 61891b8 > core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java > 71c29cd > core/src/main/java/org/apache/oozie/command/wf/ActionXCommand.java 29ccd32 > core/src/main/java/org/apache/oozie/coord/CoordUtils.java 2874bf6 > core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java f40f406 > > core/src/main/java/org/apache/oozie/executor/jpa/BundleJobsGetRunningOrPendingJPAExecutor.java > 18ceb10 > > core/src/main/java/org/apache/oozie/executor/jpa/CoordActionRemoveJPAExecutor.java > 02a4f68 > > core/src/main/java/org/apache/oozie/executor/jpa/CoordJobGetRunningActionsCountJPAExecutor.java > 274b3
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395612#comment-16395612 ] Attila Sasvari commented on OOZIE-2814: --- [~gezapeti], [~andras.piros] can you review the latest patch? Thanks. > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch, OOZIE-2814-06.patch, OOZIE-2814-07.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370112#comment-16370112
]
Attila Sasvari commented on OOZIE-3170:
---
Failed tests pass locally with {{mvn test
-Dtest=TestPartitionDependencyManagerService,TestRecoveryService,TestPartitionDependencyManagerEhcache,TestJMSAccessorService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} and they should be unrelated
(changes are only in the tools module).
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:2
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369998#comment-16369998
]
Attila Sasvari commented on OOZIE-3170:
---
Tested it on a pseudo hadoop cluster:
{code:java}
$ bin/oozie-diag-bundle-collector.sh -oozie http://localhost:11000/oozie
-output /tmp/jobs/
Checking Connection...Done
Using Temporary Directory:
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/1519129669757-0
Getting Sharelib Information...Done
Getting Configuration...Done
Getting OS Environment Variables...Done
Getting Java System Properties...Done
Getting Queue Dump...Done
Getting Thread Dump...Done
Getting Instrumentation...Done
Getting Metrics...Skipping (Metrics are unavailable)
Using Directory: /tmp/jobs/oozie-diag-bundle-1519129669082
Creating Zip File:
/tmp/jobs/oozie-diag-bundle-1519129669082/oozie-diag-bundle.zip ...Done{code}
* Verified that Diag bundle tool logs are placed in the proper directory
{code:java}
$ ls /tmp/jobs/oozie-diag-bundle-1519129669082/ jetty.log
oozie-diag-bundle.zip oozie-instrumentation.log oozie-ops.log
oozie-audit.log oozie-error.log oozie-jpa.log
oozie.log{code}
- Please adjust documentation.
- I am pretty sure test failures are unrelated, but I will run them locally
soon.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369254#comment-16369254
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks! Sorry for the late response.
- Next time, can you please submit the patch to check whether Oozie tests still
pass?
- If we change the directory structure, we should update the documentation too:
https://github.com/apache/oozie/blob/e26641db469533ac059923c7ed28a5c74f71ad20/docs/src/site/twiki/DG_CommandLineTool.twiki#L1942
- Setting the log directory to the same place as the output zip file sounds
good to me.
I plan to try the patch out with a pseudo hadoop as soon as I have some more
capacity.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch, OOZIE-3170-2.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
>
[jira] [Comment Edited] (OOZIE-2814) OYA: Update example workflows to newest schemas
[
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365983#comment-16365983
]
Attila Sasvari edited comment on OOZIE-2814 at 2/15/18 5:30 PM:
{{mvn test
-Dtest=TestHCatMessageHandler,TestDefaultConnectionContext,TestJMSSLAEventListener,TestJMSJobEventListener,TestPartitionDependencyManagerEhcache,TestRecoveryService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} passes locally.
was (Author: asasvari):
{{mvn test
-Dtest=TestHCatMessageHandler,TestDefaultConnectionContext,TestJMSSLAEventListener,TestJMSJobEventListener,TestPartitionDependencyManagerEhcache,TestRecoveryService
-DtargetJavaVersion=1.8 -DjavaVersion=1.8}} pass locally.
> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
> Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch,
> OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch,
> OOZIE-2814-05.patch, OOZIE-2814-06.patch, OOZIE-2814-07.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more
> Yarn-centric. Most, if not all, examples currently use pretty old versions
> of the schemas. We should take this opportunity to update the examples to
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows
> still work. We have some examples where we have two workflow XML files that
> do the same thing in different ways.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-07.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch, OOZIE-2814-06.patch, OOZIE-2814-07.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-06.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch, OOZIE-2814-06.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas
/java-main/workflow.xml 98e01ca92187e1c567686f6c2b4f689bb2a5ef6a examples/src/main/apps/map-reduce/job-with-config-class.properties 0b14cb749b82fcd165f8db9b1832c650e6970f61 examples/src/main/apps/map-reduce/job.properties 7115229b6a2b998f6a5eed503eec554544f67cd7 examples/src/main/apps/map-reduce/workflow-with-config-class.xml 0deab665a0175959807491f7ccfec2a47a0cea2a examples/src/main/apps/map-reduce/workflow.xml 7fe7fdbe17a941084c01fba91efed274c910f305 examples/src/main/apps/no-op/job.properties a839dbb9f01b3786e551e232e8fd75bce0d38391 examples/src/main/apps/no-op/workflow.xml a5277d0060a10457347056f225b3508b0e59d7a6 examples/src/main/apps/pig/job.properties 53e94eacf1210c54c1999617f74e24bf9dc4a80a examples/src/main/apps/pig/workflow.xml 8da2b66882e8b14f8284d7101cdedc67db3d5cb1 examples/src/main/apps/pyspark/job.properties 72e09b938a3e36f89d782f26c7e3a654b699f43d examples/src/main/apps/pyspark/workflow.xml 4768a6cb5a18099e8e8a613e47c6ec48d79a6c0f examples/src/main/apps/shell/job.properties bbc765412b1baad7f9a414814df0d9f896b17acf examples/src/main/apps/shell/workflow.xml 46566bd96e65293f7327982dbf92d8d120069577 examples/src/main/apps/sla/coordinator.xml 10e9ea3652d565de21014f37e88203645ce7e209 examples/src/main/apps/sla/job.properties 698ec6d2384f309c9c19d3df9cb14d4a9e967aae examples/src/main/apps/sla/workflow.xml 484091a8292f2532cdd60dd1f0ef2e699f0b6b67 examples/src/main/apps/spark/job.properties a509db07c6110014cc83e9ccf5480b6ba7c2371e examples/src/main/apps/spark/workflow.xml 1b1a3e889d30ef62200382f49da400ed6d3879fb examples/src/main/apps/sqoop-freeform/job.properties 306421abe136f852dcd8e25162d94d7d13bc6a57 examples/src/main/apps/sqoop-freeform/workflow.xml 1097da328056d478ea13d8108286d2390d1f8549 examples/src/main/apps/sqoop/job.properties afc4de89fa12246c7d5ce8880e857001953b089f examples/src/main/apps/sqoop/workflow.xml 0ae05a14f0617b9f2ecd458cc1b69527cf203d87 examples/src/main/apps/ssh/job.properties 540db02936a04fa64867fa27d6625abab275b265 examples/src/main/apps/ssh/workflow.xml 4b53a8bc0f59e1084feb3d4fab11abdb939344b1 examples/src/main/apps/streaming/job.properties 83ea2c35e3195a18390a4852915fc58eb57664d6 examples/src/main/apps/streaming/job.windows.properties b216d3eca61f0d8836b2b6227acf3db48ca8c645 examples/src/main/apps/streaming/workflow.xml feae1d293a70045f511c79c5489282212640dbf2 examples/src/main/apps/subwf/job.properties 3c9af19670a1336135bdefbc6de361e0b8595d08 examples/src/main/apps/subwf/workflow.xml 9873a50485d76811f9c981ed8a6bcc30e09089a0 examples/src/test/resources/localoozieexample-wf.xml c1eba55aa22446922bff291d1e8f043a74455512 Diff: https://reviews.apache.org/r/65606/diff/5/ Changes: https://reviews.apache.org/r/65606/diff/4-5/ Testing --- Thanks, Attila Sasvari
Re: Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas
> On Feb. 14, 2018, 10:50 a.m., Peter Bacsko wrote: > > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > > Lines 2349 (patched) > > <https://reviews.apache.org/r/65606/diff/4/?file=1959073#file1959073line2356> > > > > Can't we accidentally skip the PREP phase? If the WF quickly makes a > > transition to the next state, then the test will fail. We cannot, look at existing tests. - Attila --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65606/#review197501 ------- On Feb. 12, 2018, 4:19 p.m., Attila Sasvari wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65606/ > --- > > (Updated Feb. 12, 2018, 4:19 p.m.) > > > Review request for oozie. > > > Repository: oozie-git > > > Description > --- > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions of > the schemas. We should take this opportunity to update the examples to the > newest schemas added by OOZIE-2687, which should help demonstrate them. > > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. > > > Diffs > - > > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > bc0f4050bb3788988d3a5352a15220fbc4fa219d > > core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java > c236dafa5965403c33e1a400509e7c374fe2637c > core/src/main/resources/oozie-default.xml > e53b17b3cf201b4cf7bc5371503e66327eae2126 > > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > 1cc569d8d7a0e61905ad6c4e940cefb00da1343d > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > 2cdbde4d2fc86a07b35beadc56012c438baf8d54 > examples/src/main/apps/aggregator/coordinator-with-offset.xml > 4c351677f4afc6e85f807106d426bb2f922a28b1 > examples/src/main/apps/aggregator/coordinator.xml > 948edc1adbe09dfb6f7bffb5eebc82f3670658cc > examples/src/main/apps/aggregator/job-with-offset.properties > 9a28ae13e50c40f70db4b0f583a63eb42f97388d > examples/src/main/apps/aggregator/job.properties > aaf4edf7b9d030b5044610698b27a564b008f77b > examples/src/main/apps/aggregator/workflow.xml > 4a801090e31acba23ea49da1c59a8ce4b951c654 > examples/src/main/apps/bundle/job.properties > c06edeb3f42e2eba3367b9486982dabb4caf6436 > examples/src/main/apps/coord-input-logic/job.properties > 2cd137b94999c7e1f68bb54928f1c3ce2838d59d > examples/src/main/apps/coord-input-logic/workflow.xml > 80d31457eff6e78710ad1fb4d4f1b372c0d1e43c > examples/src/main/apps/cron-schedule/coordinator.xml > 48a7de76245e02397b9736a95fa0132504067f9a > examples/src/main/apps/cron-schedule/job.properties > 5b5eb43ca2bf9d8cd1b263ca3b94a552b562a20d > examples/src/main/apps/cron-schedule/workflow.xml > b53922c8f40e691cb637b91cb182172794296a04 > examples/src/main/apps/cron/coordinator.xml > 744d14e870b59d0b89f65a78fea63fffa3ff4b8e > examples/src/main/apps/cron/job.properties > 42875fd5a5ce393cd11717bcc55fc1bb91e0fcd3 > examples/src/main/apps/cron/workflow.xml > 63dff976f9e35a26b0ad09078778748d3d02bc3b > examples/src/main/apps/custom-main/job.properties > b897512a1c6246308ca17b019ce400f3ec06cd7b > examples/src/main/apps/custom-main/workflow.xml > 828dfaa0594aac0539c4e08ae8e37cb89d17eb54 > examples/src/main/apps/datelist-java-main/job.properties > e8356e3a4a2e1ae41e09f5ea124fb5360bd6e8b6 > examples/src/main/apps/datelist-java-main/workflow.xml > b59a12ebd1fc21c9e26f6a5b0348eb9c76375c03 > examples/src/main/apps/demo/job.properties > d70d3208df77e0b612005196907cecd97d7c5d21 > examples/src/main/apps/demo/job.windows.properties > ea948b20009685e1286053549fc1974669819682 > examples/src/main/apps/demo/workflow.xml > 0ceca507adc68c0674b794420c9eb733cb320879 > examples/src/main/apps/distcp/job.properties > ec9422caddcdecfa6ee9a58868cf68d7457ad4db > examples/src/main/apps/distcp/workflow.xml > 6deecb377901613a5bb28fc18bf5beff2d30f252 > examples/src/main/apps/hadoop-el/job.properties > 1d13c27056ac58ad359b5298f12ec046bb4cc61d > examples/src/main/apps/hadoop-el/workflow.xml > d4094d44136a7de2f85f5c9be042d5aab24b
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363064#comment-16363064 ] Attila Sasvari commented on OOZIE-2814: --- Can someone review this please? > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-05.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch, > OOZIE-2814-05.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas
> On Feb. 12, 2018, 8:05 p.m., András Piros wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > > Lines 266-271 (patched) > > <https://reviews.apache.org/r/65606/diff/2/?file=1958142#file1958142line266> > > > > What happens if both `resource-manager` and `job-tracker` are filled? > > I'd either separate `resourceManager` and `jobTracker` into two variables / > > fields, or have some more clean, generic labelling for the same attribute. Cannot happen. Look at the new XML schemas (it is a choice, XML validation fails if both elements are present). > On Feb. 12, 2018, 8:05 p.m., András Piros wrote: > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > > Line 510 (original), 510 (patched) > > <https://reviews.apache.org/r/65606/diff/2/?file=1958145#file1958145line510> > > > > Isn't it rather an XML validation error? The schema itself is quite > > correct. Please look at the existing code, "E0701: XML schema error, No job-tracker defined" is what is orignally defined. I just extended the error message. > On Feb. 12, 2018, 8:05 p.m., András Piros wrote: > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > > Lines 1680 (patched) > > <https://reviews.apache.org/r/65606/diff/2/?file=1958145#file1958145line1680> > > > > Would give a catchier name and remove comment. > > > > What about more test cases: > > > > * both `resource-manager` and `job-tracker` present at the same level > > * both `resource-manager` and `job-tracker` present at different levels > > * none of these are present > > Peter Cseh wrote: > I think it's better to keep both RM and JT as a single variable in the > code as they have the same meaning. JT is only there for compatiblity > reasons, but it should overwrite RM values of lower levels. - I believe XML validation is enough to protect against both resource-manager and job-tracker present at the same level. - There are existing tests that check if no job-tracker and resource-manager present. It is also validated that default job-tracker is used from oozie-site. - It is a good idea to test if action level resourcemanager address is prefered over the one defined in the global section. Can be added to TestJavaActionExecutor. - Attila --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65606/#review197301 --- On Feb. 12, 2018, 4:19 p.m., Attila Sasvari wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65606/ > --- > > (Updated Feb. 12, 2018, 4:19 p.m.) > > > Review request for oozie. > > > Repository: oozie-git > > > Description > --- > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions of > the schemas. We should take this opportunity to update the examples to the > newest schemas added by OOZIE-2687, which should help demonstrate them. > > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. > > > Diffs > - > > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > bc0f4050bb3788988d3a5352a15220fbc4fa219d > > core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java > c236dafa5965403c33e1a400509e7c374fe2637c > > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > 1cc569d8d7a0e61905ad6c4e940cefb00da1343d > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > 2cdbde4d2fc86a07b35beadc56012c438baf8d54 > examples/src/main/apps/aggregator/coordinator-with-offset.xml > 4c351677f4afc6e85f807106d426bb2f922a28b1 > examples/src/main/apps/aggregator/coordinator.xml > 948edc1adbe09dfb6f7bffb5eebc82f3670658cc > examples/src/main/apps/aggregator/job-with-offset.properties > 9a28ae13e50c40f70db4b0f583a63eb42f97388d > examples/src/main/apps/aggregator/job.properties > aaf4edf7b9d030b5044610698b27a564b008f77b > examples/src/main/apps/aggregator/workflow.xml > 4a801090e31acba23ea49da1c59a8ce4b951c654 > examples/src/main/apps/bundle/job.properties > c06edeb3f42e2eba3367b9486982dabb4c
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-04.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch, OOZIE-2814-04.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-03.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch, OOZIE-2814-03.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas
0b14cb749b82fcd165f8db9b1832c650e6970f61 examples/src/main/apps/map-reduce/job.properties 7115229b6a2b998f6a5eed503eec554544f67cd7 examples/src/main/apps/map-reduce/workflow-with-config-class.xml 0deab665a0175959807491f7ccfec2a47a0cea2a examples/src/main/apps/map-reduce/workflow.xml 7fe7fdbe17a941084c01fba91efed274c910f305 examples/src/main/apps/no-op/job.properties a839dbb9f01b3786e551e232e8fd75bce0d38391 examples/src/main/apps/no-op/workflow.xml a5277d0060a10457347056f225b3508b0e59d7a6 examples/src/main/apps/pig/job.properties 53e94eacf1210c54c1999617f74e24bf9dc4a80a examples/src/main/apps/pig/workflow.xml 8da2b66882e8b14f8284d7101cdedc67db3d5cb1 examples/src/main/apps/pyspark/job.properties 72e09b938a3e36f89d782f26c7e3a654b699f43d examples/src/main/apps/pyspark/workflow.xml 4768a6cb5a18099e8e8a613e47c6ec48d79a6c0f examples/src/main/apps/shell/job.properties bbc765412b1baad7f9a414814df0d9f896b17acf examples/src/main/apps/shell/workflow.xml 46566bd96e65293f7327982dbf92d8d120069577 examples/src/main/apps/sla/coordinator.xml 10e9ea3652d565de21014f37e88203645ce7e209 examples/src/main/apps/sla/job.properties 698ec6d2384f309c9c19d3df9cb14d4a9e967aae examples/src/main/apps/sla/workflow.xml 484091a8292f2532cdd60dd1f0ef2e699f0b6b67 examples/src/main/apps/spark/job.properties a509db07c6110014cc83e9ccf5480b6ba7c2371e examples/src/main/apps/spark/workflow.xml 1b1a3e889d30ef62200382f49da400ed6d3879fb examples/src/main/apps/sqoop-freeform/job.properties 306421abe136f852dcd8e25162d94d7d13bc6a57 examples/src/main/apps/sqoop-freeform/workflow.xml 1097da328056d478ea13d8108286d2390d1f8549 examples/src/main/apps/sqoop/job.properties afc4de89fa12246c7d5ce8880e857001953b089f examples/src/main/apps/sqoop/workflow.xml 0ae05a14f0617b9f2ecd458cc1b69527cf203d87 examples/src/main/apps/ssh/job.properties 540db02936a04fa64867fa27d6625abab275b265 examples/src/main/apps/ssh/workflow.xml 4b53a8bc0f59e1084feb3d4fab11abdb939344b1 examples/src/main/apps/streaming/job.properties 83ea2c35e3195a18390a4852915fc58eb57664d6 examples/src/main/apps/streaming/job.windows.properties b216d3eca61f0d8836b2b6227acf3db48ca8c645 examples/src/main/apps/streaming/workflow.xml feae1d293a70045f511c79c5489282212640dbf2 examples/src/main/apps/subwf/job.properties 3c9af19670a1336135bdefbc6de361e0b8595d08 examples/src/main/apps/subwf/workflow.xml 9873a50485d76811f9c981ed8a6bcc30e09089a0 examples/src/test/resources/localoozieexample-wf.xml c1eba55aa22446922bff291d1e8f043a74455512 Diff: https://reviews.apache.org/r/65606/diff/2/ Changes: https://reviews.apache.org/r/65606/diff/1-2/ Testing --- Thanks, Attila Sasvari
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360817#comment-16360817
]
Attila Sasvari commented on OOZIE-3172:
---
+1
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch,
> OOZIE-3172.003.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas
/job.properties 7115229b6a2b998f6a5eed503eec554544f67cd7 examples/src/main/apps/map-reduce/workflow-with-config-class.xml 0deab665a0175959807491f7ccfec2a47a0cea2a examples/src/main/apps/map-reduce/workflow.xml 7fe7fdbe17a941084c01fba91efed274c910f305 examples/src/main/apps/no-op/job.properties a839dbb9f01b3786e551e232e8fd75bce0d38391 examples/src/main/apps/no-op/workflow.xml a5277d0060a10457347056f225b3508b0e59d7a6 examples/src/main/apps/pig/job.properties 53e94eacf1210c54c1999617f74e24bf9dc4a80a examples/src/main/apps/pig/workflow.xml 8da2b66882e8b14f8284d7101cdedc67db3d5cb1 examples/src/main/apps/pyspark/job.properties 72e09b938a3e36f89d782f26c7e3a654b699f43d examples/src/main/apps/pyspark/workflow.xml 4768a6cb5a18099e8e8a613e47c6ec48d79a6c0f examples/src/main/apps/shell/job.properties bbc765412b1baad7f9a414814df0d9f896b17acf examples/src/main/apps/shell/workflow.xml 46566bd96e65293f7327982dbf92d8d120069577 examples/src/main/apps/sla/coordinator.xml 10e9ea3652d565de21014f37e88203645ce7e209 examples/src/main/apps/sla/job.properties 698ec6d2384f309c9c19d3df9cb14d4a9e967aae examples/src/main/apps/sla/workflow.xml 484091a8292f2532cdd60dd1f0ef2e699f0b6b67 examples/src/main/apps/spark/job.properties a509db07c6110014cc83e9ccf5480b6ba7c2371e examples/src/main/apps/spark/workflow.xml 1b1a3e889d30ef62200382f49da400ed6d3879fb examples/src/main/apps/sqoop-freeform/job.properties 306421abe136f852dcd8e25162d94d7d13bc6a57 examples/src/main/apps/sqoop-freeform/workflow.xml 1097da328056d478ea13d8108286d2390d1f8549 examples/src/main/apps/sqoop/job.properties afc4de89fa12246c7d5ce8880e857001953b089f examples/src/main/apps/sqoop/workflow.xml 0ae05a14f0617b9f2ecd458cc1b69527cf203d87 examples/src/main/apps/ssh/job.properties 540db02936a04fa64867fa27d6625abab275b265 examples/src/main/apps/ssh/workflow.xml 4b53a8bc0f59e1084feb3d4fab11abdb939344b1 examples/src/main/apps/streaming/job.properties 83ea2c35e3195a18390a4852915fc58eb57664d6 examples/src/main/apps/streaming/job.windows.properties b216d3eca61f0d8836b2b6227acf3db48ca8c645 examples/src/main/apps/streaming/workflow.xml feae1d293a70045f511c79c5489282212640dbf2 examples/src/main/apps/subwf/job.properties 3c9af19670a1336135bdefbc6de361e0b8595d08 examples/src/main/apps/subwf/workflow.xml 9873a50485d76811f9c981ed8a6bcc30e09089a0 examples/src/test/resources/localoozieexample-wf.xml c1eba55aa22446922bff291d1e8f043a74455512 Diff: https://reviews.apache.org/r/65606/diff/1/ Testing --- Thanks, Attila Sasvari
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-02.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch, > OOZIE-2814-02.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3175) PurgeService should detect and delete orphaned jobs/actions
[
https://issues.apache.org/jira/browse/OOZIE-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357179#comment-16357179
]
Attila Sasvari commented on OOZIE-3175:
---
An example SQL query to detect orphan workflow actions:
{{SELECT a.id, a.wf_id FROM WF_ACTIONS a WHERE NOT EXISTS (SELECT 1 FROM
WF_JOBS j WHERE j.id=a.wf_id);}}
> PurgeService should detect and delete orphaned jobs/actions
>
>
> Key: OOZIE-3175
> URL: https://issues.apache.org/jira/browse/OOZIE-3175
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
> Reporter: Attila Sasvari
>Priority: Major
>
> Improvement idea: the purge service could look for orphaned jobs/actions and
> delete those. It would help the cases where the user went in and only
> partially removed things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
TestPigActionBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277086>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/action/TestShellActionBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277087>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/action/TestSparkActionBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277088>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/action/TestSqoopActionBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277089>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/action/TestSshActionBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277090>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/mapping/TestGlobalMapping.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277091>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/mapping/TestParametersMapping.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277092>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/workflow/TestCredentialsBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277093>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/workflow/TestGlobalBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277094>
missing license
jobs/jobs-api/src/test/java/org/apache/oozie/jobs/api/workflow/TestParametersBuilder.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277095>
missing license
jobs/jobs-client/pom.xml
Lines 2 (patched)
<https://reviews.apache.org/r/64029/#comment277096>
missing license
jobs/jobs-client/src/test/java/org/apache/oozie/jobs/client/examples/TestCredentialsExample.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277097>
missing license
jobs/jobs-client/src/test/java/org/apache/oozie/jobs/client/examples/TestGlobalExample.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277098>
missing license
jobs/jobs-client/src/test/java/org/apache/oozie/jobs/client/examples/TestParametersExample.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277099>
missing license
minitest/src/test/java/org/apache/oozie/test/WorkflowTestCase.java
Lines 1 (patched)
<https://reviews.apache.org/r/64029/#comment277100>
missing license
- Attila Sasvari
On Jan. 8, 2018, 10:31 p.m., András Piros wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
>
> (Updated Jan. 8, 2018, 10:31 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
>
>
> Repository: oozie-git
>
>
> Description
> ---
>
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
>
>
> Diffs
> -
>
> .gitignore a4285a0d77ac2e4e8fc4afbfa21bf9fd287b3e4d
> client/pom.xml 3b67d1286499337d61c269f9dbd1ff65ba23b903
> client/src/main/java/org/apache/oozie/cli/ApiJarFactory.java PRE-CREATION
> client/src/main/java/org/apache/oozie/cli/ApiJarLoader.java PRE-CREATION
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java
> 4abc7502c0c9d8b59ded2baaed30c407ad073008
> client/src/main/java/org/apache/oozie/cli/WorkflowFactoryCompiler.java
> PRE-CREATION
> client/src/main/java/org/apache/oozie/client/OozieClient.java
> 7e41ef89c784882c26f4af0da0110101fa1a61a0
> core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java
> 38fb006980c0acb582b4f91175be1ced8a1afa79
> core/src/main/java/org/apache/oozie/local/LocalOozie.java
> 9ab646cd53223d903e387eb92aa103a7b6577d0f
> core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d
> core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java
> d4b08711223356c8d4f58a3391b6b87efa0a2b73
> core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java
> 9cdd915c02a3e34ad211cf6cced2e64f3a1196a0
> core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java
> ff50f3d121d8af7d607124424915e1bf30ffb109
> core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420
> core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java
> 7bf8fb3f0ce9253d553c7b1883666811686f6a9f
> core/src/main/resources/oozie-default.xml
> 830893fc50f0eb044df91c15ba87ed17761836d4
[jira] [Assigned] (OOZIE-3176) Oozie-core fails with checkstyle errors.
[ https://issues.apache.org/jira/browse/OOZIE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reassigned OOZIE-3176: - Assignee: Alisha Prabhu > Oozie-core fails with checkstyle errors. > > > Key: OOZIE-3176 > URL: https://issues.apache.org/jira/browse/OOZIE-3176 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 5.0.0b1 > Environment: # uname -a > Linux in-ibmibm283.persistent.co.in 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Aug > 25 11:21:22 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Alisha Prabhu >Assignee: Alisha Prabhu >Priority: Minor > Attachments: Checkstyle-erros.txt > > > There are multiple checkstyle violations in the Oozie-core module. > Maven command used is mvn clean install -DskipTests > I have attached the Checkstyle-errors.txt file, that has the errors occurred > while building. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3176) Oozie-core fails with checkstyle errors.
[
https://issues.apache.org/jira/browse/OOZIE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355416#comment-16355416
]
Attila Sasvari commented on OOZIE-3176:
---
[~alishap] many thanks for filing this JIRA, I added you to contributors and
assigned the ticket too you.
I can confirm {{mvn clean install -DskipTests
-Dmaven.javadoc.opts="-Xdoclint:none"}} fails on master because of a checkstyle
error, {{core/target/checkstyle-result.xml}} shows:
{code}
http://www.apache.org/licenses/LICENSE-2.0.;
source="com.puppycrawl.tool
s.checkstyle.checks.header.HeaderCheck"/>
{code}
WorkflowJobsBasicInfoFromParentId was introduced by OOZIE-1401.amend.003.patch.
On that day, the Oozie PreCommit build failed to add comment (JIRA was
updated).
> Oozie-core fails with checkstyle errors.
>
>
> Key: OOZIE-3176
> URL: https://issues.apache.org/jira/browse/OOZIE-3176
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0b1
> Environment: # uname -a
> Linux in-ibmibm283.persistent.co.in 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Aug
> 25 11:21:22 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Minor
> Attachments: Checkstyle-erros.txt
>
>
> There are multiple checkstyle violations in the Oozie-core module.
> Maven command used is mvn clean install -DskipTests
> I have attached the Checkstyle-errors.txt file, that has the errors occurred
> while building.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355233#comment-16355233
]
Attila Sasvari commented on OOZIE-3172:
---
[~andras.piros] thanks for submitting example workflows. Workflows succeeded,
so it looks good to me. [~gezapeti] can you answer last question?
(If it breaks Hive workflows, we can file a separate JIRA. It would be nice to
validate.)
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release Oozie 4.3.1(candidate 4)
xecutor.java:844) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLaun > cher(JavaActionExecutor.java:1095) > ... 11 more > 2018-02-06 18:09:37,661 WARN ActionStartXCommand:523 - SERVER[ > hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf] > JOB[000-180206180622504-oozie-oozi-W] > ACTION[000-180206180622504-oozie-oozi-W@python-node] Setting Action > Status to [DONE] > > > > On Tue, Feb 6, 2018 at 7:42 AM, Gézapeti Cseh <gezap...@apache.org> wrote: > > > Hi, > > > > I have created a build for Oozie 4.3.1, candidate 4. > > > > It includes all the changes in rc3 plus OOZIE-2888 to fix sharelib > > creation with Hadoop 2.7.5. and possibly other newer versions. > > > > Keys to verify the signature of the release artifact are available at > > http://www.apache.org/dist/oozie/KEYS > > > > Please download, test, and try it out: > > > > http://people.apache.org/~gezapeti/oozie-4.3.1-rc4/ > > > > The release, md5 signature, gpg signature, and rat report can be found at > > the above address. > > > > Vote closes on 02/12/2018 11:59 PM CET. > > > -- -- Attila Sasvari Software Engineer <http://www.cloudera.com/>
[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2847:
--
Affects Version/s: 5.0.0
5.0.0b1
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0, 5.0.0b1, 5.0.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMas
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353807#comment-16353807
]
Attila Sasvari commented on OOZIE-2847:
---
I decided to only push it to master. [~gezapeti] is the release manager of
4.3.1, that is in progress. He will commit OOZIE-2847-4.3.patch to branch-4.3
later and update fix version.
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.had
[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2847:
--
Fix Version/s: (was: 4.3.1)
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMas
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353797#comment-16353797
]
Attila Sasvari commented on OOZIE-2847:
---
[~dionusos] thanks for your contribution, +1. I will commit the patches to
branch-4.3 and master soon.
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:11
[jira] [Created] (OOZIE-3175) PurgeService should detect and delete orphaned jobs/actions
Attila Sasvari created OOZIE-3175: - Summary: PurgeService should detect and delete orphaned jobs/actions Key: OOZIE-3175 URL: https://issues.apache.org/jira/browse/OOZIE-3175 Project: Oozie Issue Type: Improvement Components: core Affects Versions: trunk Reporter: Attila Sasvari Improvement idea: the purge service could look for orphaned jobs/actions and delete those. It would help the cases where the user went in and only partially removed things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3174) Allow users to retain SQL file generated by OozieDBCLI
Attila Sasvari created OOZIE-3174: - Summary: Allow users to retain SQL file generated by OozieDBCLI Key: OOZIE-3174 URL: https://issues.apache.org/jira/browse/OOZIE-3174 Project: Oozie Issue Type: Improvement Components: tools Affects Versions: trunk Reporter: Attila Sasvari OozieDBCLI writes SQL statements to a temporary file. It is deleted without question at the end of the execution (see OOZIE-2727). We should allow the user to decide if they want to delete it or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353729#comment-16353729 ] Attila Sasvari commented on OOZIE-1717: --- TestHiveActionExecutor locally passes, committed to master > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2847:
--
Fix Version/s: (was: 5.0.0b1)
5.0.0
trunk
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk, 5.0.0, 4.3.1
>
> Attachments: OOZIE-2847-4.3.patch, OOZIE-2847-5.0.patch,
> OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
>
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: amend-OOZIE-1717-01.patch > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: (was: amend-OOZIE-1717-01.patch) > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: amend-OOZIE-1717-01.patch > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch, amend-OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65385: OOZIE-1717 add indices to speed up db queries
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65385/
---
(Updated Feb. 6, 2018, 1 a.m.)
Review request for oozie.
Changes
---
verify indexes - create, upgrade db
Repository: oozie-git
Description
---
Add indices to speed up SELECT operations.
Index user_name column
for purge service OOZIE-1803:
CREATE INDEX "OOZIEDB"."BUNDLE_JOBS_END_TIME_IDX" ON "BUNDLE_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."COORD_JOBS_END_TIME_IDX" ON "COORD_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."WF_JOBS_END_TIME_IDX" ON "WF_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."COORD_ACTIONS_JOB_ID_IDX" ON "COORD_ACTIONS" ("JOB_ID");
CREATE INDEX "OOZIEDB"."COORD_JOBS_BUNDLE_ID_IDX" ON "COORD_JOBS" ("BUNDLE_ID");
CREATE INDEX "OOZIEDB"."WF_ACTIONS_WF_ID_IDX" ON "WF_ACTIONS" ("WF_ID");
CREATE INDEX "OOZIEDB"."WF_JOBS_PARENT_ID_IDX" ON "WF_JOBS" ("PARENT_ID");
CREATE INDEX "OOZIEDB"."BUNDLE_ACTIONS_BUNDLE_ID_IDX" ON "BUNDLE_ACTIONS"
("BUNDLE_ID");
Composite indices to speed up operations on the web UI
Diffs (updated)
-
tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
4bb6dd8606784fc0c4eb7aaedef79c59617f65eb
Diff: https://reviews.apache.org/r/65385/diff/5/
Changes: https://reviews.apache.org/r/65385/diff/4-5/
Testing
---
Thanks,
Attila Sasvari
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: amend-OOZIE-1717-00.patch > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch, > amend-OOZIE-1717-00.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65385: OOZIE-1717 add indices to speed up db queries
> On Jan. 29, 2018, 3:20 p.m., András Piros wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
> > Lines 1156 (patched)
> > <https://reviews.apache.org/r/65385/diff/1/?file=1949528#file1949528line1156>
> >
> > Any ideas how to check automatically which indices have been created?
>
> Attila Sasvari wrote:
> I manually checked the SQL script OpenJPA's mapping tool has generated. I
> do not know a good method. JPA 2.0 (that OpenJPA 2.4.2 supports) have
> annotations for column indexes, and I verfied that the tool generated those
> indexes. Composite/multi-columns indexes are not supported. Please let me
> know what exactly and how you would like to verify about in a db agnostic and
> testable way. If that check is not abolutely necessary here or the change is
> too big, I would rather solve it in a separate JIRA.
On a second thought, you are right TestOozieDBCLI should include this kind of
check. In Derby, SYSCONGLOMERATES should contain index information according to
https://db.apache.org/derby/docs/10.8/ref/rrefsistabs39391.html It is also
possible to get such information via metadata using something like:
```
DatabaseMetaData metaData = connection.getMetaData();
ResultSet rs = metaData.getIndexInfo(null, "my_schema", "my_table", false,
true);
```
- Attila
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65385/#review196432
-------
On Feb. 5, 2018, 9:52 a.m., Attila Sasvari wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65385/
> ---
>
> (Updated Feb. 5, 2018, 9:52 a.m.)
>
>
> Review request for oozie.
>
>
> Repository: oozie-git
>
>
> Description
> ---
>
> Add indices to speed up SELECT operations.
>
>
> Index user_name column
>
> for purge service OOZIE-1803:
> CREATE INDEX "OOZIEDB"."BUNDLE_JOBS_END_TIME_IDX" ON "BUNDLE_JOBS"
> ("END_TIME");
> CREATE INDEX "OOZIEDB"."COORD_JOBS_END_TIME_IDX" ON "COORD_JOBS" ("END_TIME");
> CREATE INDEX "OOZIEDB"."WF_JOBS_END_TIME_IDX" ON "WF_JOBS" ("END_TIME");
> CREATE INDEX "OOZIEDB"."COORD_ACTIONS_JOB_ID_IDX" ON "COORD_ACTIONS"
> ("JOB_ID");
> CREATE INDEX "OOZIEDB"."COORD_JOBS_BUNDLE_ID_IDX" ON "COORD_JOBS"
> ("BUNDLE_ID");
> CREATE INDEX "OOZIEDB"."WF_ACTIONS_WF_ID_IDX" ON "WF_ACTIONS" ("WF_ID");
> CREATE INDEX "OOZIEDB"."WF_JOBS_PARENT_ID_IDX" ON "WF_JOBS" ("PARENT_ID");
> CREATE INDEX "OOZIEDB"."BUNDLE_ACTIONS_BUNDLE_ID_IDX" ON "BUNDLE_ACTIONS"
> ("BUNDLE_ID");
>
> Composite indices to speed up operations on the web UI
>
>
> Diffs
> -
>
> core/src/main/java/org/apache/oozie/BundleJobBean.java
> b42f12841c2f9cb6d508cd2ac2f393e6407d1cf0
> core/src/main/java/org/apache/oozie/CoordinatorJobBean.java
> 99c7d3c474d9e1e22e2f5a927c03d71b9292b663
> core/src/main/java/org/apache/oozie/WorkflowJobBean.java
> cd9a6df42e552a0607606a244b81978747d89b41
> tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
> a8b2234966c967f9ad9d722628bd486012b14bd3
>
>
> Diff: https://reviews.apache.org/r/65385/diff/3/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Attila Sasvari
>
>
[jira] [Reopened] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reopened OOZIE-1717: --- > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Fix Version/s: trunk > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Fix For: trunk > > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1717) Add indexes to speed up db queries
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Summary: Add indexes to speed up db queries (was: Index user_name column) > Add indexes to speed up db queries > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352389#comment-16352389 ] Attila Sasvari commented on OOZIE-1717: --- TestRecoveryService,TestJMSJobEventListener,TestPartitionDependencyManagerService,TestJMSSLAEventListener,TestHCatMessageHandler,TestDefaultConnectionContext,TestJMSAccessorService pass locally. > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352387#comment-16352387
]
Attila Sasvari commented on OOZIE-2847:
---
OOZIE-1770 is about Oozie on Yarn that migrates the launcher from MapReduce to
Yarn Application Master. It is only included in Oozie 5.0.0-beta1 and the
master branch. So, {{LauncherMapper}} only exists in 4.3.1. Related mechanism,
{{setRecoveryId()}}, on the master branch resides in
[LauncherAM|[https://github.com/apache/oozie/blob/ba665da34c23b1fa86bf1570724147e6f8c85b1d/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java#L481]|https://github.com/apache/oozie/blob/ba665da34c23b1fa86bf1570724147e6f8c85b1d/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java#L481].]
. It might have similar issue in HA mode. Anyway, it would be nice to solve it
on branch-4.3.{{}}
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.ap
[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue
[
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352300#comment-16352300
]
Attila Sasvari commented on OOZIE-2847:
---
[~dionusos] can you please rebase the patch? Thanks.
> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Denes Bodo
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a
> second try, during recovery. On the second try, Oozie is trying to see if the
> previously written job id on hdfs matches the current job id. In most
> occasion, this will match. However, in the event when Oozie launcher is
> killed right in the middle when Oozie is in the process of writing id in the
> file, the Oozie file in hdfs is created, but the id has yet to be written to
> the file. During the next recovery, Oozie will mistakenly think the id exists
> in the file while the file is actually empty, therefore throwing this
> exception:
> {noformat}
> 2015-07-10
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL
> : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code
> : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error
> Message : Hadoop job Id mismatch, action file
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
> declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID
> : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External
> Status : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name
> : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries
> : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI
> : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type
> : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started
> : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status
> : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
> : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External
> ChildIDs : null
> 2015-07-10
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main]
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
> at
> org.apache.hadoop.service.AbstractSe
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-004.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-004.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65385: OOZIE-1717 add indices to speed up db queries
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65385/
---
(Updated Feb. 5, 2018, 9:52 a.m.)
Review request for oozie.
Repository: oozie-git
Description
---
Add indices to speed up SELECT operations.
Index user_name column
for purge service OOZIE-1803:
CREATE INDEX "OOZIEDB"."BUNDLE_JOBS_END_TIME_IDX" ON "BUNDLE_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."COORD_JOBS_END_TIME_IDX" ON "COORD_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."WF_JOBS_END_TIME_IDX" ON "WF_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."COORD_ACTIONS_JOB_ID_IDX" ON "COORD_ACTIONS" ("JOB_ID");
CREATE INDEX "OOZIEDB"."COORD_JOBS_BUNDLE_ID_IDX" ON "COORD_JOBS" ("BUNDLE_ID");
CREATE INDEX "OOZIEDB"."WF_ACTIONS_WF_ID_IDX" ON "WF_ACTIONS" ("WF_ID");
CREATE INDEX "OOZIEDB"."WF_JOBS_PARENT_ID_IDX" ON "WF_JOBS" ("PARENT_ID");
CREATE INDEX "OOZIEDB"."BUNDLE_ACTIONS_BUNDLE_ID_IDX" ON "BUNDLE_ACTIONS"
("BUNDLE_ID");
Composite indices to speed up operations on the web UI
Diffs (updated)
-
core/src/main/java/org/apache/oozie/BundleJobBean.java
b42f12841c2f9cb6d508cd2ac2f393e6407d1cf0
core/src/main/java/org/apache/oozie/CoordinatorJobBean.java
99c7d3c474d9e1e22e2f5a927c03d71b9292b663
core/src/main/java/org/apache/oozie/WorkflowJobBean.java
cd9a6df42e552a0607606a244b81978747d89b41
tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
a8b2234966c967f9ad9d722628bd486012b14bd3
Diff: https://reviews.apache.org/r/65385/diff/3/
Changes: https://reviews.apache.org/r/65385/diff/2-3/
Testing
---
Thanks,
Attila Sasvari
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-01.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch, OOZIE-2814-01.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349221#comment-16349221
]
Attila Sasvari commented on OOZIE-2814:
---
[~rkanter] thanks a lot. I changed the java-main example workflow to use
{{resource-manager}} instead of {{job-tracker}}
{code:java}
${jobTracker}{code}
Result of job submission:
{code:java}
Error: E0701 : E0701: XML schema error, No job-tracker defined{code}
Workflow can be validated against the schema just fine, but
{{LiteWorkflowAppParser}} needs to be adjusted as it only tries to fetch the
child of the {{job-tracker}} element:
https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java#L424
TestLiteWorkflowAppParser is to be extended too.
> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
> Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more
> Yarn-centric. Most, if not all, examples currently use pretty old versions
> of the schemas. We should take this opportunity to update the examples to
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows
> still work. We have some examples where we have two workflow XML files that
> do the same thing in different ways.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346507#comment-16346507
]
Attila Sasvari commented on OOZIE-3172:
---
There are transitive dependencies. Spark ShareLib for example:
{code:java}
[INFO] Building Apache Oozie Share Lib Spark 5.0.0-beta1
[INFO]
[INFO]
[INFO] --- maven-dependency-plugin:2.4:tree (default-cli) @
oozie-sharelib-spark ---
[INFO] org.apache.oozie:oozie-sharelib-spark:jar:5.0.0-beta1
[INFO] +- com.google.guava:guava:jar:14.0.1:compile
[INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0:compile
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | +- org.apache.commons:commons-compress:jar1.4.1:compile
[INFO] | | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | +- com.sun.jersey:jersey-client:jar:1.9:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile{code}
I would submit the example workflows to verify it does not cause issues.
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release Oozie 4.3.1 (candidate 3)
og4j/1.2/faq.html#noconfig for
> > more info.
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1-
> > distro/oozie-4.3.1/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/
> > impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1-
> > distro/oozie-4.3.1/lib/slf4j-simple-1.6.6.jar!/org/slf4j/
> > impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > the destination path for sharelib is:
> > /user/oozie/share/lib/lib_20180130164518
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/commons/io/Charsets
> > at org.apache.hadoop.security.Credentials.(Credentials.java:222)
> > at org.apache.hadoop.mapred.JobConf.(JobConf.java:334)
> > at
> > org.apache.oozie.service.HadoopAccessorService.createJobConf(
> > HadoopAccessorService.java:349)
> > at org.apache.oozie.tools.OozieSharelibCLI.run(
> OozieSharelibCLI.java:171)
> > at org.apache.oozie.tools.OozieSharelibCLI.main(
> OozieSharelibCLI.java:67)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.commons.io.Charsets
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 5 more
> >
> > none were blockers as I was able to get the following output
> >
> > $ curl http://localhost:11000/oozie/v2/admin/status
> > {"systemMode":"NORMAL"}
> >
> >
> > On Tue, Jan 30, 2018 at 10:45 AM, Rohini Palaniswamy <roh...@apache.org>
> > wrote:
> >
> > > > the server directory is missing altogether now.
> > > That is as expected. It is supposed to be only in 5.0
> > >
> > > On Mon, Jan 29, 2018 at 12:47 PM, Artem Ervits <artemerv...@gmail.com>
> > > wrote:
> > >
> > > > the server directory is missing altogether now.
> > > >
> > > > On Mon, Jan 29, 2018 at 6:59 AM, Attila Sasvari <
> asasv...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - Verified hash, signature files, rat report
> > > > > - Built Oozie distro with hadoop.version set to 2.6.0 using the
> uber,
> > > > > hadoop-2, spark-1 profiles
> > > > > - Submitted example workflows/coordinators/bundles (except hive
> > related
> > > > > ones) on a hadoop 2.6.0 in pseudo distributed mode
> > > > > - Verified Tomcat version was 6.0.53
> > > > > - Clicked through the web UI, log retrieval worked, job DAG-s were
> > > > > rendered, version was correct.
> > > > > - Verified instrumentation, SLA pages; they are now HTML based as
> > > > expected
> > > > > (OOZIE-2533). However, json2.js is referenced by webapp/index.html
> > and
> > > it
> > > > > is not found (not a blocker in my opinion).
> > > > > - Enabled SLA, re-submitted sla coordinator example, and retrieved
> > SLA
> > > > > details for the "one-op-wf" application, graph was displayed
> > properly.
> > > > >
> > > > > I used JDK 1.8.0_121
> > > > >
> > > > > On Sun, Jan 28, 2018 at 7:09 AM, Satish Saley <
> > satishsa...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have created a build for Oozie 4.3.1, candidate 3.
> > > > > >
> > > > > > It includes all the changes decided earlier except OOZIE-3018,
> > > > OOZIE-2771
> > > > > > and
> > > > > > OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService
> which
> > we
> > > > are
> > > > > > not picking.
> > > > > > OOZIE-2771 makes changes to file created in OOZIE-2666.
> > > > > > OOZIE-3072 makes changes to files created in OOZIE-1770.
> > > > > >
> > > > > >
> > > > > > Keys to verify the signature of the release artifact are
> available
> > at
> > > > > > http://www.apache.org/dist/oozie/KEYS
> > > > > >
> > > > > > Please download, test, and try it out:
> > > > > >
> > > > > > http://people.apache.org/~satishsaley/oozie-4.3.1-rc3/
> > > > > >
> > > > > >
> > > > > > The release, md5 signature, gpg signature, and rat report can be
> > > found
> > > > at
> > > > > > the above address.
> > > > > >
> > > > > > Vote closes on 01/31/2018 11:59 PM PT.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > --
> > > > > Attila Sasvari
> > > > > Software Engineer
> > > > > <http://www.cloudera.com/>
> > > > >
> > > >
> > >
> >
>
--
--
Attila Sasvari
Software Engineer
<http://www.cloudera.com/>
[jira] [Commented] (OOZIE-3172) Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/OOZIE-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346216#comment-16346216
]
Attila Sasvari commented on OOZIE-3172:
---
{{mvn dependency:tree}} shows that multiple sharelibs are depending on
{{jackson-core-asl}} (for example Hive, Hive 2, Pig, Spark, Sqoop, Streaming).
I am a bit concerned that doing this change may cause runtime failures because
of conflicting versions. Have you submitted example workflows?
> Upgrade non-transitive Jackson dependencies from org.codehaus.jackson to
> com.fasterxml.jackson
> --
>
> Key: OOZIE-3172
> URL: https://issues.apache.org/jira/browse/OOZIE-3172
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0b1
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3172.001.patch, OOZIE-3172.002.patch
>
>
> Jackson 1.9.3 is way too old, and has several security vulnerabilities as
> well. Jackson 2.9.2 covers most of these.
> Let's switch from {{org.codehaus.jackson}} to {{com.fasterxml.jackson}} in
> Oozie's direct (non-transitive) dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345236#comment-16345236 ] Attila Sasvari commented on OOZIE-2814: --- I am not sure if it makes sense to keep older workflow schemas. Those are there in earlier releases and also under version control. Patch 00 only changes example workflows, but I guess we want to also update documentation and unit tests. > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2814: -- Attachment: OOZIE-2814-00.patch > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > Attachments: OOZIE-2814-00.patch > > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-2814) OYA: Update example workflows to newest schemas
[ https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reassigned OOZIE-2814: - Assignee: Attila Sasvari > OYA: Update example workflows to newest schemas > --- > > Key: OOZIE-2814 > URL: https://issues.apache.org/jira/browse/OOZIE-2814 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Robert Kanter > Assignee: Attila Sasvari >Priority: Blocker > > OOZIE-2687 will add in a whole set of newer schemas that are more > Yarn-centric. Most, if not all, examples currently use pretty old versions > of the schemas. We should take this opportunity to update the examples to > the newest schemas added by OOZIE-2687, which should help demonstrate them. > It may make sense to keep the older schemas too, to show that older workflows > still work. We have some examples where we have two workflow XML files that > do the same thing in different ways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari reassigned OOZIE-3170:
-
Assignee: Jason Phelps
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> Caused by: java.lang.NullPointerException
> at
> org.apache.oozie.service.ShareLibService.cacheActionKeySharelibConfList(ShareLibService.java:878)
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:132)
> ... 8 more
> 2018-01-25 10:53:58,130 INFO PartitionDependencyManagerService:520 - SERVER[]
> PartitionDependencyManagerService initialized. Dependency cache is
> org.apache.oozie.dependency.hcat.SimpleHCatDependencyCache
> 2018-01-25 10:53:58,131 FATAL Services:514 - SERVER[] Runtime Exception
> during Services Load. Check your list of {0} or {1}
> java.lang.NullPointerException
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:81)
> at
> org.apache.oozie.service.PartitionDependencyManagerService.init(PartitionDependencyManagerService.java:71)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(DiagBundleCollectorDriver.java:56)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.main(DiagBundleCollectorDriver.java:52)
> 2018-01-25 10:53:58,132 FATAL Services:514 - SERVER[] E0103: Could not load
> service classes, null
> org.apache.oozie.service.ServiceException: E0103: Could not load service
> classes, null
> at org.apache.oozie.service.Services.loadServices(Services.java:309)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.run(Di
[jira] [Commented] (OOZIE-3170) Oozie Diagnostic Bundle tool fails with NPE due to missing service class
[
https://issues.apache.org/jira/browse/OOZIE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345077#comment-16345077
]
Attila Sasvari commented on OOZIE-3170:
---
[~jphelps] many thanks for reporting this issue and reviewing related parts of
the codebase. I added you as a contributor to the project and assigned this
Jira to you.
I reproduced that the cited NPE exception is thrown. The diag bundle zip is
however generated and contain sharelib info - it is just using the
OozieClient's
[listShareLib()|https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java#L42]:
{code:java}
$ bin/oozie-diag-bundle-collector.sh -oozie http://localhost:11000/oozie
-output /tmp/jobs/
Checking Connection...Done
Using Temporary Directory:
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/1517319232457-0
Getting Sharelib Information...Done
Getting Configuration...Done
Getting OS Environment Variables...Done
Getting Java System Properties...Done
Getting Queue Dump...Done
Getting Thread Dump...Done
Getting Instrumentation...Done
Getting Metrics...Skipping (Metrics are unavailable)
Creating Zip File: /tmp/jobs/oozie-diag-bundle-1517319233190.zip...Done
$ unzip -l /tmp/jobs/oozie-diag-bundle-1517319233190.zip
68029 01-30-18 14:33 /effective-oozie-site.xml
9876 01-30-18 14:33 /instrumentation.txt
38636 01-30-18 14:33 /java-sys-props.txt
3807 01-30-18 14:33 /os-env-vars.txt
279 01-30-18 14:33 /queue-dump.txt
40032 01-30-18 14:33 /sharelib.txt
102084 01-30-18 14:33 /thread-dump.html{code}
* In fact, I am not sure all those Oozie services are really needed here to be
able to collect diagnostic information. If they are not needed they shall not
be loaded at all.
* There is also another problem. By default, logs generated by the tool appear
in the server log if you run the tool from Oozie's home directory. It can make
things very confusing for an admin or anyone who review Oozie server logs.
Setting up logging is the responsibility of the
[XLogService|[https://github.com/apache/oozie/blob/ef6d0af5edeb18fbc0259d1962ac70f8ad7c2a0c/core/src/main/java/org/apache/oozie/service/XLogService.java#L145]]
and it is started via Services.init(). It can be controlled by the
{{oozie.log.dir}} system property (e.g. \{{export
JAVA_PROPERTIES="-Doozie.log.dir=/tmp/"}} before running the tool). This is
something we should clarify in the documentation of the tool and/or change the
code/script so that logs are put in the directory where the diag bundle is
generated by default.
> Oozie Diagnostic Bundle tool fails with NPE due to missing service class
>
>
> Key: OOZIE-3170
> URL: https://issues.apache.org/jira/browse/OOZIE-3170
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3170-1.patch
>
>
>
> When I ran the below command after doing a clean build from the main branch
> {code:java}
> bin/oozie-diag-bundle-collector.sh -oozie
> http://jphelps-60-1.gce.cloudera.com:11000/oozie -output /tmp/jobs/
> {code}
> It will fail with an NPE. I apologize as I did not copy the client error, but
> the error in oozie.log is below:
> {code:java}
> 2018-01-25 10:53:58,123 ERROR ShareLibService:517 - SERVER[]
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> org.apache.oozie.service.ServiceException: E0104: Could not fully initialize
> service [org.apache.oozie.service.ShareLibService], Not able to cache
> sharelib. An Admin needs to install the sharelib with oozie-setup.sh and
> issue the 'oozie admin' CLI command to update the sharelib
> at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:144)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
> at org.apache.oozie.service.Services.setService(Services.java:372)
> at org.apache.oozie.service.Services.loadServices(Services.java:304)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.initOozieServices(DiagBundleCollectorDriver.java:153)
> at
> org.apache.oozie.tools.diag.DiagBundleCollectorDriver.setHadoopConfig(DiagBundleCollectorDriver.java:135)
> at
> org.apache.oozie.tools.d
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343484#comment-16343484
]
Attila Sasvari commented on OOZIE-1717:
---
OpenJPA 2.4.2 does not support JPA 2.1, so I had to add the following composite
indices with {{java.sql.Statement's execUpdate()}}:
{code:java}
CREATE INDEX I_WF_JOBS_STATUS_CREATED_TIME ON WF_JOBS (status, created_time)
CREATE INDEX I_COORD_ACTIONS_JOB_ID_STATUS ON COORD_ACTIONS (job_id, status)
CREATE INDEX I_COORD_JOBS_STATUS_CREATED_TIME ON COORD_JOBS (status,
created_time)
CREATE INDEX I_COORD_JOBS_STATUS_LAST_MODIFIED_TIME ON COORD_JOBS (status,
last_modified_time)
CREATE INDEX I_COORD_JOBS_PENDING_DONE_MATERIALIZATION_LAST_MODIFIED_TIME ON
COORD_JOBS (pending, done_materialization, last_modified_time)
"CREATE INDEX I_COORD_JOBS_PENDING_LAST_MODIFIED_TIME ON COORD_JOBS (pending,
last_modified_time)
CREATE INDEX I_BUNLDE_JOBS_STATUS_CREATED_TIME ON BUNDLE_JOBS (status,
created_time)
CREATE INDEX I_BUNLDE_JOBS_STATUS_LAST_MODIFIED_TIME ON BUNDLE_JOBS (status,
last_modified_time)
CREATE INDEX I_BUNLDE_ACTIONS_PENDING_LAST_MODIFIED_TIME ON BUNDLE_ACTIONS
(pending, last_modified_time){code}
After the change, the following indices are present in the oozie database:
{code:java}
> SELECT DISTINCT TABLE_NAME, COLUMN_NAME, INDEX_NAME FROM
> INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = 'oozie';
++--+--+
| TABLE_NAME | COLUMN_NAME | INDEX_NAME
|
++--+--+
| BUNDLE_ACTIONS | bundle_action_id | PRIMARY
|
| BUNDLE_ACTIONS | bundle_id | I_BNDLTNS_BUNDLE_ID
|
| BUNDLE_ACTIONS | pending |
I_BUNLDE_ACTIONS_PENDING_LAST_MODIFIED_TIME |
| BUNDLE_ACTIONS | last_modified_time |
I_BUNLDE_ACTIONS_PENDING_LAST_MODIFIED_TIME |
| BUNDLE_JOBS | id | PRIMARY
|
| BUNDLE_JOBS | created_time | I_BNDLJBS_CREATED_TIME
|
| BUNDLE_JOBS | end_time | I_BNDLJBS_END_TIME
|
| BUNDLE_JOBS | last_modified_time | I_BNDLJBS_LAST_MODIFIED_TIME
|
| BUNDLE_JOBS | status | I_BNDLJBS_STATUS
|
| BUNDLE_JOBS | suspended_time | I_BNDLJBS_SUSPENDED_TIME
|
| BUNDLE_JOBS | user_name | I_BNDLJBS_USER_NAME
|
| BUNDLE_JOBS | status |
I_BUNLDE_JOBS_STATUS_CREATED_TIME |
| BUNDLE_JOBS | created_time |
I_BUNLDE_JOBS_STATUS_CREATED_TIME |
| BUNDLE_JOBS | status |
I_BUNLDE_JOBS_STATUS_LAST_MODIFIED_TIME |
| BUNDLE_JOBS | last_modified_time |
I_BUNLDE_JOBS_STATUS_LAST_MODIFIED_TIME |
| COORD_ACTIONS | id | PRIMARY
|
| COORD_ACTIONS | created_time | I_CRD_TNS_CREATED_TIME
|
| COORD_ACTIONS | external_id | I_CRD_TNS_EXTERNAL_ID
|
| COORD_ACTIONS | job_id | I_CRD_TNS_JOB_ID
|
| COORD_ACTIONS | last_modified_time | I_CRD_TNS_LAST_MODIFIED_TIME
|
| COORD_ACTIONS | nominal_time | I_CRD_TNS_NOMINAL_TIME
|
| COORD_ACTIONS | rerun_time | I_CRD_TNS_RERUN_TIME
|
| COORD_ACTIONS | status | I_CRD_TNS_STATUS
|
| COORD_ACTIONS | job_id | I_COORD_ACTIONS_JOB_ID_STATUS
|
| COORD_ACTIONS | status | I_COORD_ACTIONS_JOB_ID_STATUS
|
| COORD_JOBS | id | PRIMARY
|
| COORD_JOBS | bundle_id | I_CRD_JBS_BUNDLE_ID
|
| COORD_JOBS | created_time | I_CRD_JBS_C
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-003.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-003.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 65385: OOZIE-1717 add indices to speed up db queries
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65385/
---
Review request for oozie.
Repository: oozie-git
Description
---
Add indices to speed up SELECT operations.
Index user_name column
for purge service OOZIE-1803:
CREATE INDEX "OOZIEDB"."BUNDLE_JOBS_END_TIME_IDX" ON "BUNDLE_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."COORD_JOBS_END_TIME_IDX" ON "COORD_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."WF_JOBS_END_TIME_IDX" ON "WF_JOBS" ("END_TIME");
CREATE INDEX "OOZIEDB"."COORD_ACTIONS_JOB_ID_IDX" ON "COORD_ACTIONS" ("JOB_ID");
CREATE INDEX "OOZIEDB"."COORD_JOBS_BUNDLE_ID_IDX" ON "COORD_JOBS" ("BUNDLE_ID");
CREATE INDEX "OOZIEDB"."WF_ACTIONS_WF_ID_IDX" ON "WF_ACTIONS" ("WF_ID");
CREATE INDEX "OOZIEDB"."WF_JOBS_PARENT_ID_IDX" ON "WF_JOBS" ("PARENT_ID");
CREATE INDEX "OOZIEDB"."BUNDLE_ACTIONS_BUNDLE_ID_IDX" ON "BUNDLE_ACTIONS"
("BUNDLE_ID");
Composite indices to speed up operations on the web UI
Diffs
-
core/src/main/java/org/apache/oozie/BundleJobBean.java
b42f12841c2f9cb6d508cd2ac2f393e6407d1cf0
core/src/main/java/org/apache/oozie/CoordinatorJobBean.java
99c7d3c474d9e1e22e2f5a927c03d71b9292b663
core/src/main/java/org/apache/oozie/WorkflowJobBean.java
cd9a6df42e552a0607606a244b81978747d89b41
tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
a8b2234966c967f9ad9d722628bd486012b14bd3
Diff: https://reviews.apache.org/r/65385/diff/1/
Testing
---
Thanks,
Attila Sasvari
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-002.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-002.patch, > OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Release Oozie 4.3.1 (candidate 3)
+1 (non-binding) - Verified hash, signature files, rat report - Built Oozie distro with hadoop.version set to 2.6.0 using the uber, hadoop-2, spark-1 profiles - Submitted example workflows/coordinators/bundles (except hive related ones) on a hadoop 2.6.0 in pseudo distributed mode - Verified Tomcat version was 6.0.53 - Clicked through the web UI, log retrieval worked, job DAG-s were rendered, version was correct. - Verified instrumentation, SLA pages; they are now HTML based as expected (OOZIE-2533). However, json2.js is referenced by webapp/index.html and it is not found (not a blocker in my opinion). - Enabled SLA, re-submitted sla coordinator example, and retrieved SLA details for the "one-op-wf" application, graph was displayed properly. I used JDK 1.8.0_121 On Sun, Jan 28, 2018 at 7:09 AM, Satish Saley <satishsa...@apache.org> wrote: > Hi, > > I have created a build for Oozie 4.3.1, candidate 3. > > It includes all the changes decided earlier except OOZIE-3018, OOZIE-2771 > and > OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService which we are > not picking. > OOZIE-2771 makes changes to file created in OOZIE-2666. > OOZIE-3072 makes changes to files created in OOZIE-1770. > > > Keys to verify the signature of the release artifact are available at > http://www.apache.org/dist/oozie/KEYS > > Please download, test, and try it out: > > http://people.apache.org/~satishsaley/oozie-4.3.1-rc3/ > > > The release, md5 signature, gpg signature, and rat report can be found at > the above address. > > Vote closes on 01/31/2018 11:59 PM PT. > -- -- Attila Sasvari Software Engineer <http://www.cloudera.com/>
Re: Review Request 65287: OOZIE-3157 - Setup truststore so that it also works in HTTP only mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65287/#review196331 --- Ship it! Ship It! - Attila Sasvari On Jan. 26, 2018, 1:16 p.m., Kinga Marton wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65287/ > --- > > (Updated Jan. 26, 2018, 1:16 p.m.) > > > Review request for oozie, András Piros, Attila Sasvari, and Peter Cseh. > > > Repository: oozie-git > > > Description > --- > > oozie.https.truststore.file is not read and used when oozie.https.enabled is > false in oozie-site xml. As a result, the Oozie server will be unable to > communicate with servers with unsigned certificate. It is a critical problem > as authentication may involve external servers (for example KMS with > self-signed certificate). Submitting a workflow in such an environment can > result in an exception like: > > `2018-01-08 10:13:51,471 WARN > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: > SERVER[myserver] KMS provider at [https://myserver:16000/kms/v1/] threw an > IOException: > javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: unable to > find valid certification path to requested target > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) > at > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) > at > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) > at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) > at > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) > at > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) > at > sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:144) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExte
Re: Review Request 65287: OOZIE-3157 - Setup truststore so that it also works in HTTP only mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65287/#review196328 --- docs/src/site/twiki/DG_QuickStart.twiki Line 149 (original), 149 (patched) <https://reviews.apache.org/r/65287/#comment275900> typo: certifacate - Attila Sasvari On Jan. 26, 2018, 1:13 p.m., Kinga Marton wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65287/ > --- > > (Updated Jan. 26, 2018, 1:13 p.m.) > > > Review request for oozie, András Piros, Attila Sasvari, and Peter Cseh. > > > Repository: oozie-git > > > Description > --- > > oozie.https.truststore.file is not read and used when oozie.https.enabled is > false in oozie-site xml. As a result, the Oozie server will be unable to > communicate with servers with unsigned certificate. It is a critical problem > as authentication may involve external servers (for example KMS with > self-signed certificate). Submitting a workflow in such an environment can > result in an exception like: > > `2018-01-08 10:13:51,471 WARN > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: > SERVER[myserver] KMS provider at [https://myserver:16000/kms/v1/] threw an > IOException: > javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: unable to > find valid certification path to requested target > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) > at > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) > at > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) > at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) > at > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) > at > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) > at > sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:144) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancing
Re: Review Request 65287: OOZIE-3157 - Setup truststore so that it also works in HTTP only mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65287/#review196327 --- docs/src/site/twiki/DG_QuickStart.twiki Lines 149 (patched) <https://reviews.apache.org/r/65287/#comment275897> If the Oozie server needs to establish secure connection with an external server with a self-signed certifacate, make sure you specify the location of a truststore that contains required certificates. docs/src/site/twiki/DG_QuickStart.twiki Lines 150 (patched) <https://reviews.apache.org/r/65287/#comment275898> It can be done by configuring =oozie.https.truststore.file= in =oozie-site.xml=, or by setting the =javax.net.ssl.trustStore= system property. - Attila Sasvari On Jan. 26, 2018, noon, Kinga Marton wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65287/ > --- > > (Updated Jan. 26, 2018, noon) > > > Review request for oozie, András Piros, Attila Sasvari, and Peter Cseh. > > > Repository: oozie-git > > > Description > --- > > oozie.https.truststore.file is not read and used when oozie.https.enabled is > false in oozie-site xml. As a result, the Oozie server will be unable to > communicate with servers with unsigned certificate. It is a critical problem > as authentication may involve external servers (for example KMS with > self-signed certificate). Submitting a workflow in such an environment can > result in an exception like: > > `2018-01-08 10:13:51,471 WARN > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: > SERVER[myserver] KMS provider at [https://myserver:16000/kms/v1/] threw an > IOException: > javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: unable to > find valid certification path to requested target > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) > at > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) > at > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) > at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) > at > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) > at > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) > at > sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:144) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > a
[jira] [Updated] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1717: -- Attachment: OOZIE-1717-01.patch > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah > Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch, OOZIE-1717-01.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340291#comment-16340291
]
Attila Sasvari commented on OOZIE-1717:
---
[~todd.farmer] thanks for the recommendation.
Oozie uses OpenJPA 2.4.2 to talk with databases (Oracle, PostgreSQL, etc. are
also supported). OpenJPA 2.4.2 supports only [JPA
2.0|http://openjpa.apache.org/]. Do you know how we can define a covering
composite index with OpenJPA, I could not find it in the documentation.
I tried the following things in org.apache.oozie.CoordinatorJobBean
{code:java}
@Index(name = "index1", columnNames = {"status, created_time"})
@Index(name = "index2", columnNames = {"status", "created_time"}){code}
but none of them resulted in a composite index. The [Table
annotation|[https://github.com/apache/oozie/blob/acdf291aef48fd0f9a5b0e3270afacb7ee4a1543/core/src/main/java/org/apache/oozie/CoordinatorJobBean.java#L158|https://github.com/apache/oozie/blob/acdf291aef48fd0f9a5b0e3270afacb7ee4a1543/core/src/main/java/org/apache/oozie/CoordinatorJobBean.java#L158]]]
in CoordinatorJobBean did not accept indices either.
I can see in
[OozieDBCLI|https://github.com/apache/oozie/blob/acdf291aef48fd0f9a5b0e3270afacb7ee4a1543/tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java#L977]
that a clustered index is created "manually". We can follow this pattern if
really needed.
It would be good to have a list for more interesting (composite) indexes.
> Index user_name column
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-1717-00.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3093) Test Scenarios for Oozie 5.0
[ https://issues.apache.org/jira/browse/OOZIE-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-3093: -- Fix Version/s: (was: 5.0.0b1) 5.0.0 > Test Scenarios for Oozie 5.0 > > > Key: OOZIE-3093 > URL: https://issues.apache.org/jira/browse/OOZIE-3093 > Project: Oozie > Issue Type: Test >Affects Versions: 5.0.0b1 >Reporter: Artem Ervits >Priority: Blocker > Fix For: 5.0.0 > > > This is umbrella jira to track test scenarios on Oozie 5.0. In my discussion > with [~gezapeti] we can have a joint effort to test different test cases. > [~clayb] please provide your input as well. > 1. Oozie HA > 2. Oozie secure cluster > 3. Run all examples > 4. Execute against every Hadoop profile > etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1987) Fix existing findbugs warnings
[ https://issues.apache.org/jira/browse/OOZIE-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-1987: -- Fix Version/s: (was: 5.0.0b1) 5.0.0 > Fix existing findbugs warnings > -- > > Key: OOZIE-1987 > URL: https://issues.apache.org/jira/browse/OOZIE-1987 > Project: Oozie > Issue Type: Bug >Reporter: Robert Kanter >Assignee: Jan Hentschel >Priority: Major > Fix For: 5.0.0 > > > Now that we have OOZIE-1793, we should actually fix the warnings that > findbugs finds in the existing codebase. It may be a good idea to split this > into subtasks or multiple patches as there are a lot of warnings: over 500 in > oozie-core alone! > If we think something is ok, but findbugs complains, we should exclude it. > There's a way to create a file that lists exclusions (Hadoop does this). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-2600) OYA: Update Documentation
[
https://issues.apache.org/jira/browse/OOZIE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2600:
--
Fix Version/s: (was: 5.0.0b1)
5.0.0
> OYA: Update Documentation
> -
>
> Key: OOZIE-2600
> URL: https://issues.apache.org/jira/browse/OOZIE-2600
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: oya, 5.0.0
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 5.0.0
>
>
> Based on the work done in OOZIE-2896, document also the way how
> {{oozie.launcher.override.\*}} and {{oozie.launcher.prepend.\*}} parameters
> function.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2942) Fix Findbugs warnings in oozie-examples
[
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2942:
--
Fix Version/s: (was: 5.0.0b1)
5.0.0
> Fix Findbugs warnings in oozie-examples
> ---
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
> Issue Type: Sub-task
> Components: build
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close
> stream on exception [org.apache.oozie.example.DateList] At
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in
> org.apache.oozie.example.DateList.main(String[])
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample,
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues
> at LocalOozieExample.java:[line 76]Path continues at
> LocalOozieExample.java:[line 77]Path continues at
> LocalOozieExample.java:[line 78]Path continues at
> LocalOozieExample.java:[line 81]Path continues at
> LocalOozieExample.java:[line 88]Path continues at
> LocalOozieExample.java:[line 89]Path continues at
> LocalOozieExample.java:[line 91]Path continues at
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail
> to close stream [org.apache.oozie.example.LocalOozieExample] At
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal
> representation by returning Repeatable.baseline
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose
> internal representation by storing an externally mutable object into
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-2975) code clean up in pig sharelib, replace Exception with more explicit, add try with resources, StringBuilder instead of StringBuffer
[ https://issues.apache.org/jira/browse/OOZIE-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2975: -- Fix Version/s: (was: 5.0.0b1) 5.0.0 > code clean up in pig sharelib, replace Exception with more explicit, add try > with resources, StringBuilder instead of StringBuffer > -- > > Key: OOZIE-2975 > URL: https://issues.apache.org/jira/browse/OOZIE-2975 > Project: Oozie > Issue Type: Improvement >Reporter: Artem Ervits >Assignee: Artem Ervits >Priority: Trivial > Labels: newbie > Fix For: 5.0.0 > > Attachments: OOZIE-2975-0.patch > > > 1. Replaced StringBuffer with StringBuilder > 2. Replaced Exception with IOException and ExecException > 3. Replaced old Hashmap notation with diamond notation > 4. Added try-with-resources notation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3169) Create a download page for releases
Attila Sasvari created OOZIE-3169: - Summary: Create a download page for releases Key: OOZIE-3169 URL: https://issues.apache.org/jira/browse/OOZIE-3169 Project: Oozie Issue Type: New Feature Components: site Reporter: Attila Sasvari Today to download a release, you need to click on the Releases tab and select a release. It will take a user to an external page where you can download the release from an Apache mirrors (e.g. [http://www.apache.org/dyn/closer.lua/oozie/5.0.0-beta1).] It would be better to create a page on the Oozie site to download releases and related hash and PGP signature files. This way a visitor could stay on the Oozie site. Examples: [http://hadoop.apache.org/releases.html] [https://flume.apache.org/releases/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65287: OOZIE-3157 - Setup truststore so that it also works in HTTP only mode
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65287/#review196112 --- server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java Lines 169 (patched) <https://reviews.apache.org/r/65287/#comment275623> What happens if a truststore password starts/ends with whitespaces on purpose? trim() would remove leading and trailing whitespaces and Oozie won't be able to open the truststore? - Attila Sasvari On Jan. 24, 2018, 12:19 p.m., Kinga Marton wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65287/ > --- > > (Updated Jan. 24, 2018, 12:19 p.m.) > > > Review request for oozie, András Piros, Attila Sasvari, and Peter Cseh. > > > Repository: oozie-git > > > Description > --- > > oozie.https.truststore.file is not read and used when oozie.https.enabled is > false in oozie-site xml. As a result, the Oozie server will be unable to > communicate with servers with unsigned certificate. It is a critical problem > as authentication may involve external servers (for example KMS with > self-signed certificate). Submitting a workflow in such an environment can > result in an exception like: > > `2018-01-08 10:13:51,471 WARN > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: > SERVER[myserver] KMS provider at [https://myserver:16000/kms/v1/] threw an > IOException: > javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: unable to > find valid certification path to requested target > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) > at > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) > at > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) > at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) > at > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) > at > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) > at > sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:144) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283) > at &g
[jira] [Commented] (OOZIE-2681) fix javadoc to compile on JDK8 again
[
https://issues.apache.org/jira/browse/OOZIE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337408#comment-16337408
]
Attila Sasvari commented on OOZIE-2681:
---
{{mvn clean deploy -Papache-release -DskipTests fails with the following while
generating Javadoc:}}
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:2.10.4:jar (attach-javadocs) on
project oozie-tools: MavenReportException: Error while generating Javadoc:
[ERROR] Exit code: 1 -
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:44:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:44:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:49:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBExportCLI.java:49:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:82:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:82:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:88:
error: self-closing element not allowed
[ERROR] *
[ERROR] ^
[ERROR]
/Users/asasvari/workspace/apache/oozie_master_push/tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java:88:
warning: empty tag
[ERROR] *
[ERROR] ^
[ERROR]
{code}
> fix javadoc to compile on JDK8 again
>
>
> Key: OOZIE-2681
> URL: https://issues.apache.org/jira/browse/OOZIE-2681
> Project: Oozie
> Issue Type: Sub-task
> Components: build
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Peter Cseh
>Priority: Critical
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2681-00.patch, OOZIE-2681-01.patch,
> OOZIE-2681-02.patch, OOZIE-2681-03.patch, OOZIE-2681-04.patch,
> OOZIE-2681.06.patch, OOZIE-2681.07.patch, OOZIE-2681.08.patch,
> OOZIE-2681.09.patch, OOZIE-2681.10.patch, OOZIE-2681.11.patch,
> OOZIE-2681.12.patch, OOZIE-2681.13.patch, OOZIE-2681.14.patch,
> OOZIE-2681.15.patch
>
>
> We have a lot of Javadoc build failures, which causes Oozie to fail to build
> on Java 8, which is more strict than Java 6 or 7 was. OOZIE-2178 fixed all
> of these that existed at that time, but we've since introduced a lot more.
> OOZIE-2487 made a workaround to prevent this from causing the build to fail,
> but we should fix these (again) and remove the OOZIE-2487 workaround.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[
https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335989#comment-16335989
]
Attila Sasvari commented on OOZIE-1717:
---
I can imagine adding index on the created_time column of COORD_JOBS , could
help too (a lot of coordinator jobs might be created). I saw this in slow
query log when going through the coordinator jobs:
{code:java}
SELECT t0.id, t0.app_name, t0.status, t0.user_name, t0.group_name,
t0.start_time, t0.end_time, t0.app_path, t0.concurrency, t0.frequency,
t0.last_action, t0.next_matd_time, t0.created_time, t0.time_unit, t0.time_zone,
t0.time_out, t0.bundle_id FROM COORD_JOBS t0 WHERE ((t0.status = 'RUNNING' OR
t0.status = 'PREP' OR t0.status = 'SUSPENDED' OR t0.status = 'RUNNINGWITHERROR'
OR t0.status = 'PREPSUSPENDED' OR t0.status = 'SUSPENDEDWITHERROR' OR t0.status
= 'PREPPAUSED' OR t0.status = 'PAUSED' OR t0.status = 'PAUSEDWITHERROR') AND
t0.status IS NOT NULL) ORDER BY t0.created_time DESC LIMIT 100, 100;{code}
> Index user_name column
> --
>
> Key: OOZIE-1717
> URL: https://issues.apache.org/jira/browse/OOZIE-1717
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
> Assignee: Attila Sasvari
>Priority: Minor
> Attachments: OOZIE-1717-00.patch
>
>
> User_name is one of the frequently used filter. Adding index should avoid
> full db scan.
> Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-1717) Index user_name column
[ https://issues.apache.org/jira/browse/OOZIE-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335934#comment-16335934 ] Attila Sasvari commented on OOZIE-1717: --- [~gezapeti] thanks for the review, I will do such changes. I will also set "4" for DB_VERSION_FOR_5_0 following existing pattern. I ran some explain query for selects that appeared in slow query log of MariaDB and after applying the new indices I saw that for example filesort was not used anymore (proper index was used). Anyway, I want to load a really large database and continue investigation (clicking though web UI and watching slow query-s). It would be nice to have a list of the most frequent queries. > Index user_name column > -- > > Key: OOZIE-1717 > URL: https://issues.apache.org/jira/browse/OOZIE-1717 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Attila Sasvari >Priority: Minor > Attachments: OOZIE-1717-00.patch > > > User_name is one of the frequently used filter. Adding index should avoid > full db scan. > Index need to be done for BUNDLE_JOBS,COORD_JOBS and WF_JOBS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
