[jira] [Commented] (OOZIE-3039) Add Sqoop2 action
[ https://issues.apache.org/jira/browse/OOZIE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644021#comment-16644021 ] Satish Subhashrao Saley commented on OOZIE-3039: Updated an initial working patch. > Add Sqoop2 action > - > > Key: OOZIE-3039 > URL: https://issues.apache.org/jira/browse/OOZIE-3039 > Project: Oozie > Issue Type: Improvement >Reporter: Hu Liu, >Assignee: Satish Subhashrao Saley >Priority: Major > Attachments: OOZIE-3039-1.patch > > > Now Sqoop2 is released and we should add support for it. > I'm glad to work on it if anyone could assign it to me. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3039) Add Sqoop2 action
[ https://issues.apache.org/jira/browse/OOZIE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley reassigned OOZIE-3039: -- Assignee: Hu Liu, (was: Satish Subhashrao Saley) > Add Sqoop2 action > - > > Key: OOZIE-3039 > URL: https://issues.apache.org/jira/browse/OOZIE-3039 > Project: Oozie > Issue Type: Improvement >Reporter: Hu Liu, >Assignee: Hu Liu, >Priority: Major > Attachments: OOZIE-3039-1.patch > > > Now Sqoop2 is released and we should add support for it. > I'm glad to work on it if anyone could assign it to me. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3039) Add Sqoop2 action
[ https://issues.apache.org/jira/browse/OOZIE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-3039: --- Attachment: OOZIE-3039-1.patch > Add Sqoop2 action > - > > Key: OOZIE-3039 > URL: https://issues.apache.org/jira/browse/OOZIE-3039 > Project: Oozie > Issue Type: Improvement >Reporter: Hu Liu, >Assignee: Satish Subhashrao Saley >Priority: Major > Attachments: OOZIE-3039-1.patch > > > Now Sqoop2 is released and we should add support for it. > I'm glad to work on it if anyone could assign it to me. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3039) Add Sqoop2 action
[ https://issues.apache.org/jira/browse/OOZIE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley reassigned OOZIE-3039: -- Assignee: Satish Subhashrao Saley (was: Hu Liu,) > Add Sqoop2 action > - > > Key: OOZIE-3039 > URL: https://issues.apache.org/jira/browse/OOZIE-3039 > Project: Oozie > Issue Type: Improvement >Reporter: Hu Liu, >Assignee: Satish Subhashrao Saley >Priority: Major > > Now Sqoop2 is released and we should add support for it. > I'm glad to work on it if anyone could assign it to me. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643738#comment-16643738
]
Robert Kanter commented on OOZIE-3136:
--
+1 on not going to log4j2 until Oozie 6. Cleaning up our logging story and
using the bridge in the meantime is probably a good idea.
Our logging is somewhat brittle because of the log streaming. IIRC, we require
log files to be rolled every hour because the log streaming makes some
assumptions about that to improve performance. Similarly, logs can be
automatically gzipped, but we also make assumptions about that and the
filenames. That's part of what the {{OozieRollingPolicy}} was for (it used to
be messier before I added it :)). So if we need to get rid of
{{OozieRollingPolicy}}, then we'll need to either replace it with something
equivalent, or refactor the log streaming code to be more flexible.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643244#comment-16643244
]
Peter Cseh commented on OOZIE-3136:
---
Additionally we have our own Policy:
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/OozieRollingPolicy.java
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3362) When killed, SSH action should kill the spawned processes on target host
Peter Orova created OOZIE-3362: -- Summary: When killed, SSH action should kill the spawned processes on target host Key: OOZIE-3362 URL: https://issues.apache.org/jira/browse/OOZIE-3362 Project: Oozie Issue Type: Improvement Reporter: Peter Orova When killing a workflow running an ssh action, currently only the process executing the ssh-wrapper.sh is killed on the target host. Upon killing, the ssh action should not only kill the process executing the wrapper script, but the one launched by that also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643106#comment-16643106
]
Peter Bacsko commented on OOZIE-3136:
-
Well, yeah, we do access log4j directly:
https://github.com/apache/oozie/blob/ba665da34c23b1fa86bf1570724147e6f8c85b1d/core/src/main/java/org/apache/oozie/service/XLogService.java#L149
https://github.com/apache/oozie/blob/ba665da34c23b1fa86bf1570724147e6f8c85b1d/core/src/main/java/org/apache/oozie/service/XLogService.java#L174-L178
We must go through the whole code to see if there's any direct calls to Log4j
classes, then we have to rewrite them appropriately...
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643098#comment-16643098
]
Peter Bacsko edited comment on OOZIE-3136 at 10/9/18 10:29 AM:
---
Regarding the bridge, check this out:
https://logging.apache.org/log4j/log4j-2.2/log4j-1.2-api/index.html
"To use the Log4j Legacy Bridge just remove all the Log4j 1.x jars from the
application and replace them with the bridge jar. Once in place all logging
that uses Log4j 1.x will be routed to Log4j 2. *However, applications that
attempt to modify legacy Log4j by adding Appenders, Filters, etc may experience
problems if they try to verify the success of these actions as these methods
are largely no-ops*."
Do we do anything like that? I mean, messing around log4j classes directly.
This must be checked.
If we directly use stuff like {{PropertyConfigurator}}, that's a no-op in the
bridge. We must also make sure that initial {{Logger.getLogger()}} calls
configure the Log4j2 based on the original log4j properties file. I just
checked this and does not seem to be the case - there is a converter class, but
looks like we have to call that manually.
So there are quite a few things to consider/validate here.
was (Author: pbacsko):
Regarding the bridge, check this out:
https://logging.apache.org/log4j/log4j-2.2/log4j-1.2-api/index.html
"To use the Log4j Legacy Bridge just remove all the Log4j 1.x jars from the
application and replace them with the bridge jar. Once in place all logging
that uses Log4j 1.x will be routed to Log4j 2. *However, applications that
attempt to modify legacy Log4j by adding Appenders, Filters, etc may experience
problems if they try to verify the success of these actions as these methods
are largely no-ops*."
Do we do anything like that? I mean, messing around log4j classes directly.
This must be checked.
If we directly use stuff like {{PropertyConfigurator}}, that's a no-op in the
bridge. We must also make sure that initial {{Logger.getlogger()}} calls
configure the Log4j2 based on the original log4j properties file. I just
checked this and does not seem to be the case - there is a converter class, but
looks like we have to call that manually.
So there are quite a few things to consider/validate here.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643098#comment-16643098
]
Peter Bacsko commented on OOZIE-3136:
-
Regarding the bridge, check this out:
https://logging.apache.org/log4j/log4j-2.2/log4j-1.2-api/index.html
"To use the Log4j Legacy Bridge just remove all the Log4j 1.x jars from the
application and replace them with the bridge jar. Once in place all logging
that uses Log4j 1.x will be routed to Log4j 2. *However, applications that
attempt to modify legacy Log4j by adding Appenders, Filters, etc may experience
problems if they try to verify the success of these actions as these methods
are largely no-ops*."
Do we do anything like that? I mean, messing around log4j classes directly.
This must be checked.
If we directly use stuff like {{PropertyConfigurator}}, that's a no-op in the
bridge. We must also make sure that initial {{Logger.getlogger()}} calls
configure the Log4j2 based on the original log4j properties file. I just
checked this and does not seem to be the case - there is a converter class, but
looks like we have to call that manually.
So there are quite a few things to consider/validate here.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3336) [persistence] Refactor entity classes to feature PK, FK, and UQ constraints
[
https://issues.apache.org/jira/browse/OOZIE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julia Kinga Marton reassigned OOZIE-3336:
-
Assignee: Julia Kinga Marton
> [persistence] Refactor entity classes to feature PK, FK, and UQ constraints
> ---
>
> Key: OOZIE-3336
> URL: https://issues.apache.org/jira/browse/OOZIE-3336
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.2.0
>
>
> When an Oozie database grows substantial in size, let's say, over a few
> hundred thousands of {{WorkflowActionBean}}, {{CoordinatorActionBean}}
> instances, we face a couple of performance issues. Here is an analysis why.
> Current Oozie JPA {{@Entity}} usage, and the resulting database DDL, suffers
> from a couple of drawback from a performance point of view:
> * {{@Id}} fields are {{String}}:
> ** leaving no space for database primary key indices to work effectively
> ** those values are calculated in case of {{WorkflowActionBean}},
> {{CoordinatorActionBean}}, and {{BundleActionBean}} instances
> * no foreign constraint is set from {{WorkflowActionBean}} to
> {{WorkflowJobBean}}, from {{CoordinatorActionBean}} to
> {{CoordinatorJobBean}}, or from {{BundleActionBean}} to {{BundleJobBean}}
> instances:
> ** have to assess JPA queries discovering parent-child relationships by hand
> ** no database indices are created, and hence, those queries that contain any
> {{JOIN}} instances are slower
> * no use of unique constraints whatsoever
> * JPA queries are created by hand instead of relying on OpenJPA
> * JPA entities are filled by hand instead of relying on OpenJPA
> Following enhancements are necessary:
> # keeping the existing {{String compositeId}} fields, let's break down the
> contents to following new fields:
> ## {{@Id long id}} - an auto-increment value that is unique across Oozie
> database
> ## {{long currentSequence}} - the sequence number of the current run since
> last Oozie server restart. The first part of the {{compositeId}}
> ## {{Timestamp serverStartupTimestamp}} - the timestamp when the Oozie server
> was last started. The second part of the {{compositeId}}
> ## {{String serverName}} - the third part of the {{compositeId}}
> ## {{String name}} - the fourth and last part of the {{compositeId}}
> ## {{compositeId}} might be calculated when an entity is loaded / persisted,
> and then stored
> # FK constraints:
> ## {{@OneToMany}} fields where we have a list of child references inside
> parent
> ## {{@ManyToOne}} fields where we have a parent reference inside child
> ## pay attention to {{FetchType}}, most of the times {{LAZY}} will be needed
> ## the containment fields should not be {{@Transient}} anymore
> # UQ constraints:
> ## on {{currentSequence}} and {{serverStartupTimestamp}}
> ## on {{currentSequence}} and {{name}}
> # new JPQL queries:
> ## to cover changed parent-child relationships
> ## to get use of each disassembled part of {{originalId}} when doing e.g.
> filtering
> # let JPA fill entities instead performing this by hand
> Following enhancements can be considered as nice-to-have:
> * upgrade to an OpenJPA version that features JPA 2.1's composite indexing
> capability
> * see whether to have an optimistic locking field using {{@Version}} instead
> of ZooKeeper based pessimistic locking would increase High Availability
> characteristics
> * refactor also SLA related entity classes
> It's necessary to have performance benchmarks with some database types like
> MySQL/MariaDB, and PostgreSQL before and after the changes for following use
> cases:
> * {{CoordinatorJobBean}} and {{WorkflowJobBean}} instances up to millions
> * {{CoordinatorActionBean}} and {{WorkflowActionBean}} instances up to tens
> of millions
> * performance for JPQLs that get a list of entities
> * performance of persisting a new entity
> * performance of querying lists of entities based on popular / possible
> filters like the ones used by {{VxJobsServlet}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643051#comment-16643051
]
Peter Bacsko commented on OOZIE-3136:
-
I do agree that we don't have to upgrade right now. That would be a much bigger
undertaking, taking at least weeks to finish and then test properly. Backward
compatibility must be preserved.
bq. if the user has a fat JAR consisting also of log4j12 classes that are
versions incompatible w/ the Oozie server
Fat jar has always been a problem and right now there's not much we can do
about it (we could mitigate it to a certain degree with better classloader
isolation, but again, that's not straightforward). We've seen conflicting
Guavas as well. Also, the YARN side is a different story, we don't even use
loggers there, just sysout. On a separate note, we shall focus on those sysouts
too, it doesn't look good :)
bq. I don't recall any issues with Oozie's logging performance
I do remember one instance and I'm sure that so do you: when we enabled full
DEBUG loggin in the tests, that completely choked log4j and upstream tests
timed out. Pretty nasty. Indeed, lot of output was generated, but still, such
things should not happen with a well-performing logging library.
Just the fact that we depend on an ancient library that hasn't been actively
maintained in the last 3-4 years is alone a good reason for an upgrade -
probably in Oozie 6.0.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643018#comment-16643018
]
Peter Cseh commented on OOZIE-3136:
---
I don't really see a compelling reason for upgrading to Log4j2 in the short
term. It has better performance characteristics, but I don't recall any issues
with Oozie's logging performance. I think moving to SLF4J is a good approach
though as it will make the future Log4J2 migration easier.
Really, with proper Slf4j usage, the log4j version shouldn't really matter.
We're using a very old logger though to support log streaming across
active-active Oozie servers in case of HA. That will be probably the hardest
part of the migration and I think we could address that in slf4j as well.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OOZIE-3327) TestConcurrentCopyFromLocal is flaky
[
https://issues.apache.org/jira/browse/OOZIE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julia Kinga Marton resolved OOZIE-3327.
---
Resolution: Cannot Reproduce
> TestConcurrentCopyFromLocal is flaky
>
>
> Key: OOZIE-3327
> URL: https://issues.apache.org/jira/browse/OOZIE-3327
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Julia Kinga Marton
>Priority: Major
>
> There are some unit tests last modified by OOZIE-2829 that [*are
> flaky*|https://builds.apache.org/job/PreCommit-OOZIE-Build/752/testReport/]:
> *
> {{org.apache.oozie.tools.TestConcurrentCopyFromLocal#testConcurrentCopyFromLocalHighThreadNr()}}
> *
> {{org.apache.oozie.tools.TestConcurrentCopyFromLocal#testConcurrentCopyFromLocalSameFileNrAndThreadNr()}}
> *
> {{org.apache.oozie.tools.TestConcurrentCopyFromLocal#testConcurrentCopyFromLocalMoreThreadsThanFiles()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3327) TestConcurrentCopyFromLocal is flaky
[
https://issues.apache.org/jira/browse/OOZIE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643014#comment-16643014
]
Julia Kinga Marton commented on OOZIE-3327:
---
I have followed the precommit results and it seems that this test case is not
flaky anymore, so I will close this issue. Feel free to reopen it, if you see
TestConcurrentCopyFromLocal failing.
> TestConcurrentCopyFromLocal is flaky
>
>
> Key: OOZIE-3327
> URL: https://issues.apache.org/jira/browse/OOZIE-3327
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Julia Kinga Marton
>Priority: Major
>
> There are some unit tests last modified by OOZIE-2829 that [*are
> flaky*|https://builds.apache.org/job/PreCommit-OOZIE-Build/752/testReport/]:
> *
> {{org.apache.oozie.tools.TestConcurrentCopyFromLocal#testConcurrentCopyFromLocalHighThreadNr()}}
> *
> {{org.apache.oozie.tools.TestConcurrentCopyFromLocal#testConcurrentCopyFromLocalSameFileNrAndThreadNr()}}
> *
> {{org.apache.oozie.tools.TestConcurrentCopyFromLocal#testConcurrentCopyFromLocalMoreThreadsThanFiles()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643002#comment-16643002
]
Andras Piros edited comment on OOZIE-3136 at 10/9/18 8:58 AM:
--
[~pbacsko] [~asalamon74] [~kmarton] HADOOP-12596 seems really a good read, as
is HBASE-10092. It seems that we indeed need to go for the {{slf4j-log4j12}}
adapter based approach because:
# in Oozie 5.x, we need that old {{.properties}} based configuration remains
working, else we would break compatibility guidelines for a minor version. In
Oozie 6.0 we may consider moving to {{log4j2}} completely, thus breaking
backwards compatibility guarantees
# as there are many Oozie users that have custom {{.properties}} settings we
need those settings remain working in Oozie 5.x
# caveats of the same type you mentioned do exist on Oozie 5.0: if the user has
a fat JAR consisting also of {{log4j12}} classes that are versions incompatible
w/ the Oozie server / sharelib's version, we can see lots of
{{ClassNotFoundException}} instances, and other types of class loading errors,
as well as strange behavior like logging doesn't work silently at all. The
general recommendation has been to either avoid packing those classes inside
user JARs, or to use {{maven-shade-plugin}} to hide those
All in all, I'm for using adapter classes and maintain {{log4j12}}
configuration compatibility throughout Oozie 5.x. [~gezapeti] [~asasvari] what
do you think?
was (Author: andras.piros):
[~pbacsko] [~asalamon74] [~kmarton] HADOOP-12596 seems really a good read, as
is HBASE-10092. It seems that we indeed need to go for the {{slf4j-log4j12}}
adapter based approach because:
# in Oozie 5.x, we need that old {{.properties}} based configuration remains
working, else we would break compatibility guidelines for a minor version. In
Oozie 6.0 we may consider moving to {{log4j2}} completely, thus breaking
backwards compatibility guarantees
# as there are many Oozie users that have custom {{.properties}} settings we
need those settings remain working in Oozie 5.x
# caveats of the same type you mentioned do exist on Oozie 5.0: if the user has
a fat JAR consisting also of {{log4j12}} classes that are versions incompatible
w/ the Oozie server / sharelib's version, we can see lots of
{{ClassNotFoundException}} instances, and other types of class loading errors,
as well as strange behavior like logging doesn't work silently at all. The
general recommendation has been to either avoid packing those classes inside
user JARs, or to use {{maven-shade-plugin}} to hide those
All in all, I'm for using adapter classes and maintain {{log4j12}}
configuration throughout Oozie 5.x. [~gezapeti] [~asasvari] what do you think?
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643002#comment-16643002
]
Andras Piros commented on OOZIE-3136:
-
[~pbacsko] [~asalamon74] [~kmarton] HADOOP-12596 seems really a good read, as
is HBASE-10092. It seems that we indeed need to go for the {{slf4j-log4j12}}
adapter based approach because:
# in Oozie 5.x, we need that old {{.properties}} based configuration remains
working, else we would break compatibility guidelines for a minor version. In
Oozie 6.0 we may consider moving to {{log4j2}} completely, thus breaking
backwards compatibility guarantees
# as there are many Oozie users that have custom {{.properties}} settings we
need those settings remain working in Oozie 5.x
# caveats of the same type you mentioned do exist on Oozie 5.0: if the user has
a fat JAR consisting also of {{log4j12}} classes that are versions incompatible
w/ the Oozie server / sharelib's version, we can see lots of
{{ClassNotFoundException}} instances, and other types of class loading errors,
as well as strange behavior like logging doesn't work silently at all. The
general recommendation has been to either avoid packing those classes inside
user JARs, or to use {{maven-shade-plugin}} to hide those
All in all, I'm for using adapter classes and maintain {{log4j12}}
configuration throughout Oozie 5.x. [~gezapeti] [~asasvari] what do you think?
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642946#comment-16642946
]
Julia Kinga Marton edited comment on OOZIE-3136 at 10/9/18 8:25 AM:
I totally agree that at some point we will need to make a complete migration.
This is cleaner path and only using this approach we will be able to use the
new functionalities introduced by Log4j2.
Since there is no way to keep compatibility, I would change the .property based
configuration to the xml configuration.
I was also thinking about waiting LOG4J2-63 but I did not see any progress
since last year.
was (Author: kmarton):
I totally agree that at some point we will need to make a complete migration.
This is cleaner and only using this approach will be able to use the new
functionalities introduced by Log4j2.
Since there is no way to keep compatibility, I would change the .property based
configuration to the xml configuration.
I was also thinking about waiting LOG4J2-63 but I did not see any progress
since last year.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x
[
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642946#comment-16642946
]
Julia Kinga Marton commented on OOZIE-3136:
---
I totally agree that at some point we will need to make a complete migration.
This is cleaner and only using this approach will be able to use the new
functionalities introduced by Log4j2.
Since there is no way to keep compatibility, I would change the .property based
configuration to the xml configuration.
I was also thinking about waiting LOG4J2-63 but I did not see any progress
since last year.
> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (90 issues) Subscriber: ooziedaily Key Summary OOZIE-3361 Document embedded XML job submission mode https://issues.apache.org/jira/browse/OOZIE-3361 OOZIE-3338 Remove SVN references https://issues.apache.org/jira/browse/OOZIE-3338 OOZIE-3326 Sqoop Action should support tez delegation tokens for hive-import https://issues.apache.org/jira/browse/OOZIE-3326 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3265 properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to appear together https://issues.apache.org/jira/browse/OOZIE-3265 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3249 [tools] Instrumentation log parser https://issues.apache.org/jira/browse/OOZIE-3249 OOZIE-3218 Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space https://issues.apache.org/jira/browse/OOZIE-3218 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3194 Oozie should set proper permissions to sharelib after upload https://issues.apache.org/jira/browse/OOZIE-3194 OOZIE-3186 Oozie is unable to use configuration linked using jceks://file/... https://issues.apache.org/jira/browse/OOZIE-3186 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3120 maven-assembly-plugin fails when bumped from 2.2.1 https://issues.apache.org/jira/browse/OOZIE-3120 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password https://issues.apache.org/jira/browse/OOZIE-2779 OOZIE-2736 Reduce the number of threads during test execution https://issues.apache.org/jira/browse/OOZIE-2736 OOZIE-2714 Detect conflicting resources during class loading https://issues.apache.org/jira/browse/OOZIE-2714 OOZIE-2694 Add logging for FsActionExecutor https://issues.apache.org/jira/browse/OOZIE-2694 OOZIE-2693 SimpleHCatDependencyCache.removeMissingDependency can throw NPE https://issues.apache.org/jira/browse/OOZIE-2693 OOZIE-2692 Oozie job submit doesn't report error message to user if there is any
