date:20170403

[jira] [Created] (OOZIE-2855) Add ability to set "X-Priority" header in email action

2017-04-03 Thread James Kinley (JIRA)

James Kinley created OOZIE-2855:
---

 Summary: Add ability to set "X-Priority" header in email action
 Key: OOZIE-2855
 URL: https://issues.apache.org/jira/browse/OOZIE-2855
 Project: Oozie
  Issue Type: New Feature
  Components: action
Reporter: James Kinley
Assignee: James Kinley






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs

2017-04-03 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953965#comment-15953965
 ] 

Hadoop QA commented on OOZIE-2701:
--

Testing JIRA OOZIE-2701

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1912
.Tests rerun: 15
.Tests failed at first run: org.apache.oozie.jms.TestJMSJobEventListener,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3766/

> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, 
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, 
> OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, 
> OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch, 
> OOZIE-2701-11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2853) Printing out Sqoop Configuration Properties may reveal passwords

2017-04-03 Thread Robert Kanter (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953815#comment-15953815
 ] 

Robert Kanter commented on OOZIE-2853:
--

Instead of doing this piecemeal (see OOZIE-2779, OOZIE-2803, etc) per action, 
could we repurpose this JIRA to mask passwords from all action types?

> Printing out Sqoop Configuration Properties may reveal passwords
> 
>
> Key: OOZIE-2853
> URL: https://issues.apache.org/jira/browse/OOZIE-2853
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> Sample from Oozie launcher's log:
> {code}
> Sqoop Configuration Properties:
> 
> ...
> oozie.sqoop.args.5=--password
> oozie.sqoop.args.6=examplPassword
> {code}
> [SqoopMain|https://github.com/apache/oozie/blob/branch-4.3/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L156]
>  prints out hadoop configuration that may contain sensitive information.
> Note: Sqoop command arguments are redacted.
> {code}
> Sqoop command arguments :
> ...
>  --password
>  
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : oozie-trunk-find-patches-available #155655

2017-04-03 Thread Apache Jenkins Server

See

Build failed in Jenkins: oozie-trunk-find-patches-available #155654

2017-04-03 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 6bac84745b9c62907e8cc6a16bad6c76ac3eb9c6 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6bac84745b9c62907e8cc6a16bad6c76ac3eb9c6
 > git rev-list 6bac84745b9c62907e8cc6a16bad6c76ac3eb9c6 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson2343519379284721100.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 599k0  599k0 0   371k  0 --:--:--  0:00:01 --:--:--  371k100 
1423k0 1423k0 0   549k  0 --:--:--  0:00:02 --:--:--  549k100 
1951k0 1951k0 0   539k  0 --:--:--  0:00:03 --:--:--  539k100 
2007k0 2007k0 0   527k  0 --:--:--  0:00:03 --:--:--  527k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs

2017-04-03 Thread Abhishek Bafna (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Bafna updated OOZIE-2701:
--
Attachment: OOZIE-2701-11.patch

[~gezapeti] Added the Doc. Please check. Thanks.

> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, 
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, 
> OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, 
> OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch, 
> OOZIE-2701-11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

2017-04-03 Thread Abhishek Bafna (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953654#comment-15953654
 ] 

Abhishek Bafna commented on OOZIE-2845:
---

+1. Committed to master. Thanks [~pbacsko] for the patch.

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 5.0.0
>
> Attachments: OOZIE-2845-001.patch, OOZIE-2845-002.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create 
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null, 
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time. 
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2844) Increase stability of Oozie actions when log4j.properties is missing or not readable

2017-04-03 Thread Andras Piros (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953552#comment-15953552
 ] 

Andras Piros commented on OOZIE-2844:
-

[~gezapeti] [~pbacsko] can you please have a look?

> Increase stability of Oozie actions when log4j.properties is missing or not 
> readable
> 
>
> Key: OOZIE-2844
> URL: https://issues.apache.org/jira/browse/OOZIE-2844
> Project: Oozie
>  Issue Type: Improvement
>  Components: action
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 4.3.0
>
> Attachments: OOZIE-2844.001.patch, OOZIE-2844.002.patch
>
>
> In several Oozie actions we face the problem that if {{log4j.properties}} is 
> not readable by current thread, or not present on the classpath, while taking 
> an {{URL.openStream()}} call afterwards, an {{IOException}} is thrown and the 
> action won't run.
> Such actions are:
> * 
> [*{{DistcpMain}}*|https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135]
> * 
> [*{{HiveMain}}*|https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146]
> * 
> [*{{PigMain}}*|https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173]
> * 
> [*{{PigMainWithOldApi}}*|https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160]
> * 
> [*{{SparkMain}}*|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430]
> * 
> [*{{SqoopMain}}*|https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124]
> Extend 
> [*{{LauncherMain.run()}}*|https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65]
>  by a convenience method that loads Hadoop's {{log4j.properties}} in 
> beforehand. In case {{URL.openStream()}} would throw {{IOException}} we 
> resort to a call to a new file {{hadoop-default-log4j.properties}} on the 
> classpath (packaged along w/ {{LauncherMain.class}}) that has the same 
> contents as the {{log4j.properties}} that is not accessible / readable / 
> present.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2844) Increase stability of Oozie actions when log4j.properties is missing or not readable

2017-04-03 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953528#comment-15953528
 ] 

Hadoop QA commented on OOZIE-2844:
--

Testing JIRA OOZIE-2844

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1907
.Tests rerun: 14
.Tests failed at first run: 
org.apache.oozie.tools.TestOozieSharelibCLI,org.apache.oozie.service.TestPartitionDependencyManagerEhcache,org.apache.oozie.service.TestPartitionDependencyManagerService,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3765/

> Increase stability of Oozie actions when log4j.properties is missing or not 
> readable
> 
>
> Key: OOZIE-2844
> URL: https://issues.apache.org/jira/browse/OOZIE-2844
> Project: Oozie
>  Issue Type: Improvement
>  Components: action
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 4.3.0
>
> Attachments: OOZIE-2844.001.patch, OOZIE-2844.002.patch
>
>
> In several Oozie actions we face the problem that if {{log4j.properties}} is 
> not readable by current thread, or not present on the classpath, while taking 
> an {{URL.openStream()}} call afterwards, an {{IOException}} is thrown and the 
> action won't run.
> Such actions are:
> * 
> [*{{DistcpMain}}*|https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135]
> * 
> [*{{HiveMain}}*|https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146]
> * 
> [*{{PigMain}}*|https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173]
> * 
> [*{{PigMainWithOldApi}}*|https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160]
> * 
> [*{{SparkMain}}*|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430]
> * 
> [*{{SqoopMain}}*|https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124]
>

Jenkins build is back to normal : oozie-trunk-precommit-build #3765

2017-04-03 Thread Apache Jenkins Server

See

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

2017-04-03 Thread Peter Bacsko (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953450#comment-15953450
 ] 

Peter Bacsko commented on OOZIE-2845:
-

ping [~abhishekbafna]

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch, OOZIE-2845-002.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create 
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null, 
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time. 
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2854) Oozie should handle transient DB problems

2017-04-03 Thread Peter Bacsko (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2854:

Description: 
There can be problems when Oozie cannot update the database properly. Recently, 
we have experienced erratic behavior with two setups:

* MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
locking which might cause a transaction to rollback if there are two or more 
parallel transaction running and one of them cannot complete because of a 
conflict.

* MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
Oozie might get "Communications link failure" exception during the failover.

The problem is that failed DB transactions later might cause a workflow (which 
are started/re-started by RecoveryService) to get stuck. It's not clear to us 
how this happens but it has to do with the fact that certain DB updates are not 
executed.

The solution is to use some sort of retry logic with exponential backoff if the 
DB update fails. We could start with a 100ms wait time which is doubled at 
every retry. The operation can be considered a failure if it still fails after 
10 attempts. These values could be configurable. We should discuss initial 
values in the scope of this JIRA.

Note that this solution is to handle *transient* failures. If the DB is down 
for a longer period of time, we have to accept that the internal state of Oozie 
is corrupted.

  was:
There can be problems when Oozie cannot update the database properly. Recently, 
we have experienced erratic behavior with two setups:

* MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
locking which might cause a transaction to rollback if there are two or more 
parallel transaction running and one of them cannot complete because of a 
conflict.

* MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
Oozie might get "Communications link failure" exception during the failover.

The problem is that failed DB transactions later might cause a workflow (which 
are started/re-started by RecoveryService) to get stuck. It's not clear to us 
how this happens but it has to do with the fact that certain DB updates are not 
executed.

The solution is to use some sort of retry logic with exponential backoff if the 
DB update fails. We could start with a 100ms wait time which is doubled at 
every retry. The operation can be considered a failure if it still fails after 
10 attempts. These values could be configurable. We should discuss initial 
values in the scope of this JIRA.

Note that this solution is to handle *transient* failures. If the DB is long 
for a longer period of time, we have to accept that the internal state of Oozie 
is corrupted.


> Oozie should handle transient DB problems
> -
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>
> There can be problems when Oozie cannot update the database properly. 
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
> locking which might cause a transaction to rollback if there are two or more 
> parallel transaction running and one of them cannot complete because of a 
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow 
> (which are started/re-started by RecoveryService) to get stuck. It's not 
> clear to us how this happens but it has to do with the fact that certain DB 
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if 
> the DB update fails. We could start with a 100ms wait time which is doubled 
> at every retry. The operation can be considered a failure if it still fails 
> after 10 attempts. These values could be configurable. We should discuss 
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is down 
> for a longer period of time, we have to accept that the internal state of 
> Oozie is corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (OOZIE-2854) Oozie should handle transient DB problems

2017-04-03 Thread Peter Bacsko (JIRA)

Peter Bacsko created OOZIE-2854:
---

 Summary: Oozie should handle transient DB problems
 Key: OOZIE-2854
 URL: https://issues.apache.org/jira/browse/OOZIE-2854
 Project: Oozie
  Issue Type: Improvement
  Components: core
Reporter: Peter Bacsko
Assignee: Peter Bacsko


There can be problems when Oozie cannot update the database properly. Recently, 
we have experienced erratic behavior with two setups:

* MySQL was set up with the Galera cluster manager. Galera uses cluster-wide 
optimistic locking which might cause a transaction to rollback if there are two 
or more parallel transaction running and one of them cannot complete because of 
a conflict.

* Another setup is MySQL with Percona XtraDB Cluster. If one of the MySQL 
instances is killed, Oozie might get "Communications link failure" exception. 

The problem is that failed DB transactions later might cause a workflow (which 
are started/re-started by RecoveryService) to get stuck. It's not clear to us 
how this happens but it has to do with the fact that certain DB updates are not 
executed.

The solution is to use some sort of retry logic with exponential backoff if the 
DB update fails. We could start with a 100ms wait time which is doubled at 
every retry. The operation can be considered a failure if it still fails after 
10 attempts. These values could be configurable. We should discuss initial 
values in the scope of this JIRA.

Note that this solution is to handle *transient* failures. If the DB is long 
for a longer period of time, we have to accept that the internal state of Oozie 
is corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2854) Oozie should handle transient DB problems

2017-04-03 Thread Peter Bacsko (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2854:

Description: 
There can be problems when Oozie cannot update the database properly. Recently, 
we have experienced erratic behavior with two setups:

* MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
locking which might cause a transaction to rollback if there are two or more 
parallel transaction running and one of them cannot complete because of a 
conflict.

* MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
Oozie might get "Communications link failure" exception during the failover.

The problem is that failed DB transactions later might cause a workflow (which 
are started/re-started by RecoveryService) to get stuck. It's not clear to us 
how this happens but it has to do with the fact that certain DB updates are not 
executed.

The solution is to use some sort of retry logic with exponential backoff if the 
DB update fails. We could start with a 100ms wait time which is doubled at 
every retry. The operation can be considered a failure if it still fails after 
10 attempts. These values could be configurable. We should discuss initial 
values in the scope of this JIRA.

Note that this solution is to handle *transient* failures. If the DB is long 
for a longer period of time, we have to accept that the internal state of Oozie 
is corrupted.

  was:
There can be problems when Oozie cannot update the database properly. Recently, 
we have experienced erratic behavior with two setups:

* MySQL was set up with the Galera cluster manager. Galera uses cluster-wide 
optimistic locking which might cause a transaction to rollback if there are two 
or more parallel transaction running and one of them cannot complete because of 
a conflict.

* Another setup is MySQL with Percona XtraDB Cluster. If one of the MySQL 
instances is killed, Oozie might get "Communications link failure" exception. 

The problem is that failed DB transactions later might cause a workflow (which 
are started/re-started by RecoveryService) to get stuck. It's not clear to us 
how this happens but it has to do with the fact that certain DB updates are not 
executed.

The solution is to use some sort of retry logic with exponential backoff if the 
DB update fails. We could start with a 100ms wait time which is doubled at 
every retry. The operation can be considered a failure if it still fails after 
10 attempts. These values could be configurable. We should discuss initial 
values in the scope of this JIRA.

Note that this solution is to handle *transient* failures. If the DB is long 
for a longer period of time, we have to accept that the internal state of Oozie 
is corrupted.


> Oozie should handle transient DB problems
> -
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>
> There can be problems when Oozie cannot update the database properly. 
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
> locking which might cause a transaction to rollback if there are two or more 
> parallel transaction running and one of them cannot complete because of a 
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow 
> (which are started/re-started by RecoveryService) to get stuck. It's not 
> clear to us how this happens but it has to do with the fact that certain DB 
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if 
> the DB update fails. We could start with a 100ms wait time which is doubled 
> at every retry. The operation can be considered a failure if it still fails 
> after 10 attempts. These values could be configurable. We should discuss 
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is long 
> for a longer period of time, we have to accept that the internal state of 
> Oozie is corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2844) Increase stability of Oozie actions when log4j.properties is missing or not readable

2017-04-03 Thread Andras Piros (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2844:

Attachment: OOZIE-2844.002.patch

Addressed patch analysis and RAT warnings.

> Increase stability of Oozie actions when log4j.properties is missing or not 
> readable
> 
>
> Key: OOZIE-2844
> URL: https://issues.apache.org/jira/browse/OOZIE-2844
> Project: Oozie
>  Issue Type: Improvement
>  Components: action
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 4.3.0
>
> Attachments: OOZIE-2844.001.patch, OOZIE-2844.002.patch
>
>
> In several Oozie actions we face the problem that if {{log4j.properties}} is 
> not readable by current thread, or not present on the classpath, while taking 
> an {{URL.openStream()}} call afterwards, an {{IOException}} is thrown and the 
> action won't run.
> Such actions are:
> * 
> [*{{DistcpMain}}*|https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135]
> * 
> [*{{HiveMain}}*|https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146]
> * 
> [*{{PigMain}}*|https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173]
> * 
> [*{{PigMainWithOldApi}}*|https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160]
> * 
> [*{{SparkMain}}*|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430]
> * 
> [*{{SqoopMain}}*|https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124]
> Extend 
> [*{{LauncherMain.run()}}*|https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65]
>  by a convenience method that loads Hadoop's {{log4j.properties}} in 
> beforehand. In case {{URL.openStream()}} would throw {{IOException}} we 
> resort to a call to a new file {{hadoop-default-log4j.properties}} on the 
> classpath (packaged along w/ {{LauncherMain.class}}) that has the same 
> contents as the {{log4j.properties}} that is not accessible / readable / 
> present.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (OOZIE-2853) Printing out Sqoop Configuration Properties may reveal passwords

2017-04-03 Thread Attila Sasvari (JIRA)

Attila Sasvari created OOZIE-2853:
-

 Summary: Printing out Sqoop Configuration Properties may reveal 
passwords
 Key: OOZIE-2853
 URL: https://issues.apache.org/jira/browse/OOZIE-2853
 Project: Oozie
  Issue Type: Bug
Reporter: Attila Sasvari


Sample from Oozie launcher's log:
{code}
Sqoop Configuration Properties:

...
oozie.sqoop.args.5=--password
oozie.sqoop.args.6=examplPassword
{code}

[SqoopMain|https://github.com/apache/oozie/blob/branch-4.3/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L156]
 prints out hadoop configuration that may contain sensitive information.

Note: Sqoop command arguments are redacted.
{code}
Sqoop command arguments :
...
 --password
 
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs

2017-04-03 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953243#comment-15953243
 ] 

Peter Cseh commented on OOZIE-2701:
---

[~abhishekbafna], could you update the documentation 
[here|https://oozie.apache.org/docs/4.3.0/DG_HCatalogIntegration.html#HCatalog_URI_Format]
 or create a separate issue for that?

+1 otherwise.


> Oozie to support Multiple HCatalog URIs
> ---
>
> Key: OOZIE-2701
> URL: https://issues.apache.org/jira/browse/OOZIE-2701
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2701-00.patch, OOZIE-2701-01.patch, 
> OOZIE-2701-02.patch, OOZIE-2701-03.patch, OOZIE-2701-04.patch, 
> OOZIE-2701-05.patch, OOZIE-2701-06.patch, OOZIE-2701-07.patch, 
> OOZIE-2701-08.patch, OOZIE-2701-09.patch, OOZIE-2701-10.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57870: OOZIE-2827 More directly view of the coordinator’s history from perspective of workflow action.

2017-04-03 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57870/#review170838
---




core/src/main/java/org/apache/oozie/command/coord/CoordWfActionInfoXCommand.java
Lines 81 (patched)


Are we sure that the default, uninitialized instance is good enough?

So let's say we store the empty bean as a placeholder. How will that be 
returned to the user?

As I can see, status will be PREP, wfId is null, id is null, everything 
else is pretty much null.

Is this sufficient for us? Won't that be misleading?



core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionGetJPAExecutor.java
Lines 82 (patched)


In this case please return null instead of creating a new bean and handle 
the null case on the caller side.



core/src/test/java/org/apache/oozie/command/coord/TestCoordWfActionInfoXCommand.java
Lines 63 (patched)


Is it possible to refactor the test so that they run as a single testcase?



core/src/test/java/org/apache/oozie/servlet/TestV2JobServlet.java
Lines 262 (patched)


How much overhead does this test have if we run all subtests as a separate 
testcase?

Having them separately would be much better.


- Peter Bacsko


On ápr. 1, 2017, 11:31 de, Alonzo Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57870/
> ---
> 
> (Updated ápr. 1, 2017, 11:31 de)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> More detailed view of the coordinator’s history can be observed from 
> perspective of  workflow action.
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 8ddb1f8 
>   core/src/main/java/org/apache/oozie/CoordinatorEngine.java 2f9f822 
>   core/src/main/java/org/apache/oozie/ErrorCode.java b03ad06 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordWfActionInfoXCommand.java
>  PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionGetJPAExecutor.java
>  0b7f50d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 03acbc1 
>   core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 6c30f5d 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 564db2a 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordWfActionInfoXCommand.java
>  PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/servlet/MockCoordinatorEngineService.java 
> 4fc8653 
>   core/src/test/java/org/apache/oozie/servlet/TestV2JobServlet.java fb203a6 
> 
> 
> Diff: https://reviews.apache.org/r/57870/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alonzo Zhou
> 
>

[jira] Subscription: Oozie Patch Available

2017-04-03 Thread jira

Issue Subscription
Filter: Oozie Patch Available (101 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2851  spelling mistakes in examples
https://issues.apache.org/jira/browse/OOZIE-2851
OOZIE-2850  Fix default callback notifications
https://issues.apache.org/jira/browse/OOZIE-2850
OOZIE-2849  Fix build warnings when assembling directories
https://issues.apache.org/jira/browse/OOZIE-2849
OOZIE-2848  Override sharelib.tgz in distro when rebuilding Oozie
https://issues.apache.org/jira/browse/OOZIE-2848
OOZIE-2846  TestSSLServerConnectorFactory fails with Jetty 9.3 and 9.4
https://issues.apache.org/jira/browse/OOZIE-2846
OOZIE-2845  Replace reflection-based code which sets variable in HiveConf
https://issues.apache.org/jira/browse/OOZIE-2845
OOZIE-2844  Increase stability of Oozie actions when log4j.properties is 
missing or not readable
https://issues.apache.org/jira/browse/OOZIE-2844
OOZIE-2843  Enhance logging inside ZKLocksService and MemoryLocksService
https://issues.apache.org/jira/browse/OOZIE-2843
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2827  More directly view of the coordinator’s history from perspective of 
workflow action.
https://issues.apache.org/jira/browse/OOZIE-2827
OOZIE-2825  Custom Authentication doc page is not well formatted
https://issues.apache.org/jira/browse/OOZIE-2825
OOZIE-2824  Fix typos in documentation
https://issues.apache.org/jira/browse/OOZIE-2824
OOZIE-2818  Can't overwrite oozie.action.max.output.data on a per-workflow basis
https://issues.apache.org/jira/browse/OOZIE-2818
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2796  oozie.action.keep.action.dir not getting noticed
https://issues.apache.org/jira/browse/OOZIE-2796
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its

[jira] [Created] (OOZIE-2855) Add ability to set "X-Priority" header in email action

[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs

[jira] [Commented] (OOZIE-2853) Printing out Sqoop Configuration Properties may reveal passwords

Jenkins build is back to normal : oozie-trunk-find-patches-available #155655

Build failed in Jenkins: oozie-trunk-find-patches-available #155654

[jira] [Updated] (OOZIE-2701) Oozie to support Multiple HCatalog URIs

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[jira] [Commented] (OOZIE-2844) Increase stability of Oozie actions when log4j.properties is missing or not readable

[jira] [Commented] (OOZIE-2844) Increase stability of Oozie actions when log4j.properties is missing or not readable

Jenkins build is back to normal : oozie-trunk-precommit-build #3765

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[jira] [Updated] (OOZIE-2854) Oozie should handle transient DB problems

[jira] [Created] (OOZIE-2854) Oozie should handle transient DB problems

[jira] [Updated] (OOZIE-2854) Oozie should handle transient DB problems

[jira] [Updated] (OOZIE-2844) Increase stability of Oozie actions when log4j.properties is missing or not readable

[jira] [Created] (OOZIE-2853) Printing out Sqoop Configuration Properties may reveal passwords

[jira] [Commented] (OOZIE-2701) Oozie to support Multiple HCatalog URIs

Re: Review Request 57870: OOZIE-2827 More directly view of the coordinator’s history from perspective of workflow action.

[jira] Subscription: Oozie Patch Available

19 matches

Site Navigation

Mail list logo

Footer information