date:20171114

[jira] Subscription: Oozie Patch Available

2017-11-14 Thread jira

Issue Subscription
Filter: Oozie Patch Available (116 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3127  Remove redundant check for user
https://issues.apache.org/jira/browse/OOZIE-3127
OOZIE-3126  Add option to allow list of users to access system config
https://issues.apache.org/jira/browse/OOZIE-3126
OOZIE-3121  bump all maven plugins to latest versions
https://issues.apache.org/jira/browse/OOZIE-3121
OOZIE-3118  fix for error: self-closing element not allowed
https://issues.apache.org/jira/browse/OOZIE-3118
OOZIE-3117  fix for warning: no @throws for
https://issues.apache.org/jira/browse/OOZIE-3117
OOZIE-3116  fix for warning: no description for @throws
https://issues.apache.org/jira/browse/OOZIE-3116
OOZIE-3115  fix for javadoc warning: empty  tag
https://issues.apache.org/jira/browse/OOZIE-3115
OOZIE-3114  fix for warning: no @return
https://issues.apache.org/jira/browse/OOZIE-3114
OOZIE-3113  Retry for ZK lock release
https://issues.apache.org/jira/browse/OOZIE-3113
OOZIE-3112  SparkConfigrationService overwrites properties provided via 
--properties-file option in SparkAction
https://issues.apache.org/jira/browse/OOZIE-3112
OOZIE-3107  org.apache.oozie.action.hadoop.TestHiveMain#testMain is flaky
https://issues.apache.org/jira/browse/OOZIE-3107
OOZIE-3105  testJMXInstrumentation from the 
org.apache.oozie.util.TestMetricsInstrumentation class is flaky
https://issues.apache.org/jira/browse/OOZIE-3105
OOZIE-3094  fix for grammar mistake
https://issues.apache.org/jira/browse/OOZIE-3094
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3083  Make improved version Info backward compatible
https://issues.apache.org/jira/browse/OOZIE-3083
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2973  Make sure Oozie works with Hadoop 3 
https://issues.apache.org/jira/browse/OOZIE-2973
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2945  Update SpotBugs to stable version after GA
https://issues.apache.org/jira/browse/OOZIE-2945
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2900  Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers 
before submission
https://issues.apache.org/jira/browse/OOZIE-2900
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action

[jira] [Updated] (OOZIE-3128) Oozie Shell Action configuration to disable Parent Process Token

2017-11-14 Thread Prabhu Joseph (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph updated OOZIE-3128:
-
Description: 
When executing pig / spark commands from Oozie Shell on secure cluster it fails 
as it picks the HADOOP_TOKEN_FILE_LOCATION of
parent process Oozie Launcher's MapTask and gets affected. Below are some of 
the issues where pig / spark command runs fine when we execute separate
but with Oozie Shell action fails.

Most of the users expect the same command which ran fine separate to run fine 
on Oozie Shell Action. And this issue will be
difficult to debug. Better to have a separate configuration for Oozie Shell 
Action
to unset the Token of parent process by default and if needed we can enable it. 
This way everyone will be aware of the Parent
process token and it's impact to their shell script.

1. Oozie Shell with Spark Command accessing Secure Hive Metastore

kinit command
spark-submit --class SparkHiveExample --master yarn-cluster --files 
/etc/spark/conf/hive-site.xml --jars 
/tmp/datanucleus-core-3.2.10.jar,/tmp/datanucleus-rdbms-3.2.9.jar,/tmp/datanucleus-api-jdo-3.2.6.jar
 spark_hive.jar

Spark Job fails with

{code}
17/05/09 12:24:29 ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at 
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at 
org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
at 
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at 
org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:204)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:249)
at 
org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:345)
at 
org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:255)
at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:459)
at 
org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
at

[jira] [Updated] (OOZIE-3128) Oozie Shell Action configuration to disable Parent Process Token

2017-11-14 Thread Prabhu Joseph (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph updated OOZIE-3128:
-
Description: 
When executing pig / spark commands from Oozie Shell on secure cluster it fails 
as it picks the HADOOP_TOKEN_FILE_LOCATION of
parent process Oozie Launcher's MapTask and gets affected. Below are some of 
the issues where pig / spark command runs fine when we execute separate
but with Oozie Shell action fails.

Most of the users expect the same command which ran fine separate to run fine 
on Oozie Shell Action. And this issue will be
difficult to debug. Better to have a separate configuration for Oozie Shell 
Action
to unset the Token of parent process by default and if needed we can enable it. 
This way everyone will be aware of the Parent
process token and it's impact to their shell script.

1. Oozie Shell with Spark Command accessing Secure HBase cluster

kinit command
spark-submit --class SparkHiveExample --master yarn-cluster --files 
/etc/spark/conf/hive-site.xml --jars 
/tmp/datanucleus-core-3.2.10.jar,/tmp/datanucleus-rdbms-3.2.9.jar,/tmp/datanucleus-api-jdo-3.2.6.jar
 spark_hive.jar

Spark Job fails with

{code}
17/05/09 12:24:29 ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at 
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at 
org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
at 
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at 
org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:204)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:249)
at 
org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:345)
at 
org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:255)
at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:459)
at 
org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
at

[jira] [Created] (OOZIE-3128) Oozie Shell Action configuration to disable Parent Process Token

2017-11-14 Thread Prabhu Joseph (JIRA)

Prabhu Joseph created OOZIE-3128:


 Summary: Oozie Shell Action configuration to disable Parent 
Process Token
 Key: OOZIE-3128
 URL: https://issues.apache.org/jira/browse/OOZIE-3128
 Project: Oozie
  Issue Type: Bug
  Components: action
Affects Versions: 4.3.0
Reporter: Prabhu Joseph


When executing pig / spark commands from Oozie Shell on secure cluster it fails 
as it picks the HADOOP_TOKEN_FILE_LOCATION of
parent process Oozie Launcher's MapTask and gets affected. Below are some of 
the issues where pig / spark command runs fine when we execute separate
but with Oozie Shell action fails.

Most of the users expect the same command which ran fine separate to run fine 
on Oozie Shell Action. And this issue will be
difficult to debug. Better to have a separate configuration for Oozie Shell 
Action
to unset the Token of parent process by default and if needed we can enable it. 
This way everyone will be aware of the Parent
process token and it's impact to their shell script.

1. Oozie Shell with Spark Command accessing Secure HBase cluster

kinit command
spark-submit --class SparkHiveExample --master yarn-cluster --files 
/etc/spark/conf/hive-site.xml --jars 
/tmp/datanucleus-core-3.2.10.jar,/tmp/datanucleus-rdbms-3.2.9.jar,/tmp/datanucleus-api-jdo-3.2.6.jar
 spark_hive.jar

Spark Job fails with

{code}
17/05/09 12:24:29 ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at 
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at 
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at 
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at 
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at 
org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
at 
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at 
org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:204)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:249)
at 
org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:345)
at 
org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:255)
at

[jira] [Commented] (OOZIE-3126) Add option to allow list of users to access system config

2017-11-14 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252741#comment-16252741
 ] 

Rohini Palaniswamy commented on OOZIE-3126:
---

+1

> Add option to allow list of users to access system config
> -
>
> Key: OOZIE-3126
> URL: https://issues.apache.org/jira/browse/OOZIE-3126
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3126-1.patch
>
>
>Currently we allow access to system config for all users. Though we mask 
> any sensitive information it is unnecessary to show it to all users. Can 
> restrict the access to admins, but services like Hue use the server side 
> config values to display UI accordingly (For eg: Server Timezone, whether SLA 
> is configured, what kind of credentials are supported, etc). So it would be 
> good to support restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Oozie 4.3.1. release

2017-11-14 Thread satish saley

Hi,

I have combined the above two lists, also included couple of JIRAs I filed
today. I will go ahead I put fix versions on these.

1. OOZIE-3095 Job submit command is not logged in audit log (puru)

2. OOZIE-3031 Coord job with only unresolved dependencies doesn't
timeout

(puru via satishsaley)

3. OOZIE-3079 Filtering coordinators returns bundle id as null
(satishsaley)

4. OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)

5. OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations

 should overwrite default values set in Hadoop's configuration files

(gezapeti)

6. OOZIE-2885 Running Spark actions should not need Hive on the
classpath

(satishsaley)

7. OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd

(satishsaley)

8. OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation
tokens

(andras.piros via gezapeti)

9. OOZIE-3028 Oozie Pig Action fails with no python dependencies
(dbist13 via

rohini)

10.  OOZIE-3018 Use Hadoop's CredentialProvider for passwords in

SchemaCheckerService (andras.piros via gezapeti)

11.  OOZIE-2984 Parse spark-defaults.conf values with spaces without
needing the

quotes (andras.piros via gezapeti)

12.  OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)

13.  OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server
start

(puru via satishsaley)

14.  OOZIE-2862 Coord change command doesn't change job to running if job
was

killed without creating any actions (puru)

15.  OOZIE-2807 Oozie gets RM delegation token even for checking job status

(satishsaley)

16.  OOZIE-2811 Add support for filtering out properties from

SparkConfigurationService (gezapeti via rkanter)

17.  OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate
sharelibs

(gezapeti via rkanter)

18.  OOZIE-2803 Mask passwords when printing out configs/args in
MapReduceMain

and SparkMain (pbacsko via rkanter)

19.  OOZIE-2799 Setting log location for spark sql on hive (satishsaley)

20.  OOZIE-2792 Hive2 action is not parsing Spark application ID from log
file

properly when Hive is on Spark (zhengxb2005 via rkanter)

21.  OOZIE-2786 Pass Oozie workflow ID and settings to Spark application

configuration (zhengxb2005 via rkanter)

22.  OOZIE-2790 log4j configuration is not passed to spark executors

(satishsaley)

23.  OOZIE-2787 Oozie distributes application jar twice making the spark job

fail (satishsaley)

24.  OOZIE-2777 Config-default.xml longer than 64k results in

java.io.UTFDataFormatException (gezapeti via harsh)

25.  OOZIE-2771 Allow retrieving keystore and truststore passwords from
Hadoop

Credential Provider (asasvari via abhishekbafna)

26.  OOZIE-2748 NPE in LauncherMapper.printArgs() (pbacsko via rkanter)

27.  OOZIE-2654 Zookeeper dependent services should not depend on

Connectionstate to be valid before cleaning up (venkatnrangan via

abhishekbafna)

28.  OOZIE-2690 OOZIE NPE while executing kill() (abhishekbafna via

jaydeepvishwakarma)

29.  OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)

30.  OOZIE-3077 SLA documentation fixes (satishsaley)

31.  OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
(dbist13 via rkanter)

32.  OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)

33.  OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
Hentschel via rkanter)

34.  OOZIE-3127 Remove redundant check for user

35.  OOZIE-3126 Add option to allow list of users to access system config

 Regards,

Satish

On Tue, Nov 14, 2017 at 3:08 PM, Rohini Palaniswamy  wrote:

> Peter,
>   OOZIE-3004 Forked action retry info is not working in your list is a
> bug fix for the feature OOZIE-2691 which is new in 5.0. So that cannot be
> included. But others are good to be included.
>
> Regards,
> Rohini
>
> On Tue, Nov 14, 2017 at 2:58 PM, Robert Kanter 
> wrote:
>
> > Sounds good to me.
> >
> > On Tue, Nov 14, 2017 at 10:09 AM, Peter Bacsko 
> > wrote:
> >
> > > Good idea.
> > >
> > > If I'm not mistaken, there's also a pending review/patch submitted by
> > > someone (apologies, but I forgot how it was) that cannot be applied to
> > the
> > > master anymore, at least not easily. It's worth checking that too
> > (perhaps
> > > there are more than one patch).
> > >
> > > Peter
> > >
> > > On Tue, Nov 14, 2017 at 5:07 PM, Peter Cseh 
> > wrote:
> > >
> > > > Hey!
> > > >
> > > > I agree, users wanting stability would appreciate a more stable 4.3.1
> > or
> > > > even 4.4.0 to be released with improvements.
> > > > I think we could include the following issues as well:
> > > > OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
> > > > OOZIE-3077 SLA documentation fixes (satishsaley)
> > > > OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
> > > > (dbist13 via rkanter)
> > > > OOZIE-2747

Re: Oozie 4.3.1. release

2017-11-14 Thread Rohini Palaniswamy

Peter,
  OOZIE-3004 Forked action retry info is not working in your list is a
bug fix for the feature OOZIE-2691 which is new in 5.0. So that cannot be
included. But others are good to be included.

Regards,
Rohini

On Tue, Nov 14, 2017 at 2:58 PM, Robert Kanter  wrote:

> Sounds good to me.
>
> On Tue, Nov 14, 2017 at 10:09 AM, Peter Bacsko 
> wrote:
>
> > Good idea.
> >
> > If I'm not mistaken, there's also a pending review/patch submitted by
> > someone (apologies, but I forgot how it was) that cannot be applied to
> the
> > master anymore, at least not easily. It's worth checking that too
> (perhaps
> > there are more than one patch).
> >
> > Peter
> >
> > On Tue, Nov 14, 2017 at 5:07 PM, Peter Cseh 
> wrote:
> >
> > > Hey!
> > >
> > > I agree, users wanting stability would appreciate a more stable 4.3.1
> or
> > > even 4.4.0 to be released with improvements.
> > > I think we could include the following issues as well:
> > > OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
> > > OOZIE-3077 SLA documentation fixes (satishsaley)
> > > OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
> > > (dbist13 via rkanter)
> > > OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)
> > > OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
> > > Hentschel via rkanter)
> > > OOZIE-3004 Forked action retry info is not working
> > > Most of these are minor improvements or user-facing fixes which would
> be
> > > nice to get through the door.
> > >
> > > BRs,
> > > gp
> > >
> > > On Tue, Nov 14, 2017 at 12:56 AM, Rohini Palaniswamy <
> > > rohini.adi...@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >I think this would be a good time to do a patch release on 4.3
> > branch.
> > > > The 5.0 release is going to be a major change with the move of
> launcher
> > > > from Mapreduce AM to Oozie AM and might take time to stabilize. So it
> > > would
> > > > be good to do one final patch release on 4.3 branch pulling in some
> > > > important fixes and Spark action related fixes (Spark 2.x support)
> that
> > > > have gone in master so far.  Satish has volunteered to be the Release
> > > > Manager.
> > > >
> > > > Below is a initial list of jiras that I identified.
> > > >
> > > >
> > > > OOZIE-3095 Job submit command is not logged in audit log (puru)
> > > > OOZIE-3031 Coord job with only unresolved dependencies doesn't
> timeout
> > > > (puru via satishsaley)
> > > > OOZIE-3079 Filtering coordinators returns bundle id as null
> > (satishsaley)
> > > > OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> > > > OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
> > > >  should overwrite default values set in Hadoop's configuration files
> > > > (gezapeti)
> > > > OOZIE-2885 Running Spark actions should not need Hive on the
> classpath
> > > > (satishsaley)
> > > > OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
> > > > (satishsaley)
> > > > OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation
> > > tokens
> > > > (andras.piros via gezapeti)
> > > > OOZIE-3028 Oozie Pig Action fails with no python dependencies
> (dbist13
> > > via
> > > > rohini)
> > > > OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
> > > > SchemaCheckerService (andras.piros via gezapeti)
> > > > OOZIE-2984 Parse spark-defaults.conf values with spaces without
> needing
> > > the
> > > > quotes (andras.piros via gezapeti)
> > > > OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
> > > > OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server
> > > start
> > > > (puru via satishsaley)
> > > > OOZIE-2862 Coord change command doesn't change job to running if job
> > was
> > > > killed without creating any actions (puru)
> > > > OOZIE-2807 Oozie gets RM delegation token even for checking job
> status
> > > > (satishsaley)
> > > > OOZIE-2811 Add support for filtering out properties from
> > > > SparkConfigurationService (gezapeti via rkanter)
> > > > OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate
> > sharelibs
> > > > (gezapeti via rkanter)
> > > > OOZIE-2803 Mask passwords when printing out configs/args in
> > MapReduceMain
> > > > and SparkMain (pbacsko via rkanter)
> > > > OOZIE-2799 Setting log location for spark sql on hive (satishsaley)
> > > > OOZIE-2792 Hive2 action is not parsing Spark application ID from log
> > file
> > > > properly when Hive is on Spark (zhengxb2005 via rkanter)
> > > > OOZIE-2786 Pass Oozie workflow ID and settings to Spark application
> > > > configuration (zhengxb2005 via rkanter)
> > > > OOZIE-2790 log4j configuration is not passed to spark executors
> > > > (satishsaley)
> > > > OOZIE-2787 Oozie distributes application jar twice making the spark
> job
> > > > fail (satishsaley)
> > > > OOZIE-2777 Config-default.xml longer than 64k results in
> > > >

Re: Oozie 4.3.1. release

2017-11-14 Thread Robert Kanter

Sounds good to me.

On Tue, Nov 14, 2017 at 10:09 AM, Peter Bacsko  wrote:

> Good idea.
>
> If I'm not mistaken, there's also a pending review/patch submitted by
> someone (apologies, but I forgot how it was) that cannot be applied to the
> master anymore, at least not easily. It's worth checking that too (perhaps
> there are more than one patch).
>
> Peter
>
> On Tue, Nov 14, 2017 at 5:07 PM, Peter Cseh  wrote:
>
> > Hey!
> >
> > I agree, users wanting stability would appreciate a more stable 4.3.1 or
> > even 4.4.0 to be released with improvements.
> > I think we could include the following issues as well:
> > OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
> > OOZIE-3077 SLA documentation fixes (satishsaley)
> > OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
> > (dbist13 via rkanter)
> > OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)
> > OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
> > Hentschel via rkanter)
> > OOZIE-3004 Forked action retry info is not working
> > Most of these are minor improvements or user-facing fixes which would be
> > nice to get through the door.
> >
> > BRs,
> > gp
> >
> > On Tue, Nov 14, 2017 at 12:56 AM, Rohini Palaniswamy <
> > rohini.adi...@gmail.com> wrote:
> >
> > > Hi folks,
> > >I think this would be a good time to do a patch release on 4.3
> branch.
> > > The 5.0 release is going to be a major change with the move of launcher
> > > from Mapreduce AM to Oozie AM and might take time to stabilize. So it
> > would
> > > be good to do one final patch release on 4.3 branch pulling in some
> > > important fixes and Spark action related fixes (Spark 2.x support) that
> > > have gone in master so far.  Satish has volunteered to be the Release
> > > Manager.
> > >
> > > Below is a initial list of jiras that I identified.
> > >
> > >
> > > OOZIE-3095 Job submit command is not logged in audit log (puru)
> > > OOZIE-3031 Coord job with only unresolved dependencies doesn't timeout
> > > (puru via satishsaley)
> > > OOZIE-3079 Filtering coordinators returns bundle id as null
> (satishsaley)
> > > OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> > > OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
> > >  should overwrite default values set in Hadoop's configuration files
> > > (gezapeti)
> > > OOZIE-2885 Running Spark actions should not need Hive on the classpath
> > > (satishsaley)
> > > OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
> > > (satishsaley)
> > > OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation
> > tokens
> > > (andras.piros via gezapeti)
> > > OOZIE-3028 Oozie Pig Action fails with no python dependencies (dbist13
> > via
> > > rohini)
> > > OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
> > > SchemaCheckerService (andras.piros via gezapeti)
> > > OOZIE-2984 Parse spark-defaults.conf values with spaces without needing
> > the
> > > quotes (andras.piros via gezapeti)
> > > OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
> > > OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server
> > start
> > > (puru via satishsaley)
> > > OOZIE-2862 Coord change command doesn't change job to running if job
> was
> > > killed without creating any actions (puru)
> > > OOZIE-2807 Oozie gets RM delegation token even for checking job status
> > > (satishsaley)
> > > OOZIE-2811 Add support for filtering out properties from
> > > SparkConfigurationService (gezapeti via rkanter)
> > > OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate
> sharelibs
> > > (gezapeti via rkanter)
> > > OOZIE-2803 Mask passwords when printing out configs/args in
> MapReduceMain
> > > and SparkMain (pbacsko via rkanter)
> > > OOZIE-2799 Setting log location for spark sql on hive (satishsaley)
> > > OOZIE-2792 Hive2 action is not parsing Spark application ID from log
> file
> > > properly when Hive is on Spark (zhengxb2005 via rkanter)
> > > OOZIE-2786 Pass Oozie workflow ID and settings to Spark application
> > > configuration (zhengxb2005 via rkanter)
> > > OOZIE-2790 log4j configuration is not passed to spark executors
> > > (satishsaley)
> > > OOZIE-2787 Oozie distributes application jar twice making the spark job
> > > fail (satishsaley)
> > > OOZIE-2777 Config-default.xml longer than 64k results in
> > > java.io.UTFDataFormatException (gezapeti via harsh)
> > > OOZIE-2771 Allow retrieving keystore and truststore passwords from
> Hadoop
> > > Credential Provider (asasvari via abhishekbafna)
> > > OOZIE-2748 NPE in LauncherMapper.printArgs() (pbacsko via rkanter)
> > > OOZIE-2654 Zookeeper dependent services should not depend on
> > > Connectionstate to be valid before cleaning up (venkatnrangan via
> > > abhishekbafna)
> > > OOZIE-2690 OOZIE NPE while executing kill() (abhishekbafna via
> > > jaydeepvishwakarma)
> > >
> > >
> > > Please reply to this

[jira] [Updated] (OOZIE-3126) Add option to allow list of users to access system config

2017-11-14 Thread Satish Subhashrao Saley (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3126:
---
Fix Version/s: 4.3.1

> Add option to allow list of users to access system config
> -
>
> Key: OOZIE-3126
> URL: https://issues.apache.org/jira/browse/OOZIE-3126
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3126-1.patch
>
>
>Currently we allow access to system config for all users. Though we mask 
> any sensitive information it is unnecessary to show it to all users. Can 
> restrict the access to admins, but services like Hue use the server side 
> config values to display UI accordingly (For eg: Server Timezone, whether SLA 
> is configured, what kind of credentials are supported, etc). So it would be 
> good to support restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3127) Remove redundant check for user

2017-11-14 Thread Satish Subhashrao Saley (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3127:
---
Attachment: OOZIE-3127-1.patch

> Remove redundant check for user
> ---
>
> Key: OOZIE-3127
> URL: https://issues.apache.org/jira/browse/OOZIE-3127
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3127-1.patch
>
>
> [validateJobConfiguration(conf);|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L95]
>  is redundant because we are checking it anyways [inside 
> here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L100]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3126) Add option to allow list of users to access system config

2017-11-14 Thread Satish Subhashrao Saley (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3126:
---
Attachment: OOZIE-3126-1.patch

> Add option to allow list of users to access system config
> -
>
> Key: OOZIE-3126
> URL: https://issues.apache.org/jira/browse/OOZIE-3126
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3126-1.patch
>
>
>Currently we allow access to system config for all users. Though we mask 
> any sensitive information it is unnecessary to show it to all users. Can 
> restrict the access to admins, but services like Hue use the server side 
> config values to display UI accordingly (For eg: Server Timezone, whether SLA 
> is configured, what kind of credentials are supported, etc). So it would be 
> good to support restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3113) Retry for ZK lock release

2017-11-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252085#comment-16252085
 ] 

Hadoop QA commented on OOZIE-3113:
--

Testing JIRA OOZIE-3113

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 0
.Tests rerun: 0
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordSubmitXCommand,org.apache.oozie.util.TestMetricsInstrumentation,org.apache.oozie.service.TestConfigurationService,org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC,org.apache.oozie.command.coord.TestCoordMaterializeTransitionXCommand,org.apache.oozie.command.coord.TestCoordActionMissingDependenciesXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.action.hadoop.TestSqoopActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/222/

> Retry for ZK lock release
> -
>
> Key: OOZIE-3113
> URL: https://issues.apache.org/jira/browse/OOZIE-3113
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3113-1.patch
>
>
> ZK Lock might not get released if release call fails. Oozie should do 
> exponential retry for some time before giving up.
> {code}
> 2017-10-25 03:07:45,787  WARN ZKLocksService:523 [pool-12-thread-74] - 
> SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] 
> JOB[123-123--oozie_saley-C] ACTION[123-123--oozie_saley-C@67] Could not 
> release lock: KeeperErrorCode = ConnectionLoss for 
> /oozie/localhost-oozie/locks/123-123--oozie_saley-C/1234__WRIT__000111
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for 
> /oozie/localhost-oozie/locks/123-123--oozie_saley-C/1234__WRIT__000111
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at 
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:239)
> at 
>

Failed: OOZIE-2869 PreCommit Build #220

2017-11-14 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2869
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/220/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.57 MB...]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/zookeeper-security-tests/src/test/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-zookeeper-security-tests ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.20.1:test (default-test) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Oozie Main .. SUCCESS [  0.578 s]
[INFO] Apache Oozie Client  SUCCESS [  1.228 s]
[INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  2.095 s]
[INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  0.650 s]
[INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.133 s]
[INFO] Apache Oozie Core .. SUCCESS [  5.797 s]
[INFO] Apache Oozie Share Lib Streaming ... SUCCESS [06:42 min]
[INFO] Apache Oozie Share Lib Pig . SUCCESS [04:05 min]
[INFO] Apache Oozie Share Lib Hive  SUCCESS [01:08 min]
[INFO] Apache Oozie Share Lib Hive 2 .. SUCCESS [  5.842 s]
[INFO] Apache Oozie Share Lib Sqoop ... SUCCESS [  2.898 s]
[INFO] Apache Oozie Examples .. SUCCESS [  5.057 s]
[INFO] Apache Oozie Share Lib Spark ... SUCCESS [  2.050 s]
[INFO] Apache Oozie Share Lib . SUCCESS [  0.016 s]
[INFO] Apache Oozie Docs .. SUCCESS [  0.025 s]
[INFO] Apache Oozie WebApp  SUCCESS [  2.374 s]
[INFO] Apache Oozie Tools . SUCCESS [  1.138 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [  1.226 s]
[INFO] Apache Oozie Server  SUCCESS [  2.849 s]
[INFO] Apache Oozie Distro  SUCCESS [  1.088 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [  1.658 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 12:34 min
[INFO] Finished at: 2017-11-14T19:44:29Z
[INFO] Final Memory: 73M/1439M
[INFO] 
bin/test-patch-20-tests: line 105: echo: write error: No space left on device
  Failure, check for details 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/test-patch/tmp/test-patch-20-tests-post.out

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 3706k0  57920 0   7292  0  0:08:40 --:--:--  0:08:40  7292
curl: (23) Failed writing body (0 != 5792)

Could not download jira-cli tool, thus no JIRA updating

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2869
Archiving artifacts
[Fast Archiver] Compressed 1.65 MB of artifacts by 39.7% relative to #216
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
61 tests failed.
FAILED:  org.apache.oozie.example.TestLocalOozieExample.testLocalOozieExampleEnd

Error Message:
Unable to initialize WebAppContext

Stack Trace:
java.io.IOException: Unable to initialize WebAppContext
at 
org.apache.oozie.example.TestLocalOozieExample.setUp(TestLocalOozieExample.java:37)
Caused by: java.io.IOException: No space left on device
at 
org.apache.oozie.example.TestLocalOozieExample.setUp(TestLocalOozieExample.java:37)


FAILED:  
org.apache.oozie.example.TestLocalOozieExample.testLocalOozieExampleKill

Error Message:
Unable to initialize WebAppContext

Stack Trace:
java.io.IOException: Unable to initialize WebAppContext
at

Failed: OOZIE-3107 PreCommit Build #219

2017-11-14 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-3107
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/219/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.63 MB...]
[ERROR]   mvn  -rf :oozie-webapp
bin/test-patch-30-distro: line 92: echo: write error: No space left on device

Reports
  Running test-patch task CLEAN
  Running test-patch task RAW_PATCH_ANALYSIS
  Running test-patch task RAT
  Running test-patch task JAVADOC
  Running test-patch task COMPILE
  Running test-patch task FINDBUGS_DIFF
mkdir: cannot create directory 
'/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff':
 No space left on device
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
tee: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/test-patch-11-findbugs-diff-report.out:
 No space left on device
bin/test-patch-11-findbugs-diff: line 179: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar:
 No such file or directory
[TRACE] FindBugs diff JAR downloaded
md5sum: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar:
 No such file or directory
bin/test-patch-11-findbugs-diff: line 187: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
grep: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
Error: Unable to access jarfile 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diffs performed
[TRACE] Checking FindBugs diffs and creating reports
find: 
'/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff':
 No such file or directory
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 1122 bytes
[TRACE] Full summary file size is 184 bytes
  Failure, check for details 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/test-patch-11-findbugs-diff-report.out

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:--

[jira] [Created] (OOZIE-3127) Remove redundant check for user

2017-11-14 Thread Satish Subhashrao Saley (JIRA)

Satish Subhashrao Saley created OOZIE-3127:
--

 Summary: Remove redundant check for user
 Key: OOZIE-3127
 URL: https://issues.apache.org/jira/browse/OOZIE-3127
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley
Assignee: Satish Subhashrao Saley
 Fix For: 4.3.1


[validateJobConfiguration(conf);|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L95]
 is redundant because we are checking it anyways [inside 
here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L100]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3126) Add option to allow list of users to access system config

2017-11-14 Thread Rohini Palaniswamy (JIRA)

Rohini Palaniswamy created OOZIE-3126:
-

 Summary: Add option to allow list of users to access system config
 Key: OOZIE-3126
 URL: https://issues.apache.org/jira/browse/OOZIE-3126
 Project: Oozie
  Issue Type: Improvement
Reporter: Rohini Palaniswamy
Assignee: Satish Subhashrao Saley


   Currently we allow access to system config for all users. Though we mask any 
sensitive information it is unnecessary to show it to all users. Can restrict 
the access to admins, but services like Hue use the server side config values 
to display UI accordingly (For eg: Server Timezone, whether SLA is configured, 
what kind of credentials are supported, etc). So it would be good to support 
restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Oozie 4.3.1. release

2017-11-14 Thread Peter Bacsko

Good idea.

If I'm not mistaken, there's also a pending review/patch submitted by
someone (apologies, but I forgot how it was) that cannot be applied to the
master anymore, at least not easily. It's worth checking that too (perhaps
there are more than one patch).

Peter

On Tue, Nov 14, 2017 at 5:07 PM, Peter Cseh  wrote:

> Hey!
>
> I agree, users wanting stability would appreciate a more stable 4.3.1 or
> even 4.4.0 to be released with improvements.
> I think we could include the following issues as well:
> OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
> OOZIE-3077 SLA documentation fixes (satishsaley)
> OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
> (dbist13 via rkanter)
> OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)
> OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
> Hentschel via rkanter)
> OOZIE-3004 Forked action retry info is not working
> Most of these are minor improvements or user-facing fixes which would be
> nice to get through the door.
>
> BRs,
> gp
>
> On Tue, Nov 14, 2017 at 12:56 AM, Rohini Palaniswamy <
> rohini.adi...@gmail.com> wrote:
>
> > Hi folks,
> >I think this would be a good time to do a patch release on 4.3 branch.
> > The 5.0 release is going to be a major change with the move of launcher
> > from Mapreduce AM to Oozie AM and might take time to stabilize. So it
> would
> > be good to do one final patch release on 4.3 branch pulling in some
> > important fixes and Spark action related fixes (Spark 2.x support) that
> > have gone in master so far.  Satish has volunteered to be the Release
> > Manager.
> >
> > Below is a initial list of jiras that I identified.
> >
> >
> > OOZIE-3095 Job submit command is not logged in audit log (puru)
> > OOZIE-3031 Coord job with only unresolved dependencies doesn't timeout
> > (puru via satishsaley)
> > OOZIE-3079 Filtering coordinators returns bundle id as null (satishsaley)
> > OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> > OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
> >  should overwrite default values set in Hadoop's configuration files
> > (gezapeti)
> > OOZIE-2885 Running Spark actions should not need Hive on the classpath
> > (satishsaley)
> > OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
> > (satishsaley)
> > OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation
> tokens
> > (andras.piros via gezapeti)
> > OOZIE-3028 Oozie Pig Action fails with no python dependencies (dbist13
> via
> > rohini)
> > OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
> > SchemaCheckerService (andras.piros via gezapeti)
> > OOZIE-2984 Parse spark-defaults.conf values with spaces without needing
> the
> > quotes (andras.piros via gezapeti)
> > OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
> > OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server
> start
> > (puru via satishsaley)
> > OOZIE-2862 Coord change command doesn't change job to running if job was
> > killed without creating any actions (puru)
> > OOZIE-2807 Oozie gets RM delegation token even for checking job status
> > (satishsaley)
> > OOZIE-2811 Add support for filtering out properties from
> > SparkConfigurationService (gezapeti via rkanter)
> > OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate sharelibs
> > (gezapeti via rkanter)
> > OOZIE-2803 Mask passwords when printing out configs/args in MapReduceMain
> > and SparkMain (pbacsko via rkanter)
> > OOZIE-2799 Setting log location for spark sql on hive (satishsaley)
> > OOZIE-2792 Hive2 action is not parsing Spark application ID from log file
> > properly when Hive is on Spark (zhengxb2005 via rkanter)
> > OOZIE-2786 Pass Oozie workflow ID and settings to Spark application
> > configuration (zhengxb2005 via rkanter)
> > OOZIE-2790 log4j configuration is not passed to spark executors
> > (satishsaley)
> > OOZIE-2787 Oozie distributes application jar twice making the spark job
> > fail (satishsaley)
> > OOZIE-2777 Config-default.xml longer than 64k results in
> > java.io.UTFDataFormatException (gezapeti via harsh)
> > OOZIE-2771 Allow retrieving keystore and truststore passwords from Hadoop
> > Credential Provider (asasvari via abhishekbafna)
> > OOZIE-2748 NPE in LauncherMapper.printArgs() (pbacsko via rkanter)
> > OOZIE-2654 Zookeeper dependent services should not depend on
> > Connectionstate to be valid before cleaning up (venkatnrangan via
> > abhishekbafna)
> > OOZIE-2690 OOZIE NPE while executing kill() (abhishekbafna via
> > jaydeepvishwakarma)
> >
> >
> > Please reply to this mail thread if you find other jiras that you think
> are
> > important and should be part of 4.3.1.
> >
> >
> > Regards,
> > Rohini
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> 
>

[jira] [Updated] (OOZIE-3079) Filtering coordinators returns bundle id as null

2017-11-14 Thread Satish Subhashrao Saley (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3079:
---
Fix Version/s: 4.3.1

> Filtering coordinators returns bundle id as null
> 
>
> Key: OOZIE-3079
> URL: https://issues.apache.org/jira/browse/OOZIE-3079
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3079-1.patch, OOZIE-3079-2.patch, 
> OOZIE-3079-3.patch
>
>
> If we filter coordinator jobs, the json reply shows (parent) bundle id as 
> null even though coordinator was started from a bundle.
> Example query - 
> http://localhost:11000/oozie/v1/jobs?filter=user=saley=coordinator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2900) Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before submission

2017-11-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251754#comment-16251754
 ] 

Hadoop QA commented on OOZIE-2900:
--

Testing JIRA OOZIE-2900

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/221/

> Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before 
> submission
> ---
>
> Key: OOZIE-2900
> URL: https://issues.apache.org/jira/browse/OOZIE-2900
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Attila Sasvari
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2900-000.patch
>
>
> We have to get tokens for oozie.launcher.mapreduce.job.hdfs-servers. Cannot 
> do cross cluster distcp without that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2900 PreCommit Build #221

2017-11-14 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2900
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/221/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 829.36 KB...]
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
-1 COMPILE
+1 HEAD compiles
-1 patch does not compile
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
-1 DISTRO
-1 distro tarball fails with the patch


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/221/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  4622k  0 --:--:-- --:--:-- --:--:-- 7224k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2900
Archiving artifacts
[Fast Archiver] Compressed 1.12 MB of artifacts by 53.2% relative to #216
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:jar:5

2017-11-14 Thread Artem Ervits (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251737#comment-16251737
 ] 

Artem Ervits commented on OOZIE-3119:
-

[~gezapeti] you're right, it was something wrong with my environment, I ran mvn 
install and then findbugs and it was good. I will close this jira.

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
> Refer to the generated Javadoc files in 
> '/Users/aervits/NetBeansProjects/OOZIE/oozie/sharelib/oozie/target/site/apidocs'
>  dir.
> at 
> org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine 
> (AbstractJavadocMojo.java:5188)
> at org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport 
> (AbstractJavadocMojo.java:2075)
> at org.apache.maven.plugin.javadoc.JavadocReport.generate 
> (JavadocReport.java:130)
> at org.apache.maven.plugin.javadoc.JavadocReport.execute 
> (JavadocReport.java:318)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Oozie Share Lib HCatalog 5.0.0-beta1
> [INFO] 
> 
> [INFO]
> [INFO] >>> maven-javadoc-plugin:2.10.4:javadoc (default-cli) > 
> generate-sources @ oozie-sharelib-hcatalog >>>
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.10.4:javadoc (default-cli) < 
> generate-sources @ oozie-sharelib-hcatalog <<<
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.544 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  2.133 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  1.708 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  FAILURE [  0.134 
> s]
> [INFO]

[jira] [Resolved] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:jar:5.

2017-11-14 Thread Artem Ervits (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits resolved OOZIE-3119.
-
Resolution: Invalid

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
> Refer to the generated Javadoc files in 
> '/Users/aervits/NetBeansProjects/OOZIE/oozie/sharelib/oozie/target/site/apidocs'
>  dir.
> at 
> org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine 
> (AbstractJavadocMojo.java:5188)
> at org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport 
> (AbstractJavadocMojo.java:2075)
> at org.apache.maven.plugin.javadoc.JavadocReport.generate 
> (JavadocReport.java:130)
> at org.apache.maven.plugin.javadoc.JavadocReport.execute 
> (JavadocReport.java:318)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Oozie Share Lib HCatalog 5.0.0-beta1
> [INFO] 
> 
> [INFO]
> [INFO] >>> maven-javadoc-plugin:2.10.4:javadoc (default-cli) > 
> generate-sources @ oozie-sharelib-hcatalog >>>
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.10.4:javadoc (default-cli) < 
> generate-sources @ oozie-sharelib-hcatalog <<<
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.544 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  2.133 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  1.708 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  FAILURE [  0.134 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SKIPPED
> [INFO] Apache Oozie Core .. SKIPPED
> [INFO] Apache Oozie Share Lib

[jira] [Updated] (OOZIE-2900) Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before submission

2017-11-14 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2900:
--
Attachment: OOZIE-2900-000.patch

OOZIE-2900-000.patch is a proof of concept. I tested copying files between 
secure clusters (cross realm authentication established) 

Issues:
- No new test cases added and it does not seem too easy to create. MiniKDC 
might be used though. 
- Trying to copy from a cluster that is not trusted (cross realm authentication 
is not establshed) results in a long running retry. I was unable to kill the 
workflow (I saw org.apache.hadoop.io.retry.RetryInvocationHandler in the thread 
dump I got with jstack).

> Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before 
> submission
> ---
>
> Key: OOZIE-2900
> URL: https://issues.apache.org/jira/browse/OOZIE-2900
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Attila Sasvari
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2900-000.patch
>
>
> We have to get tokens for oozie.launcher.mapreduce.job.hdfs-servers. Cannot 
> do cross cluster distcp without that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3121) bump all maven plugins to latest versions

2017-11-14 Thread Peter Cseh (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3121:
--
Fix Version/s: (was: 5.0.0b1)
   5.0.0

I'm retargeting this to 5.0.0 so it's not blocking the 5.0-b1 release.
I'll look into the test failures because this looks way too many.

> bump all maven plugins to latest versions
> -
>
> Key: OOZIE-3121
> URL: https://issues.apache.org/jira/browse/OOZIE-3121
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
> Fix For: 5.0.0
>
> Attachments: OOZIE-3121.patch
>
>
> bump all maven plugins to latest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:jar:5

2017-11-14 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251705#comment-16251705
 ] 

Peter Cseh commented on OOZIE-3119:
---

Have you tried executing mvn clean install -DskipTests before findbugs:findbugs 
?

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
> Refer to the generated Javadoc files in 
> '/Users/aervits/NetBeansProjects/OOZIE/oozie/sharelib/oozie/target/site/apidocs'
>  dir.
> at 
> org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine 
> (AbstractJavadocMojo.java:5188)
> at org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport 
> (AbstractJavadocMojo.java:2075)
> at org.apache.maven.plugin.javadoc.JavadocReport.generate 
> (JavadocReport.java:130)
> at org.apache.maven.plugin.javadoc.JavadocReport.execute 
> (JavadocReport.java:318)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Oozie Share Lib HCatalog 5.0.0-beta1
> [INFO] 
> 
> [INFO]
> [INFO] >>> maven-javadoc-plugin:2.10.4:javadoc (default-cli) > 
> generate-sources @ oozie-sharelib-hcatalog >>>
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.10.4:javadoc (default-cli) < 
> generate-sources @ oozie-sharelib-hcatalog <<<
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.544 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  2.133 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  1.708 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  FAILURE [  0.134 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SKIPPED
> [INFO]

[jira] [Commented] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:jar:5

2017-11-14 Thread Artem Ervits (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251706#comment-16251706
 ] 

Artem Ervits commented on OOZIE-3119:
-

the failure is sporadic and happens at different stages, just rebased to master 
and executed findbugs goal.

git: 567ce9fd2aa1da2da24f7941b218e1ef1ae0f09b

{noformat}
[INFO] --- findbugs-maven-plugin:3.0.1:findbugs (default-cli) @ 
oozie-sharelib-sqoop ---
[INFO] Fork Value is true
 [java] The following classes needed for analysis were missing:
 [java]   org.apache.oozie.action.hadoop.SqoopActionExecutor
[INFO] Done FindBugs Analysis
[INFO]
[INFO] 
[INFO] Building Apache Oozie Examples 5.0.0-beta1
[INFO] 
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Oozie Main .. SUCCESS [  0.892 s]
[INFO] Apache Oozie Client  SUCCESS [  9.502 s]
[INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  6.988 s]
[INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  3.873 s]
[INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  3.639 s]
[INFO] Apache Oozie Core .. SUCCESS [ 48.198 s]
[INFO] Apache Oozie Share Lib Streaming ... SUCCESS [  3.574 s]
[INFO] Apache Oozie Share Lib Pig . SUCCESS [  4.183 s]
[INFO] Apache Oozie Share Lib Hive  SUCCESS [  4.145 s]
[INFO] Apache Oozie Share Lib Hive 2 .. SUCCESS [  4.377 s]
[INFO] Apache Oozie Share Lib Sqoop ... SUCCESS [  3.636 s]
[INFO] Apache Oozie Examples .. FAILURE [  0.120 s]
[INFO] Apache Oozie Share Lib Spark ... SKIPPED
[INFO] Apache Oozie Share Lib . SKIPPED
[INFO] Apache Oozie Docs .. SKIPPED
[INFO] Apache Oozie WebApp  SKIPPED
[INFO] Apache Oozie Tools . SKIPPED
[INFO] Apache Oozie MiniOozie . SKIPPED
[INFO] Apache Oozie Server  SKIPPED
[INFO] Apache Oozie Distro  SKIPPED
[INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:33 min
[INFO] Finished at: 2017-11-14T11:47:17-05:00
[INFO] Final Memory: 63M/994M
[INFO] 
[ERROR] Failed to execute goal on project oozie-examples: Could not resolve 
dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0-beta1: 
Failure to find org.apache.oozie:oozie-sharelib-pig:jar:5.0.0-beta1 in 
http://repo1.maven.org/maven2 was cached in the local repository, resolution 
will not be reattempted until the update interval of central has elapsed or 
updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :oozie-examples
{noformat}

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
>

[jira] [Comment Edited] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:

2017-11-14 Thread Artem Ervits (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251691#comment-16251691
 ] 

Artem Ervits edited comment on OOZIE-3119 at 11/14/17 4:46 PM:
---

[~gezapeti] I filled out in the environment section


mvn: 3.5.2
jdk: 1.8.0.144
git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899

command I used was

{code}
mvn findbugs:findbugs
{/code}

if you still can't reproduce, try clearing your mvn cache. I reproduced this 
issue a couple of different ways including findbugs goal, let me check how else 
I can reproduce and update the thread.


was (Author: dbist13):
[~gezapeti] I filled out in the environment section


mvn: 3.5.2
jdk: 1.8.0.144
git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899

command I used was

{code}mvn findbugs:findbugs{/code}
if you still can't reproduce, try clearing your mvn cache. I reproduced this 
issue a couple of different ways including findbugs goal, let me check how else 
I can reproduce and update the thread.

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
> Refer to the generated Javadoc files in 
> '/Users/aervits/NetBeansProjects/OOZIE/oozie/sharelib/oozie/target/site/apidocs'
>  dir.
> at 
> org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine 
> (AbstractJavadocMojo.java:5188)
> at org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport 
> (AbstractJavadocMojo.java:2075)
> at org.apache.maven.plugin.javadoc.JavadocReport.generate 
> (JavadocReport.java:130)
> at org.apache.maven.plugin.javadoc.JavadocReport.execute 
> (JavadocReport.java:318)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Oozie Share Lib HCatalog 5.0.0-beta1
> [INFO] 
> 
> [INFO]
> [INFO] >>>

[jira] [Commented] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:jar:5

2017-11-14 Thread Artem Ervits (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251691#comment-16251691
 ] 

Artem Ervits commented on OOZIE-3119:
-

[~gezapeti] I filled out in the environment section


mvn: 3.5.2
jdk: 1.8.0.144
git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899

command I used was

{code}mvn findbugs:findbugs{/code}
if you still can't reproduce, try clearing your mvn cache. I reproduced this 
issue a couple of different ways including findbugs goal, let me check how else 
I can reproduce and update the thread.

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
> Refer to the generated Javadoc files in 
> '/Users/aervits/NetBeansProjects/OOZIE/oozie/sharelib/oozie/target/site/apidocs'
>  dir.
> at 
> org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine 
> (AbstractJavadocMojo.java:5188)
> at org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport 
> (AbstractJavadocMojo.java:2075)
> at org.apache.maven.plugin.javadoc.JavadocReport.generate 
> (JavadocReport.java:130)
> at org.apache.maven.plugin.javadoc.JavadocReport.execute 
> (JavadocReport.java:318)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Oozie Share Lib HCatalog 5.0.0-beta1
> [INFO] 
> 
> [INFO]
> [INFO] >>> maven-javadoc-plugin:2.10.4:javadoc (default-cli) > 
> generate-sources @ oozie-sharelib-hcatalog >>>
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.10.4:javadoc (default-cli) < 
> generate-sources @ oozie-sharelib-hcatalog <<<
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.544 
> s]
> [INFO] Apache Oozie Client

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-11-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251689#comment-16251689
 ] 

Hadoop QA commented on OOZIE-2296:
--

Testing JIRA OOZIE-2296

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [3] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:orange}0{color} There are [3] new bugs found in [tools] that would be 
nice to have fixed.
. You can find the FindBugs diff here: tools/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 0
.Tests rerun: 0
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordMaterializeTransitionXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.command.coord.TestCoordSubmitXCommand,org.apache.oozie.util.TestMetricsInstrumentation,org.apache.oozie.command.coord.TestCoordActionMissingDependenciesXCommand,org.apache.oozie.service.TestConfigurationService,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/216/

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch, 
> OOZIE-2296-012.patch, OOZIE-2296-013.patch, OOZIE-2296-014.patch, 
> OOZIE-2296-015.patch, OOZIE-2296-016.patch, OOZIE-2296-017.patch, 
> OOZIE-2296-018.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin

[jira] [Commented] (OOZIE-3119) fix for Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find org.apache.oozie:oozie-sharelib-oozie:jar:5

2017-11-14 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251641#comment-16251641
 ] 

Peter Cseh commented on OOZIE-3119:
---

I can't reproduce this issue. Can you provide some environmental information?

> fix for Could not resolve dependencies for project 
> org.apache.oozie:oozie-sharelib-hcatalog:jar:5.0.0-beta1: Failure to find 
> org.apache.oozie:oozie-sharelib-oozie:jar:5.0.0-beta1 in 
> http://repo1.maven.org/maven2 was cached in the local repositor
> -
>
> Key: OOZIE-3119
> URL: https://issues.apache.org/jira/browse/OOZIE-3119
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
> git: 5d9c6a5c9951455e6cb29a097de5f53b0fdbe899
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> {code}
> mvn javadoc:javadoc
> {code}
> {noformat}
> Command line was: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/../bin/javadoc
>  @options @packages
> Refer to the generated Javadoc files in 
> '/Users/aervits/NetBeansProjects/OOZIE/oozie/sharelib/oozie/target/site/apidocs'
>  dir.
> at 
> org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine 
> (AbstractJavadocMojo.java:5188)
> at org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport 
> (AbstractJavadocMojo.java:2075)
> at org.apache.maven.plugin.javadoc.JavadocReport.generate 
> (JavadocReport.java:130)
> at org.apache.maven.plugin.javadoc.JavadocReport.execute 
> (JavadocReport.java:318)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Oozie Share Lib HCatalog 5.0.0-beta1
> [INFO] 
> 
> [INFO]
> [INFO] >>> maven-javadoc-plugin:2.10.4:javadoc (default-cli) > 
> generate-sources @ oozie-sharelib-hcatalog >>>
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.10.4:javadoc (default-cli) < 
> generate-sources @ oozie-sharelib-hcatalog <<<
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.544 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  2.133 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  1.708 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  FAILURE [  0.134 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SKIPPED
> [INFO] Apache

[jira] [Updated] (OOZIE-2869) OYA: Introduce default mapreduce sharelib

2017-11-14 Thread Peter Cseh (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2869:
--
Attachment: OOZIE-2869.001.patch

As a first step I've defined a property to provide the possibility to 
add/remove jars from the classpath for each action type using the 
"oozie.action.mapreduce.needed.for" prefix.

> OYA: Introduce default mapreduce sharelib
> -
>
> Key: OOZIE-2869
> URL: https://issues.apache.org/jira/browse/OOZIE-2869
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Peter Cseh
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2869.001.patch
>
>
> Since Oozie On YARN doesn't fire a map only job to fire all the other 
> applications (hence the transition from {{LauncherMapper}} to 
> {{LauncherAM}}), no need to bundle the MapReduce JARs within Oozie's default 
> sharelib.
> Nevertheless, it's a common, but discouraged use case that MapReduce tasks 
> are fired using Java Action. So we need some default {{mapreduce}} sharelib 
> that can be used by customers after OOZIE-2601 will have been implemented:
> # by default, create a sharelib named {{mapreduce}} which will contain the MR 
> jars using local paths as those jars are on every machine
> # users can add mapreduce jars to any action using 
> {{oozie.action.sharelib.for.ACTION}} just like now. They could add MR jars 
> into Java, Spark or Shell actions easily



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Oozie 4.3.1. release

2017-11-14 Thread Peter Cseh

Hey!

I agree, users wanting stability would appreciate a more stable 4.3.1 or
even 4.4.0 to be released with improvements.
I think we could include the following issues as well:
OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
OOZIE-3077 SLA documentation fixes (satishsaley)
OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
(dbist13 via rkanter)
OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)
OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
Hentschel via rkanter)
OOZIE-3004 Forked action retry info is not working
Most of these are minor improvements or user-facing fixes which would be
nice to get through the door.

BRs,
gp

On Tue, Nov 14, 2017 at 12:56 AM, Rohini Palaniswamy <
rohini.adi...@gmail.com> wrote:

> Hi folks,
>I think this would be a good time to do a patch release on 4.3 branch.
> The 5.0 release is going to be a major change with the move of launcher
> from Mapreduce AM to Oozie AM and might take time to stabilize. So it would
> be good to do one final patch release on 4.3 branch pulling in some
> important fixes and Spark action related fixes (Spark 2.x support) that
> have gone in master so far.  Satish has volunteered to be the Release
> Manager.
>
> Below is a initial list of jiras that I identified.
>
>
> OOZIE-3095 Job submit command is not logged in audit log (puru)
> OOZIE-3031 Coord job with only unresolved dependencies doesn't timeout
> (puru via satishsaley)
> OOZIE-3079 Filtering coordinators returns bundle id as null (satishsaley)
> OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
>  should overwrite default values set in Hadoop's configuration files
> (gezapeti)
> OOZIE-2885 Running Spark actions should not need Hive on the classpath
> (satishsaley)
> OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
> (satishsaley)
> OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation tokens
> (andras.piros via gezapeti)
> OOZIE-3028 Oozie Pig Action fails with no python dependencies (dbist13 via
> rohini)
> OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
> SchemaCheckerService (andras.piros via gezapeti)
> OOZIE-2984 Parse spark-defaults.conf values with spaces without needing the
> quotes (andras.piros via gezapeti)
> OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
> OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
> (puru via satishsaley)
> OOZIE-2862 Coord change command doesn't change job to running if job was
> killed without creating any actions (puru)
> OOZIE-2807 Oozie gets RM delegation token even for checking job status
> (satishsaley)
> OOZIE-2811 Add support for filtering out properties from
> SparkConfigurationService (gezapeti via rkanter)
> OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate sharelibs
> (gezapeti via rkanter)
> OOZIE-2803 Mask passwords when printing out configs/args in MapReduceMain
> and SparkMain (pbacsko via rkanter)
> OOZIE-2799 Setting log location for spark sql on hive (satishsaley)
> OOZIE-2792 Hive2 action is not parsing Spark application ID from log file
> properly when Hive is on Spark (zhengxb2005 via rkanter)
> OOZIE-2786 Pass Oozie workflow ID and settings to Spark application
> configuration (zhengxb2005 via rkanter)
> OOZIE-2790 log4j configuration is not passed to spark executors
> (satishsaley)
> OOZIE-2787 Oozie distributes application jar twice making the spark job
> fail (satishsaley)
> OOZIE-2777 Config-default.xml longer than 64k results in
> java.io.UTFDataFormatException (gezapeti via harsh)
> OOZIE-2771 Allow retrieving keystore and truststore passwords from Hadoop
> Credential Provider (asasvari via abhishekbafna)
> OOZIE-2748 NPE in LauncherMapper.printArgs() (pbacsko via rkanter)
> OOZIE-2654 Zookeeper dependent services should not depend on
> Connectionstate to be valid before cleaning up (venkatnrangan via
> abhishekbafna)
> OOZIE-2690 OOZIE NPE while executing kill() (abhishekbafna via
> jaydeepvishwakarma)
>
>
> Please reply to this mail thread if you find other jiras that you think are
> important and should be part of 4.3.1.
>
>
> Regards,
> Rohini
>



-- 
Peter Cseh
Software Engineer

[jira] [Commented] (OOZIE-3002) address findbugs errors in client lib

2017-11-14 Thread Peter Bacsko (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251607#comment-16251607
 ] 

Peter Bacsko commented on OOZIE-3002:
-

It becomes even more confusing if we look at the mapping definition:

{noformat}
COORD_JOB.put("getTimeout", new Property(JsonTags.COORDINATOR_JOB_TIMEOUT, 
Integer.TYPE));
{noformat}

So we define an integer here but we still set a long value which is cast back 
to an int later.  Does anyone understand this?

> address findbugs errors in client lib
> -
>
> Key: OOZIE-3002
> URL: https://issues.apache.org/jira/browse/OOZIE-3002
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: 4.3.0
> Environment: Java 1.8.0.131
> Maven 3.5.0
> OSX 10.12.5
> commit 18694b616f419ed958230cedb2dc0431a9f9cb5d
>Reporter: Artem Ervits
>Assignee: Artem Ervits
> Attachments: OOZIE-3002-0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3002) address findbugs errors in client lib

2017-11-14 Thread Peter Bacsko (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251602#comment-16251602
 ] 

Peter Bacsko commented on OOZIE-3002:
-

I ran the findbugs-diff part locally and it returned the same errors as above:

{noformat}
 At JsonToBean.java:[line 259]: Primitive value is boxed and then immediately 
unboxed in 
org.apache.oozie.client.rest.JsonToBean$JsonInvocationHandler.parseType(Class, 
Object)
 At JsonToBean.java:[line 259]: Boxed value is unboxed and then immediately 
reboxed in 
org.apache.oozie.client.rest.JsonToBean$JsonInvocationHandler.parseType(Class, 
Object)
 At JsonToBean.java:[line 259]: Boxed value is unboxed and then immediately 
reboxed in 
org.apache.oozie.client.rest.JsonToBean$JsonInvocationHandler.parseType(Class, 
Object)
{noformat}

[~dbist13] could you explain this part?

{noformat}
else if (type == Integer.TYPE) {
return (obj != null) ? Long.valueOf((long) obj).intValue() : 
new Integer(0);
}
{noformat}

Why do we cast the incoming object to a long if we expect an integer?

I opened {{TestJsonToBean.java}} and found some weird things:

{noformat}
json.put(JsonTags.WORKFLOW_ACTION_RETRIES, (long)1);
...
json.put(JsonTags.COORDINATOR_JOB_CONCURRENCY, (long)2);
json.put(JsonTags.COORDINATOR_JOB_TIMEOUT, (long)3);
{noformat}

What's the point of all this?

> address findbugs errors in client lib
> -
>
> Key: OOZIE-3002
> URL: https://issues.apache.org/jira/browse/OOZIE-3002
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: 4.3.0
> Environment: Java 1.8.0.131
> Maven 3.5.0
> OSX 10.12.5
> commit 18694b616f419ed958230cedb2dc0431a9f9cb5d
>Reporter: Artem Ervits
>Assignee: Artem Ervits
> Attachments: OOZIE-3002-0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3122) Oozie spark action unable to handle some options in spark-defaults.conf

2017-11-14 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251571#comment-16251571
 ] 

Rohini Palaniswamy commented on OOZIE-3122:
---

Just started a discussion on dev mailing list on doing a Apache 4.3.1 release. 
Hopefully we will have a release in another month.

> Oozie spark action unable to handle some options in spark-defaults.conf
> ---
>
> Key: OOZIE-3122
> URL: https://issues.apache.org/jira/browse/OOZIE-3122
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.0
> Environment: Operating on EMR cluster (5.4.0) which includes Oozie 
> 4.3.0
>Reporter: Simon Whitelaw
>  Labels: usability
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> When using the setting 
> oozie.service.SparkConfigurationService.spark.configurations *=etc/spark/conf 
> to specify a spark-defaults.conf file for spark to use, a few options are not 
> handled properly by the oozie spark action and cause the job to fail. These 
> include the following:
> * spark.driver.extraClassPath - causes a blank --conf tag to be sent to the 
> spark submit and the spark submit will fail
> * spark.executor.extraClassPath - causes a blank --conf tag to be sent to the 
> spark submit and the spark submit will fail
> * spark.executor.extraJavaOptions with multiple options set (e.g. 
> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails 
> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 
> -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p') - causes 
> all options except for the first to be passed as singletons rather than 
> extraJavaOptions
> * spark.driver.extraJavaOptions with multiple option set - causes all options 
> except for the first to be passed as singletons rather than extraJavaOptions
> * Within the spark.driver.extraJavaOptions and 
> spark.executor.extraJavaOptions, the option -XX:OnOutOfMemoryError='kill -9 
> %p' is not parsed correctly even from the spark-opts tag and requires 
> additional double quotations within the single quote marks: '"kill -9 %p"'
> These issues were noticed running the oozie spark action on an amazon EMR 
> cluster. Workaround involves sending the options as part of the spark-opts 
> tag, but is not ideal because in EMR, the spark-defaults.conf file is 
> autogenerated for the specifications of the cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3107) org.apache.oozie.action.hadoop.TestHiveMain#testMain is flaky

2017-11-14 Thread Artem Ervits (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-3107:

Attachment: OOZIE-3107-2.patch

[~pbacsko] please review, also, found many tabs in main pom, perhaps tackle 
those in a separate jira?

> org.apache.oozie.action.hadoop.TestHiveMain#testMain is flaky
> -
>
> Key: OOZIE-3107
> URL: https://issues.apache.org/jira/browse/OOZIE-3107
> Project: Oozie
>  Issue Type: Sub-task
>  Components: tests
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
>Reporter: Artem Ervits
>Assignee: Artem Ervits
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3107-2.patch, OOZIE-3107.patch
>
>
> running testMain test in isolation fails and all subsequent runs of the test 
> will fail.
> {code}
> mvn clean -Dtest=org.apache.oozie.action.hadoop.TestHiveMain#testMain test
> {code}
> {noformat}
> [INFO] Running org.apache.oozie.action.hadoop.TestHiveMain
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 29.209 s <<< FAILURE! - in org.apache.oozie.action.hadoop.TestHiveMain
> [ERROR] testMain(org.apache.oozie.action.hadoop.TestHiveMain)  Time elapsed: 
> 29.209 s  <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Call From 
> hw12107.local/192.168.201.6 to localhost:60427 failed on connection 
> exception: java.net.ConnectException: Connection refused; For more details 
> see:  http://wiki.apache.org/hadoop/ConnectionRefused
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:146)
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:39)
> Caused by: java.net.ConnectException: Call From hw12107.local/192.168.201.6 
> to localhost:60427 failed on connection exception: java.net.ConnectException: 
> Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:146)
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:39)
> Caused by: java.net.ConnectException: Connection refused
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:146)
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:39)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR]   
> TestHiveMain>MainTestCase.testMain:40->MainTestCase.execute:31->call:39->call:146
>  » Runtime
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.529 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  3.296 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  3.961 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  1.638 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.558 
> s]
> [INFO] Apache Oozie Core .. SUCCESS [ 16.664 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SUCCESS [  4.166 
> s]
> [INFO] Apache Oozie Share Lib Pig . SUCCESS [  2.235 
> s]
> [INFO] Apache Oozie Share Lib Hive  FAILURE [ 32.805 
> s]
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:06 min
> [INFO] Finished at: 2017-11-01T10:22:57-04:00
> [INFO] Final Memory: 296M/1606M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on 
>

[jira] [Commented] (OOZIE-2585) Disable or remove test case TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and testMemoryUsageAndSpeed

2017-11-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251523#comment-16251523
 ] 

Hadoop QA commented on OOZIE-2585:
--

Testing JIRA OOZIE-2585

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Disable or remove test case 
> TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk 
> and testMemoryUsageAndSpeed
> ---
>
> Key: OOZIE-2585
> URL: https://issues.apache.org/jira/browse/OOZIE-2585
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2585-001.patch, OOZIE-2585-002.patch
>
>
> There is a problem with the test case testMemoryUsageAndSpeedOverflowToDisk.
> After 20k entries, data is spilled to the disk. However, Ehcache writes on a 
> separate thread asynchronously and all put operations return immediately. The 
> problem is that after storing the 60k entries, we try to read them back 
> instantly, with many of them are still being flushed to the storage. As long 
> as disk write in progress, these entries are unavailable.
> I didn't find a reliable way to wait for disk writes. Therefore I suggest 
> disabling this test or eliminate it altogether -- it's a performance test 
> that runs on all kinds of machines during a build, I don't think it makes too 
> much sense to keep it.
> Also testMemoryUsageAndSpeed might be prone to test failures too - these two 
> assertions might evaluate to false:
> {code}
> assertTrue((endTime - startTime) < insertTimeinMillis);
> ...
> assertTrue((System.currentTimeMillis() - endTime) < retrievalTimeinMillis);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2585 PreCommit Build #218

2017-11-14 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2585
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/218/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 680.02 KB...]
[INFO] Finished at: 2017-11-14T15:16:08Z
[INFO] Final Memory: 643M/1840M
[INFO] 
[TRACE] FindBugs output in HEAD verified and saved
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO

Applying patch

Checking patch 
core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerEhcache.java...
Checking patch 
core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java...
error: while searching for:
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

import org.apache.oozie.CoordinatorActionBean;
import org.apache.oozie.client.CoordinatorAction.Status;
import org.apache.oozie.client.rest.JsonBean;
import org.apache.oozie.dependency.hcat.HCatMessageHandler;
import org.apache.oozie.executor.jpa.BatchQueryExecutor;
import org.apache.oozie.jms.JMSConnectionInfo;
import org.apache.oozie.service.Services;
import org.apache.oozie.test.XDataTestCase;
import org.apache.oozie.util.HCatURI;
import org.apache.oozie.util.XLog;
import org.junit.Test;

/**

error: patch failed: 
core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java:21
error: 
core/src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java:
 patch does not apply
Checking patch 
src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerEhcache.java...
error: 
src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerEhcache.java:
 No such file or directory
Checking patch 
src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java...
error: 
src/test/java/org/apache/oozie/service/TestPartitionDependencyManagerService.java:
 No such file or directory
Patch failed to apply to head of branch

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3118k  0  0:00:01  0:00:01 --:--:-- 6456k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2585
Archiving artifacts
[Fast Archiver] Compressed 844.77 KB of artifacts by 64.4% relative to #214
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
No tests ran.

[jira] [Updated] (OOZIE-2585) Disable or remove test case TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and testMemoryUsageAndSpeed

2017-11-14 Thread Peter Bacsko (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2585:

Issue Type: Sub-task  (was: Bug)
Parent: OOZIE-3111

> Disable or remove test case 
> TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk 
> and testMemoryUsageAndSpeed
> ---
>
> Key: OOZIE-2585
> URL: https://issues.apache.org/jira/browse/OOZIE-2585
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2585-001.patch, OOZIE-2585-002.patch
>
>
> There is a problem with the test case testMemoryUsageAndSpeedOverflowToDisk.
> After 20k entries, data is spilled to the disk. However, Ehcache writes on a 
> separate thread asynchronously and all put operations return immediately. The 
> problem is that after storing the 60k entries, we try to read them back 
> instantly, with many of them are still being flushed to the storage. As long 
> as disk write in progress, these entries are unavailable.
> I didn't find a reliable way to wait for disk writes. Therefore I suggest 
> disabling this test or eliminate it altogether -- it's a performance test 
> that runs on all kinds of machines during a build, I don't think it makes too 
> much sense to keep it.
> Also testMemoryUsageAndSpeed might be prone to test failures too - these two 
> assertions might evaluate to false:
> {code}
> assertTrue((endTime - startTime) < insertTimeinMillis);
> ...
> assertTrue((System.currentTimeMillis() - endTime) < retrievalTimeinMillis);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2459) TestCoordActionInputCheckXCommand.testNone is flakey

2017-11-14 Thread Peter Bacsko (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2459:

Issue Type: Sub-task  (was: Bug)
Parent: OOZIE-3111

> TestCoordActionInputCheckXCommand.testNone is flakey
> 
>
> Key: OOZIE-2459
> URL: https://issues.apache.org/jira/browse/OOZIE-2459
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action, tests
>Affects Versions: 4.1.0
>Reporter: Ferenc Denes
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2459-1.patch
>
>
> TestCoordActionInputCheckXCommand.testNone is flakey.
> Also the TestCoordActionInputCheckXCommandNonUTC.testNone which is very 
> similar.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3107) org.apache.oozie.action.hadoop.TestHiveMain#testMain is flaky

2017-11-14 Thread Peter Bacsko (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251477#comment-16251477
 ] 

Peter Bacsko commented on OOZIE-3107:
-

[~dbist13] very minor: could you pls remove the unnecessary tab character from 
the patch? Just to make everyone happy.

> org.apache.oozie.action.hadoop.TestHiveMain#testMain is flaky
> -
>
> Key: OOZIE-3107
> URL: https://issues.apache.org/jira/browse/OOZIE-3107
> Project: Oozie
>  Issue Type: Sub-task
>  Components: tests
>Affects Versions: 4.3.0
> Environment: mvn: 3.5.2
> jdk: 1.8.0.144
>Reporter: Artem Ervits
>Assignee: Artem Ervits
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3107.patch
>
>
> running testMain test in isolation fails and all subsequent runs of the test 
> will fail.
> {code}
> mvn clean -Dtest=org.apache.oozie.action.hadoop.TestHiveMain#testMain test
> {code}
> {noformat}
> [INFO] Running org.apache.oozie.action.hadoop.TestHiveMain
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 29.209 s <<< FAILURE! - in org.apache.oozie.action.hadoop.TestHiveMain
> [ERROR] testMain(org.apache.oozie.action.hadoop.TestHiveMain)  Time elapsed: 
> 29.209 s  <<< ERROR!
> java.lang.RuntimeException: java.net.ConnectException: Call From 
> hw12107.local/192.168.201.6 to localhost:60427 failed on connection 
> exception: java.net.ConnectException: Connection refused; For more details 
> see:  http://wiki.apache.org/hadoop/ConnectionRefused
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:146)
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:39)
> Caused by: java.net.ConnectException: Call From hw12107.local/192.168.201.6 
> to localhost:60427 failed on connection exception: java.net.ConnectException: 
> Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:146)
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:39)
> Caused by: java.net.ConnectException: Connection refused
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:146)
>   at 
> org.apache.oozie.action.hadoop.TestHiveMain.call(TestHiveMain.java:39)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR]   
> TestHiveMain>MainTestCase.testMain:40->MainTestCase.execute:31->call:39->call:146
>  » Runtime
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main .. SUCCESS [  0.529 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  3.296 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  3.961 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  1.638 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.558 
> s]
> [INFO] Apache Oozie Core .. SUCCESS [ 16.664 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SUCCESS [  4.166 
> s]
> [INFO] Apache Oozie Share Lib Pig . SUCCESS [  2.235 
> s]
> [INFO] Apache Oozie Share Lib Hive  FAILURE [ 32.805 
> s]
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:06 min
> [INFO] Finished at: 2017-11-01T10:22:57-04:00
> [INFO] Final Memory: 296M/1606M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on 
>

[jira] [Commented] (OOZIE-2973) Make sure Oozie works with Hadoop 3

2017-11-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251465#comment-16251465
 ] 

Hadoop QA commented on OOZIE-2973:
--

Testing JIRA OOZIE-2973

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/217/

> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2973.001.patch
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2973 PreCommit Build #217

2017-11-14 Thread Apache Jenkins Server

Jira: https://issues.apache.org/jira/browse/OOZIE-2973
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/217/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 848.17 KB...]
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
-1 COMPILE
+1 HEAD compiles
-1 patch does not compile
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
-1 DISTRO
-1 distro tarball fails with the patch


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/217/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 3706k0 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3285k  0  0:00:01  0:00:01 --:--:-- 14.6M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2973
Archiving artifacts
[Fast Archiver] Compressed 1.12 MB of artifacts by 50.0% relative to #214
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Comment Edited] (OOZIE-2973) Make sure Oozie works with Hadoop 3

2017-11-14 Thread Peter Cseh (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251246#comment-16251246
 ] 

Peter Cseh edited comment on OOZIE-2973 at 11/14/17 11:15 AM:
--

First patch! (more like a poc)
# added hadoop-2 (default) and hadoop-3 profiles 
# added back hadooplibs and moved ECPolicyDisabler there
# Removed some reference to apache-directory Strings, used guava instead.
# when defined, hadoop-3 compiles with 3.0.0-beta1
# Jetty was updated to match with Hadoop 3's version to avoid collision.
# Tests are not running , MiniHDFS fails to starts up. I'm glad to accept any 
guidance about how to fix it.



was (Author: gezapeti):
First patch!
# added hadoop-2 (default) and hadoop-3 profiles 
# when defined, hadoop-3 compiles with 3.0.0-beta1
# Jetty was updated to match with Hadoop 3's version to avoid collision.
# Tests are not running , MiniHDFS fails to starts up. I'm glad to accept any 
guidance about how to fix it.


> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2973.001.patch
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3

2017-11-14 Thread Peter Cseh (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2973:
--
Attachment: OOZIE-2973.001.patch

First patch!
# added hadoop-2 (default) and hadoop-3 profiles 
# when defined, hadoop-3 compiles with 3.0.0-beta1
# Jetty was updated to match with Hadoop 3's version to avoid collision.
# Tests are not running , MiniHDFS fails to starts up. I'm glad to accept any 
guidance about how to fix it.


> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2973.001.patch
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2296) Add an Oozie diagnostic bundle tool

2017-11-14 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2296:
--
Attachment: OOZIE-2296-018.patch

rebased patch, TestDBLoadDump should be fixed

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch, 
> OOZIE-2296-012.patch, OOZIE-2296-013.patch, OOZIE-2296-014.patch, 
> OOZIE-2296-015.patch, OOZIE-2296-016.patch, OOZIE-2296-017.patch, 
> OOZIE-2296-018.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3125) TestDBLoadDump.testImportInvalidDataLeavesTablesEmpty fails

2017-11-14 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251190#comment-16251190
 ] 

Attila Sasvari commented on OOZIE-3125:
---

[~andras.piros] thanks for the review, committed to master.

> TestDBLoadDump.testImportInvalidDataLeavesTablesEmpty fails
> ---
>
> Key: OOZIE-3125
> URL: https://issues.apache.org/jira/browse/OOZIE-3125
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Minor
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3125-001.patch, OOZIE-3125-002.patch
>
>
> Running {{mvn test  
> -Dtest=TestDBLoadDump#testImportInvalidDataLeavesTablesEmpty}} results in a 
> test failure:
> {code}
> [ERROR]   
> TestDBLoadDump.testImportInvalidDataLeavesTablesEmpty:136->tryImportAndCheckPrematureExit:271
>  import should have been ended prematurely
> {code}
> {{TestDBLoadDump.testImportInvalidDataLeavesTablesEmpty}} tests that 
> processing an invalid dump file results in transactions are rolled back and 
> tables are left unpopulated (empty).
> {{tools/src/test/resources/dumpData/invalid/ooziedb_ac.json}} contains an 
> {{executionPath}} that consists of more than 1024 characters (2000). 
> {{OozieDBImportCLI.importOneInputFileToOneEntityTable}} deserialize the 
> WorkflowActionBean object from the JSON using {{final E newEntity = 
> gson.fromJson(line, entityClass);}} . However, persisting this object should 
> fail even because invariants (see {{@Column(name = "execution_path", length = 
> 1024)}} in {{WorkflowActionBean}}) do not hold. 
> In the setup method of TestDBLoadDump, {{setSqlStricEnforce()}} is used to 
> make sure strict size is enforced:
> {code}
> private void setSqlStrictEnforce(final EntityManager entityManager) {
> final String sqlStrictEnforce = "SET PROPERTY 
> \"sql.enforce_strict_size\" TRUE";
> final EntityTransaction tx = entityManager.getTransaction();
> tx.begin();
> entityManager.createNativeQuery(sqlStrictEnforce).executeUpdate();
> tx.commit();
> }
> {code}
> However it does not seem to take effect. [HSQL db's 
> documentation|http://hsqldb.org/doc/guide/dbproperties-chapt.html] says:
> {quote}  
> Management of properties has changed since version 1.8. The old SET PROPERTY 
> statement does not change a property and is ignored. The statement is 
> retained to simplify application upgrades.
> {quote}
> Running {{mvn dependency:tree}}, it turns out Oozie core uses 1.8.0.10 
> {code}
> [INFO] +- hsqldb:hsqldb:jar:1.8.0.10:compile
> {code}
> It is, however, strange that tests pass if running all the test cases in 
> TestDBLoadDump with {{mvn test -Dmaven.surefire.debug  
> -Dtest=TestDBLoadDump}}. In other words, tests are not independent of 
> execution order.
> Looking at a successfull test execution in surefire-reports 
> ({{TEST-org.apache.oozie.tools.TestDBLoadDump.xml}}) :
> {code}
>classname="org.apache.oozie.tools.TestDBLoadDump" time="25.362"/>
>classname="org.apache.oozie.tools.TestDBLoadDump" time="0.559"/>
>classname="org.apache.oozie.tools.TestDBLoadDump" time="0.708"/>
>classname="org.apache.oozie.tools.TestDBLoadDump" time="1.066"/>
>classname="org.apache.oozie.tools.TestDBLoadDump" time="1.135"/>
> {code}
> We shall make sure individual tests pass. As a first step, 
> {{TestDBLoadDump.testImportInvalidDataLeavesTablesEmpty}} is to be fixed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

47 matches

Mail list logo