[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (91 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2792  Hive2 action is not parsing Spark application ID from log file 
properly when Hive is on Spark
https://issues.apache.org/jira/browse/OOZIE-2792
OOZIE-2790  log4j configuration is not passed to spark executors
https://issues.apache.org/jira/browse/OOZIE-2790
OOZIE-2788  Fix jobs API servlet mapping for EmbeddedOozieServer
https://issues.apache.org/jira/browse/OOZIE-2788
OOZIE-2787  Oozie distributes application jar twice making the spark job fail
https://issues.apache.org/jira/browse/OOZIE-2787
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2750  Spelling errors in the log messages and exception messages
https://issues.apache.org/jira/browse/OOZIE-2750
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2739  Remove property expansion pattern from ShellMain's log4j properties 
content
https://issues.apache.org/jira/browse/OOZIE-2739
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when 
creating a Sharelib
https://issues.apache.org/jira/browse/OOZIE-2629
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2608  Comma in oozie.service.JPAService.jdbc.password value results in 
authentication error
https://issues.apache.org/jira/browse/OOZIE-2608
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues.apache.org/jira/browse/OOZIE-2585
OOZIE-2573  dataset url  which contains spaces can not be handled rightly

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859120#comment-15859120
 ] 

Hadoop QA commented on OOZIE-2787:
--

Testing JIRA OOZIE-2787

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1872
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3634/

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3634

[Apache Jenkins Server]

See 

--
[...truncated 14682 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SqoopActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2Credentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HCatCredentialHelper.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/LauncherMapperHelper.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Credentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ShellActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/PigActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HCatCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/JavaActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/DistcpActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HadoopELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProperties.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/MapReduceActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProvider.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/ActionExecutor.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859081#comment-15859081
 ] 

Xiaobin Zheng commented on OOZIE-2787:
--

[~satishsaley] Thx for the patch. Two minor suggestions:
1. java doc for 'isApplicationJar' seems outdated.
2. It would be great if we can add some simple unit test for either 'filter()' 
or 'isApplicationJar()' to ensure the behavior we want.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858946#comment-15858946
 ] 

Abhishek Bafna commented on OOZIE-2787:
---

[~satishsaley] It would be nice to document the same for oozie-spark action. 
Some thing like "How to specify application path in Oozie-Spark action". Thanks.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-1.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858502#comment-15858502
 ] 

Satish Subhashrao Saley edited comment on OOZIE-2787 at 2/8/17 10:40 PM:
-

Reopening as there is a regression.

{code}
pi.py is under oozie.wf.application.path and workflow configuration is -

pyspark example
pi.py
${testConf}
pi.py#pi-renamed.py
{code}

With the change we added in this jira, it will call run the Spark job with 
following params:

{code}
--master
yarn-cluster
--name
pyspark example
--conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.ui.view.acls=*
--queue
default
--conf
spark.executor.extraClassPath=$PWD/*
--conf
spark.driver.extraClassPath=$PWD/*
--conf
spark.yarn.security.tokens.hive.enabled=false
--conf
spark.yarn.security.tokens.hbase.enabled=false
--conf
spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--properties-file
spark-defaults.conf
--files
<>
--conf
spark.yarn.jar=hdfs://localhost/share/spark/lib/spark-assembly.jar
--verbose
hdfs://localhost/user/saley/examples/apps/spark-yarn-cluster/pi.py#pi-renamed.py
10
{code} 

The job fails saying - 
{code}
2017-02-07 21:59:24,847 [Driver] ERROR 
org.apache.spark.deploy.yarn.ApplicationMaster  - User application exited with 
status 2
2017-02-07 21:59:24,849 [Driver] INFO  
org.apache.spark.deploy.yarn.ApplicationMaster  - Final app status: FAILED, 
exitCode: 2, (reason: User application exited with status 2)
python: can't open file 'pi.py#pi-renamed.py': [Errno 2] No such file or 
directory
{code}
Spark does not understand the {{#}} sign.
Therefore, we need to pass in the direct path for the file.
But at the same time, we also need to make sure that application jar won't get 
distributed twice.

Solution - Mention the direct path for the application jar/py file if there is 
a {{#}} sign (fragment) in the path. We can do so, because file is already 
available in the launcher's local directory i.e. current directory. Also, at 
the same time remove the application jar from *--files* option. 


was (Author: satishsaley):
Reopening as there is a regression.

{code}
pi.py is under oozie.wf.application.path and workflow configuration is -

pyspark example
pi.py
${testConf}
pi.py#pi-renamed.py
{code}

With the change we added in this jira, it will call run the Spark job with 
following params:

{code}
--master
yarn-cluster
--name
pyspark example
--conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.ui.view.acls=*
--queue
default
--conf
spark.executor.extraClassPath=$PWD/*
--conf
spark.driver.extraClassPath=$PWD/*
--conf
spark.yarn.security.tokens.hive.enabled=false
--conf
spark.yarn.security.tokens.hbase.enabled=false
--conf
spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--properties-file
spark-defaults.conf
--files
<>
--conf
spark.yarn.jar=hdfs://localhost/share/spark/lib/spark-assembly.jar
--verbose
hdfs://localhost/user/saley/examples/apps/spark-yarn-cluster/pi.py#pi-renamed.py
10
{code} 

The job fails saying - 
{code}
2017-02-07 21:59:24,847 [Driver] ERROR 
org.apache.spark.deploy.yarn.ApplicationMaster  - User application exited with 
status 2
2017-02-07 21:59:24,849 [Driver] INFO  
org.apache.spark.deploy.yarn.ApplicationMaster  - Final app status: FAILED, 
exitCode: 2, (reason: User application exited with status 2)
python: can't open file 'pi.py#pi-renamed.py': [Errno 2] No such file or 
directory
{code}
Spark does not understand the {{#}} sign.
Therefore, we need to pass in the direct path for the file.
But at the same time, we also need to make sure that application jar won't get 
distributed twice.

Solution - Mention the direct path for the application jar/py file if there is 
a {{#}} sign (fragment) in the path. We can do so, because file is already 
available in the launcher's local directory i.e. current directory. Also, at 
the same time remove the application jar from *--files* option. While doing so, 
we need extra checks for PySpark dependencies otherwise those will get 
distributed multiple times. The amend patch will also distribute the files 
mentioned in  and having {{#}}.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 

Re: Oozie 4.x maintenace branch

[Robert Kanter]

Hadoop does something like this with the branch-2 (Hadoop 2.x) and trunk
(Hadoop 3.x) branches.  They even have their test-patch Jenkins thing setup
such that you can run patches against different branches.

This seems like a good idea.  Though I think we should:

   1. Define a timeframe for how long Oozie 4 would be maintained.  For
   instance, we could say Oozie 4.x will be maintained until MM/DD/YY, where
   we will re-evaluate if it should continue to be maintained to another
   future date or dropped.
   2. Define some guidelines on the sorts of things that will go into the
   Oozie 4 branch.  As Oozie 5 starts to diverge more and more from Oozie 4,
   the cost of requiring contributors to post two (possibly very different)
   versions of their patches and committers to review both will become more
   and more expensive.  Perhaps something like bug fixes only, unless the
   effort is deemed too high (which is subjective, I know).


- Robert

On Mon, Feb 6, 2017 at 2:57 AM, Peter Cseh  wrote:

> Hi all,
>
> OOZIE-1770 is (hopefully) getting close to be committed.
> It will break backward compatibility and remove huge parts of core Oozie
> code.
> There are multiple Oozie deployments out there which will be on version 4.x
> for some time.
> I'm wondering if It's a good idea to create a branch-4 along master with
> backward compatibility with 4.x to have issues fixed for a future possible
> 4.4.0 or 4.3.1. for users who don't want their Oozie upgraded to 5.x.
>
> I'm thinking about something similar to Hive's
>  HowToContribute-UnderstandingHiveBranches>
> policy
> for branch-1 and master.
>
> I don't know what's the process for this, I guess this would be a PMC
> decision, I just wanted to start the discussion about this topic.
>
> Any thoughts?
> Cheers,
> gp
>
>
> --
> Peter Cseh
> Software Engineer
> 
>

[jira] [Reopened] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley reopened OOZIE-2787:


Reopening as there is a regression.

{code}
pi.py is under oozie.wf.application.path and workflow configuration is -

pyspark example
pi.py
${testConf}
pi.py#pi-renamed.py
{code}

With the change we added in this jira, it will call run the Spark job with 
following params:

{code}
--master
yarn-cluster
--name
pyspark example
--conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.ui.view.acls=*
--queue
default
--conf
spark.executor.extraClassPath=$PWD/*
--conf
spark.driver.extraClassPath=$PWD/*
--conf
spark.yarn.security.tokens.hive.enabled=false
--conf
spark.yarn.security.tokens.hbase.enabled=false
--conf
spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--properties-file
spark-defaults.conf
--files
<>
--conf
spark.yarn.jar=hdfs://localhost/share/spark/lib/spark-assembly.jar
--verbose
hdfs://localhost/user/saley/examples/apps/spark-yarn-cluster/pi.py#pi-renamed.py
10
{code} 

The job fails saying - 
{code}
2017-02-07 21:59:24,847 [Driver] ERROR 
org.apache.spark.deploy.yarn.ApplicationMaster  - User application exited with 
status 2
2017-02-07 21:59:24,849 [Driver] INFO  
org.apache.spark.deploy.yarn.ApplicationMaster  - Final app status: FAILED, 
exitCode: 2, (reason: User application exited with status 2)
python: can't open file 'pi.py#pi-renamed.py': [Errno 2] No such file or 
directory
{code}
Spark does not understand the {{#}} sign.
Therefore, we need to pass in the direct path for the file.
But at the same time, we also need to make sure that application jar won't get 
distributed twice.

Solution - Mention the direct path for the application jar/py file if there is 
a {{#}} sign (fragment) in the path. We can do so, because file is already 
available in the launcher's local directory i.e. current directory. Also, at 
the same time remove the application jar from *--files* option. While doing so, 
we need extra checks for PySpark dependencies otherwise those will get 
distributed multiple times. The amend patch will also distribute the files 
mentioned in  and having {{#}}.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Hadoop 3 timeline

[Robert Kanter]

Hi all,

FYI, Andrew Wang (the release manager for Hadoop 3) is keeping updates on this
wiki page
,
for those who are interested.

Alpha 2 was released about a week ago.  There's going to be an Alpha 3, and
then a Beta 1.  After that, we'll have GA, which is currently targeting
July.

We've seen two issues so far with Hadoop 3:

   1. Hadoop 3 no longer issues delegation tokens in non-secure clusters (
   YARN-4126 ).  I believe
   this has been reverted in Alpha 2 via YARN-5882
   .
   2. Hadoop 3 no longer allows duplicate filenames in the distributed
   cache, even if they're the same file.  We had a problem like this long ago
   in Hadoop 2.0.2-alpha (see OOZIE-1089
   ), but the workaround
   for that doesn't seem to work here.  I believe we're still hitting this
   issue.

I imagine that Oozie 5 will be the first release that will support Hadoop 2
and Hadoop 3, so we should keep an eye on the Hadoop 3 progress.


- Robert

[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858422#comment-15858422
 ] 

Xiaobin Zheng commented on OOZIE-2792:
--

The above failure testBundleRerunInPausedWithError failed in other trunk builds 
as well. It is unrelated to this patch.

[~rkanter] Can you pls help review this patch? This is tested in a real cluster 
as well. 'externalChildIDs' field for hive2 action are displaying the Spark 
application ID properly with this patch.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858227#comment-15858227
 ] 

Abhishek Bafna commented on OOZIE-2791:
---

I tried the Oozie sharelib installation with {{-concurrency}} option with 
different number of parallel thread and It was able to install the oozie 
sharelib.

Cluster Information: 3 node cluster, build using virtual box within Mac.
The load on the cluster was not much, a bunch of MR jobs were running.
Values tried for number of threads: 50, 150, 250, 350, 450.

Thanks.

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (OOZIE-2793) Attempt to add apacheds-kerberos-codec-2.0.0-M15.jar multiple times to the distributed cache.

[Rinin (JIRA)]

Rinin created OOZIE-2793:


 Summary: Attempt to add apacheds-kerberos-codec-2.0.0-M15.jar 
multiple times to the distributed cache.
 Key: OOZIE-2793
 URL: https://issues.apache.org/jira/browse/OOZIE-2793
 Project: Oozie
  Issue Type: Bug
  Components: action, workflow
Affects Versions: 4.3.0
 Environment: emr 5.3, spark 2.1 , oozie 4.3.
Reporter: Rinin


Getting the below error while trying to execute spark submit using oozie on emr.

favorite
I am running on cluster mode. The apacheds-kerberos-codec-2.0.0-M15.jar is 
present in multiple places in oozie/share/lib/lib*/spark and 
oozie/share/lib/lib*/oozie. Is this an environmental issue ?

ava.lang.IllegalArgumentException: Attempt to add 
(hdfs://ip-***-**-**-**.ec2.internal:8020/user/oozie/share/lib/lib_20170208121307/oozie/apacheds-kerberos-codec-2.0.0-M15.jar)
 multiple times to the distributed cache.
at 
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$8.apply(Client.scala:608)
at 
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$8.apply(Client.scala:599)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
at 
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:599)
at 
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:598)
at scala.collection.immutable.List.foreach(List.scala:381)
at 
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:598)
at 
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:868)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:170)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1154)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:338)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:257)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:380)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:301)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
at 
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

FINAL REMINDER: CFP for ApacheCon closes February 11th

[Rich Bowen]

Dear Apache Enthusiast,

This is your FINAL reminder that the Call for Papers (CFP) for ApacheCon
Miami is closing this weekend - February 11th. This is your final
opportunity to submit a talk for consideration at this event.

This year, we are running several mini conferences in conjunction with
the main event, so if you're submitting for one of those events, please
pay attention to the instructions below.

Apache: Big Data
* Event information:
http://events.linuxfoundation.org/events/apache-big-data-north-america
* CFP:
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp

Apache: IoT (Internet of Things)
* Event Information: http://us.apacheiot.org/
* CFP -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp
(Indicate 'IoT' in the Target Audience field)

CloudStack Collaboration Conference
* Event information: http://us.cloudstackcollab.org/
* CFP -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp
(Indicate 'CloudStack' in the Target Audience field)

FlexJS Summit
* Event information - http://us.apacheflexjs.org/
* CFP -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp
(Indicate 'Flex' in the Target Audience field)

TomcatCon
* Event information - https://tomcat.apache.org/conference.html
* CFP -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp
(Indicate 'Tomcat' in the Target Audience field)

All other topics and projects
* Event information -
http://events.linuxfoundation.org/events/apachecon-north-america/program/about
* CFP -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp

Admission to any of these events also grants you access to all of the
others.

Thanks, and we look forward to seeing you in Miami!

-- 
Rich Bowen
VP Conferences, Apache Software Foundation
rbo...@apache.org
Twitter: @apachecon



(You are receiving this email because you are subscribed to a dev@ or
users@ list of some Apache Software Foundation project. If you do not
wish to receive email from these lists any more, you must follow that
list's unsubscription procedure. View the headers of this message for
unsubscription instructions.)

Re: Oozie UT is not stable

[Peter Cseh]

Hey,

We're aware of the unstable tests and we're working on improving on this
situation.
There were a bunch of test fixes in the last couple months, but some is
still to remain.
I can only suggest to identify the tests, open up issues for them on jira
and maybe try to look for the reason of failure if you encounter one.

gp


On Wed, Feb 8, 2017 at 10:05 AM, Dongying Jiao 
wrote:

> Hi:
> When I run oozie UTs, it is not common that all UTs can passed. Most of
> time, there are several cases failure, and the failure cases are not the
> same for each time. When run the falied case alone one by one, it will
> pass.
> I found this problem in both oozie 4.2 and oozie 4.3, I tried to run UT
> from both maven command and jenkins, the result is the same.
> Do you have any suggetions for this issue?
>
> Thanks and best regrads.
>
> Dong Ying
>



-- 
Peter Cseh
Software Engineer

Build failed in Jenkins: oozie-trunk-precommit-build #3633

[Apache Jenkins Server]

See 

Changes:

[rkanter] OOZIE-2786 Pass Oozie workflow ID and settings to Spark application

[abafna] OOZIE-2778 Copy only jetty.version related server dependencies to 
distro

--
[...truncated 14703 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/rest/JsonTags.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/rest/JsonUtils.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/rest/RestConstants.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/rest/JsonToBean.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/WorkflowJob.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/OozieClient.Metrics.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/BundleJob.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/Job.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/WorkflowAction.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/OozieClient.Metrics.Histogram.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/XOozieClient.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/BundleJob.Timeunit.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/CoordinatorJob.Timeunit.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/WorkflowJob.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/CoordinatorAction.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/SLAEvent.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/OozieClientException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/CoordinatorJob.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/OozieCLIException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/OozieCLI.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/CLIParser.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/CLIParser.Command.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/console/sla/css/images/ui-bg_glass_100_f6f6f6_1x400.png
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/console/sla/css/images/ui-bg_highlight-soft_100_ee_1x100.png
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/console/sla/css/images/ui-icons_22_256x240.png
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/console/sla/css/images/ui-bg_glass_65_ff_1x400.png
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/console/sla/css/images/ui-bg_flat_100_DEECFD_40x100.png
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/console/sla/js/table/jquery-ui-1.10.3.custom.min.js
 longer than 

[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857834#comment-15857834
 ] 

Hadoop QA commented on OOZIE-2792:
--

Testing JIRA OOZIE-2792

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that 
must be fixed.
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:red}-1{color} There are [1] new bugs found below threshold in 
[sharelib/hive2] that must be fixed.
.You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/hive2/findbugs-new.html
.The most important FindBugs errors are:
.At Hive2Main.java:[line 44]: 
org.apache.oozie.action.hadoop.Hive2Main.HIVE2_JOB_IDS_PATTERNS should be 
package protected
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1873
.Tests failed: 1
.Tests errors: 0

.The patch failed the following testcases:

.  
testBundleRerunInPausedWithError(org.apache.oozie.command.bundle.TestBundleRerunXCommand)

.Tests failing with errors:
.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3633/

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Oozie UT is not stable

[Dongying Jiao]

Hi:
When I run oozie UTs, it is not common that all UTs can passed. Most of
time, there are several cases failure, and the failure cases are not the
same for each time. When run the falied case alone one by one, it will
pass.
I found this problem in both oozie 4.2 and oozie 4.3, I tried to run UT
from both maven command and jenkins, the result is the same.
Do you have any suggetions for this issue?

Thanks and best regrads.

Dong Ying