[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860858#comment-15860858
 ] 

Satish Subhashrao Saley commented on OOZIE-2787:


Thank you Puru, Abhishek and Xiaobin for review. Committed to master.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (91 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2792  Hive2 action is not parsing Spark application ID from log file 
properly when Hive is on Spark
https://issues.apache.org/jira/browse/OOZIE-2792
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2790  log4j configuration is not passed to spark executors
https://issues.apache.org/jira/browse/OOZIE-2790
OOZIE-2787  Oozie distributes application jar twice making the spark job fail
https://issues.apache.org/jira/browse/OOZIE-2787
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2750  Spelling errors in the log messages and exception messages
https://issues.apache.org/jira/browse/OOZIE-2750
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2739  Remove property expansion pattern from ShellMain's log4j properties 
content
https://issues.apache.org/jira/browse/OOZIE-2739
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when 
creating a Sharelib
https://issues.apache.org/jira/browse/OOZIE-2629
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2608  Comma in oozie.service.JPAService.jdbc.password value results in 
authentication error
https://issues.apache.org/jira/browse/OOZIE-2608
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues.apache.org/jira/browse/OOZIE-2585
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues.apache.org/j

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491
OOZIE

[jira] [Updated] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Xiaobin Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng updated OOZIE-2792:
-
Attachment: OOZIE-2792-2.patch

OOZIE-2792-2.patch adds basic test coverage for all Action main classes that 
maintains a list of regex pattern to retrieve external IDs from log files.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860777#comment-15860777
 ] 

Hadoop QA commented on OOZIE-2787:
--

Testing JIRA OOZIE-2787

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1873
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3639/

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860660#comment-15860660
 ] 

Abhishek Bafna commented on OOZIE-2788:
---

Thank You @robert kanter and everyone for the review and committing it.

> Fix jobs API servlet mapping for EmbeddedOozieServer
> 
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not 
> allow Oozie to submit coordinator and bundle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860588#comment-15860588
 ] 

Xiaobin Zheng commented on OOZIE-2792:
--

Thx for reviewing this [~gezapeti] [~rkanter]!
Sure I'll see if there is other similar test applicable and post a new patch.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860558#comment-15860558
 ] 

Robert Kanter commented on OOZIE-2788:
--

Well that was silly :)

+1 will commit shortly.

> Fix jobs API servlet mapping for EmbeddedOozieServer
> 
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not 
> allow Oozie to submit coordinator and bundle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860557#comment-15860557
 ] 

Robert Kanter commented on OOZIE-2792:
--

This looks good.  

I know this is a little outside the scope of this patch, but it's kind of 
related and should be pretty quick to do: can you add a similar test for the 
other Action types that have a jobid pattern (e.g. 
{{PigMain.PIG_JOB_IDS_PATTERNS}}, etc)?  That's probably something we should 
have already had tests for.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860506#comment-15860506
 ] 

Purshotam Shah commented on OOZIE-2787:
---

+1 for OOZIE-2787-amend-5.patch.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-5.patch

Renaming the title for section in documentation. 

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-1525) Oozie workflow does not update status sometimes and is stuck in RUNNING state.

[Gopi Krishnan Nambiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860479#comment-15860479
 ] 

Gopi Krishnan Nambiar commented on OOZIE-1525:
--

I ran into this error recently and triaged to find that the cause for me was 
running multiple Oozie servers without HA support. I was running Oozie version 
4.0.1 which did not support HA. Do you think this could have been the reason 
for your issues too ?

> Oozie workflow does not update status sometimes and is stuck in RUNNING state.
> --
>
> Key: OOZIE-1525
> URL: https://issues.apache.org/jira/browse/OOZIE-1525
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 3.3.2
>Reporter: Eugene Shevchuk
> Attachments: OOZIE-90.patch
>
>
> Sometimes workflow is stuck in RUNNING state with the following info:
> {noformat}
> Job ID : 002-130822205833322-oozie-hdp-W
> 
> Workflow Name : wordcount-wf
> App Path  : ***
> Status: RUNNING
> Run   : 0
> User  : ***
> Group : -
> Created   : 2013-08-22 21:01 GMT
> Started   : 2013-08-22 21:01 GMT
> Last Modified : 2013-08-22 21:02 GMT
> Ended : -
> CoordAction ID: -
> Actions
> 
> ID
> StatusExt ID Ext Status Err Code
> 
> 002-130822205833322-oozie-hdp-W@:start:   
> OK-  OK -
> 
> 002-130822205833322-oozie-hdp-W@wc
> OKjob_201308221727_0287  SUCCEEDED  -
> 
> {noformat}
> In log there are some problems with jpaservice:
> {noformat}
> 2013-08-22 23:00:59,170 ERROR SignalXCommand:536 - USER[***] GROUP[-] TOKEN[] 
> APP[wordcount-wf] JOB[002-130822205833322-oozie-hdp-W] 
> ACTION[002-130822205833322-oozie-hdp-W@wc] Exception, 
>  
> org.apache.openjpa.persistence.RollbackException: The transaction has been 
> rolled back.  See the nested exceptions for details on the errors that 
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@1dadb1b6
>   at 
> org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:585)
>   at org.apache.oozie.service.JPAService.execute(JPAService.java:217)
>   at 
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:310)
>   at 
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:64)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:277)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>   at java.lang.Thread.run(Thread.java:722)
> Caused by:  
> org.apache.openjpa.persistence.PersistenceException: The transaction has been 
> rolled back.  See the nested exceptions for details on the errors that 
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@1dadb1b6
>   at 
> org.apache.openjpa.kernel.BrokerImpl.newFlushException(BrokerImpl.java:2316)
>   at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.java:2153)
>   at org.apache.openjpa.kernel.BrokerImpl.flushSafe(BrokerImpl.java:2051)
>   at 
> org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(BrokerImpl.java:1969)
>   at 
> org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:81)
>   at org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.java:1493)
>   at 
> org.apache.openjpa.kernel.DelegatingBroker.commit(DelegatingBroker.java:925)
>   at 
> org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:561)
>   ... 10 more
> Caused by:  
> org.apache.openjpa.persistence.EntityEx

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-4.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, 
> OOZIE-2787-amend-4.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860414#comment-15860414
 ] 

Hadoop QA commented on OOZIE-2787:
--

Testing JIRA OOZIE-2787

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1873
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3638/

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : oozie-trunk-precommit-build #3638

[Apache Jenkins Server]

See 

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860385#comment-15860385
 ] 

Hadoop QA commented on OOZIE-2787:
--

Testing JIRA OOZIE-2787

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1873
.Tests failed: 1
.Tests errors: 1

.The patch failed the following testcases:

.  
testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3637/

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3637

[Apache Jenkins Server]

See 

--
[...truncated 14709 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/CoordinatorAction.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/retry/ConnectionRetriableClient.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/retry/package-tree.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/retry/package-summary.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/retry/class-use/ConnectionRetriableClient.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/retry/package-frame.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/retry/package-use.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/CoordinatorJob.Timeunit.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/OozieClient.SORT_BY.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/OozieClient.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/WorkflowJob.Status.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/CoordinatorJob.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/SLAEvent.SlaAppType.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/package-use.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/BulkResponse.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/JsonToBean.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/package-tree.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/RestConstants.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/package-summary.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/JsonTags.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/JsonUtils.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/class-use/JsonToBean.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/class-use/RestConstants.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/class-use/JsonTags.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/class-use/JsonUtils.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/package-frame.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/rest/package-use.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/client/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/org/apache/oozie/package-frame.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/e

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860249#comment-15860249
 ] 

Xiaobin Zheng commented on OOZIE-2718:
--

I ran all above failing tests locally, and all of them are passing. 
[~abhishekbafna] Can you pls take a another look?

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch, OOZIE-2718-4.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-3.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2787:
---
Attachment: OOZIE-2787-amend-2.patch

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

[Satish Subhashrao Saley (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860204#comment-15860204
 ] 

Satish Subhashrao Saley commented on OOZIE-2787:


Thank you for review [~abhishekbafna]  [~zhengxb2005].
Users don't need to do any extra work, they will continue using current 
configuration. I added an example in doc. Also added some test cases for 
JarFilter.

> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, 
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, 
> OOZIE-2787-amend-1.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to 
> distributed cache. Since this gets added twice, the job fails. This is 
> observed from spark 2.1.0 which introduces a check for same file and fails 
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860159#comment-15860159
 ] 

Hadoop QA commented on OOZIE-2718:
--

Testing JIRA OOZIE-2718

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1872
.Tests failed: 6
.Tests errors: 3

.The patch failed the following testcases:

.  testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)
.  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)
.  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
.  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMainWithOldAPI)

.Tests failing with errors:
.  testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)
.  testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
.  testLoadDump(org.apache.oozie.tools.TestDBLoadDump)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3636/

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch, OOZIE-2718-4.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3636

[Apache Jenkins Server]

See 

--
[...truncated 14723 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/ActionExecutor.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/ConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSExceptionListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSSLAEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandlerException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/DependencyChecker.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/WaitingActions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/SimpleHCatDependencyCache.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/HCatDependencyCache.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/HCatMessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/WaitingAction.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/ActionDependency.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/FSURIHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandler.Conte

[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860069#comment-15860069
 ] 

Purshotam Shah commented on OOZIE-2788:
---

+1

> Fix jobs API servlet mapping for EmbeddedOozieServer
> 
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not 
> allow Oozie to submit coordinator and bundle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860016#comment-15860016
 ] 

Hadoop QA commented on OOZIE-2791:
--

Testing JIRA OOZIE-2791

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:red}-1{color} There are [2] new bugs found below threshold in 
[tools] that must be fixed.
.You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
.The most important FindBugs errors are:
.At OozieSharelibCLI.java:[line 185]: Boxing/unboxing to parse a primitive 
org.apache.oozie.tools.OozieSharelibCLI.run(String[])
.At OozieSharelibCLI.java:[lines 221-226]: Should 
org.apache.oozie.tools.OozieSharelibCLI$CopyTaskConfiguration be a _static_ 
inner class?
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1872
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3635/

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2791-01.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430

Build failed in Jenkins: oozie-trunk-precommit-build #3635

[Apache Jenkins Server]

See 

--
[...truncated 14676 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ShellActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/PigActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HCatCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/JavaActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/DistcpActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HadoopELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProperties.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/MapReduceActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProvider.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/ActionExecutor.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/ConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSExceptionListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache

[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859964#comment-15859964
 ] 

Attila Sasvari commented on OOZIE-2788:
---

+1 (nonbinding)

> Fix jobs API servlet mapping for EmbeddedOozieServer
> 
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not 
> allow Oozie to submit coordinator and bundle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859953#comment-15859953
 ] 

Xiaobin Zheng commented on OOZIE-2788:
--

+1 (nonbinding). The change looks reasonable.

> Fix jobs API servlet mapping for EmbeddedOozieServer
> 
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not 
> allow Oozie to submit coordinator and bundle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2718) Improve -dryrun for bundles

[Xiaobin Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng updated OOZIE-2718:
-
Attachment: OOZIE-2718-4.patch

[~abhishekbafna] Thx for taking a look! I updated 
TestCoordUpdateXCommand#setUpBundleAndGetCoordID() and DG_CommandLineTool.twiki.

I added 'Dryrun support for bundle job is added since 4.4.0.' to the twiki 
assuming master is pointing to 4.4.0 release. Pls help review.

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch, OOZIE-2718-4.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859918#comment-15859918
 ] 

Andras Piros commented on OOZIE-2751:
-

You're welcome [~abhishekbafna]! [~satishsaley] any news on the issue?

> LocalOozieClient is missing methods from OozieClient
> 
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
>  Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch, 
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch, 
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the 
> normal {{OozieClient}} methods instead of making REST calls to an Oozie 
> Server.  However, it currently doesn't override all methods, which means you 
> hit the {{OozieClient}} implementation, which tries to make a REST call and 
> will fail (because it's not setup for that).  For example, {{getStatus(String 
> jobId)}} isn't overridden, and doesn't work when used from 
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859789#comment-15859789
 ] 

Attila Sasvari commented on OOZIE-2791:
---

[~abhishekbafna] thanks for the additional info. We ran into this issue on a 
4-node cluster with Hadoop 2.6 (multiple components were talking to HDFS).

When I tried to reproduce the problem with {{-concurrency 150}} on my 
single-node pseudo Hadoop, I noticed that sharelib was partially installed 
(exception failures for copy tasks were logged). At the end of the execution 
there were multiple files with 0 byte size created in HDFS.

Now I have a working solution that I tested on Mac with pseudo Hadoop 2.6.0. 
Briefly, I add information about each failed copy task to a concurrent hash 
set, and retry uploading missed files using a single thread (with 
copyFromLocalFile). Before re-uploading files, we wait 1000 ms. If it fails 
again, we increase delay by the factor of 2, and decrease retry count (it is 
now hardcoded to be 5 times).  

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2791-01.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutp

[jira] [Updated] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2791:
--
Attachment: OOZIE-2791-01.patch

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2791-01.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2788) Fix jobs API servlet mapping for EmbeddedOozieServer

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859465#comment-15859465
 ] 

Abhishek Bafna commented on OOZIE-2788:
---

[~rohini], [~puru], [~satishsaley] Can someone take a look. Because this is 
breaking the oozie master while running oozie with embedded jetty. It fails 
while submitting coordinator or bundle due to missing servlet mapping. Thanks.

> Fix jobs API servlet mapping for EmbeddedOozieServer
> 
>
> Key: OOZIE-2788
> URL: https://issues.apache.org/jira/browse/OOZIE-2788
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2788-00.patch
>
>
> The {{/v1/jobs}} and {{/v2/jobs}} are mapping to {{/v0/jobs}} which does not 
> allow Oozie to submit coordinator and bundle.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859456#comment-15859456
 ] 

Abhishek Bafna commented on OOZIE-2718:
---

Few minor comments:

# Similar test code is available in the 
TestCoordUpdateXCommand#setUpBundleAndGetCoordID(), can you please take care of 
that as well.
# Update the documentation as well for the {{-dryrun}} into 
{{DG_CommandLineTool.twiki}}.

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859261#comment-15859261
 ] 

Peter Cseh commented on OOZIE-2792:
---

Thanks for the contribution [~zhengxb2005]!
Tests are passing locally.
+1 (nonbinding) for the patch.

> Hive2 action is not parsing Spark application ID from log file properly when 
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID 
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard 
> to tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859253#comment-15859253
 ] 

Peter Cseh commented on OOZIE-2718:
---

Thanks for the explanation [~zhengxb2005]! I missed that dryrun was tested 
already.
Failing tests passed locally.
+1 (nonbinding)

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)