[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (89 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2790  log4j configuration is not passed to spark executors
https://issues.apache.org/jira/browse/OOZIE-2790
OOZIE-2788  Fix jobs API servlet mapping for EmbeddedOozieServer
https://issues.apache.org/jira/browse/OOZIE-2788
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2750  Spelling errors in the log messages and exception messages
https://issues.apache.org/jira/browse/OOZIE-2750
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2739  Remove property expansion pattern from ShellMain's log4j properties 
content
https://issues.apache.org/jira/browse/OOZIE-2739
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when 
creating a Sharelib
https://issues.apache.org/jira/browse/OOZIE-2629
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2608  Comma in oozie.service.JPAService.jdbc.password value results in 
authentication error
https://issues.apache.org/jira/browse/OOZIE-2608
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues.apache.org/jira/browse/OOZIE-2585
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues.apache.org/jira/browse/OOZIE-2573
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues.apache.org/jira/browse/OOZIE-2564

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857401#comment-15857401
 ] 

Abhishek Bafna commented on OOZIE-2778:
---

+1.

Thanks [~asasvari] for working on this. Thanks [~andras.piros] for the review.
Committed to master.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Fix For: 5.0.0
>
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-07.patch, OOZIE-2778-2.patch, 
> OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark

[Xiaobin Zheng (JIRA)]

Xiaobin Zheng created OOZIE-2792:


 Summary: Hive2 action is not parsing Spark application ID from log 
file properly when Hive is on Spark
 Key: OOZIE-2792
 URL: https://issues.apache.org/jira/browse/OOZIE-2792
 Project: Oozie
  Issue Type: Bug
  Components: action
Reporter: Xiaobin Zheng
Assignee: Xiaobin Zheng
Priority: Minor


When Hive2 is on Spark, hive action is not able to parse Spark application ID 
from log file as 'externalChildID' like Spark/MR actions. This makes it hard to 
tell which job hive launches from Oozie server for a particular workflow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857032#comment-15857032
 ] 

Xiaobin Zheng commented on OOZIE-2718:
--

I don't think above failure for latest patch is related to my change. Previous 
patch with similar change was passing green.

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856982#comment-15856982
 ] 

Attila Sasvari commented on OOZIE-2778:
---

org.apache.oozie.jms.TestJMSJobEventListener.testConnectionDrop failure is 
unrelated.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-07.patch, OOZIE-2778-2.patch, 
> OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856900#comment-15856900
 ] 

Andrew Wang commented on OOZIE-2791:


I took a peek at this class, looks like that's a Java API rather than an HDFS 
one. My guess though is that {{temp.deleteOnExit}} doesn't work since the 
directory isn't empty (same for the {{temp.delete()}} before it), so we should 
handle this with a try/finally if you always want this temp dir to be cleaned 
up. There's already a call to {{FileUtils.deleteDirectory}} at the bottom that 
will do a recursive delete that could be moved into a {{finally}} block.

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This 

Build failed in Jenkins: oozie-trunk-precommit-build #3632

[Apache Jenkins Server]

See 

--
[...truncated 14708 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/WorkflowAction.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/XOozieClient.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/OozieClient.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/message/EventMessage.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/message/JobMessage.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/message/CoordinatorActionMessage.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/message/SLAMessage.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/message/WorkflowJobMessage.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/SLAEvent.SLAStatus.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/SLAEvent.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/jms/JMSMessagingUtils.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/jms/JMSHeaderConstants.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/jms/JSONMessageDeserializer.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/jms/MessageDeserializer.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/JobEvent.EventStatus.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/Event.MessageType.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/Event.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/JobEvent.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/client/event/SLAEvent.EventStatus.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/BuildInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/OozieCLI.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/CLIParser.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/OozieCLIException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/client/apidocs/src-html/org/apache/oozie/cli/CLIParser.Command.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/geronimo-annotation_1.0_spec-1.1.1.jar
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/oozie-sharelib-sqoop-4.4.0-SNAPSHOT.jar
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/oozie-sharelib-distcp-4.4.0-SNAPSHOT.jar
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/oozie-sharelib-streaming-4.4.0-SNAPSHOT.jar
 longer than 100 characters.
[WARNING] Entry: 

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856882#comment-15856882
 ] 

Hadoop QA commented on OOZIE-2718:
--

Testing JIRA OOZIE-2718

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1871
.Tests failed: 3
.Tests errors: 0

.The patch failed the following testcases:

.  
testConnectionRetryExceptionListener(org.apache.oozie.service.TestJMSAccessorService)
.  
testMemoryUsageAndSpeed(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)
.  
testMemoryUsageAndSpeed(org.apache.oozie.service.TestPartitionDependencyManagerService)

.Tests failing with errors:
.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3632/

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856878#comment-15856878
 ] 

Hadoop QA commented on OOZIE-2778:
--

Testing JIRA OOZIE-2778

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1871
.Tests failed: 1
.Tests errors: 0

.The patch failed the following testcases:

.  testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)

.Tests failing with errors:
.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3631/

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-07.patch, OOZIE-2778-2.patch, 
> OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3631

[Apache Jenkins Server]

See 

--
[...truncated 14741 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/DistcpActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HadoopELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProperties.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/MapReduceActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProvider.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/ActionExecutor.Context.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/CoordinatorEngine.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageReceiver.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/DefaultConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/ConnectionContext.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSExceptionListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSConnectionInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSJobEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/MessageHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/jms/JMSSLAEventListener.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandlerException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/URIHandler.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/DependencyChecker.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/dependency/hcat/WaitingActions.html
 longer than 100 characters.
[WARNING] Entry: 

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856848#comment-15856848
 ] 

Attila Sasvari commented on OOZIE-2791:
---

[~andrew.wang] I did not know about that. Thanks. 

I've just noticed another thing: if there is an exception during the sharelib 
installation (e.g. copy task fails), files in the temporary directory 
(https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L137)
 will not be deleted at the end. 

{code}
Error: Copy task for 
"/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/oozie5744006317396681919.dir/share/lib/hive2/hadoop-yarn-common-2.4.0.jar"
 failed with exception
...

$ ls 
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/oozie5744006317396681919.dir/share/lib/hive2/hadoop-yarn-common-2.4.0.jar
 
/var/folders/9q/f8p_r6gj0wbck49_dc092q_mgp/T/oozie5744006317396681919.dir/share/lib/hive2/hadoop-yarn-common-2.4.0.jar
{code}


> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's 

[jira] [Commented] (OOZIE-1770) Create Oozie Application Master for YARN

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856806#comment-15856806
 ] 

Robert Kanter commented on OOZIE-1770:
--

For {{launcher.lib}}, I think that's supposed to refer to this property in 
oozie-site
{code:xml}

oozie.action.ship.launcher.jar
false

Specifies whether launcher jar is shipped or not.


{code}
which makes Oozie generate a jar containing the Oozie-supplied code that 
otherwise would be in the sharelib.  This is useful for users who build their 
own sharelib instead of using the Oozie-provided one.  For instance, if you 
provided your own Pig jars instead of using the Pig sharelib, you would be 
missing Oozie classes like {{PigMain}} and {{LauncherMapper}}.  Setting this 
property to {{true}} would have Oozie provide those for you in that case.  So 
you can use any workflow, no need for a special one.  The easiest way to verify 
this is probably to go and remove the {{oozie-*}} jars from a Sharelib (e.g. 
Pig sharelib), turn on this property, and see if the (Pig) action still works.

For Oozie on MapReduce, the equivalent of launcher.xml is the Launcher Job's 
jobconf XML file, which is available in the JHS and elsewhere.  With OYA, 
that's been replaced with this launcher.xml file, which I don't think we're 
currently exposing anywhere.  Being able to get the contents of that file would 
likely be useful for debugging.  A place to make this data available could be 
the launcher AM output.  It might be better to wait to add this until we do 
OOZIE-2702 and make it DEBUG or TRACE level to reduce spam.  Another option is 
that it might be possible to include the launcher.xml file with the aggregated 
logs; IIRC, the NM will blindly process any files from that dir.

> Create Oozie Application Master for YARN
> 
>
> Key: OOZIE-1770
> URL: https://issues.apache.org/jira/browse/OOZIE-1770
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Peter Cseh
> Attachments: OOZIE-1770-001.patch, OOZIE-1770-002.patch, 
> OOZIE-1770-003.patch, OOZIE-1770-004.patch, OOZIE-1770-005.patch, 
> OOZIE-1770-006.patch, OOZIE-1770-007.patch, OOZIE-1770-008.patch, 
> OOZIE-1770-009.patch, OozieYarnAM.pdf, oya.patch, oya-rm-screenshot.jpg, OYA 
> Scoping Doc 001.pdf, Prelim OYA Scoping Doc 001.pdf, Prelim OYA Scoping Doc 
> 002.pdf, Prelim OYA Scoping Doc 003.pdf
>
>
> After the first release of oozie on hadoop 2, it will be good if users can 
> set execution engine in oozie conf, be it YARN AM or traditional MR. We can 
> target this for post oozie 4.1 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856798#comment-15856798
 ] 

Andrew Wang commented on OOZIE-2791:


I think you can do it without a new FileSystem object since there's a create 
method that takes a blockSize parameter (FileSystem is also a bit heavyweight):

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L1048

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856785#comment-15856785
 ] 

Attila Sasvari commented on OOZIE-2791:
---

[~andrew.wang] thanks for the idea. I've just created a POC based on your 
comment, and now I am evaluating it (some refactoring is needed in 
OozieSharelibCLI, [copyFolderRecursively | 
https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L255]
 could be replaced with a new method that creates a new Filesystem object with 
the proper dfs.block.size before submitting the copy job).  

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-1770) Create Oozie Application Master for YARN

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856778#comment-15856778
 ] 

Andras Piros commented on OOZIE-1770:
-

* as for benchmark results I think for this JIRA the one that has already been 
measured (some simple shell actions pre-OYA and OYA performance) would be fine, 
for the deeper measurements we could open another JIRA.
* as for re-testing that property {{launcher.lib.*}} still works, I don't know 
of an example workflow - [~rkanter] can you please advise?
* as for generated {{launcher.xml}} file content printouts, [~rkanter] can you 
please also advise?
* as for {{ApplicationHistoryServer}} (former {{TimelineService}}) usage, the 
patch hasn't changed the way of usage.
* as for {{JavaActionExecutor.setCredentialTokens()}}, here are the important 
changes - none of which changes functionality, only making code more readable:
** some of the former {{CredentialsProvider}} is extracted to 
{{CredentialsProviderFactory}}, some other parts of it are left in 
{{CredentialsProvider}}
** {{CrendentialsProvider}} is now an {{interface}}, not an {{abstract class}}
** {{Credentials.addToJobConf()}} functionality is moved to different 
{{CredentialsProvider.updateCredentials()}} implementations, and {{abstract 
class Credentials}} has been removed 

> Create Oozie Application Master for YARN
> 
>
> Key: OOZIE-1770
> URL: https://issues.apache.org/jira/browse/OOZIE-1770
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Peter Cseh
> Attachments: OOZIE-1770-001.patch, OOZIE-1770-002.patch, 
> OOZIE-1770-003.patch, OOZIE-1770-004.patch, OOZIE-1770-005.patch, 
> OOZIE-1770-006.patch, OOZIE-1770-007.patch, OOZIE-1770-008.patch, 
> OOZIE-1770-009.patch, OozieYarnAM.pdf, oya.patch, oya-rm-screenshot.jpg, OYA 
> Scoping Doc 001.pdf, Prelim OYA Scoping Doc 001.pdf, Prelim OYA Scoping Doc 
> 002.pdf, Prelim OYA Scoping Doc 003.pdf
>
>
> After the first release of oozie on hadoop 2, it will be good if users can 
> set execution engine in oozie conf, be it YARN AM or traditional MR. We can 
> target this for post oozie 4.1 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856642#comment-15856642
 ] 

Andrew Wang commented on OOZIE-2791:


One idea I actually really like is to set the dfs.block.size to the size of the 
file being uploaded. This should prevent HDFS from reserving any excess 
capacity during the write.

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2778:
--
Attachment: OOZIE-2778-07.patch

[~abhishekbafna] good point, thanks. I will pay more attention next time.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-07.patch, OOZIE-2778-2.patch, 
> OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Xiaobin Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856515#comment-15856515
 ] 

Xiaobin Zheng commented on OOZIE-2718:
--

Thx for taking a look [~gezapeti]!

There are already test cases covering dryrun: 
TestBundleSubmitXCommand.testJobXmlCommentRemoved and 
TestBundleSubmitXCommand.testDuplicateCoordName. Both of them are using 'new 
BundleSubmitXCommand(true, jobConf)' to run as 'dryrun'. Since you asked, I 
added one more assertion in 'testJobXmlCommentRemoved' in patch-3 for dryrun 
test case to be more clear.

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2718) Improve -dryrun for bundles

[Xiaobin Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobin Zheng updated OOZIE-2718:
-
Attachment: OOZIE-2718-3.patch

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch, 
> OOZIE-2718-3.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0

[Abhishek Bafna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Bafna updated OOZIE-2231:
--
Summary: Upgrade curator to latest version 2.12.0  (was: Upgrade curator to 
latest version 2.11.0)

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.11.0

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856377#comment-15856377
 ] 

Abhishek Bafna commented on OOZIE-2231:
---

Sure [~rkanter]. I will update the issue title for the same. Thanks.

> Upgrade curator to latest version 2.11.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856304#comment-15856304
 ] 

Abhishek Bafna edited comment on OOZIE-2778 at 2/7/17 4:59 PM:
---

Hi [~asasvari], one of the exclusion is at wrong place. Can you please do 
{{mvn:dependency}} output for the server module, please have a look, with the 
recent patch.

The exclusion needs to be in {{hadoop-common}} dependency. Thanks.


was (Author: abhishekbafna):
Hi [~asasvari], I think the exclusion is at wrong place. I am attaching a 
mvn:dependency output for the server module, please have a look. This is after 
the recent patch.

The exclusion needs to be in {{hadoop-common}} dependency. Thanks.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856304#comment-15856304
 ] 

Abhishek Bafna edited comment on OOZIE-2778 at 2/7/17 4:57 PM:
---

Hi [~asasvari], I think the exclusion is at wrong place. I am attaching a 
mvn:dependency output for the server module, please have a look. This is after 
the recent patch.

The exclusion needs to be in {{hadoop-common}} dependency. Thanks.


was (Author: abhishekbafna):
+1.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856304#comment-15856304
 ] 

Abhishek Bafna commented on OOZIE-2778:
---

+1.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3630

[Apache Jenkins Server]

See 

--
[...truncated 14103 lines...]
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/control/JoinActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/control/KillActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/control/StartActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProvider.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2ActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/DistcpActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SqoopActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/MapReduceActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/PigActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/OozieJobInfo.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HCatCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HbaseCredentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/LauncherMapperHelper.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/SparkActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialException.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/FsELFunctions.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ShellActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/JavaActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HiveActionExecutor.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Hive2Credentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/CredentialsProperties.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HCatCredentialHelper.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/Credentials.html
 longer than 100 characters.
[WARNING] Entry: 
oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/docs/core/apidocs/src-html/org/apache/oozie/action/hadoop/HadoopELFunctions.html
 longer than 100 characters.
[WARNING] 

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856280#comment-15856280
 ] 

Hadoop QA commented on OOZIE-2778:
--

Testing JIRA OOZIE-2778

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3630/

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855978#comment-15855978
 ] 

Attila Sasvari commented on OOZIE-2778:
---

[~abhishekbafna] you are right, I see

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855967#comment-15855967
 ] 

Hadoop QA commented on OOZIE-2778:
--

Testing JIRA OOZIE-2778

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3629

[Apache Jenkins Server]

See 

--
[...truncated 4523 lines...]
[INFO] Copying asm-5.0.1.jar to 

[INFO] Copying openjpa-lib-2.4.1.jar to 

[INFO] Copying opencsv-2.3.jar to 

[INFO] Copying jasper-runtime-5.5.23.jar to 

[INFO] Copying curator-client-2.5.0.jar to 

[INFO] Copying jetty-annotations-9.2.19.v20160908.jar to 

[INFO] Copying jersey-server-1.9.jar to 

[INFO] Copying jackson-core-2.2.2.jar to 

[INFO] 
[INFO] >>> findbugs-maven-plugin:3.0.0:check (findbug) > :findbugs @ 
oozie-server >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (findbugs) @ oozie-server ---
[INFO] Fork Value is true
[INFO] Done FindBugs Analysis
[INFO] 
[INFO] <<< findbugs-maven-plugin:3.0.0:check (findbug) < :findbugs @ 
oozie-server <<<
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:check (findbug) @ oozie-server ---
[INFO] BugInstance size is 0
[INFO] Error size is 0
[INFO] No errors/warnings found
[INFO] 
[INFO] --- xml-maven-plugin:1.0:transform (default) @ oozie-server ---
Warning:  org.apache.xerces.parsers.SAXParser: Feature 
'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not 
recognized.
[INFO] Transforming file: 

[INFO] Transformed 1 file(s).
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.9.1:check (default) @ oozie-server ---
[INFO] Starting audit...
Audit done.

[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ oozie-distro ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] >>> findbugs-maven-plugin:3.0.0:check (findbug) > :findbugs @ 
oozie-distro >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (findbugs) @ oozie-distro ---
[INFO] 
[INFO] <<< findbugs-maven-plugin:3.0.0:check (findbug) < :findbugs @ 
oozie-distro <<<
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:check (findbug) @ oozie-distro ---
[INFO] 
[INFO] --- xml-maven-plugin:1.0:transform (default) 

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855963#comment-15855963
 ] 

Peter Cseh commented on OOZIE-2718:
---

Thank you [~zhengxb2005] for picking this up!

Can you add a test in {{TestBundleSubmitXCommand}} to test dryrun?

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855959#comment-15855959
 ] 

Attila Sasvari commented on OOZIE-2778:
---

[~abhishekbafna] that was on purpose, I just want to re-trigger the QA job. I 
don't think that failures are related to the latest changes, but let me 
re-execute the test cases.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2778:
--
Attachment: OOZIE-2778-06.patch

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-06.patch, OOZIE-2778-2.patch, OOZIE-2778-3.patch, 
> OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855905#comment-15855905
 ] 

Abhishek Bafna commented on OOZIE-2778:
---

[~asasvari] I am afraid the recent patch will also fail to apply because it 
contains the same diff twice like the earlier one. There are no new changes 
compared to the last one. I think you need to check the last failed UTs locally 
for the patch [^OOZIE-2778-05.patch]. Thanks.

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-2.patch, OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2718) Improve -dryrun for bundles

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855843#comment-15855843
 ] 

Hadoop QA commented on OOZIE-2718:
--

Testing JIRA OOZIE-2718

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1871
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3628/

> Improve -dryrun for bundles
> ---
>
> Key: OOZIE-2718
> URL: https://issues.apache.org/jira/browse/OOZIE-2718
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Xiaobin Zheng
> Attachments: OOZIE-2718-1.patch, OOZIE-2718-2.patch
>
>
> When trying to do a -dryrun for Oozie:
> {code}
> error:
> Error: E1310 : E1310: Bundle Job submission Error: [null]
> {code}
> The 
> [documentation|https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Dryrun_of_Coordinator_Job]
>  states that dryruns are working for workflows and coordinators.
> {quote}
> "Dryrun a workflow (since 3.3.2) or coordinator (since 2.0) job without 
> actually executing it"
> {quote}
> The easy fix is to document that it does not work and create an appropriate 
> error message for it. It's also possible to add support for bundle dryruns. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Attila Sasvari (JIRA)]

Attila Sasvari created OOZIE-2791:
-

 Summary: ShareLib installation may fail on busy Hadoop clusters
 Key: OOZIE-2791
 URL: https://issues.apache.org/jira/browse/OOZIE-2791
 Project: Oozie
  Issue Type: Bug
Reporter: Attila Sasvari


On a busy Hadoop cluster it can happen that users cannot install properly  
Oozie ShareLib.

Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
concurrency number set high (to simulate a busy cluster):
{code}
oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
oozie-sharelib-*.tar.gz -concurrency 150
{code}

You can see a lot of errors (failed copy tasks) on the output:
{code}
Running 464 copy tasks on 150 threads

Error: Copy task failed with exception

Stack trace for the error was (for debug purposes):
--
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
/user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
could only be replicated to 0 nodes instead of minReplication (=1).  There are 
1 datanode(s) running and no node(s) are excluded in this operation.
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

at org.apache.hadoop.ipc.Client.call(Client.java:1410)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
--
...
{code}

You can see file is created but it's size is 0.
{code}
-rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
{code}

This behaviour is clearly wrong. 

In case of such an exception, we should retry copying or rollback changes. We 
should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2778) Copy only jetty.version related server dependencies to distro

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2778:
--
Attachment: OOZIE-2778-5.patch

> Copy only jetty.version related server dependencies to distro 
> --
>
> Key: OOZIE-2778
> URL: https://issues.apache.org/jira/browse/OOZIE-2778
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2778-01.patch, OOZIE-2778-05.patch, 
> OOZIE-2778-2.patch, OOZIE-2778-3.patch, OOZIE-2778-4.patch, OOZIE-2778-5.patch
>
>
> To compile JSP, org.eclipse.jdt.core is used (see 
> http://www.eclipse.org/jetty/documentation/9.2.20.v20161216/configuring-jsp.html#compiling-jsps).
> However, additional currently jasper libraries are copied to 
> embedded-oozie-server/dependency/ in the distro directory. It can cause 
> problems (e.g. NoSuchMethodException at runtime) if classes are loaded from 
> that jars. 
> Furthermore, transitive dependencies can pull in old jetty libraries (such as 
> jetty-util 6) that can cause runtime issues (at server startup).
> We can make sure with the maven assembly plugin that only jetty.version 
> related dependencies are copied to distro.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)