[jira] [Commented] (OOZIE-2795) Create lib directory or symlink for Oozie CLI during packaging

2017-04-24 Thread Abhishek Bafna (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981361#comment-15981361
 ] 

Abhishek Bafna commented on OOZIE-2795:
---

Few Points:
1. The {{/lib}} does not have the additional jars from {{/libext}} added by the 
users. The {{prepare_jetty}} executes after the jars from WEB-INF are already 
got copied.
2. line: 240 Have incorrect path into the {{ls $JETTY_LIB_DIR/$jarPath}} 
command. It should be just {{$jarPath}}.
3. We should not exit in case of a jar or library already copied. There could 
be other jars in the array which are newly added by the user into {{libext}}. 
line: 234, 245

Thanks.

> Create lib directory or symlink for Oozie CLI during packaging
> --
>
> Key: OOZIE-2795
> URL: https://issues.apache.org/jira/browse/OOZIE-2795
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
> Attachments: OOZIE-2795-001.patch, OOZIE-2795-002.patch, 
> OOZIE-2795-003.patch
>
>
> After creating an Oozie distro, lib directory is not present in the distro 
> directory (e.g. oozie-4.4.0-SNAPSHOT) in the generated tar.gz .
>  
> As a consequence, first run of theOozie CLI will fail if one tries to execute 
> it prior to server start:
> {noformat}
> $ bin/oozie
> unzip:  cannot find or open 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war,
>  
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.zip
>  or 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.ZIP.
> mv: rename 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib/*.jar
>  to 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/*.jar:
>  No such file or directory
> rmdir: 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib:
>  No such file or directory
> rmdir: 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF:
>  No such file or directory
> Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
> {noformat}
> If you start the server first however, a symlink is created (see 
> {{bin/oozie-jetty-server.sh}}), and this error will not be visible.
> Earlier {{bin/oozie}} created this library by extracting it from the WAR file 
> used for Tomcat. Comment in the script:
> {noformat}
> #Create lib directory from war if lib doesn't exist 
> {noformat}
> We could create a symlink to embedded-oozie-server/webapp/WEB-INF/lib/ or 
> copy that directory. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient

2017-04-24 Thread Abhishek Bafna (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Bafna updated OOZIE-2751:
--
Attachment: OOZIE-2751-08.patch

[~pbacsko] Addressing your review comments. Thanks.

> LocalOozieClient is missing methods from OozieClient
> 
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
>  Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch, 
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch, 
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch, 
> OOZIE-2751-08.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the 
> normal {{OozieClient}} methods instead of making REST calls to an Oozie 
> Server.  However, it currently doesn't override all methods, which means you 
> hit the {{OozieClient}} implementation, which tries to make a REST call and 
> will fail (because it's not setup for that).  For example, {{getStatus(String 
> jobId)}} isn't overridden, and doesn't work when used from 
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 54383: LocalOozieClient is missing methods from OozieClient

2017-04-24 Thread Abhishek Bafna 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54383/
---

(Updated April 24, 2017, 1:22 p.m.)


Review request for oozie.


Bugs: OOZIE-2751
https://issues.apache.org/jira/browse/OOZIE-2751


Repository: oozie-git


Description
---

LocalOozieClient is missing methods from OozieClient


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/OozieClient.java 7370808 
  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java f477531 
  core/src/main/java/org/apache/oozie/BaseEngine.java 2780ec2 
  core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/LocalOozieClient.java f734f76 
  core/src/main/java/org/apache/oozie/LocalOozieClientBundle.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/LocalOozieClientCoord.java 32b0cd0 
  core/src/main/java/org/apache/oozie/OozieClientOperationHandler.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/OozieJsonFactory.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 2c79ef0 
  core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 80c8ec4 
  core/src/test/java/org/apache/oozie/TestLocalOozieClientCoord.java 4decd52 


Diff: https://reviews.apache.org/r/54383/diff/7/

Changes: https://reviews.apache.org/r/54383/diff/6-7/


Testing
---


Thanks,

Abhishek Bafna

Re: Review Request 54383: LocalOozieClient is missing methods from OozieClient

2017-04-24 Thread Abhishek Bafna 


> On March 24, 2017, 12:24 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java
> > Lines 457 (patched)
> > 
> >
> > Extract cases to constants.
> > 
> > Optional: if it's not too much work, pls also take care of replacing 
> > the other occurrences.

The patch is already big, so it would be better to not touch other 
functionality.


> On March 24, 2017, 12:24 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/LocalOozieClientBundle.java
> > Lines 79 (patched)
> > 
> >
> > Non-typesafe cast. Do we even need to cast here at all?

This is because of super-type vs sub-type. Same as other places.


- Abhishek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54383/#review16
---


On Jan. 5, 2017, 5:34 p.m., Abhishek Bafna wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54383/
> ---
> 
> (Updated Jan. 5, 2017, 5:34 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2751
> https://issues.apache.org/jira/browse/OOZIE-2751
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> LocalOozieClient is missing methods from OozieClient
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 12c80cb 
>   core/src/main/java/org/apache/oozie/BaseEngine.java 50df897 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/LocalOozieClient.java f734f76 
>   core/src/main/java/org/apache/oozie/LocalOozieClientBundle.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/LocalOozieClientCoord.java 32b0cd0 
>   core/src/main/java/org/apache/oozie/OozieJsonFactory.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 2c79ef0 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 80c8ec4 
>   core/src/test/java/org/apache/oozie/TestLocalOozieClientCoord.java 4decd52 
> 
> 
> Diff: https://reviews.apache.org/r/54383/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Abhishek Bafna
> 
>

[jira] [Updated] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2017-04-24 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2829:
--
Attachment: OOZIE-2829-02.patch

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2017-04-24 Thread Attila Sasvari (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981110#comment-15981110
 ] 

Attila Sasvari commented on OOZIE-2829:
---

[~gezapeti] thanks for the review comments. 
1. fixed
2. fixed - also added more tests to test invalid inputs. I also extracted two 
existing integration tests to a new test class. I wanted to use JUnit 4 
annotations to check exception is thrown when the input is invalid.  
3. OK
4. I adjusted help messages of CLI command  to include more details (also 
increased width in CLIParser): 

{noformat}
usage: 
  
  OozieSharelibCLI creates or upgrade sharelib for oozie

  oozie-setup.sh create  : create a new timestamped version of 
oozie sharelib
-additional_libsAdditional sharelib 
resources. This option requires a colon-separated list of pairs of
 sharelibname and 
coma-sperated list of pathnames in the following format:
 
"sharelib_name=pathname[,pathname...][,sharelib_name2=pathname[,pathname...]...]"
 Caveats:
 * Each pathname is either 
a directory or a regular file (compressed files are not
 extracted prior to the 
upload operation).
 * Sharelibnames shall be 
specified only once.
 * Do not upload multiple 
conflicting library versions for an additional sharelib
 directory as it may cause 
runtime issues.
 Example command:
 $ oozie-setup.sh sharelib 
create -fs hdfs://localhost:9000 -locallib \
 oozie-sharelib.tar.gz 
-additional_libs share2=dir2,file2:share3=file3
-concurrencyNumber of threads to be 
used for copy operations. (default=1)
-fs URI of the fileSystem to 
create oozie share library
-locallib   Local share library 
directory
{noformat}

5. I adjusted help messages of CLI command (see previous)

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2829-01.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2795) Create lib directory or symlink for Oozie CLI during packaging

2017-04-24 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980956#comment-15980956
 ] 

Peter Cseh commented on OOZIE-2795:
---

[~abhishekbafna], are you suggesting to create the symlink in every script if 
it's not there instead of doing in build-time?

> Create lib directory or symlink for Oozie CLI during packaging
> --
>
> Key: OOZIE-2795
> URL: https://issues.apache.org/jira/browse/OOZIE-2795
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Peter Cseh
> Attachments: OOZIE-2795-001.patch, OOZIE-2795-002.patch, 
> OOZIE-2795-003.patch
>
>
> After creating an Oozie distro, lib directory is not present in the distro 
> directory (e.g. oozie-4.4.0-SNAPSHOT) in the generated tar.gz .
>  
> As a consequence, first run of theOozie CLI will fail if one tries to execute 
> it prior to server start:
> {noformat}
> $ bin/oozie
> unzip:  cannot find or open 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war,
>  
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.zip
>  or 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/oozie.war.ZIP.
> mv: rename 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib/*.jar
>  to 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/*.jar:
>  No such file or directory
> rmdir: 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF/lib:
>  No such file or directory
> rmdir: 
> /Users/asasvari/workspace/apache/oozie/distro/target/oozie-4.4.0-SNAPSHOT/lib/WEB-INF:
>  No such file or directory
> Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
> {noformat}
> If you start the server first however, a symlink is created (see 
> {{bin/oozie-jetty-server.sh}}), and this error will not be visible.
> Earlier {{bin/oozie}} created this library by extracting it from the WAR file 
> used for Tomcat. Comment in the script:
> {noformat}
> #Create lib directory from war if lib doesn't exist 
> {noformat}
> We could create a symlink to embedded-oozie-server/webapp/WEB-INF/lib/ or 
> copy that directory. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] Subscription: Oozie Patch Available

2017-04-24 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (92 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2849  Fix build warnings when assembling directories
https://issues.apache.org/jira/browse/OOZIE-2849
OOZIE-2848  Override sharelib.tgz in distro when rebuilding Oozie
https://issues.apache.org/jira/browse/OOZIE-2848
OOZIE-2846  TestSSLServerConnectorFactory fails with Jetty 9.3 and 9.4
https://issues.apache.org/jira/browse/OOZIE-2846
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2825  Custom Authentication doc page is not well formatted
https://issues.apache.org/jira/browse/OOZIE-2825
OOZIE-2824  Fix typos in documentation
https://issues.apache.org/jira/browse/OOZIE-2824
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2796  oozie.action.keep.action.dir not getting noticed
https://issues.apache.org/jira/browse/OOZIE-2796
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2651  Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML 
handling faster
https://issues.apache.org/jira/browse/OOZIE-2651
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2644  Skip queuing Notification Commands when there's nothing to notify
https://issues.apache.org/jira/browse/OOZIE-2644
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2629  Get OozieShareLibCLI to perform a final rename to destPath when