[jira] [Commented] (OOZIE-2457) Oozie log parsing regex consume more than 90% cpu

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948245#comment-15948245
 ] 

Robert Kanter commented on OOZIE-2457:
--

[~satishsaley], sorry for taking so long to do another review.  It mostly looks 
good to me.  I've left a few trivial things on ReviewBoard.  

I was also wondering if you think we need any additional unit tests for the 
changes?  I know we have a bunch of existing tests already, and them passing is 
a good sign.

> Oozie log parsing regex consume more than 90% cpu
> -
>
> Key: OOZIE-2457
> URL: https://issues.apache.org/jira/browse/OOZIE-2457
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2457-1.patch, OOZIE-2457-2.patch, 
> OOZIE-2457-3.patch, OOZIE-2457-4.patch, OOZIE-2457-5.patch, OOZIE-2457-6.patch
>
>
> http-0.0.0.0-4080-26  TID=62215  STATE=RUNNABLE  CPU_TIME=1992 (92.59%)  
> USER_TIME=1990 (92.46%) Allocted: 269156584
> java.util.regex.Pattern$Curly.match0(Pattern.java:4170)
> java.util.regex.Pattern$Curly.match(Pattern.java:4132)
> java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
> java.util.regex.Matcher.match(Matcher.java:1221)
> java.util.regex.Matcher.matches(Matcher.java:559)
> org.apache.oozie.util.XLogFilter.matches(XLogFilter.java:136)
> 
> org.apache.oozie.util.TimestampedMessageParser.parseNextLine(TimestampedMessageParser.java:145)
> 
> org.apache.oozie.util.TimestampedMessageParser.increment(TimestampedMessageParser.java:92)
> Regex 
> {code}
> (.* USER\[[^\]]*\] GROUP\[[^\]]*\] TOKEN\[[^\]]*\] APP\[[^\]]*\] 
> JOB\[000-150625114739728-oozie-puru-W\] ACTION\[[^\]]*\] .*)
> {code}
> For single line parsing we use two regex.
> 1. 
> {code}
> public ArrayList splitLogMessage(String logLine) {
> Matcher splitter = SPLITTER_PATTERN.matcher(logLine);
> if (splitter.matches()) {
> ArrayList logParts = new ArrayList();
> logParts.add(splitter.group(1));// timestamp
> logParts.add(splitter.group(2));// log level
> logParts.add(splitter.group(3));// Log Message
> return logParts;
> }
> else {
> return null;
> }
> }
> {code}
> 2.
> {code}
>  public boolean matches(ArrayList logParts) {
> if (getStartDate() != null) {
> if (logParts.get(0).substring(0, 
> 19).compareTo(getFormattedStartDate()) < 0) {
> return false;
> }
> }
> String logLevel = logParts.get(1);
> String logMessage = logParts.get(2);
> if (this.logLevels == null || 
> this.logLevels.containsKey(logLevel.toUpperCase())) {
> Matcher logMatcher = filterPattern.matcher(logMessage);
> return logMatcher.matches();
> }
> else {
> return false;
> }
> }
> {code}
> Also there is repetitive parsing  for same log message in
> {code}
> private String parseTimestamp(String line) {
> String timestamp = null;
> ArrayList logParts = filter.splitLogMessage(line);
> if (logParts != null) {
> timestamp = logParts.get(0);
> }
> return timestamp;
> }
> {code}
> where the {{line}} has already parsed using regex and we already know the 
> {{logParts}} if any.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 43970: [OOZIE-2457] Oozie log parsing regex consume more than 90% cpu

[Robert Kanter]

> On March 30, 2017, 1:39 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java
> > Lines 194-214 (patched)
> > 
> >
> > This code is almost identical to the code above.  Is there any way to 
> > combine them?  Otherwise, it can be easy for these to diverge accidently in 
> > the future.

Actually, see my other comment about parseNextLine not being needed anymore.  
In that case, there's no duplication.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43970/#review170509
---


On Feb. 27, 2017, 7:42 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43970/
> ---
> 
> (Updated Feb. 27, 2017, 7:42 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/OOZIE-2417
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/OOZIE-2417
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> This patch tries to minimize the parsing of same log line for multiple times 
> using different regex.
> It also caches the log parts once we figure out what those are, thus avoiding 
> the re-parsing of log line to get log parts whenever needed.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/util/LogLine.java PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/SimpleTimestampedMessageParser.java 
> 78cb042 
>   core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java 
> a676f4d 
>   core/src/main/java/org/apache/oozie/util/XLogFilter.java 3b49f77 
> 
> 
> Diff: https://reviews.apache.org/r/43970/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-2387) Oozie is Unable to handle Spaces in file/archive tag.

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948193#comment-15948193
 ] 

Robert Kanter commented on OOZIE-2387:
--

Is this something we should be doing inside of {{addToCache}} instead of before 
it?  It seems like it would be a good idea to do this for everything we're 
passing in given that we turn them all in to {{URI}}.  It would also be good to 
add a simple unit test that uses a space for validation.

> Oozie is Unable to handle Spaces in file/archive tag.
> -
>
> Key: OOZIE-2387
> URL: https://issues.apache.org/jira/browse/OOZIE-2387
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.1.0, 4.2.0
> Environment: Hadoop 2.6,Oozie 4.1, Hortonworks2.2
>Reporter: shiv pratap singh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2387-01.patch
>
>
> Oozie file tag is unable to handle white spaces and other special characters.
> Error - 
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:529)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:678)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
> Sample XML File --
> 
>  xmlns:ns5="uri:oozie:shell-action:0.2"
>   xmlns:ns4="uri:oozie:workflow:0.4" 
> xmlns:ns3="uri:oozie:sqoop-action:0.2"
>   xmlns:ns2="uri:oozie:hive-action:0.3" name="test-shell-space">
>   
>   
>   
>   ${wf_hadoop_resourceManager}
>   ${wf_hadoop_nameNode}
>   shell 1.sh
>   /user/map-dev/test-case-dir/shell/script/shell 
> 1.sh
>   
>   
>   
>   
>   
>   shell-decision job failed
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2585) Disable or remove test case TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and testMemoryUsageAndSpeed

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948182#comment-15948182
 ] 

Robert Kanter commented on OOZIE-2585:
--

I'm fine with getting rid of a flakey performance test, especially because it's 
testing the performance of ehcache and not Oozie itself.  

I'm not too familiar with this code, but it looks like this only applies to 
{{testMemoryUsageAndSpeed}} and {{testMemoryUsageAndSpeedOverflowToDisk}}.  The 
others seem to be checking correct behavior.  If so, I guess we should only 
remove these two.

> Disable or remove test case 
> TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk 
> and testMemoryUsageAndSpeed
> ---
>
> Key: OOZIE-2585
> URL: https://issues.apache.org/jira/browse/OOZIE-2585
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2585-001.patch, OOZIE-2585-002.patch
>
>
> There is a problem with the test case testMemoryUsageAndSpeedOverflowToDisk.
> After 20k entries, data is spilled to the disk. However, Ehcache writes on a 
> separate thread asynchronously and all put operations return immediately. The 
> problem is that after storing the 60k entries, we try to read them back 
> instantly, with many of them are still being flushed to the storage. As long 
> as disk write in progress, these entries are unavailable.
> I didn't find a reliable way to wait for disk writes. Therefore I suggest 
> disabling this test or eliminate it altogether -- it's a performance test 
> that runs on all kinds of machines during a build, I don't think it makes too 
> much sense to keep it.
> Also testMemoryUsageAndSpeed might be prone to test failures too - these two 
> assertions might evaluate to false:
> {code}
> assertTrue((endTime - startTime) < insertTimeinMillis);
> ...
> assertTrue((System.currentTimeMillis() - endTime) < retrievalTimeinMillis);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2849) Fix build warnings when assembling directories

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948177#comment-15948177
 ] 

Robert Kanter commented on OOZIE-2849:
--

I checked the maven output, and it does get rid of all of the "Assembly file" 
warnings.

However, when I listed all of the files recursively with and without the patch 
after building and did a diff, it shows that the oozie script is now missing 
when using the patch:
{noformat}
>> [5] 17:51 : oozie-git (master) :: diff /tmp/old.txt /tmp/new.txt
20d19
< distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/bin/oozie
{noformat}


> Fix build warnings when assembling directories
> --
>
> Key: OOZIE-2849
> URL: https://issues.apache.org/jira/browse/OOZIE-2849
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Trivial
> Attachments: OOZIE-2849-01.patch
>
>
> When building oozie we can observe a lot of warning
> {code}
> [WARNING] Assembly file: 
> /Users/asasvari/workspace/apache/oozie_dup/tools/target/oozie-tools-4.4.0-SNAPSHOT-tools
>  is not a regular file (it may be a directory). It cannot be attached to the 
> project build for installation or deployment.
> {code}
> This behaviour can be explained by 
> https://issues.apache.org/jira/browse/MASSEMBLY-628



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2387) Oozie is Unable to handle Spaces in file/archive tag.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947839#comment-15947839
 ] 

Hadoop QA commented on OOZIE-2387:
--

Testing JIRA OOZIE-2387

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1906
.Tests failed: 0
.Tests errors: 3

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  
testCommaSeparatedFilesAndArchives(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
.  
testCommaSeparatedFilesAndArchives(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.  
testCommaSeparatedFilesAndArchives(org.apache.oozie.action.hadoop.TestPigActionExecutor)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3750/

> Oozie is Unable to handle Spaces in file/archive tag.
> -
>
> Key: OOZIE-2387
> URL: https://issues.apache.org/jira/browse/OOZIE-2387
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.1.0, 4.2.0
> Environment: Hadoop 2.6,Oozie 4.1, Hortonworks2.2
>Reporter: shiv pratap singh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2387-01.patch
>
>
> Oozie file tag is unable to handle white spaces and other special characters.
> Error - 
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:529)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:678)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
> Sample XML File --
> 
>  xmlns:ns5="uri:oozie:shell-action:0.2"
>   xmlns:ns4="uri:oozie:workflow:0.4" 
> xmlns:ns3="uri:oozie:sqoop-action:0.2"
>   xmlns:ns2="uri:oozie:hive-action:0.3" name="test-shell-space">
>   
>   
>   
>   ${wf_hadoop_resourceManager}
>   ${wf_hadoop_nameNode}
>   shell 1.sh
>   /user/map-dev/test-case-dir/shell/script/shell 
> 1.sh
>   
>   
>   
>   
>   
>   

Build failed in Jenkins: oozie-trunk-precommit-build #3750

[Apache Jenkins Server]

See 


--
[...truncated 968.92 KB...]
[INFO] Copying curator-client-2.5.0.jar to 

[INFO] Copying jetty-annotations-9.2.19.v20160908.jar to 

[INFO] Copying jersey-server-1.9.jar to 

[INFO] Copying jackson-core-2.2.2.jar to 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-server ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-distro ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/distro.xml
[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] Building tar : 

[INFO] 
[INFO] 
[INFO] Building Apache Oozie ZooKeeper Security Tests 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ 
oozie-zookeeper-security-tests ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-zookeeper-security-tests ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ 
oozie-zookeeper-security-tests 

Build failed in Jenkins: oozie-trunk-find-patches-available #155025

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision e3590b420a0914d959f6cd555185ef04640c5801 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e3590b420a0914d959f6cd555185ef04640c5801
 > git rev-list e3590b420a0914d959f6cd555185ef04640c5801 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson5857761413842524754.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  151k0  151k0 0   136k  0 --:--:--  0:00:01 --:--:--  
136k100  151k0  151k0 0   136k  0 --:--:--  0:00:01 --:--:--  
136k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

Build failed in Jenkins: oozie-trunk-precommit-build #3749

[Apache Jenkins Server]

See 


Changes:

[gezapeti] OOZIE-2831 Update maven-project-info-reports-plugin to 2.9 (asasvari 
via

--
[...truncated 1.21 MB...]
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-server ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-distro ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/distro.xml
[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] Building tar : 

[INFO] 
[INFO] 
[INFO] Building Apache Oozie ZooKeeper Security Tests 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ 
oozie-zookeeper-security-tests ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-zookeeper-security-tests ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ 
oozie-zookeeper-security-tests ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ 
oozie-zookeeper-security-tests ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ 
oozie-zookeeper-security-tests ---
[INFO] 

[jira] [Commented] (OOZIE-2843) Enhance logging inside ZKLocksService and MemoryLocksService

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947695#comment-15947695
 ] 

Hadoop QA commented on OOZIE-2843:
--

Testing JIRA OOZIE-2843

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1907
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testOozieSharelibCLICreate(org.apache.oozie.tools.TestOozieSharelibCLI)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3749/

> Enhance logging inside ZKLocksService and MemoryLocksService
> 
>
> Key: OOZIE-2843
> URL: https://issues.apache.org/jira/browse/OOZIE-2843
> Project: Oozie
>  Issue Type: New Feature
>  Components: action, coordinator
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Minor
> Attachments: OOZIE-2843.001.patch, OOZIE-2843.002.patch, 
> OOZIE-2843.003.patch
>
>
> Sometimes when Oozie tries to acquire lock [via 
> ZooKeeper|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/ZKLocksService.java#L144]
>  or 
> [in-memory|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/MemoryLocksService.java]
>  we don't get enough information whether the action succeeded or not. Also 
> other details like retry count, timeout applied, etc. are not logged. 
> Enhancing logging here would benefit debugging of lock handling within 
> {{XCommand}} instances.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : oozie-trunk-find-patches-available #155021

[Apache Jenkins Server]

See 

Build failed in Jenkins: oozie-trunk-find-patches-available #155020

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision e3590b420a0914d959f6cd555185ef04640c5801 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e3590b420a0914d959f6cd555185ef04640c5801
 > git rev-list e3590b420a0914d959f6cd555185ef04640c5801 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson3593164965697496275.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
901120 901120 0   103k  0 --:--:-- --:--:-- --:--:--  103k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Commented] (OOZIE-2387) Oozie is Unable to handle Spaces in file/archive tag.

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947620#comment-15947620
 ] 

Attila Sasvari commented on OOZIE-2387:
---

Regarding HDFS filenames: there is an open ticket to restrict filenames: 
https://issues.apache.org/jira/browse/HDFS-240


> Oozie is Unable to handle Spaces in file/archive tag.
> -
>
> Key: OOZIE-2387
> URL: https://issues.apache.org/jira/browse/OOZIE-2387
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.1.0, 4.2.0
> Environment: Hadoop 2.6,Oozie 4.1, Hortonworks2.2
>Reporter: shiv pratap singh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2387-01.patch
>
>
> Oozie file tag is unable to handle white spaces and other special characters.
> Error - 
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:529)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:678)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
> Sample XML File --
> 
>  xmlns:ns5="uri:oozie:shell-action:0.2"
>   xmlns:ns4="uri:oozie:workflow:0.4" 
> xmlns:ns3="uri:oozie:sqoop-action:0.2"
>   xmlns:ns2="uri:oozie:hive-action:0.3" name="test-shell-space">
>   
>   
>   
>   ${wf_hadoop_resourceManager}
>   ${wf_hadoop_nameNode}
>   shell 1.sh
>   /user/map-dev/test-case-dir/shell/script/shell 
> 1.sh
>   
>   
>   
>   
>   
>   shell-decision job failed
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2387) Oozie is Unable to handle Spaces in file/archive tag.

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2387:
--
Attachment: OOZIE-2387-01.patch

> Oozie is Unable to handle Spaces in file/archive tag.
> -
>
> Key: OOZIE-2387
> URL: https://issues.apache.org/jira/browse/OOZIE-2387
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.1.0, 4.2.0
> Environment: Hadoop 2.6,Oozie 4.1, Hortonworks2.2
>Reporter: shiv pratap singh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2387-01.patch
>
>
> Oozie file tag is unable to handle white spaces and other special characters.
> Error - 
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:529)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:678)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
> Sample XML File --
> 
>  xmlns:ns5="uri:oozie:shell-action:0.2"
>   xmlns:ns4="uri:oozie:workflow:0.4" 
> xmlns:ns3="uri:oozie:sqoop-action:0.2"
>   xmlns:ns2="uri:oozie:hive-action:0.3" name="test-shell-space">
>   
>   
>   
>   ${wf_hadoop_resourceManager}
>   ${wf_hadoop_nameNode}
>   shell 1.sh
>   /user/map-dev/test-case-dir/shell/script/shell 
> 1.sh
>   
>   
>   
>   
>   
>   shell-decision job failed
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2457) Oozie log parsing regex consume more than 90% cpu

[Jan Filipiak (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947614#comment-15947614
 ] 

Jan Filipiak commented on OOZIE-2457:
-

Hi, are there any plans to change the Logging implementation entirely? Maybe 
keep a log per Bundle/Coord/Job? would a patch be welcome to maybe store those 
logs in hdfs with configurable retention times?

Don't get me wrong but I just ran into this issue today and couldn't really 
believe it.

> Oozie log parsing regex consume more than 90% cpu
> -
>
> Key: OOZIE-2457
> URL: https://issues.apache.org/jira/browse/OOZIE-2457
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2457-1.patch, OOZIE-2457-2.patch, 
> OOZIE-2457-3.patch, OOZIE-2457-4.patch, OOZIE-2457-5.patch, OOZIE-2457-6.patch
>
>
> http-0.0.0.0-4080-26  TID=62215  STATE=RUNNABLE  CPU_TIME=1992 (92.59%)  
> USER_TIME=1990 (92.46%) Allocted: 269156584
> java.util.regex.Pattern$Curly.match0(Pattern.java:4170)
> java.util.regex.Pattern$Curly.match(Pattern.java:4132)
> java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
> java.util.regex.Matcher.match(Matcher.java:1221)
> java.util.regex.Matcher.matches(Matcher.java:559)
> org.apache.oozie.util.XLogFilter.matches(XLogFilter.java:136)
> 
> org.apache.oozie.util.TimestampedMessageParser.parseNextLine(TimestampedMessageParser.java:145)
> 
> org.apache.oozie.util.TimestampedMessageParser.increment(TimestampedMessageParser.java:92)
> Regex 
> {code}
> (.* USER\[[^\]]*\] GROUP\[[^\]]*\] TOKEN\[[^\]]*\] APP\[[^\]]*\] 
> JOB\[000-150625114739728-oozie-puru-W\] ACTION\[[^\]]*\] .*)
> {code}
> For single line parsing we use two regex.
> 1. 
> {code}
> public ArrayList splitLogMessage(String logLine) {
> Matcher splitter = SPLITTER_PATTERN.matcher(logLine);
> if (splitter.matches()) {
> ArrayList logParts = new ArrayList();
> logParts.add(splitter.group(1));// timestamp
> logParts.add(splitter.group(2));// log level
> logParts.add(splitter.group(3));// Log Message
> return logParts;
> }
> else {
> return null;
> }
> }
> {code}
> 2.
> {code}
>  public boolean matches(ArrayList logParts) {
> if (getStartDate() != null) {
> if (logParts.get(0).substring(0, 
> 19).compareTo(getFormattedStartDate()) < 0) {
> return false;
> }
> }
> String logLevel = logParts.get(1);
> String logMessage = logParts.get(2);
> if (this.logLevels == null || 
> this.logLevels.containsKey(logLevel.toUpperCase())) {
> Matcher logMatcher = filterPattern.matcher(logMessage);
> return logMatcher.matches();
> }
> else {
> return false;
> }
> }
> {code}
> Also there is repetitive parsing  for same log message in
> {code}
> private String parseTimestamp(String line) {
> String timestamp = null;
> ArrayList logParts = filter.splitLogMessage(line);
> if (logParts != null) {
> timestamp = logParts.get(0);
> }
> return timestamp;
> }
> {code}
> where the {{line}} has already parsed using regex and we already know the 
> {{logParts}} if any.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2387) Oozie is Unable to handle Spaces in file/archive tag.

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947560#comment-15947560
 ] 

Attila Sasvari commented on OOZIE-2387:
---

It is the {{new URI(filepath)}} call that results in an exception in 
{{addToCache()}} in {{JavaActionExecutor}}.



> Oozie is Unable to handle Spaces in file/archive tag.
> -
>
> Key: OOZIE-2387
> URL: https://issues.apache.org/jira/browse/OOZIE-2387
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.1.0, 4.2.0
> Environment: Hadoop 2.6,Oozie 4.1, Hortonworks2.2
>Reporter: shiv pratap singh
>Assignee: Attila Sasvari
>
> Oozie file tag is unable to handle white spaces and other special characters.
> Error - 
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:529)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:678)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
> Sample XML File --
> 
>  xmlns:ns5="uri:oozie:shell-action:0.2"
>   xmlns:ns4="uri:oozie:workflow:0.4" 
> xmlns:ns3="uri:oozie:sqoop-action:0.2"
>   xmlns:ns2="uri:oozie:hive-action:0.3" name="test-shell-space">
>   
>   
>   
>   ${wf_hadoop_resourceManager}
>   ${wf_hadoop_nameNode}
>   shell 1.sh
>   /user/map-dev/test-case-dir/shell/script/shell 
> 1.sh
>   
>   
>   
>   
>   
>   shell-decision job failed
>   
>   
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2848) Override sharelib.tgz in distro when rebuilding Oozie

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947444#comment-15947444
 ] 

Hadoop QA commented on OOZIE-2848:
--

Testing JIRA OOZIE-2848

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1906
.Tests rerun: 2
.Tests failed at first run: org.apache.oozie.example.TestLocalOozieExample,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3748/

> Override sharelib.tgz in distro when rebuilding Oozie
> -
>
> Key: OOZIE-2848
> URL: https://issues.apache.org/jira/browse/OOZIE-2848
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Trivial
> Attachments: OOZIE-2848-01.patch
>
>
> Building Oozie with {{mvn package assembly:single -Dfindbugs.skip=true 
> -DskipTests -DjavaVersion=1.8 -DtargetVersion=1.7 -Puber}} twice in a row 
> results in the following error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single (default-cli) on 
> project oozie-distro: Failed to create assembly: Error creating assembly 
> archive distro: Problem copying files : 
> /Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/oozie-sharelib-4.4.0-SNAPSHOT.tar.gz
>  (Permission denied) -> [Help 1]
> {code}
> We can override the target like it is done for Oozie Client and TAR.GZ-s.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2841) Limit FindBugs diff errors in JIRA comments

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947230#comment-15947230
 ] 

Peter Cseh commented on OOZIE-2841:
---

Thanks for fixing it [~abhishekbafna]!

> Limit FindBugs diff errors in JIRA comments
> ---
>
> Key: OOZIE-2841
> URL: https://issues.apache.org/jira/browse/OOZIE-2841
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Attachments: OOZIE-2841.001.patch, OOZIE-2841.002.patch, 
> OOZIE-2841.amend.patch, OOZIE-2841.wrong.patch, 
> test-patch-reports.after.tar.gz, test-patch-reports.before.tar.gz
>
>
> In OOZIE-1986 the FindBugs diff functionality was introduced. At times, 
> however, there are cases when it's painful that we print out each and every 
> FindBugs error (the ones considered mandatory to be fixed) [*to the JIRA 
> comment*|https://issues.apache.org/jira/browse/OOZIE-2812?focusedCommentId=15937447=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937447].
> We'll limit the FindBugs errors to be displayed within a module (let's say, 
> top 5 from a module). This limitation will only be valid for 
> {{TEST-SUMMARY.jira}} and {{TEST-SUMMARY.txt}} - for 
> {{TEST-SUMMARY-FULL.jira}} and {{TEST-SUMMARY-FULL.txt}} the output remains 
> unchanged.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2843) Enhance logging inside ZKLocksService and MemoryLocksService

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2843:

Attachment: OOZIE-2843.003.patch

Fixing previous FindBugs diff.

> Enhance logging inside ZKLocksService and MemoryLocksService
> 
>
> Key: OOZIE-2843
> URL: https://issues.apache.org/jira/browse/OOZIE-2843
> Project: Oozie
>  Issue Type: New Feature
>  Components: action, coordinator
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Minor
> Attachments: OOZIE-2843.001.patch, OOZIE-2843.002.patch, 
> OOZIE-2843.003.patch
>
>
> Sometimes when Oozie tries to acquire lock [via 
> ZooKeeper|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/ZKLocksService.java#L144]
>  or 
> [in-memory|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/MemoryLocksService.java]
>  we don't get enough information whether the action succeeded or not. Also 
> other details like retry count, timeout applied, etc. are not logged. 
> Enhancing logging here would benefit debugging of lock handling within 
> {{XCommand}} instances.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2841) Limit FindBugs diff errors in JIRA comments

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947204#comment-15947204
 ] 

Abhishek Bafna commented on OOZIE-2841:
---

[~gezapeti] I have pushed the contributor change into release-log, in the this 
commit: 
https://github.com/apache/oozie/commit/7c404ad0ea4c61e90e8c86015de25ef196168c29

Thanks. 

> Limit FindBugs diff errors in JIRA comments
> ---
>
> Key: OOZIE-2841
> URL: https://issues.apache.org/jira/browse/OOZIE-2841
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Attachments: OOZIE-2841.001.patch, OOZIE-2841.002.patch, 
> OOZIE-2841.amend.patch, OOZIE-2841.wrong.patch, 
> test-patch-reports.after.tar.gz, test-patch-reports.before.tar.gz
>
>
> In OOZIE-1986 the FindBugs diff functionality was introduced. At times, 
> however, there are cases when it's painful that we print out each and every 
> FindBugs error (the ones considered mandatory to be fixed) [*to the JIRA 
> comment*|https://issues.apache.org/jira/browse/OOZIE-2812?focusedCommentId=15937447=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937447].
> We'll limit the FindBugs errors to be displayed within a module (let's say, 
> top 5 from a module). This limitation will only be valid for 
> {{TEST-SUMMARY.jira}} and {{TEST-SUMMARY.txt}} - for 
> {{TEST-SUMMARY-FULL.jira}} and {{TEST-SUMMARY-FULL.txt}} the output remains 
> unchanged.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2848) Override sharelib.tgz in distro when rebuilding Oozie

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947205#comment-15947205
 ] 

Peter Cseh commented on OOZIE-2848:
---

I've retriggered the jenkins for this one.

> Override sharelib.tgz in distro when rebuilding Oozie
> -
>
> Key: OOZIE-2848
> URL: https://issues.apache.org/jira/browse/OOZIE-2848
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Trivial
> Attachments: OOZIE-2848-01.patch
>
>
> Building Oozie with {{mvn package assembly:single -Dfindbugs.skip=true 
> -DskipTests -DjavaVersion=1.8 -DtargetVersion=1.7 -Puber}} twice in a row 
> results in the following error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single (default-cli) on 
> project oozie-distro: Failed to create assembly: Error creating assembly 
> archive distro: Problem copying files : 
> /Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/oozie-sharelib-4.4.0-SNAPSHOT.tar.gz
>  (Permission denied) -> [Help 1]
> {code}
> We can override the target like it is done for Oozie Client and TAR.GZ-s.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947154#comment-15947154
 ] 

Abhishek Bafna commented on OOZIE-2845:
---

[~pbacsko] I think 
{{HiveConf.setHiveSiteLocation(null);}}  
should be  
{{HiveConf.setHiveSiteLocation(HiveConf.class.getClassLoader().getResource("hive-site.xml"));}}

Because, we set the {{hiveSiteURL}} to the available {{hive-site.xml}} in the 
classpath.

Did we test this, on actual cluster? If not, we should I guess.

Thanks.

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create 
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null, 
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time. 
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2831) Update maven-project-info-reports-plugin to 2.9

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947193#comment-15947193
 ] 

Peter Cseh commented on OOZIE-2831:
---

+1 
Committed to master.
Thanks [~asasvari]!

> Update maven-project-info-reports-plugin to 2.9
> ---
>
> Key: OOZIE-2831
> URL: https://issues.apache.org/jira/browse/OOZIE-2831
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Trivial
> Fix For: 5.0.0
>
> Attachments: OOZIE-2831-01.patch
>
>
> Currently Oozie uses an old maven-project-info-reports-plugin (2.3). 
> Unfortunately it cannot handle Java 8 classes properly (see 
> https://issues.apache.org/jira/browse/MPIR-335).
> As a result,  org.apache.bcel.classfile.ClassFormatException are thrown when 
> using JDK8.
> {code}
> org.apache.bcel.classfile.ClassFormatException: Invalid byte tag in constant 
> pool: 18
>   at org.apache.bcel.classfile.Constant.readConstant(Constant.java:146)
>   at org.apache.bcel.classfile.ConstantPool.(ConstantPool.java:67)
>   at 
> org.apache.bcel.classfile.ClassParser.readConstantPool(ClassParser.java:222)
>   at org.apache.bcel.classfile.ClassParser.parse(ClassParser.java:136)
>   at 
> org.apache.maven.shared.jar.classes.JarClassesAnalysis.analyze(JarClassesAnalysis.java:92)
>   at 
> org.apache.maven.report.projectinfo.dependencies.Dependencies.getJarDependencyDetails(Dependencies.java:276)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.hasSealed(DependenciesRenderer.java:1374)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderSectionDependencyFileDetails(DependenciesRenderer.java:544)
>   at 
> org.apache.maven.report.projectinfo.dependencies.renderer.DependenciesRenderer.renderBody(DependenciesRenderer.java:271)
>   at 
> org.apache.maven.reporting.AbstractMavenReportRenderer.render(AbstractMavenReportRenderer.java:79)
>   at 
> org.apache.maven.report.projectinfo.DependenciesReport.executeReport(DependenciesReport.java:206)
>   at 
> org.apache.maven.reporting.AbstractMavenReport.generate(AbstractMavenReport.java:190)
>   at 
> org.apache.maven.report.projectinfo.AbstractProjectInfoReport.execute(AbstractProjectInfoReport.java:202)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>   at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>   at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> {code}
> Updating to 2.9 solves this problem.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947168#comment-15947168
 ] 

Peter Bacsko commented on OOZIE-2845:
-

Bah, that's right. I misread the original implementation. I though "null" 
refers to the value itself, but no, that's just the object instance.

I'll update the patch soon.

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create 
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null, 
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time. 
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2818) Can't overwrite oozie.action.max.output.data on a per-workflow basis

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946957#comment-15946957
 ] 

Hadoop QA commented on OOZIE-2818:
--

Testing JIRA OOZIE-2818

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1908
.Tests rerun: 46
.Tests failed at first run: 
org.apache.oozie.service.TestStatusTransitService,org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand,org.apache.oozie.util.TestMetricsInstrumentation,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3747/

> Can't overwrite oozie.action.max.output.data on a per-workflow basis
> 
>
> Key: OOZIE-2818
> URL: https://issues.apache.org/jira/browse/OOZIE-2818
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2818-01.patch
>
>
> It's not possible to raise the limit of the output on a per-action basis. 
> With the workflow:
> {quote}
> 
> 
> 
> Action failed, error 
> message\[$\{wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> $\{jobTracker}
> $\{nameNode}
> 
> 
> oozie.action.max.output.data
> 10
> 
> 
> oozie.launcher.oozie.action.max.output.data
> 10
> 
> 
> /user/admin/large_output.sh
>   10
> /user/admin/large_output.sh#large_output.sh
>   
> 
> 
> 
> 
> 
> 
> {quote}
> In the localized files in the job.xml the value of this property gets 
> overwritten:
> {quote}
> # cat 
> /yarn/nm/usercache/admin/appcache/application_1489145190857_0001/container_1489145190857_0001_01_02/job.xml
>  |grep max.outp
> oozie.launcher.oozie.action.max.output.data10programaticallyjob.xml
> oozie.action.max.output.data2048programaticallyjob.xml
> # cat 
> /yarn/nm/usercache/admin/appcache/application_1489145190857_0001/container_1489145190857_0001_01_02/action.xml
>  |grep max.outp
> oozie.launcher.oozie.action.max.output.data10programatically

Jenkins build is back to normal : oozie-trunk-precommit-build #3747

[Apache Jenkins Server]

See 

[jira] [Commented] (OOZIE-2794) Hive and Pig tests failing when multiple pre-commit jobs are running

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946922#comment-15946922
 ] 

Andras Piros commented on OOZIE-2794:
-

It's also suspected that {{bin/test-patch-11-findbugs-diff}} may also fail 
sometimes because multiple pre-commit jobs are running on the same Jenkins 
slave at the same time:
{noformat}
  Running test-patch task FINDBUGS_DIFF
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
/home/jenkins/jenkins-slave/workspace/oozie-trunk-precommit-build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
bin/test-patch-11-findbugs-diff: line 253: : No such file or directory
bin/test-patch-11-findbugs-diff: line 255: [: -le: unary operator expected
Exception in thread "main" groovy.lang.GroovyRuntimeException: Ambiguous method 
overloading for method java.io.File#.
Cannot resolve which method to invoke for [null] due to overlapping prototypes 
between:
[class java.lang.String]
[class java.net.URI]
at 
groovy.lang.MetaClassImpl.chooseMostSpecificParams(MetaClassImpl.java:3202)
...
at 
me.andrz.findbugs.diff.FindBugsDiffCli.main(FindBugsDiffCli.groovy:51)
{noformat}

> Hive and Pig tests failing when multiple pre-commit jobs are running
> 
>
> Key: OOZIE-2794
> URL: https://issues.apache.org/jira/browse/OOZIE-2794
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>
> There are multiple cases when the following tests are failing together:
> {noformat}
> . testMain(org.apache.oozie.action.hadoop.TestHiveMain)
> . testPigScript(org.apache.oozie.action.hadoop.TestPigMain)
> . testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
> . testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
> . testPigScript(org.apache.oozie.action.hadoop.TestPigMainWithOldAPI)
> . testLoadDump(org.apache.oozie.tools.TestDBLoadDump)
> {noformat}
> This issue seems to occur when multiple pre-commit jobs are running. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2729) Use MiniMRYARNCluster in tests

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946894#comment-15946894
 ] 

Andras Piros commented on OOZIE-2729:
-

Thanks [~rkanter] keeping an eye on that!

> Use MiniMRYARNCluster in tests
> --
>
> Key: OOZIE-2729
> URL: https://issues.apache.org/jira/browse/OOZIE-2729
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2729.002.patch
>
>
> {{MiniMRCluster}} is deprecated, we should replace it using 
> {{MiniYARNCluster}}.
> Also {{MiniDFSCluster}} should be created via builder.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946819#comment-15946819
 ] 

Peter Bacsko commented on OOZIE-2847:
-

In general the patch looks good, some thoughts:

* please import stuff explicitly, not {{import org.apache.hadoop.fs.*;}}
* use a hard-coded string for the recoveryID (not UUID)
* make the test name more explicit: 
{{testRecoveryWritesJobIdWhenRecoveryFileIsEmpty}}

I'll take a slightly deeper look later.

> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   

[jira] [Commented] (OOZIE-2847) Oozie Ha timing issue

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946811#comment-15946811
 ] 

Peter Bacsko commented on OOZIE-2847:
-

I think the problem description is legit, but the stack trace demonstrates a 
different issue. In this case the MR AppMaster tries to recover but it cannot 
properly parse the jhist file written by the previous (crashed/killed) app 
master.



> Oozie Ha timing issue
> -
>
> Key: OOZIE-2847
> URL: https://issues.apache.org/jira/browse/OOZIE-2847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Priority: Minor
> Attachments: OOZIE-2847.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Oozie Ha timing issue
> When Oozie is launching the mapper, it is writing a job id into a file on 
> hdfs. Let's assume the ApplicationMaster is killed, and Oozie will make a 
> second try, during recovery. On the second try, Oozie is trying to see if the 
> previously written job id on hdfs matches the current job id. In most 
> occasion, this will match. However, in the event when Oozie launcher is 
> killed right in the middle when Oozie is in the process of writing id in the 
> file, the Oozie file in hdfs is created, but the id has yet to be written to 
> the file. During the next recovery, Oozie will mistakenly think the id exists 
> in the file while the file is actually empty, therefore throwing this 
> exception: 
> {noformat}
> 2015-07-10 
> 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|
> 2015-07-10 05:56:58,137|beaver.machine|INFO|5208|1344|MainThread|Console URL  
>  : http://dal-ha21:8088/proxy/application_1436507526035_0001/
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error Code   
>  : JA018
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Error 
> Message : Hadoop job Id mismatch, action file 
> [hdfs://hdp2-ha2/user/hadoopqa/oozie-hado/003-150710041341636-oozie-hado-W/pig-node--pig/003-150710041341636-oozie-hado-W@pig-node@0]
>  declares Id [null] current Id [job_1436507526035_0001]
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External ID  
>  : job_1436507526035_0001
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|External 
> Status   : FAILED/KILLED
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Name 
>  : pig-node
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Retries  
>  : 0
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Tracker URI  
>  : dal-ha21:8032
> 2015-07-10 05:56:58,138|beaver.machine|INFO|5208|1344|MainThread|Type 
>  : pig
> 2015-07-10 05:56:58,158|beaver.machine|INFO|5208|1344|MainThread|Started  
>  : 2015-07-10 05:55:19 GMT
> 2015-07-10 05:56:58,160|beaver.machine|INFO|5208|1344|MainThread|Status   
>  : ERROR
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|Ended
>  : 2015-07-10 05:56:42 GMT
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> Stats: null
> 2015-07-10 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|External 
> ChildIDs : null
> 2015-07-10 
> 05:56:58,161|beaver.machine|INFO|5208|1344|MainThread|
> Exception:
> 2015-07-10 05:56:18,658 INFO [main] 
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file 
> system [hdfs://hdp2-ha2:8020]
> 2015-07-10 05:56:18,665 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at 
> hdfs://hdp2-ha2:8020/user/hadoopqa/.staging/job_1436507526035_0001/job_1436507526035_0001_1.jhist
> 2015-07-10 05:56:18,693 WARN [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job 
> history, aborting recovery
> java.io.IOException: Incompatible event log version: null
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:71)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1206)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1175)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1039)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> 

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946800#comment-15946800
 ] 

Peter Bacsko commented on OOZIE-2845:
-

[~abhishekbafna] [~gezapeti] please verify the patch.

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create 
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null, 
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time. 
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946797#comment-15946797
 ] 

Peter Bacsko commented on OOZIE-2845:
-

{{-1 TESTS - patch does not compile, cannot run testcases}}

Looking at the build, this is not the case, everything is passed, but there 
were problems during findbugs diff:

{code}
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
/home/jenkins/jenkins-slave/workspace/oozie-trunk-precommit-build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
bin/test-patch-11-findbugs-diff: line 253: : No such file or directory
bin/test-patch-11-findbugs-diff: line 255: [: -le: unary operator expected
Exception in thread "main" groovy.lang.GroovyRuntimeException: Ambiguous method 
overloading for method java.io.File#.
Cannot resolve which method to invoke for [null] due to overlapping prototypes 
between:
[class java.lang.String]
[class java.net.URI]
at 
groovy.lang.MetaClassImpl.chooseMostSpecificParams(MetaClassImpl.java:3202)
at 
groovy.lang.MetaClassImpl.chooseMethodInternal(MetaClassImpl.java:3154)
at groovy.lang.MetaClassImpl.chooseMethod(MetaClassImpl.java:3097)
at 
groovy.lang.MetaClassImpl.createConstructorSite(MetaClassImpl.java:3384)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallConstructorSite(CallSiteArray.java:88)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at 
me.andrz.findbugs.diff.FindBugsDiff.diffFromFilePaths(FindBugsDiff.groovy:33)
at me.andrz.findbugs.diff.FindBugsDiff$diffFromFilePaths.call(Unknown 
Source)
...
{code}

[~andras.piros] please take a look at this.

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes location of hive-site.xml in static block.
> // So this is needed so that tests like TestHiveMain that create 
> hive-site.xml don't fail.
> Field declaredField = HiveConf.class.getDeclaredField("hiveSiteURL");
> declaredField.setAccessible(true);
> declaredField.set(null, 
> HiveConf.class.getClassLoader().getResource("hive-site.xml"));
> }
> {code}
> Note that HiveConf has supported setting {{hiveSiteURL}} for a long time. 
> Even version 0.13.0 has the static setter method:
> https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1493-L1495
> Therefore reflection-based solution should be modified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (OOZIE-2845) Replace reflection-based code which sets variable in HiveConf

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946797#comment-15946797
 ] 

Peter Bacsko edited comment on OOZIE-2845 at 3/29/17 9:24 AM:
--

{{-1 TESTS - patch does not compile, cannot run testcases}}

Looking at the build, this is not the case, everything passed, but there were 
problems during findbugs diff:

{code}
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
/home/jenkins/jenkins-slave/workspace/oozie-trunk-precommit-build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
bin/test-patch-11-findbugs-diff: line 253: : No such file or directory
bin/test-patch-11-findbugs-diff: line 255: [: -le: unary operator expected
Exception in thread "main" groovy.lang.GroovyRuntimeException: Ambiguous method 
overloading for method java.io.File#.
Cannot resolve which method to invoke for [null] due to overlapping prototypes 
between:
[class java.lang.String]
[class java.net.URI]
at 
groovy.lang.MetaClassImpl.chooseMostSpecificParams(MetaClassImpl.java:3202)
at 
groovy.lang.MetaClassImpl.chooseMethodInternal(MetaClassImpl.java:3154)
at groovy.lang.MetaClassImpl.chooseMethod(MetaClassImpl.java:3097)
at 
groovy.lang.MetaClassImpl.createConstructorSite(MetaClassImpl.java:3384)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallConstructorSite(CallSiteArray.java:88)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at 
me.andrz.findbugs.diff.FindBugsDiff.diffFromFilePaths(FindBugsDiff.groovy:33)
at me.andrz.findbugs.diff.FindBugsDiff$diffFromFilePaths.call(Unknown 
Source)
...
{code}

[~andras.piros] please take a look at this.


was (Author: pbacsko):
{{-1 TESTS - patch does not compile, cannot run testcases}}

Looking at the build, this is not the case, everything is passed, but there 
were problems during findbugs diff:

{code}
[TRACE] Downloading FindBugs diff JAR from 
https://repo1.maven.org/maven2/me/andrz/findbugs/findbugs-diff/0.1.0/findbugs-diff-0.1.0-all.jar
[TRACE] FindBugs diff JAR downloaded
grep: 
/home/jenkins/jenkins-slave/workspace/oozie-trunk-precommit-build/bin/findbugs-diff-0.1.0-all.jar.md5sum:
 No such file or directory
[TRACE] FindBugs diff JAR checked, is safe to use
[TRACE] Performing FindBugs diffs
bin/test-patch-11-findbugs-diff: line 253: : No such file or directory
bin/test-patch-11-findbugs-diff: line 255: [: -le: unary operator expected
Exception in thread "main" groovy.lang.GroovyRuntimeException: Ambiguous method 
overloading for method java.io.File#.
Cannot resolve which method to invoke for [null] due to overlapping prototypes 
between:
[class java.lang.String]
[class java.net.URI]
at 
groovy.lang.MetaClassImpl.chooseMostSpecificParams(MetaClassImpl.java:3202)
at 
groovy.lang.MetaClassImpl.chooseMethodInternal(MetaClassImpl.java:3154)
at groovy.lang.MetaClassImpl.chooseMethod(MetaClassImpl.java:3097)
at 
groovy.lang.MetaClassImpl.createConstructorSite(MetaClassImpl.java:3384)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallConstructorSite(CallSiteArray.java:88)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at 
me.andrz.findbugs.diff.FindBugsDiff.diffFromFilePaths(FindBugsDiff.groovy:33)
at me.andrz.findbugs.diff.FindBugsDiff$diffFromFilePaths.call(Unknown 
Source)
...
{code}

[~andras.piros] please take a look at this.

> Replace reflection-based code which sets variable in HiveConf
> -
>
> Key: OOZIE-2845
> URL: https://issues.apache.org/jira/browse/OOZIE-2845
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: OOZIE-2845-001.patch
>
>
> In different places, we use the following reflection-based code to reset a 
> variable inside {{HiveConf.java}}
> {code}
> public static void resetHiveConfStaticVariables() throws Exception {
> // HiveConf initializes 

[jira] [Commented] (OOZIE-2818) Can't overwrite oozie.action.max.output.data on a per-workflow basis

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946789#comment-15946789
 ] 

Attila Sasvari commented on OOZIE-2818:
---

[~gezapeti] can you take a look?

> Can't overwrite oozie.action.max.output.data on a per-workflow basis
> 
>
> Key: OOZIE-2818
> URL: https://issues.apache.org/jira/browse/OOZIE-2818
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2818-01.patch
>
>
> It's not possible to raise the limit of the output on a per-action basis. 
> With the workflow:
> {quote}
> 
> 
> 
> Action failed, error 
> message\[$\{wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> $\{jobTracker}
> $\{nameNode}
> 
> 
> oozie.action.max.output.data
> 10
> 
> 
> oozie.launcher.oozie.action.max.output.data
> 10
> 
> 
> /user/admin/large_output.sh
>   10
> /user/admin/large_output.sh#large_output.sh
>   
> 
> 
> 
> 
> 
> 
> {quote}
> In the localized files in the job.xml the value of this property gets 
> overwritten:
> {quote}
> # cat 
> /yarn/nm/usercache/admin/appcache/application_1489145190857_0001/container_1489145190857_0001_01_02/job.xml
>  |grep max.outp
> oozie.launcher.oozie.action.max.output.data10programaticallyjob.xml
> oozie.action.max.output.data2048programaticallyjob.xml
> # cat 
> /yarn/nm/usercache/admin/appcache/application_1489145190857_0001/container_1489145190857_0001_01_02/action.xml
>  |grep max.outp
> oozie.launcher.oozie.action.max.output.data10programatically
> oozie.action.max.output.data10programatically
> {quote}
> I could not find a reason for this behavior, so I think it's a bug.
> The value is read from 
> [ConfigurationService|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L175]
>  and set in 
> [createLauncherConf|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L986]
>  in JavaActionExecutor. There could be a check to see if it's already set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (OOZIE-2818) Can't overwrite oozie.action.max.output.data on a per-workflow basis

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2818:
--
Attachment: OOZIE-2818-01.patch

> Can't overwrite oozie.action.max.output.data on a per-workflow basis
> 
>
> Key: OOZIE-2818
> URL: https://issues.apache.org/jira/browse/OOZIE-2818
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2818-01.patch
>
>
> It's not possible to raise the limit of the output on a per-action basis. 
> With the workflow:
> {quote}
> 
> 
> 
> Action failed, error 
> message\[$\{wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> $\{jobTracker}
> $\{nameNode}
> 
> 
> oozie.action.max.output.data
> 10
> 
> 
> oozie.launcher.oozie.action.max.output.data
> 10
> 
> 
> /user/admin/large_output.sh
>   10
> /user/admin/large_output.sh#large_output.sh
>   
> 
> 
> 
> 
> 
> 
> {quote}
> In the localized files in the job.xml the value of this property gets 
> overwritten:
> {quote}
> # cat 
> /yarn/nm/usercache/admin/appcache/application_1489145190857_0001/container_1489145190857_0001_01_02/job.xml
>  |grep max.outp
> oozie.launcher.oozie.action.max.output.data10programaticallyjob.xml
> oozie.action.max.output.data2048programaticallyjob.xml
> # cat 
> /yarn/nm/usercache/admin/appcache/application_1489145190857_0001/container_1489145190857_0001_01_02/action.xml
>  |grep max.outp
> oozie.launcher.oozie.action.max.output.data10programatically
> oozie.action.max.output.data10programatically
> {quote}
> I could not find a reason for this behavior, so I think it's a bug.
> The value is read from 
> [ConfigurationService|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L175]
>  and set in 
> [createLauncherConf|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L986]
>  in JavaActionExecutor. There could be a check to see if it's already set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (98 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2849  Fix build warnings when assembling directories
https://issues.apache.org/jira/browse/OOZIE-2849
OOZIE-2848  Override sharelib.tgz in distro when rebuilding Oozie
https://issues.apache.org/jira/browse/OOZIE-2848
OOZIE-2846  TestSSLServerConnectorFactory fails with Jetty 9.3 and 9.4
https://issues.apache.org/jira/browse/OOZIE-2846
OOZIE-2845  Replace reflection-based code which sets variable in HiveConf
https://issues.apache.org/jira/browse/OOZIE-2845
OOZIE-2843  Enhance logging inside ZKLocksService and MemoryLocksService
https://issues.apache.org/jira/browse/OOZIE-2843
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2831  Update maven-project-info-reports-plugin to 2.9
https://issues.apache.org/jira/browse/OOZIE-2831
OOZIE-2827  More directly view of the coordinator’s history from perspective of 
workflow action.
https://issues.apache.org/jira/browse/OOZIE-2827
OOZIE-2825  Custom Authentication doc page is not well formatted
https://issues.apache.org/jira/browse/OOZIE-2825
OOZIE-2824  Fix typos in documentation
https://issues.apache.org/jira/browse/OOZIE-2824
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2796  oozie.action.keep.action.dir not getting noticed
https://issues.apache.org/jira/browse/OOZIE-2796
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2785  Master is not compiling with hadoop-1 profile
https://issues.apache.org/jira/browse/OOZIE-2785
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2780  Upgrade hadoop-common version to 2.6.0
https://issues.apache.org/jira/browse/OOZIE-2780
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2769  Extend FS action to allow setrep on a file
https://issues.apache.org/jira/browse/OOZIE-2769
OOZIE-2753  Update Docs for Jetty Tomcat changes
https://issues.apache.org/jira/browse/OOZIE-2753
OOZIE-2751  LocalOozieClient is missing methods from OozieClient
https://issues.apache.org/jira/browse/OOZIE-2751
OOZIE-2746  Several tests failure in TestV2ValidateServlet.java
https://issues.apache.org/jira/browse/OOZIE-2746
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2733  change org.apache.hadoop.fs.permission.AccessControlException to 
org.apache.hadoop.security.AccessControlException
https://issues.apache.org/jira/browse/OOZIE-2733
OOZIE-2726  Flaky test due to daylight saving changes
https://issues.apache.org/jira/browse/OOZIE-2726
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2713  Provide an option to log out contents of the localized jar files
https://issues.apache.org/jira/browse/OOZIE-2713
OOZIE-2701  Oozie to support Multiple HCatalog URIs
https://issues.apache.org/jira/browse/OOZIE-2701
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2681  fix javadoc to compile on JDK8 again
https://issues.apache.org/jira/browse/OOZIE-2681
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2652  Skip trash while deleting hive table partition