[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (97 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3382  Optimize SshActionExecutor's drainBuffers method
https://issues.apache.org/jira/browse/OOZIE-3382
OOZIE-3381  [coordinator] Enhance logging of CoordElFunctions
https://issues.apache.org/jira/browse/OOZIE-3381
OOZIE-3379  Auth token cache file name should include Oozie URL
https://issues.apache.org/jira/browse/OOZIE-3379
OOZIE-3375  Can't use empty  in coordinator
https://issues.apache.org/jira/browse/OOZIE-3375
OOZIE-3367  Using && in EL expressions in oozie bundle.xml files generates 
parse errors
https://issues.apache.org/jira/browse/OOZIE-3367
OOZIE-3366  Update workflow status and subworkflow status on suspend command
https://issues.apache.org/jira/browse/OOZIE-3366
OOZIE-3364  Rerunning Oozie bundle jobs starts the coordinators in 
indeterminate order
https://issues.apache.org/jira/browse/OOZIE-3364
OOZIE-3362  When killed, SSH action should kill the spawned processes on target 
host
https://issues.apache.org/jira/browse/OOZIE-3362
OOZIE-3326  Sqoop Action should support tez delegation tokens for hive-import
https://issues.apache.org/jira/browse/OOZIE-3326
OOZIE-3320  Oozie ShellAction should support absolute bash file path
https://issues.apache.org/jira/browse/OOZIE-3320
OOZIE-3319  Log SSH action callback error output
https://issues.apache.org/jira/browse/OOZIE-3319
OOZIE-3301  Update NOTICE file
https://issues.apache.org/jira/browse/OOZIE-3301
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3266  Coord action rerun support RERUN_SKIP_NODES option
https://issues.apache.org/jira/browse/OOZIE-3266
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3137  Add support for log4j2 in HiveMain
https://issues.apache.org/jira/browse/OOZIE-3137
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3120  maven-assembly-plugin fails when bumped from 2.2.1
https://issues.apache.org/jira/browse/OOZIE-3120
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create 

[jira] [Commented] (OOZIE-3385) The situation multi user submit workflows , occasionally, occur the HDFS visitor user become another one

[LuGuangMing (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694121#comment-16694121
 ] 

LuGuangMing commented on OOZIE-3385:


@[~andras.piros] hello sir, I submit workflow by interface AuthOozieClient.doas 
to proxy user instead of CLI. I have not set property job.properties,  
user.name is my input proxy user in workflow.xml. This bug occurs when oozie 
server reading workflow.xml defined Information , my log print 
"DFSClient.toString()" info that DFSClient inner ugi is right , 
"{color:#FF}_2018-11-14 00:00:00,493 INFO 
[CallableQueue-42]org.apache.oozie.service.HadoopAccessorService(520) 
USER[platform] GROUP[-] TOKEN[] APP[myBulkload-Scheduler-CS_TTL-1539689446] 
JOB[0002497-180928143722290-oozie-root-C] 
ACTION[0002497-180928143722290-oozie-root-C@1354]_ _hdfs client user, 
DFSClient_{color}{color:#FF}_[clientName=DFSClient_NONMAPREDUCE_-515910437_325,
 ugi=platform (auth:PROXY) via oozie/nsplatfor...@dc1.fh.com 
(auth:KERBEROS)]_{color}*",* but name node check permission return user is 
wrong others, "{color:#FF}_XException, 
org.apache.oozie.command.CommandException: E0710: Could not read the workflow 
definition, Permission denied: user=dbzq04, access=READ, 
inode="/phoebus/_fileservice/users/nsplatform/platform/workflows/DataLoadWF-1427-1129/workflow.xml":platform:supergroup:-rw---{color:#33}",
 which details could be  look the attachments.{color}_{color}

> The situation multi user submit workflows , occasionally, occur the HDFS 
> visitor user become another one 
> -
>
> Key: OOZIE-3385
> URL: https://issues.apache.org/jira/browse/OOZIE-3385
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.3.1
>Reporter: LuGuangMing
>Priority: Blocker
> Attachments: oozie-server-error.log, 
> part_source_HadoopAccessorService.txt, part_source_WorkflowAppService.txt
>
>
> The situation multi user submit workflows , occasionally, occur the HDFS 
> visitor user become another one . for example, I need submit a workflow by 
> proxy user "{color:#ff}platform{color}" via user oozie (kerberos) , an 
> error occur in oozie source code  WorkflowAppService.readDefinition read 
> workflow.xml.
> *2018-11-14 00:00:00,497 ERROR 
> [CallableQueue-42]org.apache.oozie.command.wf.SubmitXCommand(517) 
> {color:#ff}USER[platform]{color} GROUP[-] TOKEN[] 
> APP[myBulkload-Scheduler-CS_TTL-1539689446] 
> JOB[0002497-180928143722290-oozie-root-C] 
> ACTION[0002497-180928143722290-oozie-root-C@1354] XException, 
> org.apache.oozie.command.CommandException: E0710: Could not read the workflow 
> definition, Permission denied: user={color:#ff}dbzq04{color}, 
> access=READ, 
> inode="/phoebus/_fileservice/users/nsplatform/platform/workflows/DataLoadWF-1427-1129/workflow.xml":{color:#ff}platform{color}:supergroup:-rw---*
> note: user  "{color:#ff}dbzq04{color}"  also submit some workflow at 
> before, but current submit the workflow of user is user 
> {color:#ff}platform. In order to prove current user is platform , I 
> insert some logs at oozie source code {color}
>  
>   
> {code:java}
> /**  org.apache.oozie.service.HadoopAccessorService   */
> public FileSystem createFileSystem(String user, final URI uri, final 
> Configuration   conf) throws HadoopAccessorException {
>   //.omit..
>  try {
>UserGroupInformation ugi = getUGI(user);
>LOG.info("current user="+ugi);  //-- my insert log, to print proxy ugi 
> info
>return ugi.doAs(new PrivilegedExceptionAction() {
>public FileSystem run() throws Exception {
> FileSystem fs = FileSystem.get(uri, conf);
> //-- my insert log, to print fs inner ugi info
> if(fs instanceof DistributedFileSystem){
>  LOG.info("hdfs client user, 
> "+((DistributedFileSystem)fs).getClient().toString());
> }
> return fs;
>}
>  });
>  }catch (InterruptedException ex) {
>throw new HadoopAccessorException(ErrorCode.E0902, ex.getMessage(), ex);
>  }catch (IOException ex) {
>throw new HadoopAccessorException(ErrorCode.E0902, ex.getMessage(), ex);
>  }
> }{code}
>  *my log print result follows:*
>  
>  2018-11-14 00:00:00,492 INFO 
> [CallableQueue-42]org.apache.oozie.service.HadoopAccessorService(520) 
> USER[platform] GROUP[-] TOKEN[] APP[myBulkload-Scheduler-CS_TTL-1539689446] 
> JOB[0002497-180928143722290-oozie-root-C] 
> ACTION[0002497-180928143722290-oozie-root-C@1354] *current user=platform 
> (auth:PROXY) via oozie/nsplatfor...@dc1.fh.com (auth:KERBEROS)*
>  2018-11-14 00:00:00,493 INFO 
> [CallableQueue-42]org.apache.oozie.service.HadoopAccessorService(520) 
> USER[platform] GROUP[-] TOKEN[] APP[myBulkload-Scheduler-CS_TTL-1539689446] 
> 

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693448#comment-16693448
 ] 

Andras Salamon commented on OOZIE-3382:
---

Looks like {{TestBufferDrainer}} is flaky.

> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there 
> are data in the buffer), or we can increase the value of the buffer size 
> (1024). 
> In the latter case the default of the buffer size could be half of the 
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an 
> additional property to specify the buffer size (to avoid memory problems 
> because of using a very big buffer). We can keep 1024 as a minimum buffer 
> size. 
> It would be also useful to refactor the code and put the buffer draining into 
> a separate class and create unit tests for the class. Using this class in 
> {{ShellMain}} to avoid code duplication would also be very useful, but we 
> have to fix OOZIE-3359 first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693430#comment-16693430
 ] 

Hadoop QA commented on OOZIE-3382:
--


Testing JIRA OOZIE-3382

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 3076
.Tests failed : 1
.Tests in error   : 0
.Tests timed out  : 0

{color:red}-1{color} [ERROR] There are [1] test failures in [core]. Listing 
only the first [5] ones
testDrainBuffersFakeCatSlowWrite:org.apache.oozie.util.TestBufferDrainer

Check console output for the full list of errors/failures
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/920/



> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle 

Failed: OOZIE-3382 PreCommit Build #920

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3382
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/920/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.90 MB...]
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2560 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
xargs: WARNING: a NUL character occurred in the input.  It cannot be passed 
through in the argument list.  Did you mean to use the --null option?
  Running test-patch task DISTRO


Testing JIRA OOZIE-3382

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 3076
Tests failed : 1
Tests in error   : 0
Tests timed out  : 0

-1 [ERROR] There are [1] test failures in [core]. Listing only the first [5] 
ones
testDrainBuffersFakeCatSlowWrite:org.apache.oozie.util.TestBufferDrainer

Check console output for the full list of errors/failures
Tests failed at first run:
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/920/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  38060 0  100  3806  0   3699  0:00:01  0:00:01 --:--:--  
3702{"self":"https://issues.apache.org/jira/rest/api/2/issue/13197752/comment/16693430","id":"16693430","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3382\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:green}+1 

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693308#comment-16693308
 ] 

Hadoop QA commented on OOZIE-3382:
--

PreCommit-OOZIE-Build started


> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there 
> are data in the buffer), or we can increase the value of the buffer size 
> (1024). 
> In the latter case the default of the buffer size could be half of the 
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an 
> additional property to specify the buffer size (to avoid memory problems 
> because of using a very big buffer). We can keep 1024 as a minimum buffer 
> size. 
> It would be also useful to refactor the code and put the buffer draining into 
> a separate class and create unit tests for the class. Using this class in 
> {{ShellMain}} to avoid code duplication would also be very useful, but we 
> have to fix OOZIE-3359 first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69408: OOZIE-3382 - Optimize SshActionExecutor's drainBuffers method

[Andras Salamon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69408/
---

(Updated Nov. 20, 2018, 2:21 p.m.)


Review request for oozie, András Piros and Kinga Marton.


Changes
---

Fixing RAT warning


Repository: oozie-git


Description
---

OOZIE-3382 - Optimize SshActionExecutor's drainBuffers method


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
3e0e3c573 
  core/src/main/java/org/apache/oozie/util/BufferDrainer.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/TestBufferDrainer.java PRE-CREATION 


Diff: https://reviews.apache.org/r/69408/diff/2/

Changes: https://reviews.apache.org/r/69408/diff/1-2/


Testing
---

Unit tests


Thanks,

Andras Salamon

[jira] [Updated] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Andras Salamon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Salamon updated OOZIE-3382:
--
Attachment: OOZIE-3382-02.patch

> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there 
> are data in the buffer), or we can increase the value of the buffer size 
> (1024). 
> In the latter case the default of the buffer size could be half of the 
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an 
> additional property to specify the buffer size (to avoid memory problems 
> because of using a very big buffer). We can keep 1024 as a minimum buffer 
> size. 
> It would be also useful to refactor the code and put the buffer draining into 
> a separate class and create unit tests for the class. Using this class in 
> {{ShellMain}} to avoid code duplication would also be very useful, but we 
> have to fix OOZIE-3359 first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693300#comment-16693300
 ] 

Andras Salamon commented on OOZIE-3382:
---

The new version of the patch fixes the RAT warning.

> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there 
> are data in the buffer), or we can increase the value of the buffer size 
> (1024). 
> In the latter case the default of the buffer size could be half of the 
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an 
> additional property to specify the buffer size (to avoid memory problems 
> because of using a very big buffer). We can keep 1024 as a minimum buffer 
> size. 
> It would be also useful to refactor the code and put the buffer draining into 
> a separate class and create unit tests for the class. Using this class in 
> {{ShellMain}} to avoid code duplication would also be very useful, but we 
> have to fix OOZIE-3359 first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693295#comment-16693295
 ] 

Andras Salamon commented on OOZIE-3382:
---

testRetryConsoleUrlForked error is unrelated, see OOZIE-3384

> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there 
> are data in the buffer), or we can increase the value of the buffer size 
> (1024). 
> In the latter case the default of the buffer size could be half of the 
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an 
> additional property to specify the buffer size (to avoid memory problems 
> because of using a very big buffer). We can keep 1024 as a minimum buffer 
> size. 
> It would be also useful to refactor the code and put the buffer draining into 
> a separate class and create unit tests for the class. Using this class in 
> {{ShellMain}} to avoid code duplication would also be very useful, but we 
> have to fix OOZIE-3359 first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3387) Optimize coordinator data input dependency search

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3387:

Description: 
During data input dependency check Oozie evaluates EL functions like 
{{coord:latest()}} using a non-optimal way which may result more than necessary 
HDFS URI checks.

1. If the {{dataset}} frequency does not match the {{uri-template}} it checks 
the same HDFS URI multiple times. For instance in the following definition:
{noformat}

${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
_SUCCESS

...

${coord:latest(0)}

{noformat}
oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It 
would be enough to check the file only once and skip the other 1439 tests.

2. If the frequency is 1 day and {{uri-template}} is definied in the following 
way:
{noformat}
${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
{noformat}
oozie will check the following directories one by one even if the some of the 
parent directories are missing:
{noformat}
2018/11/20
2018/11/19
2018/11/18
...
{noformat}
If there is no {{2018/11}} directory then it is not necessary to check all the 
{{2018/11/xx}} directories. It would be possible to reduce the number of HDFS 
URI checks.

  was:
During data input dependency check Oozie evaluates EL functions like {{ 
coord:latest}} using a non-optimal way which may result more than necessary 
HDFS URI checks.

1. If the {{dataset}} frequency does not match the {{uri-template}} it checks 
the same HDFS URI multiple times. For instance in the following definition:
{noformat}

${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
_SUCCESS

...

${coord:latest(0)}

{noformat}
oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It 
would be enough to check the file only once and skip the other 1439 tests.

2. If the frequency is 1 day and {{uri-template}} is definied in the following 
way:
{noformat}
${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
{noformat}
oozie will check the following directories one by one even if the some of the 
parent directories are missing:
{noformat}
2018/11/20
2018/11/19
2018/11/18
...
{noformat}
If there is no {{2018/11}} directory then it is not necessary to check all the 
{{2018/11/xx}} directories. It would be possible to reduce the number of HDFS 
URI checks.


> Optimize coordinator data input dependency search
> -
>
> Key: OOZIE-3387
> URL: https://issues.apache.org/jira/browse/OOZIE-3387
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Priority: Major
>
> During data input dependency check Oozie evaluates EL functions like 
> {{coord:latest()}} using a non-optimal way which may result more than 
> necessary HDFS URI checks.
> 1. If the {{dataset}} frequency does not match the {{uri-template}} it checks 
> the same HDFS URI multiple times. For instance in the following definition:
> {noformat}
>  initial-instance="2017-01-01T08:15Z" timezone="UTC">
> 
> ${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
> _SUCCESS
> 
> ...
> 
> ${coord:latest(0)}
> 
> {noformat}
> oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It 
> would be enough to check the file only once and skip the other 1439 tests.
> 2. If the frequency is 1 day and {{uri-template}} is definied in the 
> following way:
> {noformat}
> ${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
> {noformat}
> oozie will check the following directories one by one even if the some of the 
> parent directories are missing:
> {noformat}
> 2018/11/20
> 2018/11/19
> 2018/11/18
> ...
> {noformat}
> If there is no {{2018/11}} directory then it is not necessary to check all 
> the {{2018/11/xx}} directories. It would be possible to reduce the number of 
> HDFS URI checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3387) Optimize coordinator data input dependency search

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3387:

Affects Version/s: 5.1.0

> Optimize coordinator data input dependency search
> -
>
> Key: OOZIE-3387
> URL: https://issues.apache.org/jira/browse/OOZIE-3387
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Priority: Major
>
> During data input dependency check Oozie evaluates EL functions like {{ 
> coord:latest}} using a non-optimal way which may result more than necessary 
> HDFS URI checks.
> 1. If the {{dataset}} frequency does not match the {{uri-template}} it checks 
> the same HDFS URI multiple times. For instance in the following definition:
> {noformat}
>  initial-instance="2017-01-01T08:15Z" timezone="UTC">
> 
> ${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
> _SUCCESS
> 
> ...
> 
> ${coord:latest(0)}
> 
> {noformat}
> oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It 
> would be enough to check the file only once and skip the other 1439 tests.
> 2. If the frequency is 1 day and {{uri-template}} is definied in the 
> following way:
> {noformat}
> ${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
> {noformat}
> oozie will check the following directories one by one even if the some of the 
> parent directories are missing:
> {noformat}
> 2018/11/20
> 2018/11/19
> 2018/11/18
> ...
> {noformat}
> If there is no {{2018/11}} directory then it is not necessary to check all 
> the {{2018/11/xx}} directories. It would be possible to reduce the number of 
> HDFS URI checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3387) Optimize coordinator data input dependency search

[Andras Salamon (JIRA)]

Andras Salamon created OOZIE-3387:
-

 Summary: Optimize coordinator data input dependency search
 Key: OOZIE-3387
 URL: https://issues.apache.org/jira/browse/OOZIE-3387
 Project: Oozie
  Issue Type: Improvement
Reporter: Andras Salamon


During data input dependency check Oozie evaluates EL functions like {{ 
coord:latest}} using a non-optimal way which may result more than necessary 
HDFS URI checks.

1. If the {{dataset}} frequency does not match the {{uri-template}} it checks 
the same HDFS URI multiple times. For instance in the following definition:
{noformat}

${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
_SUCCESS

...

${coord:latest(0)}

{noformat}
oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It 
would be enough to check the file only once and skip the other 1439 tests.

2. If the frequency is 1 day and {{uri-template}} is definied in the following 
way:
{noformat}
${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
{noformat}
oozie will check the following directories one by one even if the some of the 
parent directories are missing:
{noformat}
2018/11/20
2018/11/19
2018/11/18
...
{noformat}
If there is no {{2018/11}} directory then it is not necessary to check all the 
{{2018/11/xx}} directories. It would be possible to reduce the number of HDFS 
URI checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693222#comment-16693222
 ] 

Hadoop QA commented on OOZIE-3382:
--


Testing JIRA OOZIE-3382

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:red}-1 RAT{color}
.{color:red}-1{color} the patch seems to introduce 1 new RAT warning(s)
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 3076
.Tests failed : 2
.Tests in error   : 0
.Tests timed out  : 0

{color:red}-1{color} [ERROR] There are [2] test failures in [core]. Listing 
only the first [5] ones
testRetryConsoleUrlForked:org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand

Check console output for the full list of errors/failures
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF1
TestPurgeXCommand#testPurgeCoordWithWFChild2
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/919/



> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is 

Failed: OOZIE-3382 PreCommit Build #919

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3382
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/919/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.89 MB...]
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2552 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
xargs: WARNING: a NUL character occurred in the input.  It cannot be passed 
through in the argument list.  Did you mean to use the --null option?
  Running test-patch task DISTRO


Testing JIRA OOZIE-3382

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
-1 RAT
-1 the patch seems to introduce 1 new RAT warning(s)
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 3076
Tests failed : 2
Tests in error   : 0
Tests timed out  : 0

-1 [ERROR] There are [2] test failures in [core]. Listing only the first [5] 
ones
testRetryConsoleUrlForked:org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand

Check console output for the full list of errors/failures
Tests failed at first run:
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF1
TestPurgeXCommand#testPurgeCoordWithWFChild2
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/919/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13197752/comment/16693222","id":"16693222","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3382\n\nCleaning local git 

[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693172#comment-16693172
 ] 

Hadoop QA commented on OOZIE-3381:
--


Testing JIRA OOZIE-3381

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3075
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeCoordWithWFChild2
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/918/



> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
>  Issue Type: Task
>  Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL 
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
>  inside an input dataset dependency, it's often the case that more 
> information is needed how many HDFS URIs are being checked for each 
> {{}}.
> Right now we don't have this information. While debugging and fine tuning 
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} 
> {{instance}}, it would be very useful to know how many HDFS roundtrips are 
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} 
> and {{CoordELFunctions#coord_futureRange_sync()}} having called 
> {{DFSClient#exists()}}. We need appropriate logging there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3381 PreCommit Build #918

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3381
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/918/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.89 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2560 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3381

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 2 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3075
Tests failed at first run:
TestPurgeXCommand#testPurgeCoordWithWFChild2
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/918/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13197397/comment/16693172","id":"16693172","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3381\n\nCleaning local git 

[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693113#comment-16693113
 ] 

Andras Piros commented on OOZIE-3379:
-

Thanks for the new patch [~zuston]! Left comments on ReviewBoard.

> Auth token cache file name should include Oozie URL
> ---
>
> Key: OOZIE-3379
> URL: https://issues.apache.org/jira/browse/OOZIE-3379
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: 5.0.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3379-1.patch, oozie-3379-2.patch, 
> oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch
>
>
> We have a program that uses the oozie client, but when the client connects to 
> multiple clusters,
> the authOozieClient class frequently requests the kdc server because the 
> authentication token cache file is invalid.
> This will cause subsequent requests in our program to be blocked, resulting 
> in unstable services.
> So, oozie client's auth token cache file name should include Oozie URL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69330: OOZIE-3379 Auth token cache file name should include Oozie URL

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69330/#review210704
---




client/src/main/java/org/apache/oozie/client/AuthOozieClient.java
Lines 73 (patched)


Can be `final`.



client/src/main/java/org/apache/oozie/client/AuthOozieClient.java
Lines 100 (patched)


Use `@VisibleForTesting`, and can be package-private.



client/src/main/java/org/apache/oozie/client/AuthOozieClient.java
Lines 101 (patched)


Why not use Guava's `BaseEncoding` like this?

```
BaseEncoding
.base64Url()
.omitPadding()
.encode("https://localhost:11443/oozie".getBytes(Charsets.UTF_8))
```



core/src/main/java/org/apache/oozie/test/EmbeddedServletContainer.java
Lines 131-133 (patched)


Can you please extract to a well-named method, or leave a code comment for 
future maintainers, or both?



core/src/main/java/org/apache/oozie/test/EmbeddedServletContainer.java
Line 128 (original), 136 (patched)


Can you please extract to a well-named method, or leave a code comment for 
future maintainers, or both?



core/src/test/java/org/apache/oozie/servlet/TestAuthFilterAuthOozieClient.java
Lines 312-336 (patched)


Can you please:

* extract to separate test method
* provide assertion messages


- András Piros


On Nov. 14, 2018, 2:02 a.m., zhang junfan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69330/
> ---
> 
> (Updated Nov. 14, 2018, 2:02 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Auth token cache file name should include Oozie URL, link to oozie-3379.
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/AuthOozieClient.java 3a8b5ab6 
>   core/src/main/java/org/apache/oozie/test/EmbeddedServletContainer.java 
> fc9d840b 
>   
> core/src/test/java/org/apache/oozie/servlet/TestAuthFilterAuthOozieClient.java
>  04fde730 
> 
> 
> Diff: https://reviews.apache.org/r/69330/diff/1/
> 
> 
> Testing
> ---
> 
> I added a test for authOozieClients with the different oozieUrl.
> I changed the code for EmbeddedServletContainer. Because this patch needs to 
> bind the cache file to oozieUrl, you need to test whether multiple different 
> clients can use the cache. In the original EmbeddedServletContainer code, the 
> port was random and could not get a given container context path. So the port 
> binding needs to be added.
> 
> 
> Thanks,
> 
> zhang junfan
> 
>

[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693098#comment-16693098
 ] 

Hadoop QA commented on OOZIE-3382:
--

PreCommit-OOZIE-Build started


> Optimize SshActionExecutor's drainBuffers method
> 
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}} 
> blocks and modified the {{drainBuffers}} method to keep draining the standard 
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is 
> running the method only reads 1024 bytes in each cycle (half a second) which 
> can take very long time if we want to drain several megabytes (for instance 
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there 
> are data in the buffer), or we can increase the value of the buffer size 
> (1024). 
> In the latter case the default of the buffer size could be half of the 
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an 
> additional property to specify the buffer size (to avoid memory problems 
> because of using a very big buffer). We can keep 1024 as a minimum buffer 
> size. 
> It would be also useful to refactor the code and put the buffer draining into 
> a separate class and create unit tests for the class. Using this class in 
> {{ShellMain}} to avoid code duplication would also be very useful, but we 
> have to fix OOZIE-3359 first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69408: OOZIE-3382 - Optimize SshActionExecutor's drainBuffers method

[Andras Salamon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69408/
---

Review request for oozie, András Piros and Kinga Marton.


Repository: oozie-git


Description
---

OOZIE-3382 - Optimize SshActionExecutor's drainBuffers method


Diffs
-

  core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
3e0e3c573 
  core/src/main/java/org/apache/oozie/util/BufferDrainer.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/TestBufferDrainer.java PRE-CREATION 


Diff: https://reviews.apache.org/r/69408/diff/1/


Testing
---

Unit tests


Thanks,

Andras Salamon

[jira] [Assigned] (OOZIE-3384) [tests] TestWorkflowActionRetryInfoXCommand#testRetryConsoleUrlForked() is flaky

[Andras Salamon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Salamon reassigned OOZIE-3384:
-

Assignee: Andras Salamon

> [tests] TestWorkflowActionRetryInfoXCommand#testRetryConsoleUrlForked() is 
> flaky
> 
>
> Key: OOZIE-3384
> URL: https://issues.apache.org/jira/browse/OOZIE-3384
> Project: Oozie
>  Issue Type: Sub-task
>  Components: tests
>Reporter: Andras Piros
>Assignee: Andras Salamon
>Priority: Major
>
> {code:java}
> junit.framework.AssertionFailedError: Expected :2 Actual :1  difference> at junit.framework.Assert.fail(Assert.java:57) at
> ...
> org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.validateRetryConsoleUrl(TestWorkflowActionRetryInfoXCommand.java:172)
>  at 
> org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.testRetryConsoleUrlForked(TestWorkflowActionRetryInfoXCommand.java:125)
>  at
>  ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OOZIE-3386) Misleading error message when workflow application does not exist

[Julia Kinga Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton reassigned OOZIE-3386:
-

Assignee: Julia Kinga Marton

> Misleading error message when workflow application does not exist
> -
>
> Key: OOZIE-3386
> URL: https://issues.apache.org/jira/browse/OOZIE-3386
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> Using 5.1.0 rc1, I tried to run an example workflow. Because of user error 
> {{oozie.wf.application.path}} in job.properties pointed to a directory in 
> HDFS that did not exist. Upon submitting the workflow, the following was 
> returned
> {code}
> bin/oozie job -oozie http://localhost:11000/oozie   -config 
> examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run 
> -DnameNode=hdfs://localhost:9000
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Error: E0307 : E0307: Runtime error [Configuration entry 
> oozie.jobs.api.generated.xml not present]
> {code}
> The server.log contained the following:
> {code}
> 2018-11-19 15:07:13,244  WARN V1JobsServlet:523 - 
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-] 
> JOB[-] ACTION[-] URL[POST 
> http://localhost:11000/oozie/v2/jobs?action=start=asasvari] 
> error[E0307], E0307: Runtime error [Configuration entry 
> oozie.jobs.api.generated.xml not present]
> org.apache.oozie.servlet.XServletException: E0307: Runtime error 
> [Configuration entry oozie.jobs.api.generated.xml not present]
>at 
> org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
>at 
> org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>at 
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
>at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
>at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
>at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
>at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at 
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
>at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>at org.eclipse.jetty.server.Server.handle(Server.java:534)
>at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
>at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
>at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>at 
> 

[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693058#comment-16693058
 ] 

Hadoop QA commented on OOZIE-3381:
--

PreCommit-OOZIE-Build started


> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
>  Issue Type: Task
>  Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL 
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
>  inside an input dataset dependency, it's often the case that more 
> information is needed how many HDFS URIs are being checked for each 
> {{}}.
> Right now we don't have this information. While debugging and fine tuning 
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} 
> {{instance}}, it would be very useful to know how many HDFS roundtrips are 
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} 
> and {{CoordELFunctions#coord_futureRange_sync()}} having called 
> {{DFSClient#exists()}}. We need appropriate logging there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693044#comment-16693044
 ] 

Andras Piros commented on OOZIE-3381:
-

Thanks for the review [~kmarton]! Updated patch based on review comments.

> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
>  Issue Type: Task
>  Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL 
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
>  inside an input dataset dependency, it's often the case that more 
> information is needed how many HDFS URIs are being checked for each 
> {{}}.
> Right now we don't have this information. While debugging and fine tuning 
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} 
> {{instance}}, it would be very useful to know how many HDFS roundtrips are 
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} 
> and {{CoordELFunctions#coord_futureRange_sync()}} having called 
> {{DFSClient#exists()}}. We need appropriate logging there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69348: OOZIE-3381 [coordinator] Enhance logging of CoordElFunctions

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69348/
---

(Updated Nov. 20, 2018, 10:54 a.m.)


Review request for oozie and Kinga Marton.


Changes
---

Update based on review comments.


Repository: oozie-git


Description
---

OOZIE-3381 [coordinator] Enhance logging of CoordElFunctions


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/command/XCommand.java 
a80444e14134ec4234e6edfef0888cf1e76566df 
  core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 
10f4f0d8d14fbb8d60a2d09a45b8f3b3b089f461 
  core/src/test/java/org/apache/oozie/command/coord/CoordELExtensions.java 
796d19cfc9cbd03211746f9b94d0f67371249c7e 
  core/src/test/java/org/apache/oozie/coord/TestOozieTimeUnitConverter.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69348/diff/2/

Changes: https://reviews.apache.org/r/69348/diff/1-2/


Testing
---

`TestCoordELFunctions`


Thanks,

András Piros

[jira] [Updated] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3381:

Attachment: OOZIE-3381.002.patch

> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
>  Issue Type: Task
>  Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL 
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
>  inside an input dataset dependency, it's often the case that more 
> information is needed how many HDFS URIs are being checked for each 
> {{}}.
> Right now we don't have this information. While debugging and fine tuning 
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} 
> {{instance}}, it would be very useful to know how many HDFS roundtrips are 
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} 
> and {{CoordELFunctions#coord_futureRange_sync()}} having called 
> {{DFSClient#exists()}}. We need appropriate logging there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3386) Misleading error message when workflow application does not exist

[Attila Sasvari (JIRA)]

Attila Sasvari created OOZIE-3386:
-

 Summary: Misleading error message when workflow application does 
not exist
 Key: OOZIE-3386
 URL: https://issues.apache.org/jira/browse/OOZIE-3386
 Project: Oozie
  Issue Type: Bug
Affects Versions: 5.1.0
Reporter: Attila Sasvari


Using 5.1.0 rc1, I tried to run an example workflow. Because of user error 
{{oozie.wf.application.path}} in job.properties pointed to a directory in HDFS 
that did not exist. Upon submitting the workflow, the following was returned
{code}
bin/oozie job -oozie http://localhost:11000/oozie   -config 
examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run 
-DnameNode=hdfs://localhost:9000
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
Error: E0307 : E0307: Runtime error [Configuration entry 
oozie.jobs.api.generated.xml not present]
{code}
The server.log contained the following:
{code}
2018-11-19 15:07:13,244  WARN V1JobsServlet:523 - 
SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
ACTION[-] URL[POST 
http://localhost:11000/oozie/v2/jobs?action=start=asasvari] error[E0307], 
E0307: Runtime error [Configuration entry oozie.jobs.api.generated.xml not 
present]
org.apache.oozie.servlet.XServletException: E0307: Runtime error [Configuration 
entry oozie.jobs.api.generated.xml not present]
   at 
org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
   at 
org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at 
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
   at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
   at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
   at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
   at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
   at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
   at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   at 
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
   at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
   at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
   at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   at org.eclipse.jetty.server.Server.handle(Server.java:534)
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   at