[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (94 issues) Subscriber: ooziedaily Key Summary OOZIE-3574 JavaAction create uncorrect fileSystem instance in addActionLibs method https://issues.apache.org/jira/browse/OOZIE-3574 OOZIE-3573 Fix oozie-default.xml descriptions https://issues.apache.org/jira/browse/OOZIE-3573 OOZIE-3569 SSH Action should add checking success file https://issues.apache.org/jira/browse/OOZIE-3569 OOZIE-3568 Have large amount of log information “WARN messages [main] openjpa.MetaData” in jetty.log need to clean https://issues.apache.org/jira/browse/OOZIE-3568 OOZIE-3567 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3567 OOZIE-3560 IDEA shows have some error in index.jsp https://issues.apache.org/jira/browse/OOZIE-3560 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apa
Failed: OOZIE-3574 PreCommit Build #1273
Jira: https://issues.apache.org/jira/browse/OOZIE-3574
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1273/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 2.15 MB...]
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[INFO] There are no new bugs found totally].
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 2536 bytes
[TRACE] Full summary file size is 1525 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Running test-patch task MODERNIZER
Running modernizer op=report
[TRACE] grep -c '.*ERROR.*Prefer'
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/reports/MODERNIZER-clean.txt
[TRACE] Modernizer errors before patch: 51
[TRACE] grep -c '.*ERROR.*Prefer'
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/reports/MODERNIZER-patch.txt
[TRACE] Modernizer errors after patch: 51
Testing JIRA OOZIE-3574
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [webapp].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3199
Tests failed at first run:
TestBlockingInputStream#testFastWritingBlockingInputStream
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch
+1 MODERNIZER
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1273/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100
34130 0 100 3413 0 2289 0:00:01 0:00:01 --:--:--
2287{"self":"https://issues.apache.org/jira/rest/api/2/issue/13275391/comment/17001623","id":"17001623","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerId=hadoopqa&avatarId=10393","32x32":"https://issue
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001623#comment-17001623
]
Hadoop QA commented on OOZIE-3574:
--
Testing JIRA OOZIE-3574
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3199
.{color:orange}Tests failed at first run:{color}
TestBlockingInputStream#testFastWritingBlockingInputStream
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1273/
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3574-v1.patch
>
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001613#comment-17001613
]
Hadoop QA commented on OOZIE-3574:
--
PreCommit-OOZIE-Build started
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3574-v1.patch
>
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3569) SSH Action should add checking success file
[
https://issues.apache.org/jira/browse/OOZIE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated OOZIE-3569:
Attachment: (was: OOZIE-3569-v1.patch)
> SSH Action should add checking success file
> ---
>
> Key: OOZIE-3569
> URL: https://issues.apache.org/jira/browse/OOZIE-3569
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3569-v1.patch
>
>
> *Phenomenon*
> Currently, {{SSH Action}} checking operation are as following:
> Firstly, check operation is to check {{Oozie}} pgid. When pgid does not
> exist, check whether there is an error file. If the error file does not
> exist, {{Oozie}} will set action status {{OK}}
> However, when {{Oozie}} pgid is killed externally, this action will be
> incorrectly determined to be successful.
> *Solution*
> In ssh-wrapper.sh, when command execution is OK, {{Oozie}} should touch a
> success empty file like touching error file.
> In {{SshActionExecutor}} check method, Oozie should add checking the success
> file existence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3569) SSH Action should add checking success file
[
https://issues.apache.org/jira/browse/OOZIE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001608#comment-17001608
]
Junfan Zhang commented on OOZIE-3569:
-
Please review it. Thanks [~asalamon74]
> SSH Action should add checking success file
> ---
>
> Key: OOZIE-3569
> URL: https://issues.apache.org/jira/browse/OOZIE-3569
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3569-v1.patch
>
>
> *Phenomenon*
> Currently, {{SSH Action}} checking operation are as following:
> Firstly, check operation is to check {{Oozie}} pgid. When pgid does not
> exist, check whether there is an error file. If the error file does not
> exist, {{Oozie}} will set action status {{OK}}
> However, when {{Oozie}} pgid is killed externally, this action will be
> incorrectly determined to be successful.
> *Solution*
> In ssh-wrapper.sh, when command execution is OK, {{Oozie}} should touch a
> success empty file like touching error file.
> In {{SshActionExecutor}} check method, Oozie should add checking the success
> file existence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3569) SSH Action should add checking success file
[
https://issues.apache.org/jira/browse/OOZIE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated OOZIE-3569:
Attachment: OOZIE-3569-v1.patch
> SSH Action should add checking success file
> ---
>
> Key: OOZIE-3569
> URL: https://issues.apache.org/jira/browse/OOZIE-3569
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3569-v1.patch
>
>
> *Phenomenon*
> Currently, {{SSH Action}} checking operation are as following:
> Firstly, check operation is to check {{Oozie}} pgid. When pgid does not
> exist, check whether there is an error file. If the error file does not
> exist, {{Oozie}} will set action status {{OK}}
> However, when {{Oozie}} pgid is killed externally, this action will be
> incorrectly determined to be successful.
> *Solution*
> In ssh-wrapper.sh, when command execution is OK, {{Oozie}} should touch a
> success empty file like touching error file.
> In {{SshActionExecutor}} check method, Oozie should add checking the success
> file existence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3570) SSH Action execution command should add the mechanism of timeout
[
https://issues.apache.org/jira/browse/OOZIE-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000865#comment-17000865
]
Andras Salamon commented on OOZIE-3570:
---
It might be possible to come up with a unit test, but right now Oozie's ssh
unit tests are not working at all: OOZIE-3391
> SSH Action execution command should add the mechanism of timeout
>
>
> Key: OOZIE-3570
> URL: https://issues.apache.org/jira/browse/OOZIE-3570
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> *Phenomenon*
> Currently, In SSH action, When the remote machine is hung, execution command
> will waiting. This will cause the number of Oozie threads to be occupied,
> which will cause delays in scheduling other tasks.
> *Solution*
> SSH Action execution command should add timeout prefix. But the {{ssh -o
> ConnectTimeout=20}} is invalid
> {code:java}
> timeout 20s ssh xx
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000858#comment-17000858
]
Andras Salamon commented on OOZIE-3574:
---
[~zuston] [~matijhs] It seems to me you have found lots of scenarios where
Oozie is not working correctly if we try to use not only HDFS but also S3,
Azure... I think I'll open an umbrella Jira about the cloud improvement and
move all the Jira there as subtasks.
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (OOZIE-3576) [Umbrella] Make Oozie cloud ready
Andras Salamon created OOZIE-3576: - Summary: [Umbrella] Make Oozie cloud ready Key: OOZIE-3576 URL: https://issues.apache.org/jira/browse/OOZIE-3576 Project: Oozie Issue Type: Improvement Reporter: Andras Salamon Several problems have been reported in the last weeks about the cloud support of Oozie. Let's open an umbrella Jira and try to collect all of them here. Oozie was designed to work with HDFS when we try to use HDFS and S3, Azure at the same time we encounter problems. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3576) [Umbrella] Make Oozie cloud ready
[ https://issues.apache.org/jira/browse/OOZIE-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000860#comment-17000860 ] Andras Salamon commented on OOZIE-3576: --- CC [~matijhs] [~zuston] > [Umbrella] Make Oozie cloud ready > - > > Key: OOZIE-3576 > URL: https://issues.apache.org/jira/browse/OOZIE-3576 > Project: Oozie > Issue Type: Improvement >Reporter: Andras Salamon >Priority: Major > > Several problems have been reported in the last weeks about the cloud support > of Oozie. Let's open an umbrella Jira and try to collect all of them here. > Oozie was designed to work with HDFS when we try to use HDFS and S3, Azure at > the same time we encounter problems. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3574:
--
Parent: OOZIE-3576
Issue Type: Sub-task (was: Bug)
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-3575:
--
Parent: OOZIE-3576
Issue Type: Sub-task (was: Improvement)
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3573) Fix oozie-default.xml descriptions
[
https://issues.apache.org/jira/browse/OOZIE-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000849#comment-17000849
]
Andras Salamon commented on OOZIE-3573:
---
Thanks [~arvaiadam] this version is much better. Precommit also likes this.
Great typo findings.
A few more suggestions:
* "multiple class" should be "multiple classes" I think.
* For the "nameNode.whitelist" and "jobTracker.whitelist" we should also add
the information, that we accept comma separated values. Something like "Comma
separated list of whitelisted job trackers for Oozie service"
* I've also realized that the block-copy error is also block-copied to our
documentation:
[https://github.com/apache/oozie/blob/master/docs/src/site/markdown/AG_HadoopConfiguration.md]
Could you please also modify this file.
> Fix oozie-default.xml descriptions
> --
>
> Key: OOZIE-3573
> URL: https://issues.apache.org/jira/browse/OOZIE-3573
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Adam Arvai
>Priority: Minor
> Attachments: OOZIE-3573-001.patch, OOZIE-3573-002.patch
>
>
> The nameNode.whitelist section in
> [oozie-default.xml|https://github.com/apache/oozie/blob/master/core/src/main/resources/oozie-default.xml]
> contains the following section:
> {noformat}
>
> oozie.service.HadoopAccessorService.nameNode.whitelist
>
>
> Whitelisted job tracker for Oozie service.
>
>
> {noformat}
> This is clearly a block-copy error, description is copied from the previous
> {{jobTracker.whitelist}} section.
> We should fix it, maybe it would be useful to check the other description
> fields in the file as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000823#comment-17000823
]
Junfan Zhang commented on OOZIE-3575:
-
Maybe we need a bigger jira to adapt to other file systems, this jira is just
one of the subtasks. :P
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3570) SSH Action execution command should add the mechanism of timeout
[
https://issues.apache.org/jira/browse/OOZIE-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000822#comment-17000822
]
Junfan Zhang commented on OOZIE-3570:
-
Yes, I think unit tests cannot reproduce this problem. Do you have any good
ideas？[~asalamon74]
> SSH Action execution command should add the mechanism of timeout
>
>
> Key: OOZIE-3570
> URL: https://issues.apache.org/jira/browse/OOZIE-3570
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> *Phenomenon*
> Currently, In SSH action, When the remote machine is hung, execution command
> will waiting. This will cause the number of Oozie threads to be occupied,
> which will cause delays in scheduling other tasks.
> *Solution*
> SSH Action execution command should add timeout prefix. But the {{ssh -o
> ConnectTimeout=20}} is invalid
> {code:java}
> timeout 20s ssh xx
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000821#comment-17000821
]
Junfan Zhang commented on OOZIE-3574:
-
[~matijhs] Thanks your response. I will recheck the patch and upload it.
In addition, Why I found this problem, because we use aws {{S3}} to store
workflow xml, but {{Oozie}} sharelib related files are still stored in
{{HDFS}}. I found that Oozie cannot support this scenario, so I modified the
current Oozie code. If there is such a demand, can we open another jira to
solve this problem, how do you think? [~asalamon74].
*The problems we encountered are as follows:*
When the workflow xml address has no scheme and uses the s3a protocol by
default, {{Oozie}} is required to provide optional parameters to support
reading the workflow xml path from {{S3A}} instead of {{HDFS}}
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000820#comment-17000820
]
Mate Juhasz commented on OOZIE-3575:
The recently opened OOZIE-3574 is actually a bug, but its strongly connecting
to OOZIE-3575
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Issue Type: Improvement (was: Bug)
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000817#comment-17000817
]
Mate Juhasz commented on OOZIE-3575:
cc: [~gezapeti] [~asalamon74] [~dionusos]
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Summary: Add credential support for cloud file systems (was: Add
credential support for cloud file storages)
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (OOZIE-3575) Add credential support for cloud file storages
Mate Juhasz created OOZIE-3575:
--
Summary: Add credential support for cloud file storages
Key: OOZIE-3575
URL: https://issues.apache.org/jira/browse/OOZIE-3575
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 5.2.0
Reporter: Mate Juhasz
Assignee: Mate Juhasz
Fix For: trunk
Oozie by default gathers delegation tokens for the nodes defined in
_mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_ in
case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where
the job related resources, which we want to access in runtime are present on
different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for
different cloud storages.
*The following scenarios should be addressed:*
Oozie should obtain delegation token in case
* the defaultFs is cloud
* the workload.xml is in cloud
* input/output/auxiliary files referred from workflow are in cloud
* (newly introduced feature) user could define filesystem credentials for the
workflow (as its done with hive/hcat etc..) -> this would allow the user to
handle the situation where Oozie could not decide which tokens are needed at
launch time by default and could also get tokens for different cloud storages
and buckets as well
Example for credentials addition:
{noformat}
filesystem
s3a://qe-s3-bucket-mst
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000791#comment-17000791
]
Mate Juhasz commented on OOZIE-3574:
Thats a great finding [~zuston]! That FileSystem instance should be created
definitely with actionLibsPath instead of appPath.
Its probably a minor change in the below code part, but I did not check in
depths.
{code:java}
FileSystem fs =
Services.get().get(HadoopAccessorService.class).createFileSystem(user,
appPath.toUri(), conf);
{code}
This could be a further enhancement for making Oozie more cloud file system
friendly.
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000713#comment-17000713
]
Andras Salamon commented on OOZIE-3574:
---
Looks like a valid problem for me, but let's ping [~dionusos] [~matijhs] they
have more Oozie S3 experience.
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
