[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency
[ https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273384#comment-17273384 ] Martin Tzvetanov Grigorov commented on TEZ-4154: Hi [~albertkohl] ! You could use `git apply -p0 file` > tez-protobuf: a single module for handling protobuf dependency > -- > > Key: TEZ-4154 > URL: https://issues.apache.org/jira/browse/TEZ-4154 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, > TEZ-4154.04.patch > > > Similarly to what I've seen in TEZ-2164, I would like to create a module that > is a single point of control of how tez handles protobuf dependency. In the > near future, when tez will upgrade to hadoop 3.3, there will be a couple of > things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf > 3.x and handle the possible mess that can come from the fact that hadoop has > relocated its protobuf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4273) Clear off staging files when TezYarnClient is unable to submit applications
[
https://issues.apache.org/jira/browse/TEZ-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273331#comment-17273331
]
Hadoop QA commented on TEZ-4273:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
59s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
46s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/97/artifact/out/Dockerfile
|
| JIRA Issue | TEZ-4273 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019541/TEZ-4273.1.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 4abcdc348aff 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 04597c7b5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/97/testReport/ |
| Max. process+thread count | 257 (vs. ulimit of 5500) |
| modules | C: tez-api U: tez-api |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/97/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Clear off staging files when TezYarnClient is unable to submit applications
> ---
>
> Key: TEZ-4273
> URL: https://issues.apache.org/jira/browse/TEZ-4273
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
[jira] [Updated] (TEZ-4273) Clear off staging files when TezYarnClient is unable to submit applications
[
https://issues.apache.org/jira/browse/TEZ-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-4273:
--
Attachment: TEZ-4273.1.patch
> Clear off staging files when TezYarnClient is unable to submit applications
> ---
>
> Key: TEZ-4273
> URL: https://issues.apache.org/jira/browse/TEZ-4273
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: TEZ-4273.1.patch
>
>
> Currently it leaves behind few resources like "tez-conf.pb" etc when
> exception is encountered during app submission. This causes issues in
> cluster, when apps continue to submit queries continuously.
> {noformat}
> drwx-- - hive supergroup 0 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042
> -rw-r--r-- 3 hive supergroup 135519 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez-conf.pb
> -rw-r--r-- 3 hive supergroup 1056 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez.session.local-resources.pb
>
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1611791897439_0042 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has X applications from user hive cannot accept submission of application:
> application_1611791897439_0042
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322)
> ~[hadoop-yarn-client-3.x:?]
> at
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77)
> ~[tez-api-0.9.x.jar:0.9.x]
> at org.apache.tez.client.TezClient.start(TezClient.java:405)
> ~[tez-api-0.9.x.jar:0.9.x]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:535)
> ~[hive-exec-3.x]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4273) Clear off staging files when TezYarnClient is unable to submit applications
[
https://issues.apache.org/jira/browse/TEZ-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273262#comment-17273262
]
Rajesh Balamohan edited comment on TEZ-4273 at 1/28/21, 4:00 AM:
-
Lines of interest:
[https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L403]
[https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L470]
When exception happens in submitApplication, TezClient should clear off staging
directory.
was (Author: rajesh.balamohan):
Lines of interest:
[https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L1123]
[https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L470]
When exception happens in submitApplication, TezClient should clear off staging
directory.
> Clear off staging files when TezYarnClient is unable to submit applications
> ---
>
> Key: TEZ-4273
> URL: https://issues.apache.org/jira/browse/TEZ-4273
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Currently it leaves behind few resources like "tez-conf.pb" etc when
> exception is encountered during app submission. This causes issues in
> cluster, when apps continue to submit queries continuously.
> {noformat}
> drwx-- - hive supergroup 0 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042
> -rw-r--r-- 3 hive supergroup 135519 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez-conf.pb
> -rw-r--r-- 3 hive supergroup 1056 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez.session.local-resources.pb
>
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1611791897439_0042 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has X applications from user hive cannot accept submission of application:
> application_1611791897439_0042
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322)
> ~[hadoop-yarn-client-3.x:?]
> at
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77)
> ~[tez-api-0.9.x.jar:0.9.x]
> at org.apache.tez.client.TezClient.start(TezClient.java:405)
> ~[tez-api-0.9.x.jar:0.9.x]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:535)
> ~[hive-exec-3.x]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4273) Clear off staging files when TezYarnClient is unable to submit applications
[
https://issues.apache.org/jira/browse/TEZ-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273262#comment-17273262
]
Rajesh Balamohan edited comment on TEZ-4273 at 1/28/21, 3:59 AM:
-
Lines of interest:
[https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L1123]
[https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L470]
When exception happens in submitApplication, TezClient should clear off staging
directory.
was (Author: rajesh.balamohan):
Lines of interest:
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L403
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L408
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L470
When exception happens in submitApplication, TezClient should clear off staging
directory.
> Clear off staging files when TezYarnClient is unable to submit applications
> ---
>
> Key: TEZ-4273
> URL: https://issues.apache.org/jira/browse/TEZ-4273
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Currently it leaves behind few resources like "tez-conf.pb" etc when
> exception is encountered during app submission. This causes issues in
> cluster, when apps continue to submit queries continuously.
> {noformat}
> drwx-- - hive supergroup 0 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042
> -rw-r--r-- 3 hive supergroup 135519 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez-conf.pb
> -rw-r--r-- 3 hive supergroup 1056 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez.session.local-resources.pb
>
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1611791897439_0042 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has X applications from user hive cannot accept submission of application:
> application_1611791897439_0042
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322)
> ~[hadoop-yarn-client-3.x:?]
> at
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77)
> ~[tez-api-0.9.x.jar:0.9.x]
> at org.apache.tez.client.TezClient.start(TezClient.java:405)
> ~[tez-api-0.9.x.jar:0.9.x]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:535)
> ~[hive-exec-3.x]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4273) Clear off staging files when TezYarnClient is unable to submit applications
[
https://issues.apache.org/jira/browse/TEZ-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273262#comment-17273262
]
Rajesh Balamohan commented on TEZ-4273:
---
Lines of interest:
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L403
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L408
https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L470
When exception happens in submitApplication, TezClient should clear off staging
directory.
> Clear off staging files when TezYarnClient is unable to submit applications
> ---
>
> Key: TEZ-4273
> URL: https://issues.apache.org/jira/browse/TEZ-4273
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Currently it leaves behind few resources like "tez-conf.pb" etc when
> exception is encountered during app submission. This causes issues in
> cluster, when apps continue to submit queries continuously.
> {noformat}
> drwx-- - hive supergroup 0 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042
> -rw-r--r-- 3 hive supergroup 135519 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez-conf.pb
> -rw-r--r-- 3 hive supergroup 1056 2021-01-28 01:58
> /tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez.session.local-resources.pb
>
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1611791897439_0042 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has X applications from user hive cannot accept submission of application:
> application_1611791897439_0042
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322)
> ~[hadoop-yarn-client-3.x:?]
> at
> org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77)
> ~[tez-api-0.9.x.jar:0.9.x]
> at org.apache.tez.client.TezClient.start(TezClient.java:405)
> ~[tez-api-0.9.x.jar:0.9.x]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:535)
> ~[hive-exec-3.x]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TEZ-4273) Clear off staging files when TezYarnClient is unable to submit applications
Rajesh Balamohan created TEZ-4273:
-
Summary: Clear off staging files when TezYarnClient is unable to
submit applications
Key: TEZ-4273
URL: https://issues.apache.org/jira/browse/TEZ-4273
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
Currently it leaves behind few resources like "tez-conf.pb" etc when exception
is encountered during app submission. This causes issues in cluster, when apps
continue to submit queries continuously.
{noformat}
drwx-- - hive supergroup 0 2021-01-28 01:58
/tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042
-rw-r--r-- 3 hive supergroup 135519 2021-01-28 01:58
/tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez-conf.pb
-rw-r--r-- 3 hive supergroup 1056 2021-01-28 01:58
/tmp/hive/hive/_tez_session_dir/806cd302-abd0-4694-be4a-b2b2473e75a8/.tez/application_1611791897439_0042/tez.session.local-resources.pb
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1611791897439_0042 to YARN :
org.apache.hadoop.security.AccessControlException: Queue root.default already
has X applications from user hive cannot accept submission of application:
application_1611791897439_0042
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322)
~[hadoop-yarn-client-3.x:?]
at
org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77)
~[tez-api-0.9.x.jar:0.9.x]
at org.apache.tez.client.TezClient.start(TezClient.java:405)
~[tez-api-0.9.x.jar:0.9.x]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:535)
~[hive-exec-3.x]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4236) Investigate if DAGAppMaster's DAGClientServer is really need to be started/used in local mode
[ https://issues.apache.org/jira/browse/TEZ-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273255#comment-17273255 ] Ashutosh Chauhan commented on TEZ-4236: --- +1 > Investigate if DAGAppMaster's DAGClientServer is really need to be > started/used in local mode > - > > Key: TEZ-4236 > URL: https://issues.apache.org/jira/browse/TEZ-4236 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4236.01.patch, TEZ-4236.02.patch, TEZ-4236.03.patch, > TEZ-4236.04.patch > > > For a customer, we found that they try to run hive on tez local mode tests, > and due to strict firewall rules and security policy, the DagClientServer > struggles to start or respond to connections (think about corporate laptops > here). Instead of investigating the network-related workarounds, let's find > another way and decide if DAGAppMaster really needs this server. > LocalClient has an embedded > [DAGAppMaster|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L65], > so at the first sight, it doesn't make sense to rely on the network > environment in terms of AM usage. Hopefully, AM can become more lightweight > in the long term if we try to eliminate pointless network usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-3985) Correctness: Throw a clear exception for DMEs sent during cleanup
[ https://issues.apache.org/jira/browse/TEZ-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273204#comment-17273204 ] Rajesh Balamohan edited comment on TEZ-3985 at 1/27/21, 11:53 PM: -- Committed the patch. Thanks for the contribution [~jmarhuen] . Thanks [~gopalv] , [~ashutoshc] . was (Author: rajesh.balamohan): Thanks for the contribution [~jmarhuen] . Thanks [~gopalv] , [~ashutoshc] . > Correctness: Throw a clear exception for DMEs sent during cleanup > - > > Key: TEZ-3985 > URL: https://issues.apache.org/jira/browse/TEZ-3985 > Project: Apache Tez > Issue Type: Bug >Reporter: Gopal Vijayaraghavan >Assignee: Jaume M >Priority: Major > Fix For: 0.10.1 > > Attachments: TEZ-3985.1.patch, TEZ-3985.2.patch, TEZ-3985.3.patch, > TEZ-3985.3.patch, TEZ-3985.4.patch, TEZ-3985.5.patch, TEZ-3985.6.patch, > TEZ-3985.7.patch > > > If a DME is sent during cleanup, that implies that the .close() of the > LogicalIOProcessorRuntimeTask did not succeed and therefore these events are > an error condition. > These events should not be sent and more importantly should be received by > the AM. > Throw a clear exception, in case of this & allow the developers to locate the > extraneous event from the backtrace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-3985) Correctness: Throw a clear exception for DMEs sent during cleanup
[ https://issues.apache.org/jira/browse/TEZ-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273197#comment-17273197 ] Ashutosh Chauhan commented on TEZ-3985: --- +1 > Correctness: Throw a clear exception for DMEs sent during cleanup > - > > Key: TEZ-3985 > URL: https://issues.apache.org/jira/browse/TEZ-3985 > Project: Apache Tez > Issue Type: Bug >Reporter: Gopal Vijayaraghavan >Assignee: Jaume M >Priority: Major > Attachments: TEZ-3985.1.patch, TEZ-3985.2.patch, TEZ-3985.3.patch, > TEZ-3985.3.patch, TEZ-3985.4.patch, TEZ-3985.5.patch, TEZ-3985.6.patch, > TEZ-3985.7.patch > > > If a DME is sent during cleanup, that implies that the .close() of the > LogicalIOProcessorRuntimeTask did not succeed and therefore these events are > an error condition. > These events should not be sent and more importantly should be received by > the AM. > Throw a clear exception, in case of this & allow the developers to locate the > extraneous event from the backtrace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4266) Make Tez Inherit From Apache POM
[
https://issues.apache.org/jira/browse/TEZ-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273008#comment-17273008
]
Tez CI commented on TEZ-4266:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} tez-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} root generated 0 new + 4 unchanged - 34 fixed = 4
total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} tez-dist in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 30s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-89/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/89 |
| JIRA Issue | TEZ-4266 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile |
| uname | Linux 0e1db26fe74a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 7374b69ed |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| unit |
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-89/4/artifact/out/patch-unit-root.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-89/4/testReport/ |
| Max. process+thread count | 1678 (vs. ulimit of 5500) |
| modules | C: tez-dist . U: . |
| Console output |
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-89/4/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Make Tez Inherit From Apache POM
>
>
> Key: TEZ-4266
> URL: https://issues.apache.org/jira/browse/TEZ-4266
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Moll
[jira] [Created] (TEZ-4272) Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED org.apache.hadoop.service.ServiceStateException: java.lang.ExceptionInInitializerError
Nomen'Aina created TEZ-4272: --- Summary: Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED org.apache.hadoop.service.ServiceStateException: java.lang.ExceptionInInitializerError Key: TEZ-4272 URL: https://issues.apache.org/jira/browse/TEZ-4272 Project: Apache Tez Issue Type: Bug Affects Versions: 0.10.0, 0.9.2 Reporter: Nomen'Aina Attachments: log.txt Hello to all. Currently I am trying to install Apache tez by following step by step, the instructions on the official website, and when I run the command: $HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount /tmp/input/file.txt /tmp/outout I get the following errors: Service org.apache.tez.dag.app.rm.TaskSchedulerManager failed in state STARTED java.lang.NullPointerException at org.apache.tez.dag.app.rm.TaskSchedulerManager.serviceStart(TaskSchedulerManager.java:653) and Error starting DAGAppMaster org.apache.hadoop.service.ServiceStateException: java.lang.ExceptionInInitializerError hadoop version:3.3.0 / tez version: 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4240) Remove SHA-256 from Tez
[
https://issues.apache.org/jira/browse/TEZ-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272688#comment-17272688
]
Hadoop QA commented on TEZ-4240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
35s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/96/artifact/out/Dockerfile
|
| JIRA Issue | TEZ-4240 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019477/TEZ-4240.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 9f0f0f54b390 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 7374b69ed |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/96/testReport/ |
| Max. process+thread count | 215 (vs. ulimit of 5500) |
| modules | C: tez-api U: tez-api |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/96/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Remove SHA-256 from Tez
> ---
>
> Key: TEZ-4240
> URL: https://issues.apache.org/jira/browse/TEZ-4240
> Project: Apache Tez
> Issue Type: Improvement
>Affects Versions: 0.9.2
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.10.1, 0.9.3
>
>
[jira] [Updated] (TEZ-4240) Remove SHA-256 from Tez
[ https://issues.apache.org/jira/browse/TEZ-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4240: -- Attachment: TEZ-4240.02.patch > Remove SHA-256 from Tez > --- > > Key: TEZ-4240 > URL: https://issues.apache.org/jira/browse/TEZ-4240 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.2 >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.1, 0.9.3 > > Attachments: TEZ-4240.01.patch, TEZ-4240.02.patch > > > SHA-256 is being deprecated, and it's recommended to be replaced by at least > SHA-384. Tez uses SHA-256 for resource validation, and even if it doesn't > seem to be a direct vulnerability, it's better to upgrade it in order to > remain FIPS compliant. -- This message was sent by Atlassian Jira (v8.3.4#803005)
