Failed: TEZ-3901 PreCommit Build #2738
Jira: https://issues.apache.org/jira/browse/TEZ-3901
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2738/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4.55 KB...]
patching file tez-tools/analyzers/pom.xml
==
==
Determining number of patched javac warnings.
==
==
/home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -Ptest-patch >
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/../patchprocess/patchJavacWarnings.txt
2>&1
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913341/TEZ-3901.1.patch
against master revision c34e46c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2738//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Updated] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3901: -- Attachment: TEZ-3901.1.patch > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
Eric Wohlstadter created TEZ-3901: - Summary: Add hadoop3 profile for upgrade to Jersey 1.19 Key: TEZ-3901 URL: https://issues.apache.org/jira/browse/TEZ-3901 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter >From [~harishjp]: "DAGAppMaster fails to start when using hadoop3 and ATSv15, because TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez packages jersey-client 1.9 with it. There are incompatible changes between them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in older hadoop and 1.19 in hadoop3." This patch includes some copy and paste of the hadoop28 profile to a hadoop3 profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389039#comment-16389039 ] Eric Wohlstadter commented on TEZ-3900: --- [~sershe] Tez is still defaulting to Hadoop 2.7.0. I've tested Tez with Guava 19.0 on Hadoop 3.0.0, so there should be no problem there. Do you think there is any reason Guava 19.0 would conflict with Hadoop 2.7.x, 2.8.x? If not, LGTM. If it might cause conflicts, we can put Guava deps into version profiles, i.e. -Phadoop28, -Phadoop3, etc.. > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389023#comment-16389023
]
Eric Wohlstadter commented on TEZ-3874:
---
[~jlowe]
Thanks again for the feedback. I think I've addressed it TEZ-3874.5.patch.
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
>
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3874.1.patch, TEZ-3874.3.patch, TEZ-3874.4.patch,
> TEZ-3874.5.patch, TEZ-3874.patch.2
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389018#comment-16389018
]
Eric Wohlstadter commented on TEZ-3874:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913265/TEZ-3874.5.patch
against master revision c34e46c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2736//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2736//console
This message is automatically generated.
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
>
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3874.1.patch, TEZ-3874.3.patch, TEZ-3874.4.patch,
> TEZ-3874.5.patch, TEZ-3874.patch.2
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: TEZ-3900 PreCommit Build #2737
Jira: https://issues.apache.org/jira/browse/TEZ-3900
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2737/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.43 KB...]
[INFO] Final Memory: 96M/1384M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913310/TEZ-3900.patch
against master revision c34e46c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2737//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2737//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] Compressed 3.53 MB of artifacts by 10.6% relative to #2736
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388901#comment-16388901 ] Sergey Shelukhin commented on TEZ-3900: --- [~ewohlstadter] [~hagleitn] can you take a look? > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-3900: -- Attachment: TEZ-3900.patch > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3900) upgrade to a recent guava version
Sergey Shelukhin created TEZ-3900: - Summary: upgrade to a recent guava version Key: TEZ-3900 URL: https://issues.apache.org/jira/browse/TEZ-3900 Project: Apache Tez Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Success: TEZ-3874 PreCommit Build #2736
Jira: https://issues.apache.org/jira/browse/TEZ-3874
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2736/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.38 KB...]
[INFO]
[INFO] Total time: 55:57 min
[INFO] Finished at: 2018-03-06T22:24:33Z
[INFO] Final Memory: 90M/1281M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913265/TEZ-3874.5.patch
against master revision c34e46c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2736//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2736//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
==
==
Finished build.
==
==
Archiving artifacts
[Fast Archiver] Compressed 3.53 MB of artifacts by 11.5% relative to #2734
[description-setter] Description set: TEZ-3874
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3874:
--
Attachment: TEZ-3874.5.patch
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
>
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3874.1.patch, TEZ-3874.3.patch, TEZ-3874.4.patch,
> TEZ-3874.5.patch, TEZ-3874.patch.2
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: TEZ-3874 PreCommit Build #2735
Jira: https://issues.apache.org/jira/browse/TEZ-3874
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2735/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4.97 KB...]
patching file tez-api/src/main/java/org/apache/tez/dag/api/DAG.java
patching file tez-api/src/test/java/org/apache/tez/client/TestTezClient.java
patching file tez-common/src/test/java/org/apache/tez/common/TestTezUtils.java
==
==
Determining number of patched javac warnings.
==
==
/home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -Ptest-patch >
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/../patchprocess/patchJavacWarnings.txt
2>&1
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913263/TEZ-3874.4.patch
against master revision c34e46c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2735//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Updated] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3874:
--
Attachment: TEZ-3874.4.patch
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
>
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3874.1.patch, TEZ-3874.3.patch, TEZ-3874.4.patch,
> TEZ-3874.patch.2
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3892) getClient API for TezClient
[
https://issues.apache.org/jira/browse/TEZ-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388451#comment-16388451
]
Gopal V commented on TEZ-3892:
--
LGTM - +1
> getClient API for TezClient
> ---
>
> Key: TEZ-3892
> URL: https://issues.apache.org/jira/browse/TEZ-3892
> Project: Apache Tez
> Issue Type: New Feature
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3892.1.patch, TEZ-3892.2.patch
>
>
> This is a proposed opt-in feature.
> Tez AM already supports long-lived sessions, if desired a AM session can live
> indefinitely.
> However, new clients cannot connect to a long-lived AM session through the
> standard TezClient API.
> TezClient API only provides a "start" method to initiate a connection, which
> always allocates a new AM from YARN.
> # For interactive BI use-cases, this startup time can be significant.
> # Hive is implementing a HiveServer2 High Availability feature.
> ** When the singleton HS2 master server fails, the HS2 client is quickly
> redirected to a pre-warmed HS2 backup.
> # For the failover to complete quickly end-to-end, a Tez AM must also be
> pre-warmed and ready to accept connections.
> For more information, see design for:
> https://issues.apache.org/jira/browse/HIVE-18281.
>
> Anticipated changes:
> # A getClient{{(ApplicationId)}} method is added to TezClient. The
> functionality is similar to {{start}}
> ** Code related to launching a new AM from the RM is factored out.
> ** Since {{start}} and getClient will share some code, this code is
> refactored into reusable helper methods.
> ** A usage example is added to {{org/apache/tez/examples}}
> # It is not a goal of this JIRA to ensure that running Tez DAGs can be
> recovered by a client using the getClient API. The goal is only for
> maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3892) getClient API for TezClient
[
https://issues.apache.org/jira/browse/TEZ-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388447#comment-16388447
]
Sergey Shelukhin commented on TEZ-3892:
---
+1, but I'm also not a Tez committer. cc [~hagleitn] can you +1?
> getClient API for TezClient
> ---
>
> Key: TEZ-3892
> URL: https://issues.apache.org/jira/browse/TEZ-3892
> Project: Apache Tez
> Issue Type: New Feature
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3892.1.patch, TEZ-3892.2.patch
>
>
> This is a proposed opt-in feature.
> Tez AM already supports long-lived sessions, if desired a AM session can live
> indefinitely.
> However, new clients cannot connect to a long-lived AM session through the
> standard TezClient API.
> TezClient API only provides a "start" method to initiate a connection, which
> always allocates a new AM from YARN.
> # For interactive BI use-cases, this startup time can be significant.
> # Hive is implementing a HiveServer2 High Availability feature.
> ** When the singleton HS2 master server fails, the HS2 client is quickly
> redirected to a pre-warmed HS2 backup.
> # For the failover to complete quickly end-to-end, a Tez AM must also be
> pre-warmed and ready to accept connections.
> For more information, see design for:
> https://issues.apache.org/jira/browse/HIVE-18281.
>
> Anticipated changes:
> # A getClient{{(ApplicationId)}} method is added to TezClient. The
> functionality is similar to {{start}}
> ** Code related to launching a new AM from the RM is factored out.
> ** Since {{start}} and getClient will share some code, this code is
> refactored into reusable helper methods.
> ** A usage example is added to {{org/apache/tez/examples}}
> # It is not a goal of this JIRA to ensure that running Tez DAGs can be
> recovered by a client using the getClient API. The goal is only for
> maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3892) getClient API for TezClient
[
https://issues.apache.org/jira/browse/TEZ-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388367#comment-16388367
]
Prasanth Jayachandran commented on TEZ-3892:
changes look good to me.. would be good to have someone for tez take another
stab..
> getClient API for TezClient
> ---
>
> Key: TEZ-3892
> URL: https://issues.apache.org/jira/browse/TEZ-3892
> Project: Apache Tez
> Issue Type: New Feature
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3892.1.patch, TEZ-3892.2.patch
>
>
> This is a proposed opt-in feature.
> Tez AM already supports long-lived sessions, if desired a AM session can live
> indefinitely.
> However, new clients cannot connect to a long-lived AM session through the
> standard TezClient API.
> TezClient API only provides a "start" method to initiate a connection, which
> always allocates a new AM from YARN.
> # For interactive BI use-cases, this startup time can be significant.
> # Hive is implementing a HiveServer2 High Availability feature.
> ** When the singleton HS2 master server fails, the HS2 client is quickly
> redirected to a pre-warmed HS2 backup.
> # For the failover to complete quickly end-to-end, a Tez AM must also be
> pre-warmed and ready to accept connections.
> For more information, see design for:
> https://issues.apache.org/jira/browse/HIVE-18281.
>
> Anticipated changes:
> # A getClient{{(ApplicationId)}} method is added to TezClient. The
> functionality is similar to {{start}}
> ** Code related to launching a new AM from the RM is factored out.
> ** Since {{start}} and getClient will share some code, this code is
> refactored into reusable helper methods.
> ** A usage example is added to {{org/apache/tez/examples}}
> # It is not a goal of this JIRA to ensure that running Tez DAGs can be
> recovered by a client using the getClient API. The goal is only for
> maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
