[jira] [Assigned] (TEZ-2161) Support CRDT aggregation models for counters
[ https://issues.apache.org/jira/browse/TEZ-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned TEZ-2161: - Assignee: Eric Wohlstadter > Support CRDT aggregation models for counters > - > > Key: TEZ-2161 > URL: https://issues.apache.org/jira/browse/TEZ-2161 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Eric Wohlstadter >Priority: Major > > Some counters such as last event received time need to be handled different > to say bytes read counters. Bytes reads requires a summation across all tasks > within a vertex. The received time requires doing a max() across all the > tasks. First event received time would likely need a min(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Success: TEZ-3874 PreCommit Build #2741
Jira: https://issues.apache.org/jira/browse/TEZ-3874
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2741/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 341.00 KB...]
[INFO]
[INFO] Total time: 56:40 min
[INFO] Finished at: 2018-03-08T23:05:49Z
[INFO] Final Memory: 84M/1392M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12913644/TEZ-3874.6.patch
against master revision 82d73b3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2741//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2741//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
==
==
Finished build.
==
==
Archiving artifacts
[Fast Archiver] Compressed 3.54 MB of artifacts by 14.1% relative to #2736
[description-setter] Description set: TEZ-3874
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[
https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391985#comment-16391985
]
Jonathan Eagles commented on TEZ-3900:
--
http://tez.apache.org/install.html
- Prefer tez tarball tarball install to hdfs and specify subdirectory so that
tez classpath is reliably after hive classpath. This avoids two versions of
dependencies in the same directory which is not defined what order they are
resolved in the classpath.
Example:
{noformat:title=tez-site.xml}
tez.lib.uris
${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz#tez
tez.lib.uris.classpath
./tez/*,./tez/lib/*
{noformat}
http://tez.apache.org/install.html
> upgrade to a recent guava version
> -
>
> Key: TEZ-3900
> URL: https://issues.apache.org/jira/browse/TEZ-3900
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3900.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3900) upgrade to a recent guava version
[ https://issues.apache.org/jira/browse/TEZ-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391955#comment-16391955 ] Sergey Shelukhin commented on TEZ-3900: --- Where can one view the best practices? :) We were deploying Guava 11 because of the dependency version in Tez, and are hitting issue in Hive due to lack of API compat (Hive is on 19). We are going to try deploying 19 for everything and see if it works. > upgrade to a recent guava version > - > > Key: TEZ-3900 > URL: https://issues.apache.org/jira/browse/TEZ-3900 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3900.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3904) an API to update tokens for Tez AM and the DAG
Sergey Shelukhin created TEZ-3904: - Summary: an API to update tokens for Tez AM and the DAG Key: TEZ-3904 URL: https://issues.apache.org/jira/browse/TEZ-3904 Project: Apache Tez Issue Type: Bug Reporter: Sergey Shelukhin Nothing is permanent in this world, lest of all delegation tokens. The current way around token expiration (the one where you cannot keep renewing anymore) in Hive when Tez AM is used in session mode is to cycle Tez AM. It may happen though that a query is running at that time, and so the AM cannot be restarted with new tokens. We let the query run its course and it usually dies because it tries to do something with an expired token. To get around that, we cycle AMs a few hours before tokens are going to expire. However, that is still not ideal because it puts an upper bound on safe Hive query runtime (a query longer than 3 hours with current config may fail due to an expired token if its timing is unlucky), and also precludes setting tokens to expire much faster than the standard 7-day time frame. There should be a mechanism to replace tokens in the AM, including for a running DAG. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
[
https://issues.apache.org/jira/browse/TEZ-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3874:
--
Attachment: TEZ-3874.6.patch
> NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in
> Configuration
>
>
> Key: TEZ-3874
> URL: https://issues.apache.org/jira/browse/TEZ-3874
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3874.1.patch, TEZ-3874.3.patch, TEZ-3874.4.patch,
> TEZ-3874.5.patch, TEZ-3874.6.patch, TEZ-3874.patch.2
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> "yarn.resourcemanager.zk-address" is deprecated in favor of
> "hadoop.zk.address" for Hadoop 2.9+.
> Configuration base class does't auto-translate the deprecation. Only
> YarnConfiguration applies the translation.
> In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
> "yarn.resourcemanager.zk-address" is present in the Configuration.
> {code}
> for (Entry entry : amConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(amConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
> Even though Tez is not specifically looking for the deprecated property,
> {{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
> any of the merged xml property resources.
> {{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
> will trigger NPE.
> Suggested solution is to change to:
> {code}
> YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
> for (Entry entry : wrappedConf) {
> PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
> kvp.setKey(entry.getKey());
> kvp.setValue(wrappedConf.get(entry.getKey()));
> builder.addConfKeyValues(kvp);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
