[jira] [Updated] (TEZ-3375) Logging with slf4j way instead of "+" to reduce logging overhead
[
https://issues.apache.org/jira/browse/TEZ-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3375:
Description: Currently, Tez ATSv1.5 uses concatenation of Strings instead
using placeholders {} which slfj4 provides. It increases the cost of creating
objects.
> Logging with slf4j way instead of "+" to reduce logging overhead
>
>
> Key: TEZ-3375
> URL: https://issues.apache.org/jira/browse/TEZ-3375
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Tsuyoshi Ozawa
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3375.001.patch
>
>
> Currently, Tez ATSv1.5 uses concatenation of Strings instead using
> placeholders {} which slfj4 provides. It increases the cost of creating
> objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3375) Logging with slf4j way instead of "+" to reduce logging overhead
[ https://issues.apache.org/jira/browse/TEZ-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3375: Attachment: TEZ-3375.001.patch Attaching a patch. > Logging with slf4j way instead of "+" to reduce logging overhead > > > Key: TEZ-3375 > URL: https://issues.apache.org/jira/browse/TEZ-3375 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Tsuyoshi Ozawa >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3375.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3375) Logging with slf4j way instead of "+" to reduce logging overhead
Tsuyoshi Ozawa created TEZ-3375: --- Summary: Logging with slf4j way instead of "+" to reduce logging overhead Key: TEZ-3375 URL: https://issues.apache.org/jira/browse/TEZ-3375 Project: Apache Tez Issue Type: Sub-task Reporter: Tsuyoshi Ozawa Assignee: Tsuyoshi Ozawa -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393231#comment-15393231
]
Hitesh Shah commented on TEZ-3359:
--
Comments:
{code}
} else {
1063 // Validate and set value from tezConf.
1064 logLevel =
tezConf.get(TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL);
1065 if (logLevel != null) {
1066if (!HistoryLogLevel.validateLogLevel(logLevel)) {
1067 throw new IllegalArgumentException(
1068 "Config: " +
TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL +
1069 " is set to invalid value: " + logLevel);
1070}
1071PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
1072kvp.setKey(TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL);
1073kvp.setValue(logLevel);
1074confProtoBuilder.addConfKeyValues(kvp);
1075 }
1076}
{code}
- Shouldnt only validation be needed here given that the DAG will fall back
to the log level from the AM conf if there is no config set at the dag level (
i.e. no need to copy the am config value into the dag conf)?
{code}
// Set value in dag, should override tez conf value.
404 dag.setHistoryLogLevel(HistoryLogLevel.VERTEX);
405 dagPlan = dag.createDag(tezConf, null, null, null, false);
406 Assert.assertEquals("VERTEX",
findKVP(DAGPlan.newBuilder(dagPlan).getDagConf(),
407 TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL));
{code}
- could be augmented with
dag.setConf(TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL, "invalid") to test
all code paths
Patch mostly looks good.
Also, noticed that grouping of dags does not account for dag ids starting from
1 instead of 0. Maybe another sub task for that?
> Add granular log levels for HistoryLoggingService.
> --
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch,
> TEZ-3359.04.patch, TEZ-3359.05.patch, TEZ-3359.06.patch, TEZ-3359.07.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by having more granular control over the
> loglevel. Ex: disable task level logs, or log only until VERTEX and so on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
[ https://issues.apache.org/jira/browse/TEZ-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3374: --- Assignee: Tsuyoshi Ozawa > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. > - > > Key: TEZ-3374 > URL: https://issues.apache.org/jira/browse/TEZ-3374 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3374.001.patch > > > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from > tez.history.timeline.num-dags-per-group to > tez.history.logging.timeline.num-dags-per-group, so that its consistent with > the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
[ https://issues.apache.org/jira/browse/TEZ-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3374: Attachment: TEZ-3374.001.patch > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. > - > > Key: TEZ-3374 > URL: https://issues.apache.org/jira/browse/TEZ-3374 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > Attachments: TEZ-3374.001.patch > > > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from > tez.history.timeline.num-dags-per-group to > tez.history.logging.timeline.num-dags-per-group, so that its consistent with > the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
[ https://issues.apache.org/jira/browse/TEZ-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393216#comment-15393216 ] Tsuyoshi Ozawa commented on TEZ-3374: - Attaching first patch. > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. > - > > Key: TEZ-3374 > URL: https://issues.apache.org/jira/browse/TEZ-3374 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > Attachments: TEZ-3374.001.patch > > > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from > tez.history.timeline.num-dags-per-group to > tez.history.logging.timeline.num-dags-per-group, so that its consistent with > the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[ https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3359: --- Attachment: TEZ-3359.07.patch Created new patch with --no-prefix > Add granular log levels for HistoryLoggingService. > -- > > Key: TEZ-3359 > URL: https://issues.apache.org/jira/browse/TEZ-3359 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch, > TEZ-3359.04.patch, TEZ-3359.05.patch, TEZ-3359.06.patch, TEZ-3359.07.patch > > > We publish too many events to ATS, this increases the file size for ATS. > Reduce data size logged into ATS by having more granular control over the > loglevel. Ex: disable task level logs, or log only until VERTEX and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
Harish Jaiprakash created TEZ-3374: -- Summary: Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. Key: TEZ-3374 URL: https://issues.apache.org/jira/browse/TEZ-3374 Project: Apache Tez Issue Type: Sub-task Reporter: Harish Jaiprakash Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from tez.history.timeline.num-dags-per-group to tez.history.logging.timeline.num-dags-per-group, so that its consistent with the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[ https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3359: --- Attachment: TEZ-3359.06.patch Attaching the correct patch. Fixing bugs and tests pointed out by [~hitesh]. > Add granular log levels for HistoryLoggingService. > -- > > Key: TEZ-3359 > URL: https://issues.apache.org/jira/browse/TEZ-3359 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch, > TEZ-3359.04.patch, TEZ-3359.05.patch, TEZ-3359.06.patch > > > We publish too many events to ATS, this increases the file size for ATS. > Reduce data size logged into ATS by having more granular control over the > loglevel. Ex: disable task level logs, or log only until VERTEX and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3317 PreCommit Build #1878
Jira: https://issues.apache.org/jira/browse/TEZ-3317
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1878/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4138 lines...]
[INFO] Total time: 58:28 min
[INFO] Finished at: 2016-07-26T00:12:05+00:00
[INFO] Final Memory: 78M/1348M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819974/TEZ-3317.001.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1878//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1878//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 707: /home/jenkins/tools/jiracli/latest/jira: Permission denied
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 708: /home/jenkins/tools/jiracli/latest/jira: Permission denied
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.28 MB of artifacts by 29.5% relative to #1875
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392831#comment-15392831
]
Chris K Wensel commented on TEZ-3369:
-
Cascading simply needs is a way to navigate status/counters from a DAG, to a
Vertex to all the Tasks for the Vertex.
The method `getVertexID` only exists because it isn't available on the
`VertexStatus` object and is required to find the Vertex children (Tasks) via
ATS.
We would very much prefer to not go through the ATS if there is a more
efficient route, but have the fallback option to ATS of the current dag manager
is no longer around. see TEZ-2981 for further information.
> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID,
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That
> way the cascading code wouldn't need updating if the uri changed / json
> format changed, it would end up being updated in these clients as well. I
> suspect adding this to DAGClient would require more work as it'll also need
> to be supported by the RPCClient and I don't think there are the relevant
> protos and such available.
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient
> (currently it just implements Closeable). This will not require any changes
> on the Cascading side as DAGClientTimelineImpl will continue to be a
> DAGClient.
> Full stack trace:
> {code}
> Exception in thread "flow
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
>
[jira] [Commented] (TEZ-3338) Support classloader isolation
[ https://issues.apache.org/jira/browse/TEZ-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392801#comment-15392801 ] Siddharth Seth commented on TEZ-3338: - bq. No. It doesn't need to be. But we should be able to make a decision what needs to be user classpath and what should be system classpath. Specifically it would be changes in TezYARNUtils.getFrameworkClasspath() and code around it (we would have another environment variable for the user classpath). Hope that makes sense. The main reason I say this is that user's can setup the classpath and provide archives which could contain files in locations other than the single user classpath. I guess it'll work as long as the user classpath is setup under this one location as the base directory. > Support classloader isolation > - > > Key: TEZ-3338 > URL: https://issues.apache.org/jira/browse/TEZ-3338 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma > > HADOOP-10893 and MAPREDUCE-1700 provide classloader isolation at both client > side and container side for MR. We should add the same support for Tez. Given > we use hadoop command to launch Tez, it appears the client side has been > taken care of. Only the container side support is needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3360) Tez Custom Shuffle Handler Documentation
[ https://issues.apache.org/jira/browse/TEZ-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3360: - Attachment: TEZ-3360.2.patch > Tez Custom Shuffle Handler Documentation > > > Key: TEZ-3360 > URL: https://issues.apache.org/jira/browse/TEZ-3360 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3360.1.patch, TEZ-3360.2.patch > > > Provide setup instructions and API documentation for the Tez Custom Shuffle > Handler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392723#comment-15392723
]
Piyush Narang commented on TEZ-3369:
[~hitesh] thanks for getting back. Based on what I understand after short
forays into the Cascading code, the extra APIs that Cascading is using right
now are in
[TimelineClient|https://github.com/cwensel/cascading/blob/wip-3.2/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java]:
{code}
public interface TimelineClient
{
String getVertexID( String vertexName ) throws IOException, TezException;
Iterator getVertexChildren( String vertexID, int limit, String
startTaskID ) throws IOException, TezException;
TaskStatus getVertexChild( String taskID ) throws TezException;
}
{code}
They have an implementation
[TezTimelineClient|https://github.com/cwensel/cascading/blob/wip-3.2/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
that implements this interface as well as the DAGClient interface. For the
DAGClient APIs, they mostly seem to defer to the dag client object that is
passed in (I think this is returned from Tez). For the TimelineClient calls
(getVertexID, getVertexChildren, getVertexChild) they end up using the http
based endpoints taking advantage of the code in DAGClientTimelineImpl (which
implements the http client in Tez).
If its just these 3 calls, do you have a sense of how long it would take to
implement (I'm not too familiar with the rpc implementation)? One option is to
go ahead with my suggested approach 2 (making DAGClientInternal extend
DAGClient) and file a follow up jira so that we can track doing this the right
way and still be unblocked?
If someone with more Cascading chops ([~cwensel]?) could chime in if I'm
missing something / if there's other APIs that Cascading wants apart from these
3, that would be great. I can also spin up a new thread on Cascading dev where
we could ask this.
> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
>
[jira] [Comment Edited] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392645#comment-15392645
]
Hitesh Shah edited comment on TEZ-3369 at 7/25/16 8:52 PM:
---
Some general background info ( not having looked at the patch yet ):
- DAGClient is a user-facing API.
- DAGClientImpl is the impl for this user-facing API.
- DAGClientInternal was created as an abtract class for providing different
impls i.e. one which talks to the AM and the other which talks to ATS/Timeline.
Keeping DAGClientInternal separate from DAGClient was intentional due its
internal facing nature.
was (Author: hitesh):
Some general background info ( not having looked at the patch yet ):
- DAGClient is a user-facing API.
- DAGClientImpl is the impl for this user-facing API.
- DAGClientInternal was created as an abtract class for providing different
impls i.e. one which talks to the AM and the other which talks to ATS. Keeping
DAGClientInternal separate from DAGClient was intentional due its internal
facing nature.
> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID,
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That
> way the cascading code wouldn't need updating if the uri changed / json
> format changed, it would end up being updated in these clients as well. I
> suspect adding this to DAGClient would require more work as it'll also need
> to be supported by the RPCClient and I don't think there are the relevant
> protos and such available.
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient
> (currently it just implements Closeable).
[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392653#comment-15392653
]
Hitesh Shah commented on TEZ-3369:
--
The last time we had a similar discussion around the Cascading integration, we
had asked for a bunch of APIs that the Cascading folks were using ( as some of
them were marked @private ) to come up with a potential list to make part of
the base framework APIs.
This seems like a good opportunity to re-look at that and see which APIs make
sense to add and maybe at the same time look at which APIs Cascading is using
and make a call on ensuring better support for them or changing Cascading to
not use them.
> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID,
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That
> way the cascading code wouldn't need updating if the uri changed / json
> format changed, it would end up being updated in these clients as well. I
> suspect adding this to DAGClient would require more work as it'll also need
> to be supported by the RPCClient and I don't think there are the relevant
> protos and such available.
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient
> (currently it just implements Closeable). This will not require any changes
> on the Cascading side as DAGClientTimelineImpl will continue to be a
> DAGClient.
> Full stack trace:
> {code}
> Exception in thread "flow
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
>
[jira] [Commented] (TEZ-3369) Fixing Tez's DAGClients to work with Cascading
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392645#comment-15392645
]
Hitesh Shah commented on TEZ-3369:
--
Some general background info ( not having looked at the patch yet ):
- DAGClient is a user-facing API.
- DAGClientImpl is the impl for this user-facing API.
- DAGClientInternal was created as an abtract class for providing different
impls i.e. one which talks to the AM and the other which talks to ATS. Keeping
DAGClientInternal separate from DAGClient was intentional due its internal
facing nature.
> Fixing Tez's DAGClients to work with Cascading
> --
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID,
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That
> way the cascading code wouldn't need updating if the uri changed / json
> format changed, it would end up being updated in these clients as well. I
> suspect adding this to DAGClient would require more work as it'll also need
> to be supported by the RPCClient and I don't think there are the relevant
> protos and such available.
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient
> (currently it just implements Closeable). This will not require any changes
> on the Cascading side as DAGClientTimelineImpl will continue to be a
> DAGClient.
> Full stack trace:
> {code}
> Exception in thread "flow
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> at
[jira] [Updated] (TEZ-3360) Tez Custom Shuffle Handler Documentation
[ https://issues.apache.org/jira/browse/TEZ-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3360: - Attachment: TEZ-3360.1.patch > Tez Custom Shuffle Handler Documentation > > > Key: TEZ-3360 > URL: https://issues.apache.org/jira/browse/TEZ-3360 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3360.1.patch > > > Provide setup instructions and API documentation for the Tez Custom Shuffle > Handler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3360) Tez Custom Shuffle Handler Documentation
[ https://issues.apache.org/jira/browse/TEZ-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned TEZ-3360: Assignee: Jonathan Eagles > Tez Custom Shuffle Handler Documentation > > > Key: TEZ-3360 > URL: https://issues.apache.org/jira/browse/TEZ-3360 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > > Provide setup instructions and API documentation for the Tez Custom Shuffle > Handler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3359 PreCommit Build #1877
Jira: https://issues.apache.org/jira/browse/TEZ-3359
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1877/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 86 lines...]
patching file tez-api/src/test/java/org/apache/tez/dag/api/TestDAG.java
patching file
tez-dag/src/main/java/org/apache/tez/dag/history/HistoryEventHandler.java
patching file
tez-dag/src/main/java/org/apache/tez/dag/history/HistoryEventType.java
patching file
tez-plugins/tez-yarn-timeline-history-with-fs/src/test/java/org/apache/tez/dag/history/ats/acls/TestATSHistoryV15.java
==
==
Determining number of patched javac warnings.
==
==
/home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -Ptest-patch >
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/../patchprocess/patchJavacWarnings.txt
2>&1
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819990/TEZ-3359.05.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1877//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 707: /home/jenkins/tools/jiracli/latest/jira: Permission denied
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 708: /home/jenkins/tools/jiracli/latest/jira: Permission denied
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 756.66 KB of artifacts by 46.5% relative to #1875
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Updated] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[ https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3359: --- Description: We publish too many events to ATS, this increases the file size for ATS. Reduce data size logged into ATS by having more granular control over the loglevel. Ex: disable task level logs, or log only until VERTEX and so on. (was: We publish too many events to ATS, this increases the file size for ATS. Reduce data size logged into ATS by: * Having a more granular control over the loglevel, disable task level logs, or all logs and so on. * Disable logging counters.) > Add granular log levels for HistoryLoggingService. > -- > > Key: TEZ-3359 > URL: https://issues.apache.org/jira/browse/TEZ-3359 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch, > TEZ-3359.04.patch, TEZ-3359.05.patch > > > We publish too many events to ATS, this increases the file size for ATS. > Reduce data size logged into ATS by having more granular control over the > loglevel. Ex: disable task level logs, or log only until VERTEX and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392422#comment-15392422
]
Hitesh Shah edited comment on TEZ-3359 at 7/25/16 6:16 PM:
---
Comments:
{code}
// Copy historyLogLevel from tezConf into dagConf if its not overridden in
dagConf.
1053String logLevel =
this.dagConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1054if (logLevel == null) {
1055 logLevel = tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1056}
if (logLevel != null) {
1058 if (!HistoryLogLevel.validateLogLevel(logLevel)) {
1059throw new IllegalArgumentException("Config: " +
TezConfiguration.TEZ_HISTORY_LOGLEVEL +
1060" is set to invalid value: " + logLevel);
1061 }
1062 PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
1063 kvp.setKey(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1064 kvp.setValue(tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL));
1065 confProtoBuilder.addConfKeyValues(kvp);
1066}
{code}
- why is the AM log level overriding the dag conf provided level? line 1064?
- Please enhance unit tests to catch this bug.
Patch 05 seems incomplete. New files missing?
was (Author: hitesh):
Comments:
{code}
// Copy historyLogLevel from tezConf into dagConf if its not overridden in
dagConf.
1053String logLevel =
this.dagConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1054if (logLevel == null) {
1055 logLevel = tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1056}
if (logLevel != null) {
1058 if (!HistoryLogLevel.validateLogLevel(logLevel)) {
1059throw new IllegalArgumentException("Config: " +
TezConfiguration.TEZ_HISTORY_LOGLEVEL +
1060" is set to invalid value: " + logLevel);
1061 }
1062 PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
1063 kvp.setKey(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1064 kvp.setValue(tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL));
1065 confProtoBuilder.addConfKeyValues(kvp);
1066}
{code}
- why is the AM log level overriding the dag conf provided level? line 1064?
- Please enhance unit tests to catch this bug.
Patch 05 seems incomplete.
> Add granular log levels for HistoryLoggingService.
> --
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch,
> TEZ-3359.04.patch, TEZ-3359.05.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs,
> or all logs and so on.
> * Disable logging counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392422#comment-15392422
]
Hitesh Shah commented on TEZ-3359:
--
Comments:
{code}
// Copy historyLogLevel from tezConf into dagConf if its not overridden in
dagConf.
1053String logLevel =
this.dagConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1054if (logLevel == null) {
1055 logLevel = tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1056}
if (logLevel != null) {
1058 if (!HistoryLogLevel.validateLogLevel(logLevel)) {
1059throw new IllegalArgumentException("Config: " +
TezConfiguration.TEZ_HISTORY_LOGLEVEL +
1060" is set to invalid value: " + logLevel);
1061 }
1062 PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
1063 kvp.setKey(TezConfiguration.TEZ_HISTORY_LOGLEVEL);
1064 kvp.setValue(tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL));
1065 confProtoBuilder.addConfKeyValues(kvp);
1066}
{code}
- why is the AM log level overriding the dag conf provided level? line 1064?
- Please enhance unit tests to catch this bug.
Patch 05 seems incomplete.
> Add granular log levels for HistoryLoggingService.
> --
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch,
> TEZ-3359.04.patch, TEZ-3359.05.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs,
> or all logs and so on.
> * Disable logging counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
[ https://issues.apache.org/jira/browse/TEZ-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392405#comment-15392405 ] Hitesh Shah commented on TEZ-3371: -- [~Sreenath] to be clear, this issue seems only related to the AM polling for previously completed dags - is that a correct assumption? The jira description does not seem to be quite useful in terms of indicating what is the issue being addressed. If yes, shouldnt we special case this only for the scenario where the AM returned dag not found? Also, does it make sense to add a separate webservice endpoint in the AM to return the status of past completed dags to say that they are completed and what their status was instead of stopping polling based on some edge case conditions? Also, what if AM is running a different DAG but there is an ATS lag in that ATS does not have the final updated status, what happens now given that ATS has incomplete data but the AM has zero data about the dag? > Tez UI: At times polling is not stopping for completed DAGs > --- > > Key: TEZ-3371 > URL: https://issues.apache.org/jira/browse/TEZ-3371 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3371.1.patch > > > - Right now polling is very depended on app state > -- At least when a page is loaded/refreshed, if DAG status is complete asper > ATS, we shouldn't start polling. > -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[ https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3359: --- Attachment: TEZ-3359.04.patch Addressed new comments and findbugs issue. > Add granular log levels for HistoryLoggingService. > -- > > Key: TEZ-3359 > URL: https://issues.apache.org/jira/browse/TEZ-3359 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch, > TEZ-3359.04.patch > > > We publish too many events to ATS, this increases the file size for ATS. > Reduce data size logged into ATS by: > * Having a more granular control over the loglevel, disable task level logs, > or all logs and so on. > * Disable logging counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392296#comment-15392296
]
Hitesh Shah edited comment on TEZ-3359 at 7/25/16 5:21 PM:
---
For some reason, the pre-commit did not update jira:
>From the console logs:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819890/TEZ-3359.03.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1874//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1874//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1874//console
was (Author: hitesh):
For some reason, the pre-commit did not update jira:
>From the console logs:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819890/TEZ-3359.03.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
> Add granular log levels for HistoryLoggingService.
> --
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs,
> or all logs and so on.
> * Disable logging counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392327#comment-15392327
]
Hitesh Shah commented on TEZ-3359:
--
Patch mostly looks good.
Comments:
{code}
if (!this.dagConf.containsKey(TezConfiguration.TEZ_HISTORY_LOGLEVEL) &&
1054tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL) != null) {
1055 if (!HistoryLogLevel.validateLogLevel(tezConf)) {
1056throw new IllegalArgumentException("Config: " +
TezConfiguration.TEZ_HISTORY_LOGLEVEL +
1057" is set to invalid value: " +
tezConf.get(TezConfiguration.TEZ_HISTORY_LOGLEVEL));
1058 }
{code}
- there is still a hole in checks if someone invokes DAG::setConf(string,
string) with the key as history log level.
- the simpler check will be to just validate the value if set in dag conf and
if not set, validate the value set in the config object ( assume default val if
not present obviously ).
{code}
private static HistoryLogLevel getLogLevel(Configuration conf, HistoryLogLevel
defaultValue,
50HistoryLogLevel invalidValue) {
{code}
- maybe less code re-use will help here to reduce confusing code esp. in the
use of "invalidValue". Lets keep the validation code separate and then the
getLogLevel() should never catch IllegalArg? Does that make sense / agree?
> Add granular log levels for HistoryLoggingService.
> --
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs,
> or all logs and so on.
> * Disable logging counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392296#comment-15392296
]
Hitesh Shah commented on TEZ-3359:
--
For some reason, the pre-commit did not update jira:
>From the console logs:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819890/TEZ-3359.03.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
> Add granular log levels for HistoryLoggingService.
> --
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs,
> or all logs and so on.
> * Disable logging counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TEZ-3316) Speculative execution starts too early due to 0 progress
[ https://issues.apache.org/jira/browse/TEZ-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla resolved TEZ-3316. -- Resolution: Duplicate Closing this as a duplicate of TEZ-3317. > Speculative execution starts too early due to 0 progress > > > Key: TEZ-3316 > URL: https://issues.apache.org/jira/browse/TEZ-3316 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles > > Don't know at this point if this is a tez or a PigProcessor issue. There is > some setProgress chain that is keeping task progress from being correctly > reported. Task status is always zero, so as soon as the first task finishes, > tasks up to the speculation limit are always launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3317) Speculative execution starts too early due to 0 progress
[
https://issues.apache.org/jira/browse/TEZ-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated TEZ-3317:
-
Attachment: TEZ-3317.001.patch
Attaching an initial prototype patch that addresses MRInput and
UnorderedKVInput. Added {{getProgress}} at Input level and formulated the
progress for the Unordered case to be : (totalBytesReadSofar/TotalfileBytes) *
(numofCompletedInput/numofTotalInputs). Requesting comments/ideas on how to
tackle different Readers for calculating and propagating progress to their
inputs. Processors with multiple Inputs would need some more work to report
correct/weighted progress.
> Speculative execution starts too early due to 0 progress
>
>
> Key: TEZ-3317
> URL: https://issues.apache.org/jira/browse/TEZ-3317
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Kuhu Shukla
> Attachments: TEZ-3317.001.patch
>
>
> Don't know at this point if this is a tez or a PigProcessor issue. There is
> some setProgress chain that is keeping task progress from being correctly
> reported. Task status is always zero, so as soon as the first task finishes,
> tasks up to the speculation limit are always launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
[ https://issues.apache.org/jira/browse/TEZ-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392207#comment-15392207 ] Sreenath Somarajapuram commented on TEZ-3371: - [~jeagles] Both Tez UI V1 & V2 are using different tech internally. Will give another patch for v1 in branch-0.8, that could be easily backported. > Tez UI: At times polling is not stopping for completed DAGs > --- > > Key: TEZ-3371 > URL: https://issues.apache.org/jira/browse/TEZ-3371 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3371.1.patch > > > - Right now polling is very depended on app state > -- At least when a page is loaded/refreshed, if DAG status is complete asper > ATS, we shouldn't start polling. > -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (TEZ-3317) Speculative execution starts too early due to 0 progress
[ https://issues.apache.org/jira/browse/TEZ-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla reopened TEZ-3317: -- Assignee: Kuhu Shukla Re-opening this issue whilst closing TEZ-3316. > Speculative execution starts too early due to 0 progress > > > Key: TEZ-3317 > URL: https://issues.apache.org/jira/browse/TEZ-3317 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > > Don't know at this point if this is a tez or a PigProcessor issue. There is > some setProgress chain that is keeping task progress from being correctly > reported. Task status is always zero, so as soon as the first task finishes, > tasks up to the speculation limit are always launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
[ https://issues.apache.org/jira/browse/TEZ-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3371: Target Version/s: 0.9.0, 0.8.5 (was: 0.9.0) > Tez UI: At times polling is not stopping for completed DAGs > --- > > Key: TEZ-3371 > URL: https://issues.apache.org/jira/browse/TEZ-3371 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3371.1.patch > > > - Right now polling is very depended on app state > -- At least when a page is loaded/refreshed, if DAG status is complete asper > ATS, we shouldn't start polling. > -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-3371 PreCommit Build #1875
Jira: https://issues.apache.org/jira/browse/TEZ-3371
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1875/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4158 lines...]
[INFO] Tez SUCCESS [ 0.028 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 54:43 min
[INFO] Finished at: 2016-07-25T15:34:47+00:00
[INFO] Final Memory: 90M/1189M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819941/TEZ-3371.1.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1875//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1875//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 707: /home/jenkins/tools/jiracli/latest/jira: Permission denied
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 708: /home/jenkins/tools/jiracli/latest/jira: Permission denied
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3371
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Resolved] (TEZ-3355) Tez Custom Shuffle Handler POC
[ https://issues.apache.org/jira/browse/TEZ-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-3355. -- Resolution: Fixed Fix Version/s: TEZ-3334 > Tez Custom Shuffle Handler POC > -- > > Key: TEZ-3355 > URL: https://issues.apache.org/jira/browse/TEZ-3355 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: TEZ-3334 > > Attachments: TEZ-3355.1.patch, TEZ-3355.2.patch > > > This jira is effectively a starter jira to port the mapreduce shuffle handler > into the tez namespace. > Answers needed for this jira to be finished: > - how to package this artifact without polluting the tez runtime distribution > - what directory/artifact to place this code in > - how to minimize reliance on hadoop internals > - what to do with previous port in tez-ext-service-tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
[ https://issues.apache.org/jira/browse/TEZ-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392073#comment-15392073 ] Jonathan Eagles commented on TEZ-3371: -- I see similar issues for 0.7.x branch. Is this patch straightforward to backport? > Tez UI: At times polling is not stopping for completed DAGs > --- > > Key: TEZ-3371 > URL: https://issues.apache.org/jira/browse/TEZ-3371 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3371.1.patch > > > - Right now polling is very depended on app state > -- At least when a page is loaded/refreshed, if DAG status is complete asper > ATS, we shouldn't start polling. > -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
[ https://issues.apache.org/jira/browse/TEZ-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391954#comment-15391954 ] Sreenath Somarajapuram edited comment on TEZ-3371 at 7/25/16 2:29 PM: -- The patch ensures that - on page load/refresh - of a multi-DAG job - if DAG is complete then polling is not started for DAG, vertex, task & attempt entities even if app is running. [~hitesh] Please help in reviewing the patch. was (Author: sreenath): The patch ensures that - on page load - of a multi-DAG - if DAG is complete then polling is not started for DAG, vertex, task & attempt entities. [~hitesh] Please help in reviewing the patch. > Tez UI: At times polling is not stopping for completed DAGs > --- > > Key: TEZ-3371 > URL: https://issues.apache.org/jira/browse/TEZ-3371 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3371.1.patch > > > - Right now polling is very depended on app state > -- At least when a page is loaded/refreshed, if DAG status is complete asper > ATS, we shouldn't start polling. > -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3373) TezClient test case for history logging.
Harish Jaiprakash created TEZ-3373: -- Summary: TezClient test case for history logging. Key: TEZ-3373 URL: https://issues.apache.org/jira/browse/TEZ-3373 Project: Apache Tez Issue Type: Sub-task Reporter: Harish Jaiprakash A comprehensive test in tez-tests for history logging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3372) Disable logging of counters.
Harish Jaiprakash created TEZ-3372: -- Summary: Disable logging of counters. Key: TEZ-3372 URL: https://issues.apache.org/jira/browse/TEZ-3372 Project: Apache Tez Issue Type: Sub-task Reporter: Harish Jaiprakash Remove counters data so that size of the history packet is smaller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
[ https://issues.apache.org/jira/browse/TEZ-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3371: Attachment: TEZ-3371.1.patch The patch ensures that - on page load - of a multi-DAG - if DAG is complete then polling is not started for DAG, vertex, task & attempt entities. [~hitesh] Please help in reviewing the patch. > Tez UI: At times polling is not stopping for completed DAGs > --- > > Key: TEZ-3371 > URL: https://issues.apache.org/jira/browse/TEZ-3371 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3371.1.patch > > > - Right now polling is very depended on app state > -- At least when a page is loaded/refreshed, if DAG status is complete asper > ATS, we shouldn't start polling. > -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3371) Tez UI: At times polling is not stopping for completed DAGs
Sreenath Somarajapuram created TEZ-3371: --- Summary: Tez UI: At times polling is not stopping for completed DAGs Key: TEZ-3371 URL: https://issues.apache.org/jira/browse/TEZ-3371 Project: Apache Tez Issue Type: Bug Affects Versions: 0.8.3 Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram - Right now polling is very depended on app state -- At least when a page is loaded/refreshed, if DAG status is complete asper ATS, we shouldn't start polling. -- Right now, we start polling even on page load/refresh after DAG complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3359 PreCommit Build #1874
Jira: https://issues.apache.org/jira/browse/TEZ-3359
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1874/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 15218 lines...]
[INFO]
[INFO] Total time: 56:21 min
[INFO] Finished at: 2016-07-25T11:16:29+00:00
[INFO] Final Memory: 75M/1097M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12819890/TEZ-3359.03.patch
against master revision 97fa44f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1874//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1874//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1874//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 707: /home/jenkins/tools/jiracli/latest/jira: Permission denied
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/build-tools/test-patch.sh:
line 708: /home/jenkins/tools/jiracli/latest/jira: Permission denied
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.71 MB of artifacts by 23.6% relative to #1871
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3359) Add granular log levels for HistoryLoggingService.
[ https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3359: --- Attachment: TEZ-3359.03.patch * Addressed findbug issue. * Addressed all code review comments. * Adding 2 more test cases. > Add granular log levels for HistoryLoggingService. > -- > > Key: TEZ-3359 > URL: https://issues.apache.org/jira/browse/TEZ-3359 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch > > > We publish too many events to ATS, this increases the file size for ATS. > Reduce data size logged into ATS by: > * Having a more granular control over the loglevel, disable task level logs, > or all logs and so on. > * Disable logging counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
