[jira] [Commented] (TEZ-3334) Tez Custom Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372143#comment-15372143 ] Siddharth Seth commented on TEZ-3334: - [~jeagles]. Don't think there's any other template at the moment. LLAP has a custom shuffle handler - which is essentially a stripped down version of the MR shuffle handler, but adds one more component to path processing (the dag id). A new shuffle handler would ideally handle MR, Tez and LLAP (which currently hosts it's shuffle service, but will likely move it out soon). Some of the issues which would be good to fix eventually Multiple partition ranges in a single request (main intent of this jira per the description?) Handling UnsortedOutput - multiple files / files merged in a different manner. Improvements to broadcast shuffle handling Better error reporting. It would be really nice if we could build something out which potentially understand file formats beyond the current index files + merged IFiles. > Tez Custom Shuffle Handler > -- > > Key: TEZ-3334 > URL: https://issues.apache.org/jira/browse/TEZ-3334 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles > > For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom > shuffle handler could help reduce the number of fetches and could more > efficiently fetch data. In particular if a reducer is fetching 100 pieces > serially from the same mapper it could do this in one fetch call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3331: - Attachment: TEZ-3331.wip.3.patch Removed use of isTracked() > Add operation specific HDFS counters for Tez UI > --- > > Key: TEZ-3331 > URL: https://issues.apache.org/jira/browse/TEZ-3331 > Project: Apache Tez > Issue Type: Bug >Reporter: Jitendra Nath Pandey > Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.3.patch, > TEZ-3331.wip.patch > > > Hadoop has added several operation specific counters in the FileSystem > statistics (HADOOP-13065). These counters are useful to track file system > operations more granularly. It would be great to track these counters for Tez > and expose them via UI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371927#comment-15371927 ] Hitesh Shah commented on TEZ-1248: -- Committing shortly. > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-1248 PreCommit Build #1846
Jira: https://issues.apache.org/jira/browse/TEZ-1248
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1846/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4118 lines...]
[INFO] Tez ... SUCCESS [ 0.024 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 55:42 min
[INFO] Finished at: 2016-07-11T23:14:58+00:00
[INFO] Final Memory: 78M/1096M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12817248/TEZ-1248.1.patch
against master revision 608e15e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1846//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1846//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
32ba56ce213539cb60be527fa8fd0589b5353315 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-1248
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[
https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371870#comment-15371870
]
TezQA commented on TEZ-1248:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12817248/TEZ-1248.1.patch
against master revision 608e15e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1846//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1846//console
This message is automatically generated.
> Reduce slow-start should special case 1 reducer runs
>
>
> Key: TEZ-1248
> URL: https://issues.apache.org/jira/browse/TEZ-1248
> Project: Apache Tez
> Issue Type: Improvement
>Affects Versions: 0.5.0
> Environment: 20 node cluster running tez
>Reporter: Gopal V
>Assignee: Zhiyuan Yang
>Priority: Critical
> Attachments: TEZ-1248.1.patch
>
>
> Reducer slow-start has a performance problem for the small cases where there
> is just 1 reducer for a case with a single wave.
> Tez knows the split count and wave count, being able to determine if the
> cluster has enough spare capacity to run the reducer earlier for lower
> latency in a N-mapper -> 1 reducer case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3330) Error on avro M/R job with Tez: missing configuration property
[
https://issues.apache.org/jira/browse/TEZ-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371859#comment-15371859
]
Hitesh Shah commented on TEZ-3330:
--
For now, can you try adding the configs in question into tez-site.xml and see
if that gets you past the error?
> Error on avro M/R job with Tez: missing configuration property
> --
>
> Key: TEZ-3330
> URL: https://issues.apache.org/jira/browse/TEZ-3330
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Manuel Godbert
>
> I tried running the simple avro M/R job MapredColorCount, that I found in the
> examples of avro release 1.7.7.
> It failed with the following trace:
> {code}
> errorMessage=Shuffle Runner
> Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
> Error while doing final merge
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:378)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:337)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at org.apache.avro.mapred.AvroJob.getMapOutputSchema(AvroJob.java:78)
> at
> org.apache.avro.mapred.AvroKeyComparator.setConf(AvroKeyComparator.java:39)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.tez.runtime.library.common.ConfigUtils.getIntermediateInputKeyComparator(ConfigUtils.java:133)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.finalMerge(MergeManager.java:915)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.close(MergeManager.java:540)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:376)
> ... 6 more
> {code}
> Digging a bit I saw that during shuffle Tez can't access some of the
> configuration properties of the job. In our example it is the
> avro.output.schema that is missing.
> With some more complicated code I could get one step further and a similar
> issue happened when the valuesIterator for the reducer was being built:
> {code}
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at org.apache.avro.mapred.AvroJob.getMapOutputSchema(AvroJob.java:78)
> at
> org.apache.avro.mapred.AvroSerialization.getDeserializer(AvroSerialization.java:53)
> at
> org.apache.hadoop.io.serializer.SerializationFactory.getDeserializer(SerializationFactory.java:90)
> at
> org.apache.tez.runtime.library.common.ValuesIterator.(ValuesIterator.java:80)
> at
> org.apache.tez.runtime.library.input.OrderedGroupedKVInput.createValuesIterator(OrderedGroupedKVInput.java:287)
> {code}
> I am using HDP2.4, Tez 0.7.0, avro 1.7.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371706#comment-15371706 ] Tsuyoshi Ozawa commented on TEZ-3303: - [~sseth] do you have any comments? > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch, TEZ-3303.003.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371679#comment-15371679 ] Zhiyuan Yang commented on TEZ-1248: --- Never mind.. I thought you are talking about previous code change that causes this problem... > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371654#comment-15371654 ] Zhiyuan Yang commented on TEZ-1248: --- Thanks for the review! By 'simple code change', which jira are you referring to? > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371643#comment-15371643 ] Bikas Saha edited comment on TEZ-1248 at 7/11/16 9:16 PM: -- lgtm. seems like a simple code change whose side-effect produces the results for this jira :P +1. Thanks for the fix! was (Author: bikassaha): lgtm. seems like a simple code change whose side-effect produces the results for this jira :P > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371643#comment-15371643 ] Bikas Saha commented on TEZ-1248: - lgtm. seems like a simple code change whose side-effect produces the results for this jira :P > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371588#comment-15371588 ] Zhiyuan Yang commented on TEZ-1248: --- The patched uploaded rounds up the float when calculating number of tasks to schedule, so that single reducer gets scheduled as long as source tasks completion percentage becomes more than min slow start fraction. > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371582#comment-15371582 ] Zhiyuan Yang commented on TEZ-1248: --- Please help review the patch. [~hitesh], [~sseth], [~gopalv], [~rajesh.balamohan]. > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-1248: -- Attachment: TEZ-1248.1.patch > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371550#comment-15371550
]
Hitesh Shah edited comment on TEZ-3331 at 7/11/16 8:32 PM:
---
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
The former is an incompatible change but may provide a more cleaner approach
for different FS impls if we can make the group name a bit more understandable.
was (Author: hitesh):
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
The former is an incompatible change but may provide a more cleaner approach
for different FS impls.
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-3331:
-
Attachment: TEZ-3331.wip.2.patch
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3331) Add operation specific HDFS counters for Tez UI
[
https://issues.apache.org/jira/browse/TEZ-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371550#comment-15371550
]
Hitesh Shah edited comment on TEZ-3331 at 7/11/16 8:32 PM:
---
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
The former is an incompatible change but may provide a more cleaner approach
for different FS impls.
was (Author: hitesh):
Some minor whitespace cleanup.
A couple of points to note:
Using "TezCounter counter = counters.findCounter(stats.getScheme(),
fsCounter.name()" results in counters:
{code}
file
BYTES_READ=12075
BYTES_WRITTEN=10401
hdfs
BYTES_READ=23513
BYTES_WRITTEN=14163
OP_CREATE=1
OP_GET_FILE_STATUS=4
OP_OPEN=8
OP_RENAME=1
READ_OPS=12
WRITE_OPS=2
{code}
compared to the current patch:
{code}
File System Counters
FILE_BYTES_READ=12075
FILE_BYTES_WRITTEN=10401
HDFS_BYTES_READ=23513
HDFS_BYTES_WRITTEN=14163
HDFS_READ_OPS=12
HDFS_WRITE_OPS=2
HDFS_OP_CREATE=1
HDFS_OP_GET_FILE_STATUS=4
HDFS_OP_OPEN=8
HDFS_OP_RENAME=1
{code}
> Add operation specific HDFS counters for Tez UI
> ---
>
> Key: TEZ-3331
> URL: https://issues.apache.org/jira/browse/TEZ-3331
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
> Attachments: TEZ-3331.wip.2.patch, TEZ-3331.wip.patch
>
>
> Hadoop has added several operation specific counters in the FileSystem
> statistics (HADOOP-13065). These counters are useful to track file system
> operations more granularly. It would be great to track these counters for Tez
> and expose them via UI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-1248: -- Attachment: (was: TEZ-1248.1.patch) > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1248) Reduce slow-start should special case 1 reducer runs
[ https://issues.apache.org/jira/browse/TEZ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-1248: -- Attachment: TEZ-1248.1.patch > Reduce slow-start should special case 1 reducer runs > > > Key: TEZ-1248 > URL: https://issues.apache.org/jira/browse/TEZ-1248 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: 20 node cluster running tez >Reporter: Gopal V >Assignee: Zhiyuan Yang >Priority: Critical > Attachments: TEZ-1248.1.patch > > > Reducer slow-start has a performance problem for the small cases where there > is just 1 reducer for a case with a single wave. > Tez knows the split count and wave count, being able to determine if the > cluster has enough spare capacity to run the reducer earlier for lower > latency in a N-mapper -> 1 reducer case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1869) Exclude tez-ui war / other dependencies from tez.tar.gz
[ https://issues.apache.org/jira/browse/TEZ-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371336#comment-15371336 ] Jonathan Eagles commented on TEZ-1869: -- I think this change makes sense, but feel leaving tez-ui in tez-ui/target/ directory is not the best long term. > Exclude tez-ui war / other dependencies from tez.tar.gz > --- > > Key: TEZ-1869 > URL: https://issues.apache.org/jira/browse/TEZ-1869 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-1869.1.txt > > > tez.tar.gz is meant to be uploaded to HDFS and configured in tez.lib.uris. > The war file, along with any dependencies should not be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1869) Exclude tez-ui war / other dependencies from tez.tar.gz
[ https://issues.apache.org/jira/browse/TEZ-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371293#comment-15371293 ] Hitesh Shah edited comment on TEZ-1869 at 7/11/16 6:00 PM: --- I think we need the ui war for now in the minimal archive as we use that for creating the binary release. Also, not sure if not having it in the full tarball buys us much from a perf viewpoint. Will likely need it in the longer term if we decide to build a UI into the AM. was (Author: hitesh): I think we need the ui war for now in the minimal archive as we use that for creating the binary release. > Exclude tez-ui war / other dependencies from tez.tar.gz > --- > > Key: TEZ-1869 > URL: https://issues.apache.org/jira/browse/TEZ-1869 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-1869.1.txt > > > tez.tar.gz is meant to be uploaded to HDFS and configured in tez.lib.uris. > The war file, along with any dependencies should not be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1869) Exclude tez-ui war / other dependencies from tez.tar.gz
[ https://issues.apache.org/jira/browse/TEZ-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371293#comment-15371293 ] Hitesh Shah commented on TEZ-1869: -- I think we need the ui war for now in the minimal archive as we use that for creating the binary release. > Exclude tez-ui war / other dependencies from tez.tar.gz > --- > > Key: TEZ-1869 > URL: https://issues.apache.org/jira/browse/TEZ-1869 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-1869.1.txt > > > tez.tar.gz is meant to be uploaded to HDFS and configured in tez.lib.uris. > The war file, along with any dependencies should not be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3334) Tez Custom Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371277#comment-15371277 ] Bikas Saha commented on TEZ-3334: - +1. The new YARN aux service isolation work should make this easier to deploy alongside the existing MR shuffle while we iron things out. > Tez Custom Shuffle Handler > -- > > Key: TEZ-3334 > URL: https://issues.apache.org/jira/browse/TEZ-3334 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles > > For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom > shuffle handler could help reduce the number of fetches and could more > efficiently fetch data. In particular if a reducer is fetching 100 pieces > serially from the same mapper it could do this in one fetch call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3334) Tez Custom Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371247#comment-15371247 ] Jonathan Eagles commented on TEZ-3334: -- [~sseth], [~gopalv] is there an example that can be pointed to for a reference implementation? In addition, a rough design will help to make sure an implementation is sufficiently generic. > Tez Custom Shuffle Handler > -- > > Key: TEZ-3334 > URL: https://issues.apache.org/jira/browse/TEZ-3334 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles > > For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom > shuffle handler could help reduce the number of fetches and could more > efficiently fetch data. In particular if a reducer is fetching 100 pieces > serially from the same mapper it could do this in one fetch call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3334) Tez Custom Shuffle Handler
Jonathan Eagles created TEZ-3334: Summary: Tez Custom Shuffle Handler Key: TEZ-3334 URL: https://issues.apache.org/jira/browse/TEZ-3334 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom shuffle handler could help reduce the number of fetches and could more efficiently fetch data. In particular if a reducer is fetching 100 pieces serially from the same mapper it could do this in one fetch call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3205) Update tez poms to remove hadoop 2.2/2.4 profiles
[ https://issues.apache.org/jira/browse/TEZ-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371181#comment-15371181 ] Siddharth Seth commented on TEZ-3205: - Good points. I'll update the patch when I get a chance. > Update tez poms to remove hadoop 2.2/2.4 profiles > - > > Key: TEZ-3205 > URL: https://issues.apache.org/jira/browse/TEZ-3205 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth > Attachments: TEZ-3205.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1869) Exclude tez-ui war / other dependencies from tez.tar.gz
[ https://issues.apache.org/jira/browse/TEZ-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371180#comment-15371180 ] Siddharth Seth commented on TEZ-1869: - Does anyone see this as a problem ? Will close it as Won't fix otherwise. > Exclude tez-ui war / other dependencies from tez.tar.gz > --- > > Key: TEZ-1869 > URL: https://issues.apache.org/jira/browse/TEZ-1869 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-1869.1.txt > > > tez.tar.gz is meant to be uploaded to HDFS and configured in tez.lib.uris. > The war file, along with any dependencies should not be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3330) Error on avro M/R job with Tez: missing configuration property
[
https://issues.apache.org/jira/browse/TEZ-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371177#comment-15371177
]
Siddharth Seth commented on TEZ-3330:
-
That makes sense. Maybe we should consider removing the filtering completely..
> Error on avro M/R job with Tez: missing configuration property
> --
>
> Key: TEZ-3330
> URL: https://issues.apache.org/jira/browse/TEZ-3330
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Manuel Godbert
>
> I tried running the simple avro M/R job MapredColorCount, that I found in the
> examples of avro release 1.7.7.
> It failed with the following trace:
> {code}
> errorMessage=Shuffle Runner
> Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
> Error while doing final merge
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:378)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:337)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at org.apache.avro.mapred.AvroJob.getMapOutputSchema(AvroJob.java:78)
> at
> org.apache.avro.mapred.AvroKeyComparator.setConf(AvroKeyComparator.java:39)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.tez.runtime.library.common.ConfigUtils.getIntermediateInputKeyComparator(ConfigUtils.java:133)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.finalMerge(MergeManager.java:915)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.close(MergeManager.java:540)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:376)
> ... 6 more
> {code}
> Digging a bit I saw that during shuffle Tez can't access some of the
> configuration properties of the job. In our example it is the
> avro.output.schema that is missing.
> With some more complicated code I could get one step further and a similar
> issue happened when the valuesIterator for the reducer was being built:
> {code}
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at org.apache.avro.mapred.AvroJob.getMapOutputSchema(AvroJob.java:78)
> at
> org.apache.avro.mapred.AvroSerialization.getDeserializer(AvroSerialization.java:53)
> at
> org.apache.hadoop.io.serializer.SerializationFactory.getDeserializer(SerializationFactory.java:90)
> at
> org.apache.tez.runtime.library.common.ValuesIterator.(ValuesIterator.java:80)
> at
> org.apache.tez.runtime.library.input.OrderedGroupedKVInput.createValuesIterator(OrderedGroupedKVInput.java:287)
> {code}
> I am using HDP2.4, Tez 0.7.0, avro 1.7.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TEZ-3321) Changes for 0.8.4 release
[ https://issues.apache.org/jira/browse/TEZ-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved TEZ-3321. - Resolution: Done > Changes for 0.8.4 release > - > > Key: TEZ-3321 > URL: https://issues.apache.org/jira/browse/TEZ-3321 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3321.part1.txt, TEZ-3321.part2.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3333) Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370939#comment-15370939
]
TezQA commented on TEZ-:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12817173/TEZ-.1.patch
against master revision 608e15e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1845//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1845//console
This message is automatically generated.
> Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
> ---
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-.1.patch
>
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
> - Data not available messages must be displayed in Graphical and swimlane
> views when required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3333 PreCommit Build #1845
Jira: https://issues.apache.org/jira/browse/TEZ-
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1845/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4123 lines...]
[INFO] Tez ... SUCCESS [ 0.042 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 58:34 min
[INFO] Finished at: 2016-07-11T15:11:37+00:00
[INFO] Final Memory: 75M/1030M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12817173/TEZ-.1.patch
against master revision 608e15e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1845//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1845//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
35470dc017660d48bf96fc5fa3eb93212ad8cc01 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3333) Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated TEZ-:
Summary: Tez UI: Handle cases where Vertex/Task/Task Attempt data is
missing (was: Tez UI: Ensure that the UI is equipped for ATS performance
changes)
> Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
> ---
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-.1.patch
>
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
> - Data not available messages must be displayed in Graphical and swimlane
> views when required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3333) Tez UI: Ensure that the UI is equipped for ATS performance changes with Hive-LLAP
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated TEZ-:
Summary: Tez UI: Ensure that the UI is equipped for ATS performance changes
with Hive-LLAP (was: Tez UI: Ensure that the UI is equiped for BUG-60112
change)
> Tez UI: Ensure that the UI is equipped for ATS performance changes with
> Hive-LLAP
> -
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-.1.patch
>
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
> - Data not available messages must be displayed in Graphical and swimlane
> views when required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3333) Tez UI: Ensure that the UI is equipped for ATS performance changes
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated TEZ-:
Summary: Tez UI: Ensure that the UI is equipped for ATS performance changes
(was: Tez UI: Ensure that the UI is equipped for ATS performance changes with
Hive-LLAP)
> Tez UI: Ensure that the UI is equipped for ATS performance changes
> --
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-.1.patch
>
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
> - Data not available messages must be displayed in Graphical and swimlane
> views when required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3333) Tez UI: Ensure that the UI is equiped for BUG-60112 change
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated TEZ-:
Description:
Observations (When vertex, task & attempt details are not available):
# All the tables will just display a “No records available!” message, and work
as expected. But
# Graphical and swimlane views will cause error.
# Stats in DAG details page will be displayed with all vertex/task counts 0.
Expected behavior:
-When ever data is not available, all pages must display a “No {} available!”
message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
- When ever vertex data is not available in DAG details page, display a "No
vertices available!" message.
- Stats table in DAG details page must be hidden when vertex data is not
available.
- Data not available messages must be displayed in Graphical and swimlane views
when required.
was:
Observations (When vertex, task & attempt details are not available):
# All the tables will just display a “No records available!” message, and work
as expected. But
# Graphical and swimlane views will cause error.
# Stats in DAG details page will be displayed with all vertex/task counts 0.
Expected behavior:
-When ever data is not available, all pages must display a “No {} available!”
message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
- When ever vertex data is not available in DAG details page, display a "No
vertices available!" message.
- Stats table in DAG details page must be hidden when vertex data is not
available.
> Tez UI: Ensure that the UI is equiped for BUG-60112 change
> --
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-.1.patch
>
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
> - Data not available messages must be displayed in Graphical and swimlane
> views when required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3333) Tez UI: Ensure that the UI is equiped for BUG-60112 change
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated TEZ-:
Attachment: TEZ-.1.patch
Attaching a patch to meet the expected behaviors.
[~hitesh] Please help in reviewing.
> Tez UI: Ensure that the UI is equiped for BUG-60112 change
> --
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-.1.patch
>
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
> - Data not available messages must be displayed in Graphical and swimlane
> views when required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3333) Tez UI: Ensure that the UI is equiped for BUG-60112 change
[
https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated TEZ-:
Description:
Observations (When vertex, task & attempt details are not available):
# All the tables will just display a “No records available!” message, and work
as expected. But
# Graphical and swimlane views will cause error.
# Stats in DAG details page will be displayed with all vertex/task counts 0.
Expected behavior:
-When ever data is not available, all pages must display a “No {} available!”
message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
- When ever vertex data is not available in DAG details page, display a "No
vertices available!" message.
- Stats table in DAG details page must be hidden when vertex data is not
available.
was:
Observations (When vertex, task & attempt details are not available):
# All the tables will just display a “No records available!” message, and work
as expected. But
# Graphical and swimlane views will cause error.
# Stats in DAG details page will be displayed with all vertex/task counts 0.
Expected behavior:
- When ever data is not available, all pages must display a “No {recordType}
available!” message. Where recordType could be Vertices, Tasks, Task Attempts
etc...
- Stats table in DAG details page must be hidden when vertex data is not
available.
> Tez UI: Ensure that the UI is equiped for BUG-60112 change
> --
>
> Key: TEZ-
> URL: https://issues.apache.org/jira/browse/TEZ-
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
>
> Observations (When vertex, task & attempt details are not available):
> # All the tables will just display a “No records available!” message, and
> work as expected. But
> # Graphical and swimlane views will cause error.
> # Stats in DAG details page will be displayed with all vertex/task counts 0.
> Expected behavior:
> -When ever data is not available, all pages must display a “No {} available!”
> message. Where recordType could be Vertices, Tasks, Task Attempts etc...-
> - When ever vertex data is not available in DAG details page, display a "No
> vertices available!" message.
> - Stats table in DAG details page must be hidden when vertex data is not
> available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-3333) Tez UI: Ensure that the UI is equiped for BUG-60112 change
Sreenath Somarajapuram created TEZ-:
---
Summary: Tez UI: Ensure that the UI is equiped for BUG-60112 change
Key: TEZ-
URL: https://issues.apache.org/jira/browse/TEZ-
Project: Apache Tez
Issue Type: Bug
Reporter: Sreenath Somarajapuram
Assignee: Sreenath Somarajapuram
Observations (When vertex, task & attempt details are not available):
# All the tables will just display a “No records available!” message, and work
as expected. But
# Graphical and swimlane views will cause error.
# Stats in DAG details page will be displayed with all vertex/task counts 0.
Expected behavior:
- When ever data is not available, all pages must display a “No {recordType}
available!” message. Where recordType could be Vertices, Tasks, Task Attempts
etc...
- Stats table in DAG details page must be hidden when vertex data is not
available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
