[jira] [Commented] (TEZ-3402) SplitGrouper: Integer overflow
[
https://issues.apache.org/jira/browse/TEZ-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408998#comment-15408998
]
Tsuyoshi Ozawa commented on TEZ-3402:
-
[~aplusplus]
{quote}
If you see List interface, its size() method returns an int value,
{quote}
You're right. Thanks for the clarification. I will think how to solve it.
> SplitGrouper: Integer overflow
> --
>
> Key: TEZ-3402
> URL: https://issues.apache.org/jira/browse/TEZ-3402
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Gopal V
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3402.001.patch
>
>
> Bad configs triggers integer overflow. This is a 5Tb query which tries to
> group by max-size of 4096.
> {code}
> // splits too small to work. Need to override with size.
> int newDesiredNumSplits = (int)(totalLength/minLengthPerGroup) + 1;
> {code}
> {code}
> diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed,
> vertex=vertex_1470081722620_0072_3_00 [Map 2],
> java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
> at java.util.ArrayList.(ArrayList.java:156)
> at
> org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3402) SplitGrouper: Integer overflow
[
https://issues.apache.org/jira/browse/TEZ-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3402:
Attachment: TEZ-3402.001.patch
> SplitGrouper: Integer overflow
> --
>
> Key: TEZ-3402
> URL: https://issues.apache.org/jira/browse/TEZ-3402
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Gopal V
> Attachments: TEZ-3402.001.patch
>
>
> Bad configs triggers integer overflow. This is a 5Tb query which tries to
> group by max-size of 4096.
> {code}
> // splits too small to work. Need to override with size.
> int newDesiredNumSplits = (int)(totalLength/minLengthPerGroup) + 1;
> {code}
> {code}
> diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed,
> vertex=vertex_1470081722620_0072_3_00 [Map 2],
> java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
> at java.util.ArrayList.(ArrayList.java:156)
> at
> org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3375) Logging with slf4j way instead of "+" to reduce logging overhead
[
https://issues.apache.org/jira/browse/TEZ-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3375:
Description: Currently, Tez ATSv1.5 uses concatenation of Strings instead
using placeholders {} which slfj4 provides. It increases the cost of creating
objects.
> Logging with slf4j way instead of "+" to reduce logging overhead
>
>
> Key: TEZ-3375
> URL: https://issues.apache.org/jira/browse/TEZ-3375
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Tsuyoshi Ozawa
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3375.001.patch
>
>
> Currently, Tez ATSv1.5 uses concatenation of Strings instead using
> placeholders {} which slfj4 provides. It increases the cost of creating
> objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3375) Logging with slf4j way instead of "+" to reduce logging overhead
[ https://issues.apache.org/jira/browse/TEZ-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3375: Attachment: TEZ-3375.001.patch Attaching a patch. > Logging with slf4j way instead of "+" to reduce logging overhead > > > Key: TEZ-3375 > URL: https://issues.apache.org/jira/browse/TEZ-3375 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Tsuyoshi Ozawa >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3375.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3375) Logging with slf4j way instead of "+" to reduce logging overhead
Tsuyoshi Ozawa created TEZ-3375: --- Summary: Logging with slf4j way instead of "+" to reduce logging overhead Key: TEZ-3375 URL: https://issues.apache.org/jira/browse/TEZ-3375 Project: Apache Tez Issue Type: Sub-task Reporter: Tsuyoshi Ozawa Assignee: Tsuyoshi Ozawa -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
[ https://issues.apache.org/jira/browse/TEZ-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3374: --- Assignee: Tsuyoshi Ozawa > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. > - > > Key: TEZ-3374 > URL: https://issues.apache.org/jira/browse/TEZ-3374 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3374.001.patch > > > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from > tez.history.timeline.num-dags-per-group to > tez.history.logging.timeline.num-dags-per-group, so that its consistent with > the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
[ https://issues.apache.org/jira/browse/TEZ-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3374: Attachment: TEZ-3374.001.patch > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. > - > > Key: TEZ-3374 > URL: https://issues.apache.org/jira/browse/TEZ-3374 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > Attachments: TEZ-3374.001.patch > > > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from > tez.history.timeline.num-dags-per-group to > tez.history.logging.timeline.num-dags-per-group, so that its consistent with > the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3374) Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
[ https://issues.apache.org/jira/browse/TEZ-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393216#comment-15393216 ] Tsuyoshi Ozawa commented on TEZ-3374: - Attaching first patch. > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name. > - > > Key: TEZ-3374 > URL: https://issues.apache.org/jira/browse/TEZ-3374 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > Attachments: TEZ-3374.001.patch > > > Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP name from > tez.history.timeline.num-dags-per-group to > tez.history.logging.timeline.num-dags-per-group, so that its consistent with > the other config names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374379#comment-15374379 ] Tsuyoshi Ozawa commented on TEZ-3303: - Thanks for your review and your committing :-) > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Fix For: 0.9.0 > > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch, TEZ-3303.003.02.patch, TEZ-3303.003.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3334) Tez Custom Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3334: Issue Type: New Feature (was: Bug) > Tez Custom Shuffle Handler > -- > > Key: TEZ-3334 > URL: https://issues.apache.org/jira/browse/TEZ-3334 > Project: Apache Tez > Issue Type: New Feature >Reporter: Jonathan Eagles > > For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom > shuffle handler could help reduce the number of fetches and could more > efficiently fetch data. In particular if a reducer is fetching 100 pieces > serially from the same mapper it could do this in one fetch call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371706#comment-15371706 ] Tsuyoshi Ozawa commented on TEZ-3303: - [~sseth] do you have any comments? > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch, TEZ-3303.003.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367391#comment-15367391 ] Tsuyoshi Ozawa commented on TEZ-3303: - Thank you, Ming. [~sseth] could you also check the patch? > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch, TEZ-3303.003.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3303: Attachment: TEZ-3303.003.patch Good catch, thanks [~mingma] for the precise review! Updating the patch. > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch, TEZ-3303.003.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3313) ATSFileParser : Wrong args passed in VersionInfo
[ https://issues.apache.org/jira/browse/TEZ-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360817#comment-15360817 ] Tsuyoshi Ozawa commented on TEZ-3313: - +1(non-binding), We can implement TEZ-3319 after this issue. > ATSFileParser : Wrong args passed in VersionInfo > > > Key: TEZ-3313 > URL: https://issues.apache.org/jira/browse/TEZ-3313 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3313.1.patch, TEZ-3313.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3313) ATSFileParser : Wrong args passed in VersionInfo
[ https://issues.apache.org/jira/browse/TEZ-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3313: Assignee: Rajesh Balamohan > ATSFileParser : Wrong args passed in VersionInfo > > > Key: TEZ-3313 > URL: https://issues.apache.org/jira/browse/TEZ-3313 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3313.1.patch, TEZ-3313.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360727#comment-15360727 ] Tsuyoshi Ozawa commented on TEZ-3303: - [~sseth] [~mingma] could you take a look? > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3303: Attachment: TEZ-3303.002.patch > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch, > TEZ-3303.002.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360194#comment-15360194 ] Tsuyoshi Ozawa commented on TEZ-3303: - The test failures don't occur on my local. so kicking the CI again. > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3303: Attachment: TEZ-3303.002.patch > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359570#comment-15359570 ] Tsuyoshi Ozawa commented on TEZ-3303: - [~sseth] [~mingma] thanks for your clarification. Make sense. I'll update the patch to use precise stats instead of rough stats. > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358196#comment-15358196 ] Tsuyoshi Ozawa commented on TEZ-3303: - [~sseth] could you take a look? > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3287) Have UnorderedPartitionedKVWriter honor tez.runtime.empty.partitions.info-via-events.enabled
[ https://issues.apache.org/jira/browse/TEZ-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358068#comment-15358068 ] Tsuyoshi Ozawa edited comment on TEZ-3287 at 6/30/16 11:57 PM: --- [~lmv] thanks for your taking a look! The patch is targeting master, not for branch-8. After merging this into master, I can backport it to branch-0.8. BTW, I'm thinking that there is no relationship between "hive.tez.auto.reducer.parallelism=true" and this jira. Let me know if I'm wrong. was (Author: ozawa): [~lmv] thanks for your taking a look! The patch is targeting master, not for branch-8. After merging this into master, I can backport it to branch-0.8. > Have UnorderedPartitionedKVWriter honor > tez.runtime.empty.partitions.info-via-events.enabled > > > Key: TEZ-3287 > URL: https://issues.apache.org/jira/browse/TEZ-3287 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3287.001.patch > > > The ordered partitioned output allows applications to specify if empty > partition stats should be included as part of DataMovementEvent via a > configuration. It seems unordered partitioned output should honor that > configuration as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3287) Have UnorderedPartitionedKVWriter honor tez.runtime.empty.partitions.info-via-events.enabled
[ https://issues.apache.org/jira/browse/TEZ-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358068#comment-15358068 ] Tsuyoshi Ozawa commented on TEZ-3287: - [~lmv] thanks for your taking a look! The patch is targeting master, not for branch-8. After merging this into master, I can backport it to branch-0.8. > Have UnorderedPartitionedKVWriter honor > tez.runtime.empty.partitions.info-via-events.enabled > > > Key: TEZ-3287 > URL: https://issues.apache.org/jira/browse/TEZ-3287 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3287.001.patch > > > The ordered partitioned output allows applications to specify if empty > partition stats should be included as part of DataMovementEvent via a > configuration. It seems unordered partitioned output should honor that > configuration as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2962) Use per partition stats in shuffle vertex manager auto parallelism
[ https://issues.apache.org/jira/browse/TEZ-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357961#comment-15357961 ] Tsuyoshi Ozawa commented on TEZ-2962: - This can be done after TEZ-3303. > Use per partition stats in shuffle vertex manager auto parallelism > -- > > Key: TEZ-2962 > URL: https://issues.apache.org/jira/browse/TEZ-2962 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Priority: Critical > > The original code used output size sent by completed tasks. Recently per > partition stats have been added that provide granular information. Using > partition stats may be more accurate and also remove the duplicate counting > of data size in partition stats and per task overall. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3303: Attachment: TEZ-3303.001.patch Attaching a patch to consume more precise partition stats. > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3303.001.patch > > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats
[ https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3303: --- Assignee: Tsuyoshi Ozawa > Have ShuffleVertexManager consume more precise partition stats > -- > > Key: TEZ-3303 > URL: https://issues.apache.org/jira/browse/TEZ-3303 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > > TEZ-3216 adds the support for more precise partition stats. > ShuffleVertexManager should be updated to consume the more precise partition > stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3320) Java implementation of bitonic merge sort
[ https://issues.apache.org/jira/browse/TEZ-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3320: --- Assignee: Tsuyoshi Ozawa > Java implementation of bitonic merge sort > - > > Key: TEZ-3320 > URL: https://issues.apache.org/jira/browse/TEZ-3320 > Project: Apache Tez > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Assignee: Tsuyoshi Ozawa > > Pure java cache-aware bitonic merge sort without JNI can solve the bottleneck > of sort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3320) Java implementation of bitonic merge sort
Tsuyoshi Ozawa created TEZ-3320: --- Summary: Java implementation of bitonic merge sort Key: TEZ-3320 URL: https://issues.apache.org/jira/browse/TEZ-3320 Project: Apache Tez Issue Type: Improvement Reporter: Tsuyoshi Ozawa Pure java cache-aware bitonic merge sort without JNI can solve the bottleneck of sort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3319) tez-history-parser should not have its own Version class
[ https://issues.apache.org/jira/browse/TEZ-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357458#comment-15357458 ] Tsuyoshi Ozawa commented on TEZ-3319: - I agree with the solution :-) > tez-history-parser should not have its own Version class > > > Key: TEZ-3319 > URL: https://issues.apache.org/jira/browse/TEZ-3319 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Rajesh Balamohan >Priority: Critical > > This will hopefully restrict problems such as TEZ-3313 to a single > implementation > \cc [~rajesh.balamohan] [~ozawa] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3313) ATSFileParser : Wrong args passed in VersionInfo
[ https://issues.apache.org/jira/browse/TEZ-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352204#comment-15352204 ] Tsuyoshi Ozawa commented on TEZ-3313: - Thanks [~rajesh.balamohan] for your contribution. I have 2 comments: 1. What's the difference between revision and version? We should write it down on docs. 2. Why not making the order of arguments for constructors same as org.apache.tez.common.VersionInfo? (String version, String revision, String buildTime) looks to be better since it's more consistent with org.apache.tez.common.VersionInfo. Thoughts? > ATSFileParser : Wrong args passed in VersionInfo > > > Key: TEZ-3313 > URL: https://issues.apache.org/jira/browse/TEZ-3313 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3313.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2608) UnorderedPartitionedKVWriter should override write(Object key, Iterable values)
[
https://issues.apache.org/jira/browse/TEZ-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-2608:
Attachment: TEZ-2608.002.patch
Attaching v2 patch based on Sid's comment.
> UnorderedPartitionedKVWriter should override write(Object key,
> Iterable values)
>
>
> Key: TEZ-2608
> URL: https://issues.apache.org/jira/browse/TEZ-2608
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-2608.001.patch, TEZ-2608.002.patch
>
>
> From BaseUnorderedPartitionedKVWriter:
> {code}
> @Override
> public void write(Object key, Iterable values) throws IOException {
> //TODO: UnorderedPartitionedKVWriter should override this method later.
> Iterator it = values.iterator();
> while(it.hasNext()) {
> write(key, it.next());
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2608) UnorderedPartitionedKVWriter should override write(Object key, Iterable values)
[
https://issues.apache.org/jira/browse/TEZ-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351533#comment-15351533
]
Tsuyoshi Ozawa commented on TEZ-2608:
-
{quote}
Another useful bit to look at is whether the write(Key, Iterable UnorderedPartitionedKVWriter should override write(Object key,
> Iterable values)
>
>
> Key: TEZ-2608
> URL: https://issues.apache.org/jira/browse/TEZ-2608
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-2608.001.patch, TEZ-2608.002.patch
>
>
> From BaseUnorderedPartitionedKVWriter:
> {code}
> @Override
> public void write(Object key, Iterable values) throws IOException {
> //TODO: UnorderedPartitionedKVWriter should override this method later.
> Iterator it = values.iterator();
> while(it.hasNext()) {
> write(key, it.next());
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3287) Have UnorderedPartitionedKVWriter honor tez.runtime.empty.partitions.info-via-events.enabled
[ https://issues.apache.org/jira/browse/TEZ-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351409#comment-15351409 ] Tsuyoshi Ozawa commented on TEZ-3287: - [~bikassaha] could you review this? > Have UnorderedPartitionedKVWriter honor > tez.runtime.empty.partitions.info-via-events.enabled > > > Key: TEZ-3287 > URL: https://issues.apache.org/jira/browse/TEZ-3287 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3287.001.patch > > > The ordered partitioned output allows applications to specify if empty > partition stats should be included as part of DataMovementEvent via a > configuration. It seems unordered partitioned output should honor that > configuration as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3284) Synchronization for every write in UnorderdKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3284:
Assignee: Jonathan Eagles
> Synchronization for every write in UnorderdKVWriter
> ---
>
> Key: TEZ-3284
> URL: https://issues.apache.org/jira/browse/TEZ-3284
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Assignee: Jonathan Eagles
>Priority: Critical
> Labels: Performance
> Attachments: TEZ-3284.1.patch, TEZ-3284.2.patch
>
>
> {code}
> baos = new ByteArrayOutputStream();
> dos = new DataOutputStream(baos);
> keySerializer.open(dos);
> valSerializer.open(dos);
> {code}
> This is a known performance issue as documented in HADOOP-10694
> Both ByteArrayOutputStream::write() and DataOutputStream::write() have lock
> prefix calls in them, because they are object synchronized methods.
> Recommended solution is to replicate the Hive NonSync impls similar to
> HADOOP-10694
> {code}
> TezTaskRunner [RUNNABLE]
> *** java.io.DataOutputStream.write(byte[], int, int) DataOutputStream.java:107
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Writable)
> TezBytesWritableSerialization.java:123
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Object)
> TezBytesWritableSerialization.java:110
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object, int) UnorderedPartitionedKVWriter.java:295
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object) UnorderedPartitionedKVWriter.java:257
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(Object,
> Object) TezProcessor.java:232
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.collect(BytesWritable,
> Writable) VectorReduceSinkCommonOperator.java:432
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.process(Object,
> int) VectorReduceSinkCommonOperator.java:397
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(Object,
> int) VectorSelectOperator.java:144
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(Object,
> int) VectorFilterOperator.java:121
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(Object, int)
> TableScanOperator.java:130
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable)
> VectorMapOperator.java:796
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object)
> MapRecordSource.java:86
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord()
> MapRecordSource.java:70
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run()
> MapRecordProcessor.java:361
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(Map,
> Map) TezProcessor.java:172
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(Map, Map)
> TezProcessor.java:160
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run()
> LogicalIOProcessorRuntimeTask.java:370
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run()
> TaskRunner2Callable.java:73
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run()
> TaskRunner2Callable.java:61
> java.security.AccessController.doPrivileged(PrivilegedExceptionAction,
> AccessControlContext) AccessController.java (native)
> javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction)
> Subject.java:422
> org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
> UserGroupInformation.java:1657
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal()
> TaskRunner2Callable.java:61
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal()
> TaskRunner2Callable.java:37
> org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
> java.util.concurrent.FutureTask.run() FutureTask.java:266
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3311) ATSFileParser resource leak: InputStream should be closed
[ https://issues.apache.org/jira/browse/TEZ-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348961#comment-15348961 ] Tsuyoshi Ozawa commented on TEZ-3311: - [~rajesh.balamohan] thank you for the review. You're right, I checked ZipFile.getInputStream just only call ensureOpen function internally, and it doesn't touch any file descriptor. Closing this as Not a problem. > ATSFileParser resource leak: InputStream should be closed > -- > > Key: TEZ-3311 > URL: https://issues.apache.org/jira/browse/TEZ-3311 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa >Priority: Minor > Labels: newbie > Attachments: TEZ-3311.001.patch > > > Discovered in a coverity scan that inputstream is never closed: > 189InputStream inputStream = atsZipFile.getInputStream(zipEntry); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2582) Consider removing DataInputBuffer sync overheads in pipelinedsorter
[ https://issues.apache.org/jira/browse/TEZ-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348945#comment-15348945 ] Tsuyoshi Ozawa commented on TEZ-2582: - OK, thanks! > Consider removing DataInputBuffer sync overheads in pipelinedsorter > --- > > Key: TEZ-2582 > URL: https://issues.apache.org/jira/browse/TEZ-2582 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2582) Consider removing DataInputBuffer sync overheads in pipelinedsorter
[ https://issues.apache.org/jira/browse/TEZ-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348928#comment-15348928 ] Tsuyoshi Ozawa commented on TEZ-2582: - [~rajesh.balamohan] HADOOP-10694 fixes this, but do you mean we should have same class at Tez side on this issue? > Consider removing DataInputBuffer sync overheads in pipelinedsorter > --- > > Key: TEZ-2582 > URL: https://issues.apache.org/jira/browse/TEZ-2582 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2146) TaskCounter.OUTPUT_BYTES_PHYSICAL is not populated in PipelinedSorter
[ https://issues.apache.org/jira/browse/TEZ-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa resolved TEZ-2146. - Resolution: Duplicate Closing this since TEZ-2575 fixes this issue. > TaskCounter.OUTPUT_BYTES_PHYSICAL is not populated in PipelinedSorter > - > > Key: TEZ-2146 > URL: https://issues.apache.org/jira/browse/TEZ-2146 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan > > TaskCounter.OUTPUT_BYTES_PHYSICAL (fileOutputByteCounter to be precise) is > populated in DefaultSorter but not in PipelinedSorter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3287) Have UnorderedPartitionedKVWriter honor tez.runtime.empty.partitions.info-via-events.enabled
[ https://issues.apache.org/jira/browse/TEZ-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3287: Attachment: TEZ-3287.001.patch Attaching a first patch to honor t.r.e.partitions.info-via-events.enabed in UnorderPartitiionedKVWriter. > Have UnorderedPartitionedKVWriter honor > tez.runtime.empty.partitions.info-via-events.enabled > > > Key: TEZ-3287 > URL: https://issues.apache.org/jira/browse/TEZ-3287 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ming Ma > Attachments: TEZ-3287.001.patch > > > The ordered partitioned output allows applications to specify if empty > partition stats should be included as part of DataMovementEvent via a > configuration. It seems unordered partitioned output should honor that > configuration as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3284) Synchronization for every write in UnorderdKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348807#comment-15348807
]
Tsuyoshi Ozawa commented on TEZ-3284:
-
[~jeagles] is this jira working in progress? The patch introduces non
synchronized data buffer, and it looks to be ready for merge to me.
> Synchronization for every write in UnorderdKVWriter
> ---
>
> Key: TEZ-3284
> URL: https://issues.apache.org/jira/browse/TEZ-3284
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Priority: Critical
> Labels: Performance
> Attachments: TEZ-3284.1.patch, TEZ-3284.2.patch
>
>
> {code}
> baos = new ByteArrayOutputStream();
> dos = new DataOutputStream(baos);
> keySerializer.open(dos);
> valSerializer.open(dos);
> {code}
> This is a known performance issue as documented in HADOOP-10694
> Both ByteArrayOutputStream::write() and DataOutputStream::write() have lock
> prefix calls in them, because they are object synchronized methods.
> Recommended solution is to replicate the Hive NonSync impls similar to
> HADOOP-10694
> {code}
> TezTaskRunner [RUNNABLE]
> *** java.io.DataOutputStream.write(byte[], int, int) DataOutputStream.java:107
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Writable)
> TezBytesWritableSerialization.java:123
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Object)
> TezBytesWritableSerialization.java:110
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object, int) UnorderedPartitionedKVWriter.java:295
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object) UnorderedPartitionedKVWriter.java:257
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(Object,
> Object) TezProcessor.java:232
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.collect(BytesWritable,
> Writable) VectorReduceSinkCommonOperator.java:432
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.process(Object,
> int) VectorReduceSinkCommonOperator.java:397
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(Object,
> int) VectorSelectOperator.java:144
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(Object,
> int) VectorFilterOperator.java:121
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(Object, int)
> TableScanOperator.java:130
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable)
> VectorMapOperator.java:796
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object)
> MapRecordSource.java:86
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord()
> MapRecordSource.java:70
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run()
> MapRecordProcessor.java:361
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(Map,
> Map) TezProcessor.java:172
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(Map, Map)
> TezProcessor.java:160
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run()
> LogicalIOProcessorRuntimeTask.java:370
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run()
> TaskRunner2Callable.java:73
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run()
> TaskRunner2Callable.java:61
> java.security.AccessController.doPrivileged(PrivilegedExceptionAction,
> AccessControlContext) AccessController.java (native)
> javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction)
> Subject.java:422
> org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
> UserGroupInformation.java:1657
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal()
> TaskRunner2Callable.java:61
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal()
> TaskRunner2Callable.java:37
> org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
> java.util.concurrent.FutureTask.run() FutureTask.java:266
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3311) ATSFileParser resource leak: InputStream should be closed
[ https://issues.apache.org/jira/browse/TEZ-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3311: Attachment: TEZ-3311.001.patch > ATSFileParser resource leak: InputStream should be closed > -- > > Key: TEZ-3311 > URL: https://issues.apache.org/jira/browse/TEZ-3311 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa >Priority: Minor > Labels: newbie > Attachments: TEZ-3311.001.patch > > > Discovered in a coverity scan that inputstream is never closed: > 189InputStream inputStream = atsZipFile.getInputStream(zipEntry); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3311) ATSFileParser resource leak: InputStream should be closed
[ https://issues.apache.org/jira/browse/TEZ-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3311: Attachment: (was: TEZ-3311.001.patch) > ATSFileParser resource leak: InputStream should be closed > -- > > Key: TEZ-3311 > URL: https://issues.apache.org/jira/browse/TEZ-3311 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa >Priority: Minor > Labels: newbie > > Discovered in a coverity scan that inputstream is never closed: > 189InputStream inputStream = atsZipFile.getInputStream(zipEntry); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3311) ATSFileParser resource leak: InputStream should be closed
[ https://issues.apache.org/jira/browse/TEZ-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3311: Attachment: TEZ-3311.001.patch Attaching a first patch to close the inputStream. > ATSFileParser resource leak: InputStream should be closed > -- > > Key: TEZ-3311 > URL: https://issues.apache.org/jira/browse/TEZ-3311 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Minor > Labels: newbie > Attachments: TEZ-3311.001.patch > > > Discovered in a coverity scan that inputstream is never closed: > 189InputStream inputStream = atsZipFile.getInputStream(zipEntry); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3272) Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
[
https://issues.apache.org/jira/browse/TEZ-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339094#comment-15339094
]
Tsuyoshi Ozawa commented on TEZ-3272:
-
This patch only changes pom.xml, so we don't need to add any test.
> Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
> -
>
> Key: TEZ-3272
> URL: https://issues.apache.org/jira/browse/TEZ-3272
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3272.001.patch
>
>
> The pom.xml includes the following classes. It will be useful to include
> AMContainerImpl and AMNodeImpl.
> {noformat}
> org.apache.tez.dag.app.dag.impl.DAGImpl,
> org.apache.tez.dag.app.dag.impl.VertexImpl,
> org.apache.tez.dag.app.dag.impl.TaskImpl,
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339049#comment-15339049 ] Tsuyoshi Ozawa commented on TEZ-3009: - [~hitesh] could you check this? Jason's patch LGTM, while my review is non-binding one. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa resolved TEZ-3014. - Resolution: Fixed Closing this for now. Feel free to reopen this. > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3113) massive increase of run time using PipelinedSorter rather than DefaultSorter
[
https://issues.apache.org/jira/browse/TEZ-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339044#comment-15339044
]
Tsuyoshi Ozawa commented on TEZ-3113:
-
> Observed behaviour of PipelinedSorter is that several hundred thousand
> different files are put flat in the same per-TezChild local temporary
> directories, and thing become very slow (not alleging any causality)
I think this behaviour can cause file-system level lock contention if lots
threads access.
One possible solution is partitioning by adding upper limits per directory or
changing directory structure. [~rajesh.balamohan] what do you think?
> massive increase of run time using PipelinedSorter rather than DefaultSorter
>
>
> Key: TEZ-3113
> URL: https://issues.apache.org/jira/browse/TEZ-3113
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.2
> Environment: scalding 0.15-SNAPSHOT per
> https://github.com/twitter/scalding/pull/1446
> cascading 3.1.0-wip-54
> tez-0.8.2
> OpenJDK 8 on AMD64
> Hadoop 2.6.0 (YARN, HDFS); Apache distribution
> Debian Linux 8
> 8 * Intel Core i7-3770K
>Reporter: Cyrille Chépélov
>
> While running a (fairly complex) scalding DAG that was working fine using
> tez-0.6.2, now under tez-0.8.2, the run time became suddenly extremely large.
> Reverting "tez.runtime.sorter.class" -> "LEGACY" restored proper behaviour.
> Difficulties can be traced to this shape of code:
> {code:scala}
> val x: TypedPipe[(String, String)] = ??? // get *LARGE* dataset
> x
> .group
> .mapValues(x => 1L)
> .sum
> .write(TypedTsvHeader("foo.tsv", ('key, 'count)))
> {code}
> where the incoming data contains many, many different keys. Observed
> behaviour of PipelinedSorter is that several hundred thousand different files
> are put flat in the same per-TezChild local temporary directories, and thing
> become very slow (not alleging any causality).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3252) [Umbrella] Enable support for Hadoop-3.x
[ https://issues.apache.org/jira/browse/TEZ-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339040#comment-15339040 ] Tsuyoshi Ozawa commented on TEZ-3252: - I think we need to check dependencies here. > [Umbrella] Enable support for Hadoop-3.x > - > > Key: TEZ-3252 > URL: https://issues.apache.org/jira/browse/TEZ-3252 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah > > Placeholder umbrella to track the various issues/tasks discovered to get full > stable functionality against hadoop-3.x once it is released in a stable form. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3272) Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
[
https://issues.apache.org/jira/browse/TEZ-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3272:
Attachment: TEZ-3272.001.patch
Attaching a first patch.
I confirmed with this patch that AMContainer and AMNode state machines are
visualized.
> Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
> -
>
> Key: TEZ-3272
> URL: https://issues.apache.org/jira/browse/TEZ-3272
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3272.001.patch
>
>
> The pom.xml includes the following classes. It will be useful to include
> AMContainerImpl and AMNodeImpl.
> {noformat}
> org.apache.tez.dag.app.dag.impl.DAGImpl,
> org.apache.tez.dag.app.dag.impl.VertexImpl,
> org.apache.tez.dag.app.dag.impl.TaskImpl,
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (TEZ-3272) Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
[
https://issues.apache.org/jira/browse/TEZ-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa reassigned TEZ-3272:
---
Assignee: Tsuyoshi Ozawa
> Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
> -
>
> Key: TEZ-3272
> URL: https://issues.apache.org/jira/browse/TEZ-3272
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Tsuyoshi Ozawa
>
> The pom.xml includes the following classes. It will be useful to include
> AMContainerImpl and AMNodeImpl.
> {noformat}
> org.apache.tez.dag.app.dag.impl.DAGImpl,
> org.apache.tez.dag.app.dag.impl.VertexImpl,
> org.apache.tez.dag.app.dag.impl.TaskImpl,
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3302) Add a version of processorContext.waitForAllInputsReady and waitForAnyInputReady with a timeout
[ https://issues.apache.org/jira/browse/TEZ-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3302: Attachment: TEZ-3302.002.patch Oops, I attached wrong patch. Fixing it and added test. > Add a version of processorContext.waitForAllInputsReady and > waitForAnyInputReady with a timeout > --- > > Key: TEZ-3302 > URL: https://issues.apache.org/jira/browse/TEZ-3302 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3302.002.patch > > > This is useful when a Processor needs to check on whether it has been aborted > or not, and the interrupt that is sent in as part of the 'Task kill' process > has been swallowed by some other entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3302) Add a version of processorContext.waitForAllInputsReady and waitForAnyInputReady with a timeout
[ https://issues.apache.org/jira/browse/TEZ-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3302: Attachment: (was: TEZ-3302.001.patch) > Add a version of processorContext.waitForAllInputsReady and > waitForAnyInputReady with a timeout > --- > > Key: TEZ-3302 > URL: https://issues.apache.org/jira/browse/TEZ-3302 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > > This is useful when a Processor needs to check on whether it has been aborted > or not, and the interrupt that is sent in as part of the 'Task kill' process > has been swallowed by some other entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3302) Add a version of processorContext.waitForAllInputsReady and waitForAnyInputReady with a timeout
[ https://issues.apache.org/jira/browse/TEZ-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3302: Attachment: TEZ-3302.001.patch Attaching first patch. [~sseth], could you check it? > Add a version of processorContext.waitForAllInputsReady and > waitForAnyInputReady with a timeout > --- > > Key: TEZ-3302 > URL: https://issues.apache.org/jira/browse/TEZ-3302 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-3302.001.patch > > > This is useful when a Processor needs to check on whether it has been aborted > or not, and the interrupt that is sent in as part of the 'Task kill' process > has been swallowed by some other entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3302) Add a version of processorContext.waitForAllInputsReady and waitForAnyInputReady with a timeout
[ https://issues.apache.org/jira/browse/TEZ-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3302: --- Assignee: Tsuyoshi Ozawa > Add a version of processorContext.waitForAllInputsReady and > waitForAnyInputReady with a timeout > --- > > Key: TEZ-3302 > URL: https://issues.apache.org/jira/browse/TEZ-3302 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > > This is useful when a Processor needs to check on whether it has been aborted > or not, and the interrupt that is sent in as part of the 'Task kill' process > has been swallowed by some other entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3237) Corrupted shuffle transfers to disk are not detected during transfer
[ https://issues.apache.org/jira/browse/TEZ-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281573#comment-15281573 ] Tsuyoshi Ozawa commented on TEZ-3237: - [~rajesh.balamohan] make sense to me. > Corrupted shuffle transfers to disk are not detected during transfer > > > Key: TEZ-3237 > URL: https://issues.apache.org/jira/browse/TEZ-3237 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3237.001.patch > > > When a shuffle transfer is larger than the single transfer limit it gets > written straight to disk during the transfer. Unfortunately there are no > checksum validations performed during that transfer, so if the data is > corrupted at the source or during transmit it goes undetected. Only later > when the task tries to consume the transferred data is the error detected, > but at that point it's too late to blame the source task for the error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3237) Corrupted shuffle transfers to disk are not detected during transfer
[
https://issues.apache.org/jira/browse/TEZ-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281020#comment-15281020
]
Tsuyoshi Ozawa commented on TEZ-3237:
-
Thanks Jason for taking important change!
{quote}
Would it be good to have the fast path (i.e without checksumming) as an option
as well?
{quote}
Sounds good, but I don't know whether it useful to make it configurable in real
use case - in our use cases, we use checksum all time.
> Corrupted shuffle transfers to disk are not detected during transfer
>
>
> Key: TEZ-3237
> URL: https://issues.apache.org/jira/browse/TEZ-3237
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3237.001.patch
>
>
> When a shuffle transfer is larger than the single transfer limit it gets
> written straight to disk during the transfer. Unfortunately there are no
> checksum validations performed during that transfer, so if the data is
> corrupted at the source or during transmit it goes undetected. Only later
> when the task tries to consume the transferred data is the error detected,
> but at that point it's too late to blame the source task for the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2607) SIMD-based bitonic merge sorting
[ https://issues.apache.org/jira/browse/TEZ-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213712#comment-15213712 ] Tsuyoshi Ozawa commented on TEZ-2607: - I resumed to work on this. Maybe I will be able to post the first patch here next week. > SIMD-based bitonic merge sorting > > > Key: TEZ-2607 > URL: https://issues.apache.org/jira/browse/TEZ-2607 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Tsuyoshi Ozawa >Assignee: Tsuyoshi Ozawa > Attachments: map_phase.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-987) Need a way to measure container reuse
[ https://issues.apache.org/jira/browse/TEZ-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-987: -- Assignee: Tsuyoshi Ozawa > Need a way to measure container reuse > - > > Key: TEZ-987 > URL: https://issues.apache.org/jira/browse/TEZ-987 > Project: Apache Tez > Issue Type: New Feature >Reporter: Rohini Palaniswamy >Assignee: Tsuyoshi Ozawa > > It is very important to know about the amount of container reuse to measure > the effectiveness of Tez. Currently nothing is available (AM API, counters, > etc) to get those stats programatically. Only way is probably to scrape the > AM logs (haven't tried it though). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188803#comment-15188803
]
Tsuyoshi Ozawa commented on TEZ-3148:
-
[~hitesh] Make sense. Changed to handle TA_TEZ_EVENT_UPDATE only in KILLED,
KILL_IN_PROGRESS, FAIL_IN_PROGRESS. Added tests by Zhiyuan's comment.
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3148.001.patch, TEZ-3148.002.patch, am.log.gz,
> dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3148:
Attachment: TEZ-3148.002.patch
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3148.001.patch, TEZ-3148.002.patch, am.log.gz,
> dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188676#comment-15188676
]
Tsuyoshi Ozawa commented on TEZ-3148:
-
[~aplusplus] Thanks a lot for the feedback! Yes, I'll add test cases in the
next patch.
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3148.001.patch, am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3096) Statemachine: TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS fails
[
https://issues.apache.org/jira/browse/TEZ-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184011#comment-15184011
]
Tsuyoshi Ozawa commented on TEZ-3096:
-
I've also uploaded the patch, so I appreciate if you take a look. Thanks!
> Statemachine: TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS fails
> ---
>
> Key: TEZ-3096
> URL: https://issues.apache.org/jira/browse/TEZ-3096
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Zhiyuan Yang
>
> Tasks are failing exactly 300ms into running due to a FileSystem error.
> {code}
> 2016-02-04 05:05:56,853 [ERROR] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: Can't handle this event at current state for
> attempt_1454544113740_0027_1_00_03_3
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:795)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:120)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2180)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2165)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
> at java.lang.Thread.run(Thread.java:745)
> 2016-02-04 05:05:56,903 [ERROR] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: Can't handle this event at current state for
> attempt_1454544113740_0027_1_00_00_3
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:795)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:120)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2180)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2165)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3096) Statemachine: TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS fails
[
https://issues.apache.org/jira/browse/TEZ-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184009#comment-15184009
]
Tsuyoshi Ozawa commented on TEZ-3096:
-
[~gopalv] [~aplusplus] maybe duplicated issue of TEZ-3148?
> Statemachine: TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS fails
> ---
>
> Key: TEZ-3096
> URL: https://issues.apache.org/jira/browse/TEZ-3096
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Zhiyuan Yang
>
> Tasks are failing exactly 300ms into running due to a FileSystem error.
> {code}
> 2016-02-04 05:05:56,853 [ERROR] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: Can't handle this event at current state for
> attempt_1454544113740_0027_1_00_03_3
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:795)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:120)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2180)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2165)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
> at java.lang.Thread.run(Thread.java:745)
> 2016-02-04 05:05:56,903 [ERROR] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: Can't handle this event at current state for
> attempt_1454544113740_0027_1_00_00_3
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_TEZ_EVENT_UPDATE at KILL_IN_PROGRESS
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:795)
> at
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:120)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2180)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2165)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3010) Container task acquisition has no retries for errors
[ https://issues.apache.org/jira/browse/TEZ-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3010: Attachment: TEZ-3010.001.patch Attaching first patch to add retry semantics to TezChild. This patch is created on the patch of TEZ-3009. > Container task acquisition has no retries for errors > > > Key: TEZ-3010 > URL: https://issues.apache.org/jira/browse/TEZ-3010 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe > Attachments: TEZ-3010.001.patch > > > There's no retries for errors that occur during task acquisition. If any > error occurs the container will just shut down, resulting in task attempt > failures if a task attempt happened to be assigned to the container by the > AM. The container should try harder to obtain the task before giving up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-3148:
Attachment: TEZ-3148.001.patch
Adding transition after starting task attempt to handle TA_TEZ_EVENT_UPDATE
without exception. I think this can happen when network delays.
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: TEZ-3148.001.patch, am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2756) MergeManager close should not try merging files on close if invoked after a shuffle exception
[ https://issues.apache.org/jira/browse/TEZ-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179517#comment-15179517 ] Tsuyoshi Ozawa commented on TEZ-2756: - [~rajesh.balamohan] thanks Rajesh for the reviewing and committing this. > MergeManager close should not try merging files on close if invoked after a > shuffle exception > - > > Key: TEZ-2756 > URL: https://issues.apache.org/jira/browse/TEZ-2756 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Fix For: 0.8.3 > > Attachments: TEZ-2756.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2954) Container launch timeouts should count towards node blacklisting
[ https://issues.apache.org/jira/browse/TEZ-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179515#comment-15179515 ] Tsuyoshi Ozawa commented on TEZ-2954: - I think the patch includes TEZ-925. [~sseth] could you take a look? > Container launch timeouts should count towards node blacklisting > > > Key: TEZ-2954 > URL: https://issues.apache.org/jira/browse/TEZ-2954 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2954.001.patch > > > Currently, only task failures count towards blacklisting. A container timing > out should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2954) Container launch timeouts should count towards node blacklisting
[ https://issues.apache.org/jira/browse/TEZ-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-2954: --- Assignee: Tsuyoshi Ozawa > Container launch timeouts should count towards node blacklisting > > > Key: TEZ-2954 > URL: https://issues.apache.org/jira/browse/TEZ-2954 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2954.001.patch > > > Currently, only task failures count towards blacklisting. A container timing > out should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2954) Container launch timeouts should count towards node blacklisting
[ https://issues.apache.org/jira/browse/TEZ-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2954: Attachment: TEZ-2954.001.patch Attaching a first patch to consider container-launch timeout towards node blacklisting. > Container launch timeouts should count towards node blacklisting > > > Key: TEZ-2954 > URL: https://issues.apache.org/jira/browse/TEZ-2954 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth > Attachments: TEZ-2954.001.patch > > > Currently, only task failures count towards blacklisting. A container timing > out should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2756) MergeManager close should not try merging files on close if invoked after a shuffle exception
[ https://issues.apache.org/jira/browse/TEZ-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179041#comment-15179041 ] Tsuyoshi Ozawa commented on TEZ-2756: - [~sseth] could you take a look? > MergeManager close should not try merging files on close if invoked after a > shuffle exception > - > > Key: TEZ-2756 > URL: https://issues.apache.org/jira/browse/TEZ-2756 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2756.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2580) Remove VertexManagerPlugin#setVertexParallelism with VertexManagerPlugin#reconfigureVertex
[ https://issues.apache.org/jira/browse/TEZ-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179039#comment-15179039 ] Tsuyoshi Ozawa commented on TEZ-2580: - Make sense. Pending this for that it's done. > Remove VertexManagerPlugin#setVertexParallelism with > VertexManagerPlugin#reconfigureVertex > -- > > Key: TEZ-2580 > URL: https://issues.apache.org/jira/browse/TEZ-2580 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Tsuyoshi Ozawa >Priority: Blocker > Attachments: TEZ-2580.001.patch > > > This was deprecated in 0.7. Should be replaced with reconfigureVertex() - > change of name - to make it consistent with other reconfigureVertex() API's. > Should be done just close to release to enabled Hive to continue to build/use > master of Tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[ https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2988: Attachment: TEZ-2988.002.patch [~hitesh] thanks for your review. Completely agree, updating a patch. > DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked > earlier > - > > Key: TEZ-2988 > URL: https://issues.apache.org/jira/browse/TEZ-2988 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2988.001.patch, TEZ-2988.002.patch > > > If the session is already shutting down, DAGAppMaster::shutdownTezAM() should > not block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[ https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2988: Attachment: TEZ-2988.001.patch Attaching v1 patch. [~hitesh], could you take a look? > DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked > earlier > - > > Key: TEZ-2988 > URL: https://issues.apache.org/jira/browse/TEZ-2988 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah > Attachments: TEZ-2988.001.patch > > > If the session is already shutting down, DAGAppMaster::shutdownTezAM() should > not block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3120: Attachment: (was: TEZ-3120.001.patch) > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2580) Remove VertexManagerPlugin#setVertexParallelism with VertexManagerPlugin#reconfigureVertex
[ https://issues.apache.org/jira/browse/TEZ-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2580: Attachment: TEZ-2580.001.patch Attaching a patch to remove setVertexParallelism API. > Remove VertexManagerPlugin#setVertexParallelism with > VertexManagerPlugin#reconfigureVertex > -- > > Key: TEZ-2580 > URL: https://issues.apache.org/jira/browse/TEZ-2580 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Priority: Blocker > Attachments: TEZ-2580.001.patch > > > This was deprecated in 0.7. Should be replaced with reconfigureVertex() - > change of name - to make it consistent with other reconfigureVertex() API's. > Should be done just close to release to enabled Hive to continue to build/use > master of Tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2756) MergeManager close should not try merging files on close if invoked after a shuffle exception
[ https://issues.apache.org/jira/browse/TEZ-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2756: Attachment: TEZ-2756.001.patch Attaching v1 patch. > MergeManager close should not try merging files on close if invoked after a > shuffle exception > - > > Key: TEZ-2756 > URL: https://issues.apache.org/jira/browse/TEZ-2756 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Siddharth Seth > Attachments: TEZ-2756.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1944) OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
[
https://issues.apache.org/jira/browse/TEZ-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171028#comment-15171028
]
Tsuyoshi Ozawa commented on TEZ-1944:
-
Can we close this as duplicated problem of TEZ-1911?
> OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
> ---
>
> Key: TEZ-1944
> URL: https://issues.apache.org/jira/browse/TEZ-1944
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> ETL workload at 10 TB scale
> {code}
> $HIVE_HOME/bin/hive -hiveconf tez.runtime.io.sort.factor=200 --hiveconf
> hive.tez.auto.reducer.parallelism=true --hiveconf
> tez.am.heartbeat.interval-ms.max=20 --hiveconf tez.runtime.io.sort.mb=1200
> --hiveconf tez.runtime.sort.threads=2 --hiveconf hive.tez.container.size=4096
> --hiveconf tez.runtime.shuffle.memory-to-memory.enable=true --hiveconf
> tez.runtime.shuffle.memory-to-memory.segments=4
> create table testData as select
> ss_sold_date_sk,ss_sold_time_sk,ss_item_sk,ss_customer_sk,ss_quantity,ss_sold_date
> from store_sales distribute by ss_sold_date;
> {code}
> Tasks starts throwing OOM with the following exception.
> {code}
> aused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:56)
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:46)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.(MapOutput.java:81)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.createMemoryMapOutput(MapOutput.java:133)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.unconditionalReserve(MergeManager.java:397)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.access$300(MergeManager.java:74)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$IntermediateMemoryToMemoryMerger.merge(MergeManager.java:530)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
> , errorMessage=Shuffle Runner
> Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
> error in shuffle in MemToMemMerger [Map_1]
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:349)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:328)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Created this to track issues related to memory to memory merging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171026#comment-15171026 ] Tsuyoshi Ozawa commented on TEZ-3009: - Kicking CI again. [~sseth], the patch looks good to me(non-binding). Could you check it? > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171024#comment-15171024 ] Tsuyoshi Ozawa commented on TEZ-3014: - Linking to TEZ-3114 and TEZ-3115 since they seems to be related. > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171023#comment-15171023 ] Tsuyoshi Ozawa commented on TEZ-3120: - Opps, overlooking that assignee is Sidd. Feel free to use the patch. Thanks > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3120: Attachment: TEZ-3120.001.patch Attaching a patch to remove the deprecated method. > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[ https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3128: Attachment: TEZ-3128.004.patch Attaching v04 patch to remove a needless comment. > Avoid stopping containers on the AM shutdown thread > --- > > Key: TEZ-3128 > URL: https://issues.apache.org/jira/browse/TEZ-3128 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.0-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3128.001.patch, TEZ-3128.002.patch, > TEZ-3128.003.patch, TEZ-3128.004.patch, amJstack > > > During an AM shutdown, the TaskCommunicator is also shutdown and it tries to > stop containers in the shutdown thread itself. This can cause the AM shutdown > to block if NMs are not available. > This likely affects 0.7 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[ https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169083#comment-15169083 ] Tsuyoshi Ozawa commented on TEZ-3128: - Attached v03 patch to address the comment. > Avoid stopping containers on the AM shutdown thread > --- > > Key: TEZ-3128 > URL: https://issues.apache.org/jira/browse/TEZ-3128 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.0-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3128.001.patch, TEZ-3128.002.patch, > TEZ-3128.003.patch, amJstack > > > During an AM shutdown, the TaskCommunicator is also shutdown and it tries to > stop containers in the shutdown thread itself. This can cause the AM shutdown > to block if NMs are not available. > This likely affects 0.7 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[ https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167944#comment-15167944 ] Tsuyoshi Ozawa commented on TEZ-3128: - [~sseth] [~hitesh] could you check the patch? > Avoid stopping containers on the AM shutdown thread > --- > > Key: TEZ-3128 > URL: https://issues.apache.org/jira/browse/TEZ-3128 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.0-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3128.001.patch, TEZ-3128.002.patch, amJstack > > > During an AM shutdown, the TaskCommunicator is also shutdown and it tries to > stop containers in the shutdown thread itself. This can cause the AM shutdown > to block if NMs are not available. > This likely affects 0.7 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[ https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3128: Attachment: TEZ-3128.002.patch Sidd and Hitesh, thank for the clarification. Created the patch to make the kill faster when shutdown by making the interval of NMProxy's retry and the timeout of creating connection smaller. > Avoid stopping containers on the AM shutdown thread > --- > > Key: TEZ-3128 > URL: https://issues.apache.org/jira/browse/TEZ-3128 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.0-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3128.001.patch, TEZ-3128.002.patch, amJstack > > > During an AM shutdown, the TaskCommunicator is also shutdown and it tries to > stop containers in the shutdown thread itself. This can cause the AM shutdown > to block if NMs are not available. > This likely affects 0.7 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3067) Links to tez configs documentation should be bubbled up to top-level release page
[ https://issues.apache.org/jira/browse/TEZ-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158007#comment-15158007 ] Tsuyoshi Ozawa commented on TEZ-3067: - Thanks [~hitesh] for your committing and reviewing. > Links to tez configs documentation should be bubbled up to top-level release > page > -- > > Key: TEZ-3067 > URL: https://issues.apache.org/jira/browse/TEZ-3067 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Labels: newbie > Fix For: 0.8.3 > > Attachments: TEZ-3067.001.patch, TEZ-3067.002.patch > > > http://tez.apache.org/releases/0.8.2/tez-api-javadocs/configs/TezConfiguration.html > is hidden away in the api docs. Would you useful to update the release > template to add direct links to the config docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[
https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158004#comment-15158004
]
Tsuyoshi Ozawa commented on TEZ-3128:
-
[~hitesh] [~sseth] Thank you for pointing.
{quote}
dagappmaster shuts down yarn scheduler service but it does not kill containers
on shutdown - just releases them via amrmclient
TezTaskCommunicatorImpl on stop() does nothing to kill containers.
{quote}
Right, that's why I thought the place I fixed was what you mentioned. Could you
help me to clarify where to fix?
> Avoid stopping containers on the AM shutdown thread
> ---
>
> Key: TEZ-3128
> URL: https://issues.apache.org/jira/browse/TEZ-3128
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.0-alpha
>Reporter: Siddharth Seth
>Assignee: Tsuyoshi Ozawa
> Labels: newbie
> Attachments: TEZ-3128.001.patch
>
>
> During an AM shutdown, the TaskCommunicator is also shutdown and it tries to
> stop containers in the shutdown thread itself. This can cause the AM shutdown
> to block if NMs are not available.
> This likely affects 0.7 as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[ https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3128: Attachment: TEZ-3128.001.patch Attaching a patch to make the YarnTaskSchedulerService's timeout for shutdown smaller. > Avoid stopping containers on the AM shutdown thread > --- > > Key: TEZ-3128 > URL: https://issues.apache.org/jira/browse/TEZ-3128 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.0-alpha >Reporter: Siddharth Seth > Labels: newbie > Attachments: TEZ-3128.001.patch > > > During an AM shutdown, the TaskCommunicator is also shutdown and it tries to > stop containers in the shutdown thread itself. This can cause the AM shutdown > to block if NMs are not available. > This likely affects 0.7 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3128) Avoid stopping containers on the AM shutdown thread
[ https://issues.apache.org/jira/browse/TEZ-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3128: --- Assignee: Tsuyoshi Ozawa > Avoid stopping containers on the AM shutdown thread > --- > > Key: TEZ-3128 > URL: https://issues.apache.org/jira/browse/TEZ-3128 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.0-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3128.001.patch > > > During an AM shutdown, the TaskCommunicator is also shutdown and it tries to > stop containers in the shutdown thread itself. This can cause the AM shutdown > to block if NMs are not available. > This likely affects 0.7 as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3067) Links to tez configs documentation should be bubbled up to top-level release page
[ https://issues.apache.org/jira/browse/TEZ-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3067: Attachment: TEZ-3067.002.patch [~hitesh] thanks for your review. Updating a patch to add the links to the runtime configuration. > Links to tez configs documentation should be bubbled up to top-level release > page > -- > > Key: TEZ-3067 > URL: https://issues.apache.org/jira/browse/TEZ-3067 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3067.001.patch, TEZ-3067.002.patch > > > http://tez.apache.org/releases/0.8.2/tez-api-javadocs/configs/TezConfiguration.html > is hidden away in the api docs. Would you useful to update the release > template to add direct links to the config docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3067) Links to tez configs documentation should be bubbled up to top-level release page
[ https://issues.apache.org/jira/browse/TEZ-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156403#comment-15156403 ] Tsuyoshi Ozawa commented on TEZ-3067: - [~hitesh] could you take a look? > Links to tez configs documentation should be bubbled up to top-level release > page > -- > > Key: TEZ-3067 > URL: https://issues.apache.org/jira/browse/TEZ-3067 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3067.001.patch > > > http://tez.apache.org/releases/0.8.2/tez-api-javadocs/configs/TezConfiguration.html > is hidden away in the api docs. Would you useful to update the release > template to add direct links to the config docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3067) Links to tez configs documentation should be bubbled up to top-level release page
[ https://issues.apache.org/jira/browse/TEZ-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3067: Attachment: TEZ-3067.001.patch Add a link to the page of configuration. Before 0.8.1-alpha, the links don't exist, so this patch only adds the links for 0.8.1-alpha and 0.8.2. > Links to tez configs documentation should be bubbled up to top-level release > page > -- > > Key: TEZ-3067 > URL: https://issues.apache.org/jira/browse/TEZ-3067 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Labels: newbie > Attachments: TEZ-3067.001.patch > > > http://tez.apache.org/releases/0.8.2/tez-api-javadocs/configs/TezConfiguration.html > is hidden away in the api docs. Would you useful to update the release > template to add direct links to the config docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3067) Links to tez configs documentation should be bubbled up to top-level release page
[ https://issues.apache.org/jira/browse/TEZ-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa reassigned TEZ-3067: --- Assignee: Tsuyoshi Ozawa > Links to tez configs documentation should be bubbled up to top-level release > page > -- > > Key: TEZ-3067 > URL: https://issues.apache.org/jira/browse/TEZ-3067 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Labels: newbie > > http://tez.apache.org/releases/0.8.2/tez-api-javadocs/configs/TezConfiguration.html > is hidden away in the api docs. Would you useful to update the release > template to add direct links to the config docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1670) Add tests for all converter functions in HistoryEventTimelineConversion
[ https://issues.apache.org/jira/browse/TEZ-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1469#comment-1469 ] Tsuyoshi Ozawa commented on TEZ-1670: - Hi Tatsuya, thank you for taking this issue. Submitting a patch instead of him. Committers, could you can add Tatsuya to contributor list? > Add tests for all converter functions in HistoryEventTimelineConversion > --- > > Key: TEZ-1670 > URL: https://issues.apache.org/jira/browse/TEZ-1670 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Minor > Labels: newbie > Attachments: TEZ-1670.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2932) Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на русском [07.11.2015]
[ https://issues.apache.org/jira/browse/TEZ-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa resolved TEZ-2932. - Resolution: Invalid > Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на > русском [07.11.2015] > --- > > Key: TEZ-2932 > URL: https://issues.apache.org/jira/browse/TEZ-2932 > Project: Apache Tez > Issue Type: Bug >Reporter: lhokour > > Смотреть Демоны Да Винчи 3 сезон 2 серия на сайте - http://bit.ly/1OYk13V > Смотреть Демоны Да Винчи 3 сезон 2 серия на сайте - http://bit.ly/1OYk13V > Смотреть Демоны Да Винчи 3 сезон 2 серия на сайте - http://bit.ly/1OYk13V > Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на > русском [07.11.2015] > Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на > русском [07.11.2015] > Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на > русском [07.11.2015] > Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на > русском [07.11.2015] > Демоны Да Винчи 3 сезон 2 серия смотреть сериал онлайн в качестве HD720 на > русском [07.11.2015] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2849) Implement Specific Workaround for JDK-8026049 & JDK-8073093
[
https://issues.apache.org/jira/browse/TEZ-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated TEZ-2849:
Attachment: TEZ-2849.2.patch
[~gopalv] Attaching a patch rebased on trunk. Unfortunately, it fails with
UnsupportedOperationException.
{quote}
15/10/06 23:54:42 INFO examples.TeraSortWithSession: DAG diagnostics: [Vertex
failed, vertexName=TeraSortMapper, vertexId=vertex_1444175589504_0001_2_00,
diagnostics=[Task failed, taskId=task_1444175589504_0001_2_00_16,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:
attempt_1444175589504_0001_2_00_16_0:java.lang.UnsupportedOperationException
at java.nio.ByteBuffer.array(ByteBuffer.java:959)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.compareKeys(PipelinedSorter.java:828)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.compare(PipelinedSorter.java:847)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:99)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.sort(PipelinedSorter.java:794)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.flush(PipelinedSorter.java:535)
at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:182)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:375)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:79)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{quote}
> Implement Specific Workaround for JDK-8026049 & JDK-8073093
>
>
> Key: TEZ-2849
> URL: https://issues.apache.org/jira/browse/TEZ-2849
> Project: Apache Tez
> Issue Type: Sub-task
>Affects Versions: 0.8.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: TEZ-2849.1.patch, TEZ-2849.2.patch
>
>
> The CPU performance overhead of the sort operation is tied to the JIT
> handling of the aliased byte buffers without an explicit alignment (i.e a
> byte[] buffer referenced as an IntBuffer).
> https://bugs.openjdk.java.net/browse/JDK-8026049
> +
> https://bugs.openjdk.java.net/browse/JDK-8073093
> Workaround that JIT limitation while keeping in mind the modern CPU
> micro-code optimizations around alignment.
> http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2849) Implement Specific Workaround for JDK-8026049 & JDK-8073093
[ https://issues.apache.org/jira/browse/TEZ-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940866#comment-14940866 ] Tsuyoshi Ozawa commented on TEZ-2849: - [~gopalv] thanks for your patch! I'll take a benchmark with your patch in this weekend. Please wait a moment. > Implement Specific Workaround for JDK-8026049 & JDK-8073093 > > > Key: TEZ-2849 > URL: https://issues.apache.org/jira/browse/TEZ-2849 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Gopal V >Assignee: Gopal V > Attachments: TEZ-2849.1.patch > > > The CPU performance overhead of the sort operation is tied to the JIT > handling of the aliased byte buffers without an explicit alignment (i.e a > byte[] buffer referenced as an IntBuffer). > https://bugs.openjdk.java.net/browse/JDK-8026049 > + > https://bugs.openjdk.java.net/browse/JDK-8073093 > Workaround that JIT limitation while keeping in mind the modern CPU > micro-code optimizations around alignment. > http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2607) SIMD-based bitonic merge sorting
[ https://issues.apache.org/jira/browse/TEZ-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712747#comment-14712747 ] Tsuyoshi Ozawa commented on TEZ-2607: - I've found that we need a hack in a tricky way to embed SIMD-based bitonic merge into Tez code. I'm designing it, but we can split the bottleneck into 2 phases - sorting phase(Text#compare in the figure), doing this on this jira, and swapping phase(IntBuffer#get/IntBuffer#put), creating TEZ-2746 for addressing the issue. SIMD-based bitonic merge sorting Key: TEZ-2607 URL: https://issues.apache.org/jira/browse/TEZ-2607 Project: Apache Tez Issue Type: Sub-task Reporter: Tsuyoshi Ozawa Assignee: Tsuyoshi Ozawa Attachments: map_phase.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2607) SIMD-based bitonic merge sorting
[ https://issues.apache.org/jira/browse/TEZ-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699070#comment-14699070 ] Tsuyoshi Ozawa commented on TEZ-2607: - Implemented bitonic_algorithm with [~maropu]. https://github.com/oza/bitonic_sort Flash report of micro benchmark is as follows: ||algorithm||speed(million sort per sec)|| |qsort(C)|5.9883126432| |bitonic_sort(C)|29.1652639347| I've started to work integrate this code with Tez. SIMD-based bitonic merge sorting Key: TEZ-2607 URL: https://issues.apache.org/jira/browse/TEZ-2607 Project: Apache Tez Issue Type: Sub-task Reporter: Tsuyoshi Ozawa Assignee: Tsuyoshi Ozawa Attachments: map_phase.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2602) Throwing EOFException when launching MR job
[
https://issues.apache.org/jira/browse/TEZ-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14638342#comment-14638342
]
Tsuyoshi Ozawa commented on TEZ-2602:
-
Thanks Gopal for review and thanks Rajesh for the fix.
Throwing EOFException when launching MR job
---
Key: TEZ-2602
URL: https://issues.apache.org/jira/browse/TEZ-2602
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0, 0.7.0, 0.8.0
Reporter: Tsuyoshi Ozawa
Assignee: Rajesh Balamohan
Fix For: 0.8.0
Attachments: TEZ-2602.1.patch, TEZ-2602.WIP.1.patch
{quote}
$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar
wordcount -Dmapreduce.framework.name=yarn-tez -Dmapr
ed.reduce.tasks=15 -Dtez.runtime.sort.threads=1 wc10g tezwc10g5
15/07/07 13:24:30 INFO client.RMProxy: Connecting to ResourceManager at
/127.0.0.1:8081
15/07/07 13:24:30 INFO client.AHSProxy: Connecting to Application History
server at /0.0.0.0:10200
15/07/07 13:24:30 INFO mapreduce.Job: The url to track the job:
http://ip-172-31-4-8.ap-northeast-1.compute.internal:8088/proxy/application_1435943097882_0019/
15/07/07 13:24:30 INFO mapreduce.Job: Running job: job_1435943097882_0019
15/07/07 13:24:31 INFO mapreduce.Job: Job job_1435943097882_0019 running in
uber mode : false
15/07/07 13:24:31 INFO mapreduce.Job: map 0% reduce 0%
15/07/07 13:24:59 INFO mapreduce.Job: Job job_1435943097882_0019 failed with
state FAILED due to: Vertex failed, vertexName=initialmap,
vertexId=vertex_1435943097882_0019_1_00, diagnostics=[Task failed,
taskId=task_1435943097882_0019_1_00_05, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running task:java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at org.apache.hadoop.io.Text.readWithKnownLength(Text.java:319)
at org.apache.hadoop.io.Text.readFields(Text.java:291)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at
org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:142)
at
org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121)
at
org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170)
at
org.apache.tez.mapreduce.combine.MRCombiner.runNewCombiner(MRCombiner.java:191)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:115)
at
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.runCombineProcessor(ExternalSorter.java:285)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.spill(PipelinedSorter.java:463)
at
org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:219)
at
