[jira] [Commented] (TEZ-3991) Unmanaged tez sessions
[
https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809481#comment-16809481
]
TezQA commented on TEZ-3991:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s{color}
| {color:red} tez-dag generated 1 new + 7 unchanged - 1 fixed = 8 total (was 8)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 39s{color}
| {color:red} root generated 1 new + 39 unchanged - 1 fixed = 40 total (was 40)
{color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} tez-api: The patch generated 47 new + 131
unchanged - 4 fixed = 178 total (was 135) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} tez-common: The patch generated 0 new + 69 unchanged
- 5 fixed = 69 total (was 74) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} tez-dag: The patch generated 38 new + 138
unchanged - 0 fixed = 176 total (was 138) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} tez-tests: The patch generated 23 new + 5
unchanged - 0 fixed = 28 total (was 5) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} root: The patch generated 108 new + 343
unchanged - 9 fixed = 451 total (was 352) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
59s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 15s{color}
| {color:red} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m
20s{color} | {color:green} tez-tests in the patch
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Attachment: TEZ-3991.3.patch > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > Labels: Kubernetes > Attachments: TEZ-3991.1.patch, TEZ-3991.2.patch, TEZ-3991.3.patch > > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3991) Unmanaged tez sessions
[
https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809374#comment-16809374
]
TezQA commented on TEZ-3991:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s{color}
| {color:red} tez-dag generated 1 new + 7 unchanged - 1 fixed = 8 total (was 8)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 40s{color}
| {color:red} root generated 1 new + 39 unchanged - 1 fixed = 40 total (was 40)
{color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} tez-api: The patch generated 47 new + 130
unchanged - 4 fixed = 177 total (was 134) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} tez-common: The patch generated 0 new + 69 unchanged
- 5 fixed = 69 total (was 74) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} tez-dag: The patch generated 38 new + 138
unchanged - 0 fixed = 176 total (was 138) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} tez-tests: The patch generated 23 new + 5
unchanged - 0 fixed = 28 total (was 5) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} root: The patch generated 108 new + 342
unchanged - 9 fixed = 450 total (was 351) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
16s{color} | {color:red} tez-dag generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
10s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
13s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m
[jira] [Assigned] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned TEZ-1348: --- Assignee: Todd Lipcon > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809292#comment-16809292 ] Todd Lipcon commented on TEZ-1348: -- It seems that by changing the code to ensure that the tez working directory is created on Local FS, then tez local-mode can work even if defaultFS is a remote cluster. This is actually useful, for example when testing Hive against a pseudo-distributed HDFS, if you don't want to also start a pseudo-distributed YARN. I'll work on a patch for this. > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Priority: Critical > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Attachment: TEZ-3991.2.patch > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > Labels: Kubernetes > Attachments: TEZ-3991.1.patch, TEZ-3991.2.patch > > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4059) TezSplitGrouper doesn't respect tez.grouping.max-size if the originalSplit has bigger splits
[ https://issues.apache.org/jira/browse/TEZ-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808607#comment-16808607 ] Chenren Shao commented on TEZ-4059: --- after checking more, it looks like it is working as intended. > TezSplitGrouper doesn't respect tez.grouping.max-size if the originalSplit > has bigger splits > > > Key: TEZ-4059 > URL: https://issues.apache.org/jira/browse/TEZ-4059 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 > Environment: Tez 0.9.1, Hive 2.3.4, Hadoop 2.8.5 (EMR 5.20.0), >Reporter: Chenren Shao >Priority: Minor > Attachments: split-generation-log.txt > > > In using Tez 0.9.1, Hive 2.3.4, Hadoop 2.8.5 (EMR 5.20.0), I found if we have > below scenario, TezSplitGrouper is unable to respect tez.grouping.max-size. > On line 238, Let us say, a customer-defined inputformat generate a M number > of splits, but its split size is greater than tez.grouping.max-size, then > line 248 is executed, and a newDesiredNumSplits is calculated based on > tez.grouping.max-size, N. (N is greater than M), desiredNumSplits is set to > be N. > However, when it goes to line 273, since the N is greater than M now, it > executes line 274, it uses originalSplits M anyway and basically ignores the > N setting. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-4059) TezSplitGrouper doesn't respect tez.grouping.max-size if the originalSplit has bigger splits
[ https://issues.apache.org/jira/browse/TEZ-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenren Shao resolved TEZ-4059. --- Resolution: Not A Problem > TezSplitGrouper doesn't respect tez.grouping.max-size if the originalSplit > has bigger splits > > > Key: TEZ-4059 > URL: https://issues.apache.org/jira/browse/TEZ-4059 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 > Environment: Tez 0.9.1, Hive 2.3.4, Hadoop 2.8.5 (EMR 5.20.0), >Reporter: Chenren Shao >Priority: Minor > Attachments: split-generation-log.txt > > > In using Tez 0.9.1, Hive 2.3.4, Hadoop 2.8.5 (EMR 5.20.0), I found if we have > below scenario, TezSplitGrouper is unable to respect tez.grouping.max-size. > On line 238, Let us say, a customer-defined inputformat generate a M number > of splits, but its split size is greater than tez.grouping.max-size, then > line 248 is executed, and a newDesiredNumSplits is calculated based on > tez.grouping.max-size, N. (N is greater than M), desiredNumSplits is set to > be N. > However, when it goes to line 273, since the N is greater than M now, it > executes line 274, it uses originalSplits M anyway and basically ignores the > N setting. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
