[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level
[
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943208#comment-16943208
]
Hadoop QA commented on YARN-9760:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 41s{color} | {color:orange} root: The patch generated 19 new + 534 unchanged
- 3 fixed = 553 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
1s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m
41s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 30s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
52s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}338m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9760 |
| JIRA Patch URL |
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[
https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943069#comment-16943069
]
Hadoop QA commented on YARN-5106:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 26 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
25s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
12s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 12s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 77 new + 340 unchanged - 34 fixed = 417 total (was 374) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m
19s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 24m
54s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
40s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:080e9d0f9b3 |
| JIRA Issue | YARN-5106 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981994/YARN-5106-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 754c89827dc5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / f1fe3ab |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-9792) Document examples of SchedulerConf with Node Labels
[ https://issues.apache.org/jira/browse/YARN-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943065#comment-16943065 ] Hudson commented on YARN-9792: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17438 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17438/]) YARN-9792. Document examples of SchedulerConf with Node Labels. (sunilg: rev 0d2d6f92679a0c1e169fd90706bc5ded06d088b7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md > Document examples of SchedulerConf with Node Labels > --- > > Key: YARN-9792 > URL: https://issues.apache.org/jira/browse/YARN-9792 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9792-001.patch, YARN-9792-002.patch, > YARN-9792-003.patch > > > Document examples of SchedulerConf with Node Labels. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[
https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942966#comment-16942966
]
Hadoop QA commented on YARN-5106:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 27 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
56s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
3s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 82 new + 325 unchanged - 35 fixed = 407 total (was 360) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 8s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 24m
55s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
41s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}243m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | YARN-5106 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981982/YARN-5106-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 764f8747e539 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942957#comment-16942957
]
Shane Kumpf commented on YARN-9860:
---
{quote}It looks like we are reverting to our old habit of using 0 for true. It
would be more consistent to use is_feature_enabled() method to determine if
service_mode is enabled, and reduce some code debris.
{quote}
Good points, I agree these comments should be addressed.
{quote}Container-executor already dup the container run output into stdout and
stderr log files with proper user permission for entrypoint mode because the
log files are initialized as user who runs the container executor rather than
the user in the container. It works for both secure and non-secure mode. I fail
to see the need to craft logging mechanism for the given reasoning for service
mode. Let me know if I missed something.
{quote}
Excellent, I can confirm this is working exactly how we'd want. I overlooked
this before. Seems logging isn't an issue after all. Thanks for pointing that
out!
I did retest the patch today and it is still working as expected.
With the patch applied in my dev VM, below is the ps and logs from the official
postgres image running under YARN with zero changes!
*ps:*
{code:java}
root@centos7-0:/# ps -ef
UIDPID PPID C STIME TTY TIME CMD
postgres 1 0 0 16:13 ?00:00:00 postgres
postgres53 1 0 16:13 ?00:00:00 postgres: checkpointer
postgres54 1 0 16:13 ?00:00:00 postgres: background writer
postgres55 1 0 16:13 ?00:00:00 postgres: walwriter
postgres56 1 0 16:13 ?00:00:00 postgres: autovacuum launcher
postgres57 1 0 16:13 ?00:00:00 postgres: stats collector
postgres58 1 0 16:13 ?00:00:00 postgres: logical replication
launcher
root59 0 4 16:14 pts/000:00:00 bash
root6459 0 16:14 pts/000:00:00 ps -ef
{code}
*Logs:*
{code:java}
[root@y7001 ~]# yarn logs -applicationId application_1570018164872_0005
-containerId container_1570018164872_0005_01_02
2019-10-02 16:26:40,269 INFO client.RMProxy: Connecting to ResourceManager at
y7001.yns.foo.com/192.168.70.211:9104
Container: container_1570018164872_0005_01_02 on y7001.yns.foo.com:9105
LogAggregationType: LOCAL
===
LogType:stdout.txt
LogLastModifiedTime:Wed Oct 02 16:13:31 + 2019
LogLength:2638
LogContents:
Launching docker container...
Docker run command: /usr/bin/docker run
--name=container_1570018164872_0005_01_02 --net=host -v
/tmp/hadoop-yarn/nm-local-dir/filecache/13/httpd-proxy.conf:/etc/httpd/conf.d/httpd-proxy.conf:ro
--cgroup-parent=/hadoop-yarn/container_1570018164872_0005_01_02
--cap-drop=ALL --cap-add=SYS_CHROOT --cap-add=MKNOD --cap-add=SETFCAP
--cap-add=SETPCAP --cap-add=FSETID --cap-add=CHOWN --cap-add=AUDIT_WRITE
--cap-add=SETGID --cap-add=NET_RAW --cap-add=FOWNER --cap-add=SETUID
--cap-add=DAC_OVERRIDE --cap-add=KILL --cap-add=NET_BIND_SERVICE
--hostname=centos7-0.skumpftest.hadoopuser.ynsdev --env-file
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1570018164872_0005/container_1570018164872_0005_01_02/docker.container_1570018164872_0005_01_028354800474788286290.env
library/postgres
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start2019-10-02 16:13:31.231 UTC [43] LOG: listening
on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-02 16:13:31.253 UTC [44] LOG: database system was shut down at
2019-10-02 16:13:30 UTC
2019-10-02 16:13:31.259 UTC [43] LOG: database system is ready to accept
connections
done
server started
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down...2019-10-02 16:13:31.322 UTC [43] LOG:
received fast shutdown request
.2019-10-02 16:13:31.325 UTC [43] LOG: aborting any active transactions
2019-10-02 16:13:31.329 UTC [43] LOG: background worker "logical replication
[jira] [Updated] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-9760: --- Attachment: YARN-9760.03.patch > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch, YARN-9760.02.patch, > YARN-9760.03.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942921#comment-16942921
]
Eric Yang commented on YARN-9860:
-
{quote}That code does the opposite of what you stated, user is only passed when
service mode is NOT enabled, which is what we want.{quote}
It looks like we are reverting to our old habit of using 0 for true. It would
be more consistent to use is_feature_enabled() method to determine if
service_mode is enabled, and reduce some code debris.
{quote}You are spot on that this will be an issue. The challenge is that if we
mount the read-write log dirs into the container and the container user isn't
the user YARN expects, the writes could fail or YARN may be unable to clean up
the logs. I talked with Craig on this a bit and he had some interesting
thoughts on how we might handle it with fuse. For the sake of this patch, I
didn't want to get bogged down in the details there, given this has enough
going already. Could we address logging in a follow up? In the meantime, with
debug delay enabled, doing a docker logs on the exited container will allow
admins to take a look, since the output redirection typically done by YARN is
dropped in Service Mode.{quote}
Container-executor already
[dup|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L1982]
the container run output into stdout and stderr log files with proper user
permission for entrypoint mode because the log files are initialized as user
who runs the container executor rather than the user in the container. It
works for both secure and non-secure mode. I fail to see the need to craft
logging mechanism for the given reasoning for service mode. Let me know if I
missed something.
> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service
> Mode".
> Service Mode - Run the container as defined by the image, but still allow for
> injecting configuration.
> Background:
> Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as
> defined in the image. However, still requires modification to official images
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>
> Implementation:
> Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
> Must be requested at runtime - (example:
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
> Entrypoint mode is default enabled for this mode (If Service Mode is
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set
> to true)
> Writable log mount will not be added - stdout logging may still work
> with entrypoint mode - remove the writable bind mounts
> User and groups will not be propagated (now: docker run --user nobody
> --group-add=nobody , after: docker run )
> Read-only resources mounted at the file level, files get chmod 777,
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-5106: --- Attachment: YARN-5106-branch-3.1.001.patch > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942875#comment-16942875 ] Szilard Nemeth commented on YARN-5106: -- Hi [~zsiegl]! You can upload 2 patches at the same time, however, jenkins will pick up the latest only. Just reupload one patch at a time, wait for Jenkins to pick it up then upload the next one. This is the workflow, unfortunately. Anyway, latest trunk patch (009) produced unit test failures, please check them! > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106.001.patch, YARN-5106.002.patch, > YARN-5106.003.patch, YARN-5106.004.patch, YARN-5106.005.patch, > YARN-5106.006.patch, YARN-5106.007.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.008.patch, YARN-5106.009.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[
https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942857#comment-16942857
]
Hadoop QA commented on YARN-5106:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 27 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 24s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 77 new + 296 unchanged - 34 fixed = 373 total (was 330) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 28s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m
14s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-5106 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981971/YARN-5106.009.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1bb92951baea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f1ba9bf |
| maven | version:
[jira] [Commented] (YARN-9870) Remove unused function from OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942849#comment-16942849 ] Abhishek Modi commented on YARN-9870: - Thanks [~elgoiri] for review. Committed to trunk. > Remove unused function from OpportunisticContainerAllocatorAMService > > > Key: YARN-9870 > URL: https://issues.apache.org/jira/browse/YARN-9870 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > Attachments: YARN-9870.001.patch, YARN-9870.002.patch > > > Code clean up of OpportunisticContainerAllocatorAMService and removal of > unused functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942833#comment-16942833
]
Eric Badger commented on YARN-9860:
---
{quote}
You are spot on that this will be an issue. The challenge is that if we mount
the read-write log dirs into the container and the container user isn't the
user YARN expects, the writes could fail or YARN may be unable to clean up the
logs. I talked with Craig on this a bit and he had some interesting thoughts on
how we might handle it with fuse. For the sake of this patch, I didn't want to
get bogged down in the details there, given this has enough going already.
Could we address logging in a follow up? In the meantime, with debug delay
enabled, doing a docker logs on the exited container will allow admins to take
a look, since the output redirection typically done by YARN is dropped in
Service Mode.
{quote}
I'm ok having a followup for this. My comment was mostly for my own
information. I'm not opposed to the feature, I just think it might be a little
bit difficult to deal with if there are configuration issues or other issues
that cause the containers to fail. I think that debug delay to not remove
containers for awhile after completion could be a useful debugging tool,
though.
> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service
> Mode".
> Service Mode - Run the container as defined by the image, but still allow for
> injecting configuration.
> Background:
> Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as
> defined in the image. However, still requires modification to official images
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>
> Implementation:
> Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
> Must be requested at runtime - (example:
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
> Entrypoint mode is default enabled for this mode (If Service Mode is
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set
> to true)
> Writable log mount will not be added - stdout logging may still work
> with entrypoint mode - remove the writable bind mounts
> User and groups will not be propagated (now: docker run --user nobody
> --group-add=nobody , after: docker run )
> Read-only resources mounted at the file level, files get chmod 777,
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942783#comment-16942783 ] Zoltan Siegl commented on YARN-5106: The branch-3.1 and branch-3.2 patches seem okay. Reuploaded them also for a fresh build. > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106.001.patch, YARN-5106.002.patch, > YARN-5106.003.patch, YARN-5106.004.patch, YARN-5106.005.patch, > YARN-5106.006.patch, YARN-5106.007.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.008.patch, YARN-5106.009.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9870) Remove unused function from OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942782#comment-16942782 ] Hudson commented on YARN-9870: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17436 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17436/]) YARN-9870. Remove unused function from (abmodi: rev 61a8436004294db0224dd20aa3997fa2e3136114) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java > Remove unused function from OpportunisticContainerAllocatorAMService > > > Key: YARN-9870 > URL: https://issues.apache.org/jira/browse/YARN-9870 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > Attachments: YARN-9870.001.patch, YARN-9870.002.patch > > > Code clean up of OpportunisticContainerAllocatorAMService and removal of > unused functions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-5106: --- Attachment: YARN-5106-branch-3.2.001.patch YARN-5106-branch-3.1.001.patch > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106.001.patch, YARN-5106.002.patch, > YARN-5106.003.patch, YARN-5106.004.patch, YARN-5106.005.patch, > YARN-5106.006.patch, YARN-5106.007.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.008.patch, YARN-5106.009.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942699#comment-16942699
]
Shane Kumpf commented on YARN-9860:
---
{quote}Can you give more clear definition of service mode?
{quote}
To simplify it the best I can, in Service Mode, YARN does not set the user
(--user and --group-add) when running the container. The rest of the changes
are in support of dropping the user in this mode. A simple use case where this
is needed is running the official postgres image without modification. Note
that this mode is disabled by default to limit any security implications.
{quote}I don't understand the reason to add --user= parameter only when service
mode is enabled.
{quote}
That code does the opposite of what you stated, user is only passed when
service mode is NOT enabled, which is what we want.
{quote}If there are no log directories, how would you attack debugging
container failures?
{quote}
You are spot on that this will be an issue. The challenge is that if we mount
the read-write log dirs into the container and the container user isn't the
user YARN expects, the writes could fail or YARN may be unable to clean up the
logs. I talked with Craig on this a bit and he had some interesting thoughts on
how we might handle it with fuse. For the sake of this patch, I didn't want to
get bogged down in the details there, given this has enough going already.
Could we address logging in a follow up? In the meantime, with debug delay
enabled, doing a {{docker logs}} on the exited container will allow admins to
take a look, since the output redirection typically done by YARN is dropped in
Service Mode.
I've done extensive testing of an earlier version of this patch and it
addresses the use case and works as expected. I'm going to do some additional
testing today with the patch here to make sure there are no regressions.
> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service
> Mode".
> Service Mode - Run the container as defined by the image, but still allow for
> injecting configuration.
> Background:
> Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as
> defined in the image. However, still requires modification to official images
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>
> Implementation:
> Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
> Must be requested at runtime - (example:
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
> Entrypoint mode is default enabled for this mode (If Service Mode is
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set
> to true)
> Writable log mount will not be added - stdout logging may still work
> with entrypoint mode - remove the writable bind mounts
> User and groups will not be propagated (now: docker run --user nobody
> --group-add=nobody , after: docker run )
> Read-only resources mounted at the file level, files get chmod 777,
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942697#comment-16942697 ] Zoltan Siegl commented on YARN-5106: There was a missing import in one of the test files. Fixed that and uploaded a new patch. > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106.001.patch, YARN-5106.002.patch, > YARN-5106.003.patch, YARN-5106.004.patch, YARN-5106.005.patch, > YARN-5106.006.patch, YARN-5106.007.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.008.patch, YARN-5106.009.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-5106: --- Attachment: YARN-5106.009.patch > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106.001.patch, YARN-5106.002.patch, > YARN-5106.003.patch, YARN-5106.004.patch, YARN-5106.005.patch, > YARN-5106.006.patch, YARN-5106.007.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.008.patch, YARN-5106.009.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942679#comment-16942679 ] Zoltan Siegl commented on YARN-5106: Checking up on this. > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106.001.patch, YARN-5106.002.patch, > YARN-5106.003.patch, YARN-5106.004.patch, YARN-5106.005.patch, > YARN-5106.006.patch, YARN-5106.007.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.008.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9830) Improve ContainerAllocationExpirer it blocks scheduling
[
https://issues.apache.org/jira/browse/YARN-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942631#comment-16942631
]
Hadoop QA commented on YARN-9830:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 28s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
47s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 36s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9830 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981952/YARN-9830.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 698bcf671e54 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-9862) yarn-services-core test timeout
[
https://issues.apache.org/jira/browse/YARN-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942544#comment-16942544
]
Prabhu Joseph commented on YARN-9862:
-
Thanks [~eyang].
> yarn-services-core test timeout
> ---
>
> Key: YARN-9862
> URL: https://issues.apache.org/jira/browse/YARN-9862
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9862-001.patch, YARN-9862-002.patch
>
>
> yarn-services-core test timeout. The default timeout of 15minute is not
> sufficient as the TestYarnNativeServices has 14 testcases each with timeout
> of 200 seconds which is not completing within 15minutes.
> {code}
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test)
> on project hadoop-yarn-services-core: There was a timeout or other error in
> the fork -> [Help 1]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running
> org.apache.hadoop.yarn.service.timelineservice.TestServiceTimelinePublisher
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.222
> s - in
> org.apache.hadoop.yarn.service.timelineservice.TestServiceTimelinePublisher
> [INFO] Running org.apache.hadoop.yarn.service.monitor.probe.TestDefaultProbe
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.138
> s - in org.apache.hadoop.yarn.service.monitor.probe.TestDefaultProbe
> [INFO] Running org.apache.hadoop.yarn.service.monitor.TestServiceMonitor
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.317
> s - in org.apache.hadoop.yarn.service.monitor.TestServiceMonitor
> [INFO] Running org.apache.hadoop.yarn.service.TestServiceAM
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 30.337 s - in org.apache.hadoop.yarn.service.TestServiceAM
> [INFO] Running
> org.apache.hadoop.yarn.service.TestDefaultUpgradeComponentsFinder
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.533
> s - in org.apache.hadoop.yarn.service.TestDefaultUpgradeComponentsFinder
> [INFO] Running org.apache.hadoop.yarn.service.TestServiceManager
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.33
> s - in org.apache.hadoop.yarn.service.TestServiceManager
> [INFO] Running org.apache.hadoop.yarn.service.utils.TestFilterUtils
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.799
> s - in org.apache.hadoop.yarn.service.utils.TestFilterUtils
> [INFO] Running org.apache.hadoop.yarn.service.utils.TestServiceApiUtil
> [INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.468
> s - in org.apache.hadoop.yarn.service.utils.TestServiceApiUtil
> [INFO] Running org.apache.hadoop.yarn.service.utils.TestCoreFileSystem
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.699
> s - in org.apache.hadoop.yarn.service.utils.TestCoreFileSystem
> [INFO] Running org.apache.hadoop.yarn.service.provider.TestProviderUtils
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.76 s
> - in org.apache.hadoop.yarn.service.provider.TestProviderUtils
> [INFO] Running
> org.apache.hadoop.yarn.service.provider.TestAbstractProviderService
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.814
> s - in org.apache.hadoop.yarn.service.provider.TestAbstractProviderService
> [INFO] Running
> org.apache.hadoop.yarn.service.component.TestComponentDecommissionInstances
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 108.719 s - in
> org.apache.hadoop.yarn.service.component.TestComponentDecommissionInstances
> [INFO] Running
> org.apache.hadoop.yarn.service.component.TestComponentRestartPolicy
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.108
> s - in org.apache.hadoop.yarn.service.component.TestComponentRestartPolicy
> [INFO] Running org.apache.hadoop.yarn.service.component.TestComponent
> [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.731
> s - in org.apache.hadoop.yarn.service.component.TestComponent
> [INFO] Running
> org.apache.hadoop.yarn.service.component.instance.TestComponentInstance
> [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.866
> s - in
