[jira] [Updated] (YARN-1526) change owner before setting setguid
[ https://issues.apache.org/jira/browse/YARN-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-1526: -- Attachment: create-order-chown.txt Patch enhanced. Do not optimize out chown if euid/eguid matches desired file mode. File might be created in inherit group directory change owner before setting setguid --- Key: YARN-1526 URL: https://issues.apache.org/jira/browse/YARN-1526 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.3.0 Environment: BSD Reporter: Radim Kolar Attachments: create-order-chown.txt, create-order.txt if nodemgr work directory has copy group from parent flag (often set on /tmp), chmod fails to set gid bit because owners group does not match caller group. to chown first, then chmod -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1526) change owner before setting setguid
Radim Kolar created YARN-1526: - Summary: change owner before setting setguid Key: YARN-1526 URL: https://issues.apache.org/jira/browse/YARN-1526 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 2.3.0 Environment: BSD Reporter: Radim Kolar if nodemgr work directory has copy group from parent flag (often set on /tmp), chmod fails to set gid bit because owners group does not match caller group. to chown first, then chmod -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1519) check if sysconf is implemented before using it
Radim Kolar created YARN-1519: - Summary: check if sysconf is implemented before using it Key: YARN-1519 URL: https://issues.apache.org/jira/browse/YARN-1519 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0, 2.3.0 Reporter: Radim Kolar If sysconf value _SC_GETPW_R_SIZE_MAX is not implemented, it leads to segfault because invalid pointer gets passed to libc function. fix: enforce minimum value 1024, same method is used in hadoop-common native code. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (YARN-1519) check if sysconf is implemented before using it
[ https://issues.apache.org/jira/browse/YARN-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-1519: -- Attachment: nodemgr-sysconf.txt fixed reported case and also removed compiler warning in while loop. check if sysconf is implemented before using it --- Key: YARN-1519 URL: https://issues.apache.org/jira/browse/YARN-1519 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0, 2.3.0 Reporter: Radim Kolar Attachments: nodemgr-sysconf.txt If sysconf value _SC_GETPW_R_SIZE_MAX is not implemented, it leads to segfault because invalid pointer gets passed to libc function. fix: enforce minimum value 1024, same method is used in hadoop-common native code. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1327) Fix nodemgr native compilation problems on FreeBSD9
[ https://issues.apache.org/jira/browse/YARN-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846155#comment-13846155 ] Radim Kolar commented on YARN-1327: --- can anybody look at this? it breaks hadoop on freeBSD. Fix nodemgr native compilation problems on FreeBSD9 --- Key: YARN-1327 URL: https://issues.apache.org/jira/browse/YARN-1327 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0 Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 3.0.0, 2.3.0 Attachments: nodemgr-portability.txt There are several portability problems preventing from compiling native component on freebsd. 1. libgen.h is not included. correct function prototype is there but linux glibc has workaround to define it for user if libgen.h is not directly included. Include this file directly. 2. query max size of login name using sysconf. it follows same code style like rest of code using sysconf too. 3. cgroups are linux only feature, make conditional compile and return error if mount_cgroup is attempted on non linux OS 4. do not use posix function setpgrp() since it clashes with same function from BSD 4.2, use equivalent function. After inspecting glibc sources its just shortcut to setpgid(0,0) These changes makes it compile on both linux and freebsd. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (YARN-1327) Fix nodemgr native compilation problems on FreeBSD9
Radim Kolar created YARN-1327: - Summary: Fix nodemgr native compilation problems on FreeBSD9 Key: YARN-1327 URL: https://issues.apache.org/jira/browse/YARN-1327 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0 Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 2.2.1, 3.0.0 There are several portability problems preventing from compiling native component on freebsd. 1. libgen.h is not included. correct function prototype is there but linux glibc has workaround to define it for user if libgen.h is not directly included. Include this file directly. 2. query max size of login name using sysconf. it follows same code style like rest of code using sysconf too. 3. cgroups are linux only feature, make conditional compile and return error if mount_cgroup is attempted on non linux OS 4. do not use posix function setpgrp() since it clashes with same function from BSD 4.2, use equivalent function. After inspecting glibc sources its just shortcut to setpgid(0,0) These changes makes it compile on both linux and freebsd. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-1327) Fix nodemgr native compilation problems on FreeBSD9
[ https://issues.apache.org/jira/browse/YARN-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-1327: -- Attachment: nodemgr-portability.txt Fix nodemgr native compilation problems on FreeBSD9 --- Key: YARN-1327 URL: https://issues.apache.org/jira/browse/YARN-1327 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0 Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 3.0.0, 2.2.1 Attachments: nodemgr-portability.txt There are several portability problems preventing from compiling native component on freebsd. 1. libgen.h is not included. correct function prototype is there but linux glibc has workaround to define it for user if libgen.h is not directly included. Include this file directly. 2. query max size of login name using sysconf. it follows same code style like rest of code using sysconf too. 3. cgroups are linux only feature, make conditional compile and return error if mount_cgroup is attempted on non linux OS 4. do not use posix function setpgrp() since it clashes with same function from BSD 4.2, use equivalent function. After inspecting glibc sources its just shortcut to setpgid(0,0) These changes makes it compile on both linux and freebsd. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (YARN-446) Container killed before hprof dumps profile.out
Radim Kolar created YARN-446: Summary: Container killed before hprof dumps profile.out Key: YARN-446 URL: https://issues.apache.org/jira/browse/YARN-446 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Radim Kolar If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-446) Container killed before hprof dumps profile.out
[ https://issues.apache.org/jira/browse/YARN-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13591937#comment-13591937 ] Radim Kolar commented on YARN-446: -- My original idea is to have 2nd timeout value for use cases where profiling is enabled on particular container. Container killed before hprof dumps profile.out --- Key: YARN-446 URL: https://issues.apache.org/jira/browse/YARN-446 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Radim Kolar If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-297) Improve hashCode implementations for PB records
[ https://issues.apache.org/jira/browse/YARN-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546159#comment-13546159 ] Radim Kolar commented on YARN-297: -- bigger numbers will give hashCodes larger spread and it will less likely to generate collision if you sum hashCodes. look at source code for java.lang.ThreadLocal Improve hashCode implementations for PB records --- Key: YARN-297 URL: https://issues.apache.org/jira/browse/YARN-297 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Xuan Gong Attachments: YARN.297.1.patch As [~hsn] pointed out in YARN-2, we use very small primes in all our hashCode implementations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-297) Improve hashCode implementations for PB records
[ https://issues.apache.org/jira/browse/YARN-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540617#comment-13540617 ] Radim Kolar commented on YARN-297: -- You also use initial (seed) value 1 or -1. Best is to use different prime per class. Improve hashCode implementations for PB records --- Key: YARN-297 URL: https://issues.apache.org/jira/browse/YARN-297 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Xuan Gong As [~hsn] pointed out in YARN-2, we use very small primes in all our hashCode implementations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540142#comment-13540142 ] Radim Kolar commented on YARN-2: do not use seed 1 in hashCode(), for best results both numbers (seed and multiplier) should be large primes. 31 is not large. Enhance CS to schedule accounting for both memory and cpu cores --- Key: YARN-2 URL: https://issues.apache.org/jira/browse/YARN-2 Project: Hadoop YARN Issue Type: New Feature Components: capacityscheduler, scheduler Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536273#comment-13536273 ] Radim Kolar commented on YARN-223: -- currently every system (windows and linux) which have resource calculator has process tree. if hypothetical system has just resource calculator but not resource process tree then it will miss just some stats currently sent to RM but not used anywhere. Most important function - watching container resources will be still missing without process tree. Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-211) Allow definition of max-active-applications per queue
[ https://issues.apache.org/jira/browse/YARN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13535528#comment-13535528 ] Radim Kolar commented on YARN-211: -- need review. Allow definition of max-active-applications per queue - Key: YARN-211 URL: https://issues.apache.org/jira/browse/YARN-211 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Radim Kolar Assignee: Radim Kolar Attachments: capacity-maxactive.txt, max-running.txt In some cases, automatic max-active is not enough, especially if you need less active tasks in given queue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534397#comment-13534397 ] Radim Kolar commented on YARN-223: -- its private api @InterfaceAudience.Private @InterfaceStability.Unstable Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-223: - Attachment: pstree-update6.txt reupload patch to test QA bot Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-228) move yarn.app.mapreduce.am.resource.mb to yarn-site.xml
[ https://issues.apache.org/jira/browse/YARN-228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar resolved YARN-228. -- Resolution: Invalid move yarn.app.mapreduce.am.resource.mb to yarn-site.xml --- Key: YARN-228 URL: https://issues.apache.org/jira/browse/YARN-228 Project: Hadoop YARN Issue Type: Improvement Reporter: Radim Kolar Priority: Minor property yarn.app.mapreduce.am.resource.mb is configured in mapred-site.xml. It should be moved to yarn-site.xml to be consistent with rest of yarn.* properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-228) move yarn.app.mapreduce.am.resource.mb to yarn-site.xml
[ https://issues.apache.org/jira/browse/YARN-228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504720#comment-13504720 ] Radim Kolar commented on YARN-228: -- or is there any reason why it should belong to mapred-site? move yarn.app.mapreduce.am.resource.mb to yarn-site.xml --- Key: YARN-228 URL: https://issues.apache.org/jira/browse/YARN-228 Project: Hadoop YARN Issue Type: Improvement Reporter: Radim Kolar Priority: Minor property yarn.app.mapreduce.am.resource.mb is configured in mapred-site.xml. It should be moved to yarn-site.xml to be consistent with rest of yarn.* properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-246) qa bot check
[ https://issues.apache.org/jira/browse/YARN-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-246: - Attachment: murmur3-3.txt qa bot check Key: YARN-246 URL: https://issues.apache.org/jira/browse/YARN-246 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Attachments: murmur3-3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar reassigned YARN-223: Assignee: Radim Kolar Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-211) Allow definition of max-active-applications per queue
[
https://issues.apache.org/jira/browse/YARN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502777#comment-13502777
]
Radim Kolar commented on YARN-211:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554679/max-running.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1801//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1801//console
This message is automatically generated.
Allow definition of max-active-applications per queue
-
Key: YARN-211
URL: https://issues.apache.org/jira/browse/YARN-211
Project: Hadoop YARN
Issue Type: Improvement
Components: capacityscheduler
Reporter: Radim Kolar
Assignee: Radim Kolar
Attachments: capacity-maxactive.txt, max-running.txt
In some cases, automatic max-active is not enough, especially if you need
less active tasks in given queue
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-210) Make TestQueueParsing unit test to work again
[ https://issues.apache.org/jira/browse/YARN-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502790#comment-13502790 ] Radim Kolar commented on YARN-210: -- Hadoop QA added a comment - 22/Nov/12 15:09 +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554680/capacity-unit.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1802//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1802//console This message is automatically generated. Make TestQueueParsing unit test to work again - Key: YARN-210 URL: https://issues.apache.org/jira/browse/YARN-210 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 3.0.0 Reporter: Radim Kolar Assignee: Radim Kolar Attachments: capacity-unit.txt revieve testMaxCapacity() from TestQueueParsing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-239) Make link in Aggregation is not enabled. Try the nodemanager at
[ https://issues.apache.org/jira/browse/YARN-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502865#comment-13502865 ] Radim Kolar commented on YARN-239: -- it was not fixed. Make link in Aggregation is not enabled. Try the nodemanager at - Key: YARN-239 URL: https://issues.apache.org/jira/browse/YARN-239 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.0-alpha Reporter: Radim Kolar Priority: Trivial if log aggregation is disabled message is displayed *Aggregation is not enabled. Try the nodemanager at reavers.com:9006* It would be helpfull to make link to nodemanager clickable. This message is located in /hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java but i could not figure out how to make link in hamlet framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-239) Make link in Aggregation is not enabled. Try the nodemanager at
[ https://issues.apache.org/jira/browse/YARN-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502940#comment-13502940 ] Radim Kolar commented on YARN-239: -- why not use JSP? Its way easier to work with. Make link in Aggregation is not enabled. Try the nodemanager at - Key: YARN-239 URL: https://issues.apache.org/jira/browse/YARN-239 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.0-alpha Reporter: Radim Kolar Priority: Trivial if log aggregation is disabled message is displayed *Aggregation is not enabled. Try the nodemanager at reavers.com:9006* It would be helpfull to make link to nodemanager clickable. This message is located in /hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java but i could not figure out how to make link in hamlet framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-223: - Attachment: pstree-update4.txt Migrated Task from ResourceCalculator to PsTree API. Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update2.txt, pstree-update3.txt, pstree-update4.txt, pstree-update.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-223: - Attachment: pstree-update6.txt Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-223: - Attachment: (was: pstree-update.txt) Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-223: - Attachment: (was: pstree-update3.txt) Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502494#comment-13502494 ] Radim Kolar commented on YARN-223: -- Just commit it before YARN-234 patch goes in. YARN-234 contains ResouceCalculator API change because of not understanding usage of getProcResourceValues. As this patch removes it, no changes are needed anymore. Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502534#comment-13502534 ] Radim Kolar commented on YARN-223: -- will leave it as it is. version 6 passed QA bot checks. Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update4.txt, pstree-update6.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-234) YARN support for Windows process tree and resource calculation
[ https://issues.apache.org/jira/browse/YARN-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501615#comment-13501615 ] Radim Kolar commented on YARN-234: -- Also i have YARN-223 to change interface a bit, its minor change. So wait until its committed. YARN support for Windows process tree and resource calculation -- Key: YARN-234 URL: https://issues.apache.org/jira/browse/YARN-234 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: YARN-234-branch-trunk-win.patch Port the Windows process tree code to YARN for full support of task management and resource calculations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500432#comment-13500432 ] Radim Kolar commented on YARN-223: -- problems with old code: if you are using some system resources such as file handles in native library implementation of process tree then java can create kinda lot of objects of processTree type which are not garbage collected and temporary leaks system resources. If you start chaining references to original object, then you will have allocated system resources just once but you will create long chain on objects which could not be GC'd by java until all of them are unreferenced at end of container life. I tried both and didn't liked it, after inspection of code calling psTree it was discovered that simply updating object will be enough because code do not keeps old copy around while creating a new one. Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-97) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/YARN-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar resolved YARN-97. - Resolution: Not A Problem nodemanager depends on /bin/bash Key: YARN-97 URL: https://issues.apache.org/jira/browse/YARN-97 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-223) Change processTree interface to work better with native code
Radim Kolar created YARN-223: Summary: Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-223) Change processTree interface to work better with native code
[ https://issues.apache.org/jira/browse/YARN-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-223: - Attachment: pstree-update.txt Change processTree interface to work better with native code Key: YARN-223 URL: https://issues.apache.org/jira/browse/YARN-223 Project: Hadoop YARN Issue Type: Bug Reporter: Radim Kolar Priority: Critical Attachments: pstree-update.txt Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-211) Allow definition of max-active-tasks per queue
[ https://issues.apache.org/jira/browse/YARN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495309#comment-13495309 ] Radim Kolar commented on YARN-211: -- maximum-applications is number of submitted applications to queue. This patch limits number of concurrently running applications. Allow definition of max-active-tasks per queue -- Key: YARN-211 URL: https://issues.apache.org/jira/browse/YARN-211 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Radim Kolar Attachments: capacity-maxactive.txt In some cases, automatic max-active is not enough, especially if you need less active tasks in given queue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-211) Allow definition of max-active-applications per queue
[ https://issues.apache.org/jira/browse/YARN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495350#comment-13495350 ] Radim Kolar commented on YARN-211: -- I know that it is currently controlled by percent and it can be configured per queue. This is unsuitable if you need to limit number of running applications to certain low number to work around primitive cluster scheduling in hadoop. For example if you have nodes with 2 GPU units, then create queue with maximum-running-tasks 1 or 2 and submit applications to that queue, there is better chance that they will not run concurrently on one node. Using percentage for configuration is tricky, its difficult to predict target effect and it changes with number of nodes in cluster. Allow definition of max-active-applications per queue - Key: YARN-211 URL: https://issues.apache.org/jira/browse/YARN-211 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Radim Kolar Attachments: capacity-maxactive.txt In some cases, automatic max-active is not enough, especially if you need less active tasks in given queue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495354#comment-13495354 ] Radim Kolar commented on YARN-2: Can you modify Resource to be more generic and to become Name/value map. like memory:,cores:2 Enhance CS to schedule accounting for both memory and cpu cores --- Key: YARN-2 URL: https://issues.apache.org/jira/browse/YARN-2 Project: Hadoop YARN Issue Type: New Feature Components: capacityscheduler, scheduler Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-211) Allow definition of max-active-applications per queue
[ https://issues.apache.org/jira/browse/YARN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495355#comment-13495355 ] Radim Kolar commented on YARN-211: -- with this patch, you can still use percent to configure it. But it can be overridden by number. Allow definition of max-active-applications per queue - Key: YARN-211 URL: https://issues.apache.org/jira/browse/YARN-211 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Radim Kolar Attachments: capacity-maxactive.txt In some cases, automatic max-active is not enough, especially if you need less active tasks in given queue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-210) Make TestQueueParsing unit test to work again
Radim Kolar created YARN-210: Summary: Make TestQueueParsing unit test to work again Key: YARN-210 URL: https://issues.apache.org/jira/browse/YARN-210 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 3.0.0 Reporter: Radim Kolar revieve testMaxCapacity() from TestQueueParsing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-210) Make TestQueueParsing unit test to work again
[ https://issues.apache.org/jira/browse/YARN-210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-210: - Attachment: capacity-unit.txt Make TestQueueParsing unit test to work again - Key: YARN-210 URL: https://issues.apache.org/jira/browse/YARN-210 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 3.0.0 Reporter: Radim Kolar Attachments: capacity-unit.txt revieve testMaxCapacity() from TestQueueParsing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-211) Allow definition of max-active-tasks per queue
[ https://issues.apache.org/jira/browse/YARN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-211: - Attachment: capacity-maxactive.txt Allow definition of max-active-tasks per queue -- Key: YARN-211 URL: https://issues.apache.org/jira/browse/YARN-211 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Radim Kolar Attachments: capacity-maxactive.txt In some cases, automatic max-active is not enough, especially if you need less active tasks in given queue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-211) Allow definition of max-active-tasks per queue
Radim Kolar created YARN-211: Summary: Allow definition of max-active-tasks per queue Key: YARN-211 URL: https://issues.apache.org/jira/browse/YARN-211 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Reporter: Radim Kolar Attachments: capacity-maxactive.txt In some cases, automatic max-active is not enough, especially if you need less active tasks in given queue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-178) Fix custom ProcessTree instance creation
Radim Kolar created YARN-178: Summary: Fix custom ProcessTree instance creation Key: YARN-178 URL: https://issues.apache.org/jira/browse/YARN-178 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 0.23.5 Reporter: Radim Kolar Priority: Critical Attachments: pstree-instance.txt 1. In current pluggable resourcecalculatorprocesstree is not passed root process id to custom implementation making it unusable. 2. pstree do not extend Configured as it should Added constructor with pid argument with testsuite. Also added test that pstree is correctly configured. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-178) Fix custom ProcessTree instance creation
[ https://issues.apache.org/jira/browse/YARN-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-178: - Attachment: pstree-instance.txt Fix custom ProcessTree instance creation Key: YARN-178 URL: https://issues.apache.org/jira/browse/YARN-178 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 0.23.5 Reporter: Radim Kolar Priority: Critical Attachments: pstree-instance.txt 1. In current pluggable resourcecalculatorprocesstree is not passed root process id to custom implementation making it unusable. 2. pstree do not extend Configured as it should Added constructor with pid argument with testsuite. Also added test that pstree is correctly configured. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-178) Fix custom ProcessTree instance creation
[ https://issues.apache.org/jira/browse/YARN-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar reassigned YARN-178: Assignee: Radim Kolar Fix custom ProcessTree instance creation Key: YARN-178 URL: https://issues.apache.org/jira/browse/YARN-178 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0, 0.23.5 Reporter: Radim Kolar Assignee: Radim Kolar Priority: Critical Attachments: pstree-instance2.txt, pstree-instance.txt 1. In current pluggable resourcecalculatorprocesstree is not passed root process id to custom implementation making it unusable. 2. pstree do not extend Configured as it should Added constructor with pid argument with testsuite. Also added test that pstree is correctly configured. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463645#comment-13463645 ] Radim Kolar commented on YARN-57: - Fine. You both agree, then merge it already. Plugable process tree - Key: YARN-57 URL: https://issues.apache.org/jira/browse/YARN-57 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Environment: FreeBSD 64 bit Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 3.0.0 Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, plugable-pstree-3.txt, plugable-pstree-4.txt, plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, plugable-pstree.txt, pstree-0.23-merge-2.txt, pstree-0.23-merge.txt Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-97) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/YARN-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461477#comment-13461477 ] Radim Kolar commented on YARN-97: - adding /bin/bash as requirement is fine. I do not have plans to fix this patch because i cant even get mvn test working on bsd. It dies with some obscure error. nodemanager depends on /bin/bash Key: YARN-97 URL: https://issues.apache.org/jira/browse/YARN-97 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-57: Attachment: pstree-0.23-merge-2.txt patch for 0.23 regenerated Plugable process tree - Key: YARN-57 URL: https://issues.apache.org/jira/browse/YARN-57 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Environment: FreeBSD 64 bit Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 3.0.0 Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, plugable-pstree-3.txt, plugable-pstree-4.txt, plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, plugable-pstree.txt, pstree-0.23-merge-2.txt, pstree-0.23-merge.txt Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-57: Attachment: pstree-0.23-merge.txt I need merge to 0.23, attaching patch with conflict resolved Plugable process tree - Key: YARN-57 URL: https://issues.apache.org/jira/browse/YARN-57 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Environment: FreeBSD 64 bit Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 3.0.0 Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, plugable-pstree-3.txt, plugable-pstree-4.txt, plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, plugable-pstree.txt, pstree-0.23-merge.txt Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated YARN-57: Attachment: plugable-pstree-trunk-merge-8.txt one file was missing from patch because git do not tracks directory renames Plugable process tree - Key: YARN-57 URL: https://issues.apache.org/jira/browse/YARN-57 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Environment: FreeBSD 64 bit Reporter: Radim Kolar Assignee: Radim Kolar Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, plugable-pstree-3.txt, plugable-pstree-4.txt, plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-8.txt, plugable-pstree.txt Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-57) Plugable process tree
[ https://issues.apache.org/jira/browse/YARN-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448024#comment-13448024 ] Radim Kolar commented on YARN-57: - where is log with javac errors? Plugable process tree - Key: YARN-57 URL: https://issues.apache.org/jira/browse/YARN-57 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Environment: FreeBSD 64 bit Reporter: Radim Kolar Assignee: Radim Kolar Attachments: plugable-pstree-1.txt, plugable-pstree-2.txt, plugable-pstree-3.txt, plugable-pstree-4.txt, plugable-pstree-4-with-whitespace.txt, plugable-pstree-5-with-whitespace.txt, plugable-pstree-6-typofix.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree-trunk-merge-7.txt, plugable-pstree.txt Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
