[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050931#comment-17050931 ] Zhenyu Zheng commented on YARN-9897: [~ganeshraju] Hi, ARM CI has been added here [https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/] it runs across all Hadoop components > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10041) Should not use AbstractPath to create unix domain socket
[ https://issues.apache.org/jira/browse/YARN-10041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999839#comment-16999839 ] Zhenyu Zheng commented on YARN-10041: - [~tangzhankun] Liu Shang has provided a patch > Should not use AbstractPath to create unix domain socket > > > Key: YARN-10041 > URL: https://issues.apache.org/jira/browse/YARN-10041 > Project: Hadoop YARN > Issue Type: Bug > Components: client > Environment: X86/ARM > OS: ubuntu 1804 > java: java8 >Reporter: zhao bo >Priority: Major > > This issue hits by a very coincidental scene. That 's happend when we test on > ARM. > The test case is: > org.apache.hadoop.yarn.csi.client.TestCsiClient.testIdentityService > > The step is: > If we make the hadoop source code dir to a very deep dir path, this case > would be pass at the first time running, but always fail in the following > tries. > The official jenkins doesn't cover this, because it runs on Docker container > and just run test 1 time. So it looks like alway pass. > > The key point is the UNIX domain socket path exceed the limit of > UNIX_PATH_MAX(108). Please see [1] > > This issue is very difficult to locate, as it will always return binding > failed when we exec the test. > > Also I saw the hadoop code in trunk branch, the code use the AbsolutePath to > create the UNIX DOMAIN SOCKET file. The source code is [2]. So that can not > forbid to hit this issue. That's good to provide a second way to set the > socket path to '/tmp' or any place when exec this test. > [1] > [https://serverfault.com/questions/641347/check-if-a-path-exceeds-maximum-for-unix-domain-socket] > [2] > [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/client/TestCsiClient.java#L48] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960233#comment-16960233 ] Zhenyu Zheng commented on YARN-9897: [~eyang] Thanks alot for the help and suggestion, we have already contacted infra team, and we are now waiting for some new Aarch64 server to be in place for donating, we hope it could be done in next week, so let's wait and see. Thanks again for the help. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959555#comment-16959555 ] Zhenyu Zheng commented on YARN-9897: [~eyang] Thanks for the support, do you have any suggestion about how we should try to promote this idea? And we have added a job in openlabtesting.org so that people can see what we are trying to do. The job are defined using ansbile playbooks: https://github.com/theopenlab/openlab-zuul-jobs/blob/master/playbooks/hadoop-yarn-unit-test-arm64/run.yaml as you can see from the script, we have done few walk-arounds: 1. protocbuf does not support aarch64 but later version supports, we have proposed to upgrade it and it is ongoing now: https://issues.apache.org/jira/browse/HADOOP-13363 in our script, we just cherry-picked the patch made in protobuf to make it work and packed a local package, this won't be needed anymore if the upgrade is done. 2.phantomjs does not support aarch64 and we have contacted the author and seems the project is not maintained anymore, so we also downloaded the source code and built local package, this walk-around could also be removed if we do similar things with leveldbjni. 3.netty does not have a support aarch64 and someone is working on it: https://github.com/netty/netty/issues/8279, so we also downloaded the source code and compile locally, in order to remove this walk-around, we could first upload the arm64 package to openlabtesting maven repo and then switch back to official one once they got aarch64 supported. 4.protoc-gen-grpc-java lack aarch64 support, we have not yet contact grpc team yet, and we are using aajisaka's package, we can also remove this walk-around by upload it to openlabtesting maven repo or contact grpc team to see what we can do. Here is the job panel for the job http://status.openlabtesting.org/job/hadoop-yarn-unit-test-arm64 it is a periodic job and will be running at 10.00 UTC everyday so you can check the logs latter by click ``build-history`` and thte ``result`` section for the build. One more thing I want to mention that openlabtesting.org is just the platform that we are using to testing now, we are willing to connect it to the current CI system, but it is not mandantory, we are also able to provide servers directly to the current CI system if people thinks it is better. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959347#comment-16959347 ] Zhenyu Zheng commented on YARN-9897: [~eyang] Thanks alot for the discussion and review, it really helped us alot. And what do you think about adding a general CI? It will keep proving that YARN is OK to run on ARM? I think it might be very useful if YARN can run/manage resources for both arch. We will try to run our tests on a our public OpenLab site, so that you can see full log and scripts we used. I will update the link latter. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957487#comment-16957487 ] Zhenyu Zheng commented on YARN-9897: [~eyang]BTW, we have actually started to run tests and debug for about a month now and we can pass all YARN tests with only a few fixs like: https://issues.apache.org/jira/browse/HADOOP-16614 (it is only a possible proposal, we are open for discussions) > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956777#comment-16956777 ] Zhenyu Zheng commented on YARN-9897: Some updates, our team has succesfully donated ARM resources and setup an ARM CI for Apache Spark: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/ it will set to periodic job and then PR trigger when we think it is stable enough. And it includes some basic YARN tests, and it seems OK. I really hope we can do the same for YARN. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated YARN-9897: --- Attachment: hadoop_build.log > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956643#comment-16956643 ] Zhenyu Zheng commented on YARN-9897: [~eyang]Hi, the log is in the attachment field. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > Attachments: hadoop_build.log > > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9897) Add an Aarch64 CI for YARN
[ https://issues.apache.org/jira/browse/YARN-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated YARN-9897: --- Description: As YARN is the resource manager of Hadoop and there are large number of other software that also uses YARN for resource management. The capability of running YARN on platforms with different architecture and managing hardware resources with different architecture could be very important and useful. Aarch64(ARM) architecture is currently the dominate architecture in small devices like phone, IOT devices, security cameras, drones etc. With the increasing compuiting capability and the increasing connection speed like 5G network, there could be greate posibility and opportunity for world chaging inovations and new market if we can managing and make use of those devices as well. Currently, all YARN CIs are based on x86 architecture and we have been performing tests on Aarch64 and proposing possible solutions for problems we have meet, like: https://issues.apache.org/jira/browse/HADOOP-16614 we have done all YARN tests and it turns out there are only a few problems, and we can provide possible solutions for discussion. We want to propose to add an Aarch64 CI for YARN to promote the support for YARN on Aarch64 platforms. We are willing to provide machines to the current CI system and manpower to mananging the CI and fxing problems that occours. was: As YARN is the resource manager of Hadoop and there are large number of other software that also uses YARN for resource management. The capability of running YARN on platforms with different architecture and managing hardware resources with different architecture could be very important and useful. Aarch64(ARM) architecture is currently the dominate architecture in small devices like phone, IOT devices, security cameras, drones etc. With the increasing compuiting capability and the increasing connection speed like 5G network, there could be greate posibility and opportunity for world chaging inovations and new market if we can managing and make use of those devices as well. Currently, all YARN CIs are based on x86 architecture and we have been performing tests on Aarch64 and proposing possible solutions for problems we have meet, like: https://issues.apache.org/jira/browse/HADOOP-16614 we have done all YARN tests and it turns out there are only a few problems, and we can provide possible solutions for discussion. We want to propose to add an Aarch64 CI for YARN to promote the support for YARN on Aarch64 platforms. > Add an Aarch64 CI for YARN > -- > > Key: YARN-9897 > URL: https://issues.apache.org/jira/browse/YARN-9897 > Project: Hadoop YARN > Issue Type: Improvement > Components: build, test >Reporter: Zhenyu Zheng >Priority: Major > > As YARN is the resource manager of Hadoop and there are large number of other > software that also uses YARN for resource management. The capability of > running YARN on platforms with different architecture and managing hardware > resources with different architecture could be very important and useful. > Aarch64(ARM) architecture is currently the dominate architecture in small > devices like phone, IOT devices, security cameras, drones etc. With the > increasing compuiting capability and the increasing connection speed like 5G > network, there could be greate posibility and opportunity for world chaging > inovations and new market if we can managing and make use of those devices as > well. > Currently, all YARN CIs are based on x86 architecture and we have been > performing tests on Aarch64 and proposing possible solutions for problems we > have meet, like: > https://issues.apache.org/jira/browse/HADOOP-16614 > we have done all YARN tests and it turns out there are only a few problems, > and we can provide possible solutions for discussion. > We want to propose to add an Aarch64 CI for YARN to promote the support for > YARN on Aarch64 platforms. We are willing to provide machines to the current > CI system and manpower to mananging the CI and fxing problems that occours. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9897) Add an Aarch64 CI for YARN
Zhenyu Zheng created YARN-9897: -- Summary: Add an Aarch64 CI for YARN Key: YARN-9897 URL: https://issues.apache.org/jira/browse/YARN-9897 Project: Hadoop YARN Issue Type: Improvement Components: build, test Reporter: Zhenyu Zheng As YARN is the resource manager of Hadoop and there are large number of other software that also uses YARN for resource management. The capability of running YARN on platforms with different architecture and managing hardware resources with different architecture could be very important and useful. Aarch64(ARM) architecture is currently the dominate architecture in small devices like phone, IOT devices, security cameras, drones etc. With the increasing compuiting capability and the increasing connection speed like 5G network, there could be greate posibility and opportunity for world chaging inovations and new market if we can managing and make use of those devices as well. Currently, all YARN CIs are based on x86 architecture and we have been performing tests on Aarch64 and proposing possible solutions for problems we have meet, like: https://issues.apache.org/jira/browse/HADOOP-16614 we have done all YARN tests and it turns out there are only a few problems, and we can provide possible solutions for discussion. We want to propose to add an Aarch64 CI for YARN to promote the support for YARN on Aarch64 platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
