[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008748#comment-17008748 ] Vinayakumar B commented on HADOOP-16358: Thanks Everyone > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 3.3.0 > > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002352#comment-17002352 ] Ayush Saxena commented on HADOOP-16358: --- [~Kevin_Zheng] I have removed it. From the next time, it won't be there. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002228#comment-17002228 ] Zhenyu Zheng commented on HADOOP-16358: --- [~ayushtkn] Thanks a lot for the help, I saw there is a: # ERROR: File 'out/email-report.txt' does not exist in the ML notification, can we get rid of it? > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002214#comment-17002214 ] Ayush Saxena commented on HADOOP-16358: --- I have corrected the Mail issue, it will send the mail now to dev mailing list. Regarding the UNSTABLE, it seems correct Failed will come in case of script failure only. Let me know if something else is pending here. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002098#comment-17002098 ] Zhenyu Zheng commented on HADOOP-16358: --- [~ayushtkn] Hi again, I saw you manually triggered once ARM CI, so can you also help us a little bit about the current problem? Our current problem is that the job is always in "UNSTABLE" status, we have tried serveral ways, any idea how could we make it to "Sucess/Failure" status? > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002044#comment-17002044 ] Zhenyu Zheng commented on HADOOP-16358: --- [~ayushtkn] Hi, thanks for the reply, we have only a small thing to fix so that we can send mail to ML, I will mark this as resolved once we done that today. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001627#comment-17001627 ] Ayush Saxena commented on HADOOP-16358: --- Seems like done, ARM build is there at: [https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/] If nothing remaining here we can close this ticket. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956771#comment-16956771 ] Zhenyu Zheng commented on HADOOP-16358: --- Some updates, our team has succesfully donated ARM resources and setup an ARM CI for Apache Spark: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/ it will set to periodic job and then PR trigger when we think it is stable enough. I really hope we can do the same for Hadoop. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949947#comment-16949947 ] Zhenyu Zheng commented on HADOOP-16358: --- It has been quite a while since anyone replied on this issue, I would like to provide some updates on this. We have identified some of the current blocker of making Hadoop successful running on Aarch64 platform and sent out ML for discussion: http://hadoop-common.472056.n3.nabble.com/DISCUSS-ARM-aarch64-support-for-Hadoop-td4190435.html and there was discussions about protocbuf upgrade and Vinayakumar, Duo Zhang and Akira Ajisaka have done alot about protocbuf and the progress seems well: https://issues.apache.org/jira/browse/HADOOP-13363 At the same time, we also keep trying to build and test Hadoop components on ARM machines and finding possible solutions, like: https://issues.apache.org/jira/browse/HADOOP-16614 We hope that this could prove that we are willing to debugging and maintaining ARM related stuff in Hadoop comunity in long term. So, I would like to propose to push this again, we want to add an ARM CI to the current CI system, we could donate servers to the existing CI system, it could be starting with a periodic job or maybe just for one of the components, and we can see how it goes, and judge on the next steps. This could also be good for others that are interested on solving problems for Aarch64 platform as they can see the log and pointing out possible causes and provide solutions, this could speed up the whole process. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904887#comment-16904887 ] Zhenyu Zheng commented on HADOOP-16358: --- Hi All, I would like to pushing this forward again and update some of our progress. To be clear again, our goal is to make Big Data related projects more capable to be running on ARM platforms, Openlab is just one possible tool that we are using, we are willing to donate ARM resources. We have now a running job for building Hadoop trunk on ARM in OpenLab http://status.openlabtesting.org/builds?job_name=hadoop-build-arm64 we discovered some problem and made some work-arounds to made the build successful, and we are try to pushing some of the fix like: https://issues.apache.org/jira/browse/HADOOP-13363 We are also tring to run full tests in our platform and yet we are still debuging. As for the mainainers for ARM related stuff, we will have a team doing it for upstream. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865437#comment-16865437 ] RuiChen commented on HADOOP-16358: -- Zhenyu Zheng and I work for [OpenLab|https://openlabtesting.org/] (a community to do open source project testing, this is OpenLab [Charter|https://github.com/theopenlab/governance/blob/master/CHARTER.md]). We hope to help more open source software worked on ARM, Hadoop is important one. We have a developer team, some ARM resources and [OpenLab CI infrastructure|http://status.openlabtesting.org/] (based on Zuul) to involve the target. I also saw Hadoop team already had a CI infrastructure run on Jenkins, OpenLab can share some ARM testing resources to Hadoop Jenkins CI system in order to avoiding to change the current development habit. It's not necessary to change Jenkins CI to OpenLab CI in Hadoop (OpenLab CI is valuable for some projects that don't have CI infrastructure) OpenLab team will work with Hadoop team to do the following things: # Enable ARM resources in Hadoop Jenkins CI # Enable Hadoop ARM testing jobs (Compile, UT, functional tests) # Some developers to maintain ARM CI jobs and handle testing failures Rome wasn't built in a day, so we plan to start a simple target to add ARM build job for Hadoop, to verify Hadoop and its subprojects can be compiled on ARM successfully, OpenLab's developers will maintain build CI jobs and address the CI issues, then we can add more complex test cases on ARM, like: unit tests and functional tests, step by step, it's a long term works. Of course, welcome another developers join to maintain the ARM CI. We will start a topic to discuss these ARM works in Hadoop mail list to get more feedback and make a community consensus. Thank you paying attention. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861142#comment-16861142 ] Wei-Chiu Chuang commented on HADOOP-16358: -- I wonder if we should reach out to Apache Infra to set up Jenkins or CI credentials. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861098#comment-16861098 ] Allen Wittenauer commented on HADOOP-16358: --- bq. Yetus is the main test runner we use for the runs; it choreographs the jenkins runs for PRs. So any changes would have to go in there -talk to Allen Wittenauer Best place for Apache Yetus support is d...@yetus.apache.org . I'm not really working on "not ASF" stuff lately. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860966#comment-16860966 ] Steve Loughran commented on HADOOP-16358: - Yetus is the main test runner we use for the runs; it choreographs the jenkins runs for PRs. So any changes would have to go in there -talk to [~aw] I don't think anyone has any fundamental issue with ARM, it's just that there's always been ongoing difficulties trying to get the native code to work properly across different CPUs, especially big-endian: you can search for Power, Sun sparc and past ARM JIRAs to see that history. A big issue here is getting people to care when a change breaks the Arm build: if we don't care that things break then its going to atrophy in exactly the same way that the windows test runs did. And frankly, we aren't being proactive about dealing with jenkins run failures today https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/ So: not sure what to do here. If someone really wants to do an ARM release, then they're going to have to get involved enough not just to set up the test run, but get it actually working, then own the problem of making sure it still works. Is there someone we can recruit for that Personal note: not a fan of Zuul. Not going to block anything, but not going to put any effort in either. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
