[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002228#comment-17002228 ] Zhenyu Zheng commented on HADOOP-16358: --- [~ayushtkn] Thanks a lot for the help, I saw there is a: # ERROR: File 'out/email-report.txt' does not exist in the ML notification, can we get rid of it? > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002098#comment-17002098 ] Zhenyu Zheng commented on HADOOP-16358: --- [~ayushtkn] Hi again, I saw you manually triggered once ARM CI, so can you also help us a little bit about the current problem? Our current problem is that the job is always in "UNSTABLE" status, we have tried serveral ways, any idea how could we make it to "Sucess/Failure" status? > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002044#comment-17002044 ] Zhenyu Zheng commented on HADOOP-16358: --- [~ayushtkn] Hi, thanks for the reply, we have only a small thing to fix so that we can send mail to ML, I will mark this as resolved once we done that today. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16766) Increase timeout for RPCCallBenckmark.testBenchmarkWithProto()
[ https://issues.apache.org/jira/browse/HADOOP-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng resolved HADOOP-16766. --- Resolution: Invalid > Increase timeout for RPCCallBenckmark.testBenchmarkWithProto() > -- > > Key: HADOOP-16766 > URL: https://issues.apache.org/jira/browse/HADOOP-16766 > Project: Hadoop Common > Issue Type: Wish >Reporter: Zhenyu Zheng >Priority: Trivial > > Currently the timeout setting for > org.apache.hadoop.ipc.TestRPCCallBenchmark.testBenchmarkWithProto() is 20 > seconds, and on our ARM64 machine with 8 cores, it tooks about 60 seconds to > finish the tests, so I want to propose to increase the timeout setting to 80 > seconds, this will not affect tests on other platforms as they will finish in > less than 20 secs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16766) Increase timeout for RPCCallBenckmark.testBenchmarkWithProto()
[ https://issues.apache.org/jira/browse/HADOOP-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HADOOP-16766: -- Summary: Increase timeout for RPCCallBenckmark.testBenchmarkWithProto() (was: Increase timeout for RPCCallBenckmarik.testBenchmarkWithProto()) > Increase timeout for RPCCallBenckmark.testBenchmarkWithProto() > -- > > Key: HADOOP-16766 > URL: https://issues.apache.org/jira/browse/HADOOP-16766 > Project: Hadoop Common > Issue Type: Wish >Reporter: Zhenyu Zheng >Priority: Trivial > > Currently the timeout setting for > org.apache.hadoop.ipc.TestRPCCallBenchmark.testBenchmarkWithProto() is 20 > seconds, and on our ARM64 machine with 8 cores, it tooks about 60 seconds to > finish the tests, so I want to propose to increase the timeout setting to 80 > seconds, this will not affect tests on other platforms as they will finish in > less than 20 secs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16766) Increase timeout for RPCCallBenckmarik.testBenchmarkWithProto()
Zhenyu Zheng created HADOOP-16766: - Summary: Increase timeout for RPCCallBenckmarik.testBenchmarkWithProto() Key: HADOOP-16766 URL: https://issues.apache.org/jira/browse/HADOOP-16766 Project: Hadoop Common Issue Type: Wish Reporter: Zhenyu Zheng Currently the timeout setting for org.apache.hadoop.ipc.TestRPCCallBenchmark.testBenchmarkWithProto() is 20 seconds, and on our ARM64 machine with 8 cores, it tooks about 60 seconds to finish the tests, so I want to propose to increase the timeout setting to 80 seconds, this will not affect tests on other platforms as they will finish in less than 20 secs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16723) [Umbrella] Support ARM/aarch64
[ https://issues.apache.org/jira/browse/HADOOP-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978891#comment-16978891 ] Zhenyu Zheng edited comment on HADOOP-16723 at 11/21/19 1:59 AM: - Hi Wei-Chiu, Thanks for doing this. Actually, our team member has succesfully donated a ARM server to Apache Infra few days ago, and has done some basic adjustment, the server might be OK to use, at least for some test POC. And I have also done some test related things, I've manually tested those YARN test suits which was skipped last time due to some previous test failures, and only 2 of the tests failed to run, so the overall result will be 2 failure and 13 errors for all YARN tests. 13 Errors are all "Apache Hadoop YARN TimelineService HBase tests" and I was able to identify the root cause for last time, it is because Hbase still uses protoc 2.5 and protoc 3.5 which is different from trunk Hadoop, so I manually built HBase 1.4.8 and 2.0.2(which was used by Hadoop according to pom), And the test still fails but maybe due to another reasons. I was planning to send out a mail and mention all the issues and walk-arounds and also upload all the test logs and surefire-reports for failed tests, for wider discussion and ask for advises, but I'm on business travels this week, and I can do it on Friday when I go back to office. was (Author: kevin_zheng): Hi Wei-Chiu, Thanks for doing this. Actually, our team member has succesfully donated a ARM server to Apache Infra few days ago, and has done some basic adjustment, the server might be OK to use, at least for some test POC. And I have also done some test related things, I've manually tested those YARN test suits which was skipped last time due to some previous test failures, and only 2 of the tests failed to run, so the overall result will be 2 failure and 13 errors for all YARN tests. 13 Errors are all "Apache Hadoop YARN TimelineService HBase tests" and I was able to identify the root cause for last time, it is because Hbase still uses protoc 2.5 and protoc 3.5 which is different from trunk Hadoop, so I manually built HBase 1.4.8 and 2.0.2(which was used by Hadoop according to pom), And the test still fails but maybe due to another reasons. I was planning to send out a mail and mention all the issues and walk-arounds and also upload all the test logs and surefire-reports for failed tests, but I'm on business travels this week, and I can do it on Friday when I go back to office. > [Umbrella] Support ARM/aarch64 > -- > > Key: HADOOP-16723 > URL: https://issues.apache.org/jira/browse/HADOOP-16723 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Wei-Chiu Chuang >Priority: Major > > I don't think there's an umbrella jira to link all efforts of ARM/aarch64 > support in Hadoop. So here it is for ease of tracking purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16723) [Umbrella] Support ARM/aarch64
[ https://issues.apache.org/jira/browse/HADOOP-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978891#comment-16978891 ] Zhenyu Zheng commented on HADOOP-16723: --- Hi Wei-Chiu, Thanks for doing this. Actually, our team member has succesfully donated a ARM server to Apache Infra few days ago, and has done some basic adjustment, the server might be OK to use, at least for some test POC. And I have also done some test related things, I've manually tested those YARN test suits which was skipped last time due to some previous test failures, and only 2 of the tests failed to run, so the overall result will be 2 failure and 13 errors for all YARN tests. 13 Errors are all "Apache Hadoop YARN TimelineService HBase tests" and I was able to identify the root cause for last time, it is because Hbase still uses protoc 2.5 and protoc 3.5 which is different from trunk Hadoop, so I manually built HBase 1.4.8 and 2.0.2(which was used by Hadoop according to pom), And the test still fails but maybe due to another reasons. I was planning to send out a mail and mention all the issues and walk-arounds and also upload all the test logs and surefire-reports for failed tests, but I'm on business travels this week, and I can do it on Friday when I go back to office. > [Umbrella] Support ARM/aarch64 > -- > > Key: HADOOP-16723 > URL: https://issues.apache.org/jira/browse/HADOOP-16723 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Wei-Chiu Chuang >Priority: Major > > I don't think there's an umbrella jira to link all efforts of ARM/aarch64 > support in Hadoop. So here it is for ease of tracking purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956771#comment-16956771 ] Zhenyu Zheng commented on HADOOP-16358: --- Some updates, our team has succesfully donated ARM resources and setup an ARM CI for Apache Spark: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/ it will set to periodic job and then PR trigger when we think it is stable enough. I really hope we can do the same for Hadoop. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949947#comment-16949947 ] Zhenyu Zheng commented on HADOOP-16358: --- It has been quite a while since anyone replied on this issue, I would like to provide some updates on this. We have identified some of the current blocker of making Hadoop successful running on Aarch64 platform and sent out ML for discussion: http://hadoop-common.472056.n3.nabble.com/DISCUSS-ARM-aarch64-support-for-Hadoop-td4190435.html and there was discussions about protocbuf upgrade and Vinayakumar, Duo Zhang and Akira Ajisaka have done alot about protocbuf and the progress seems well: https://issues.apache.org/jira/browse/HADOOP-13363 At the same time, we also keep trying to build and test Hadoop components on ARM machines and finding possible solutions, like: https://issues.apache.org/jira/browse/HADOOP-16614 We hope that this could prove that we are willing to debugging and maintaining ARM related stuff in Hadoop comunity in long term. So, I would like to propose to push this again, we want to add an ARM CI to the current CI system, we could donate servers to the existing CI system, it could be starting with a periodic job or maybe just for one of the components, and we can see how it goes, and judge on the next steps. This could also be good for others that are interested on solving problems for Aarch64 platform as they can see the log and pointing out possible causes and provide solutions, this could speed up the whole process. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904887#comment-16904887 ] Zhenyu Zheng edited comment on HADOOP-16358 at 8/12/19 6:41 AM: Hi All, I would like to pushing this forward again and update some of our progress. To be clear again, our goal is to make Big Data related projects more capable to be running on ARM platforms, Openlab is just one possible tool that we are using, we are willing to donate ARM resources. We have now a running job for building Hadoop trunk on ARM in OpenLab http://status.openlabtesting.org/builds?job_name=hadoop-build-arm64 we discovered some problem and made some work-arounds to made the build successful, and we are try to pushing some of the fix like: https://issues.apache.org/jira/browse/HADOOP-13363 We are also tring to run full tests in our platform and yet we are still debuging. As for the mainainers for ARM related stuff, we will have a team doing it for upstream. And yet we already have some progress in Flink community: https://issues.apache.org/jira/browse/INFRA-18822 So I hope we could push this forward in Hadoop too. was (Author: kevin_zheng): Hi All, I would like to pushing this forward again and update some of our progress. To be clear again, our goal is to make Big Data related projects more capable to be running on ARM platforms, Openlab is just one possible tool that we are using, we are willing to donate ARM resources. We have now a running job for building Hadoop trunk on ARM in OpenLab http://status.openlabtesting.org/builds?job_name=hadoop-build-arm64 we discovered some problem and made some work-arounds to made the build successful, and we are try to pushing some of the fix like: https://issues.apache.org/jira/browse/HADOOP-13363 We are also tring to run full tests in our platform and yet we are still debuging. As for the mainainers for ARM related stuff, we will have a team doing it for upstream. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16358) Add an ARM CI for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904887#comment-16904887 ] Zhenyu Zheng commented on HADOOP-16358: --- Hi All, I would like to pushing this forward again and update some of our progress. To be clear again, our goal is to make Big Data related projects more capable to be running on ARM platforms, Openlab is just one possible tool that we are using, we are willing to donate ARM resources. We have now a running job for building Hadoop trunk on ARM in OpenLab http://status.openlabtesting.org/builds?job_name=hadoop-build-arm64 we discovered some problem and made some work-arounds to made the build successful, and we are try to pushing some of the fix like: https://issues.apache.org/jira/browse/HADOOP-13363 We are also tring to run full tests in our platform and yet we are still debuging. As for the mainainers for ARM related stuff, we will have a team doing it for upstream. > Add an ARM CI for Hadoop > > > Key: HADOOP-16358 > URL: https://issues.apache.org/jira/browse/HADOOP-16358 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Zhenyu Zheng >Priority: Major > > Now the CI of Hadoop is handled by jenkins. While the tests are running under > x86 ARCH, the ARM arch has not being considered. This leads an problem that > we don't have a way to test every pull request that if it'll break the Hadoop > deployment on ARM or not. > We should add a CI system that support ARM arch. Using it, Hadoop can > officially support arm release in the future. Here I'd like to introduce > OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open > source CI system that can test any open source software on either x86 or arm > ARCH, it's mainly used by github projects. Now some > [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] > has integrated it already. Such as containerd (a graduated CNCF project, the > arm build will be triggerd in every PR, > [https://github.com/containerd/containerd/pulls]), terraform and so on. > OpenLab uses the open source CI software [Zuul > |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by > OpenStack community as well. integrating with OpneLab is quite easy using its > github app. All config info is open source as well. > If apache Hadoop community has interested with it, we can help for the > integration. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898475#comment-16898475 ] Zhenyu Zheng commented on HADOOP-13363: --- Any update on this one? I noticed that the Hadoop-trunk-Commit job has been failling for over a month probably relaed to this issue. According to the latest successful build log: https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull the os version was ubuntu 14.04 and for the jobs that are failling now such as: https://builds.apache.org/job/Hadoop-trunk-Commit/17020/console, the os version is 18.04. I'm not very familiar with the version changing for the jobs but I did a little search, according to: https://packages.ubuntu.com/search?keywords=protobuf-compiler=names & https://packages.ubuntu.com/search?suite=default=all=any=libprotoc-dev=names it both said that the version of libprotc-dev and protobuf-compiler available for ubuntu 18.04 is 3.0.0 Will this support the idea that it is time to upgrade protobuf version? We cannot stay at 14.04 forever right? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884886#comment-16884886 ] Zhenyu Zheng commented on HADOOP-13363: --- Sounds good, as I posted, seems some of the component uses protobuf v3(like grpc) and most others uses protobuf v2, this kind of multi-version might cause unpredictable problems, right? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881916#comment-16881916 ] Zhenyu Zheng edited comment on HADOOP-13363 at 7/10/19 10:04 AM: - Steve Loughran, Hi, thanks for the reply, I've tested what you have suggested, and encountered some problem: When I try with protobuf=2.6.1 using -D option I failed to build trunk branch due to: 2019-07-10 09:43:03.803715 | ubuntu-xenial-arm64 | [ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default) on project hadoop-yarn-csi: protoc did not exit cleanly. Review output for more information. -> [Help 1] 2019-07-10 09:43:03.804110 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.805475 | ubuntu-xenial-arm64 | [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 2019-07-10 09:43:03.806682 | ubuntu-xenial-arm64 | [ERROR] Re-run Maven using the -X switch to enable full debug logging. 2019-07-10 09:43:03.806989 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.808426 | ubuntu-xenial-arm64 | [ERROR] For more information about the errors and possible solutions, please read the following articles: 2019-07-10 09:43:03.809936 | ubuntu-xenial-arm64 | [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException 2019-07-10 09:43:03.810460 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.811584 | ubuntu-xenial-arm64 | [ERROR] After correcting the problems, you can resume the build with the command 2019-07-10 09:43:03.812624 | ubuntu-xenial-arm64 | [ERROR] mvn -rf :hadoop-yarn-csi Maybe this is due to that grpc requires protobuf 3.6.1? (https://github.com/apache/hadoop/blob/993dc8726b7d40ac832ae3e23b64e8541b62c4bd/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml#L29) And when I try to use protobuf 3.6.1 directly, something else failed to build: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile(default-compile)on project hadoop-hdfs: Compilation failure [ERROR] /home/zuul/src/github.com/ZhengZhenyu/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java:[468,39] incompatible types: org.apache.hadoop.hdfs.server.namenode.FsImageProto.FileSummary cannot be converted to com.google.protobuf.GeneratedMessage[m [ERROR] Help [ERROR] To see the full stack trace of the errors, re-run Maven with the switch. [ERROR] Re-run Maven using the [1m-X[m switch to enable full debug logging. [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] Help 1 http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-hdfs seems due to this part of the code is not compatiable for protobuf v3? Any suggestion on these? was (Author: kevin_zheng): Steve Loughran, Hi, thanks for the reply, I've tested what you have suggested, and encountered some problem: When I try with protobuf=2.6.1 using -D option I failed to build trunk branch due to: 2019-07-10 09:43:03.803715 | ubuntu-xenial-arm64 | [ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default) on project hadoop-yarn-csi: protoc did not exit cleanly. Review output for more information. -> [Help 1] 2019-07-10 09:43:03.804110 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.805475 | ubuntu-xenial-arm64 | [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 2019-07-10 09:43:03.806682 | ubuntu-xenial-arm64 | [ERROR] Re-run Maven using the -X switch to enable full debug logging. 2019-07-10 09:43:03.806989 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.808426 | ubuntu-xenial-arm64 | [ERROR] For more information about the errors and possible solutions, please read the following articles: 2019-07-10 09:43:03.809936 | ubuntu-xenial-arm64 | [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException 2019-07-10 09:43:03.810460 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.811584 | ubuntu-xenial-arm64 | [ERROR] After correcting the problems, you can resume the build with the command 2019-07-10 09:43:03.812624 | ubuntu-xenial-arm64 | [ERROR] mvn -rf :hadoop-yarn-csi Maybe this is due to that grpc requires protobuf 3.6.1? (https://github.com/apache/hadoop/blob/993dc8726b7d40ac832ae3e23b64e8541b62c4bd/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml#L29) And when I try to use protobuf 3.6.1 directly, something else failed to build: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile(default-compile)on project hadoop-hdfs: Compilation failure [ERROR]
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881916#comment-16881916 ] Zhenyu Zheng commented on HADOOP-13363: --- Steve Loughran, Hi, thanks for the reply, I've tested what you have suggested, and encountered some problem: When I try with protobuf=2.6.1 using -D option I failed to build trunk branch due to: 2019-07-10 09:43:03.803715 | ubuntu-xenial-arm64 | [ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default) on project hadoop-yarn-csi: protoc did not exit cleanly. Review output for more information. -> [Help 1] 2019-07-10 09:43:03.804110 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.805475 | ubuntu-xenial-arm64 | [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 2019-07-10 09:43:03.806682 | ubuntu-xenial-arm64 | [ERROR] Re-run Maven using the -X switch to enable full debug logging. 2019-07-10 09:43:03.806989 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.808426 | ubuntu-xenial-arm64 | [ERROR] For more information about the errors and possible solutions, please read the following articles: 2019-07-10 09:43:03.809936 | ubuntu-xenial-arm64 | [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException 2019-07-10 09:43:03.810460 | ubuntu-xenial-arm64 | [ERROR] 2019-07-10 09:43:03.811584 | ubuntu-xenial-arm64 | [ERROR] After correcting the problems, you can resume the build with the command 2019-07-10 09:43:03.812624 | ubuntu-xenial-arm64 | [ERROR] mvn -rf :hadoop-yarn-csi Maybe this is due to that grpc requires protobuf 3.6.1? (https://github.com/apache/hadoop/blob/993dc8726b7d40ac832ae3e23b64e8541b62c4bd/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml#L29) And when I try to use protobuf 3.6.1 directly, something else failed to build: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile(default-compile)on project hadoop-hdfs: Compilation failure [ERROR] /home/zuul/src/github.com/ZhengZhenyu/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java:[468,39] incompatible types: org.apache.hadoop.hdfs.server.namenode.FsImageProto.FileSummary cannot be converted to com.google.protobuf.GeneratedMessage[m [ERROR] Help [ERROR] To see the full stack trace of the errors, re-run Maven with the switch. [ERROR] Re-run Maven using the [1m-X[m switch to enable full debug logging. [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] Help 1 http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-hdfs seems due to this part of the code is not compatiable for protobuf v3? Any suggestion on these? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880015#comment-16880015 ] Zhenyu Zheng edited comment on HADOOP-13363 at 7/8/19 3:59 AM: --- Any updates on this? We have meet similar issues when build Hadoop on AArch64 platform was (Author: kevin_zheng): Any updates on this? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Tsuyoshi Ozawa >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880015#comment-16880015 ] Zhenyu Zheng commented on HADOOP-13363: --- Any updates on this? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Tsuyoshi Ozawa >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16358) Add an ARM CI for Hadoop
Zhenyu Zheng created HADOOP-16358: - Summary: Add an ARM CI for Hadoop Key: HADOOP-16358 URL: https://issues.apache.org/jira/browse/HADOOP-16358 Project: Hadoop Common Issue Type: Improvement Reporter: Zhenyu Zheng Now the CI of Hadoop is handled by jenkins. While the tests are running under x86 ARCH, the ARM arch has not being considered. This leads an problem that we don't have a way to test every pull request that if it'll break the Hadoop deployment on ARM or not. We should add a CI system that support ARM arch. Using it, Hadoop can officially support arm release in the future. Here I'd like to introduce OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open source CI system that can test any open source software on either x86 or arm ARCH, it's mainly used by github projects. Now some [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml] has integrated it already. Such as containerd (a graduated CNCF project, the arm build will be triggerd in every PR, [https://github.com/containerd/containerd/pulls]), terraform and so on. OpenLab uses the open source CI software [Zuul |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by OpenStack community as well. integrating with OpneLab is quite easy using its github app. All config info is open source as well. If apache Hadoop community has interested with it, we can help for the integration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
