Re: Some updates for Hadoop on ARM and next steps

[Vinayakumar B]

Also note that.. Nothing gets deployed back to maven central repo from this
job. So no interference with other jobs and nodes as well.

-Vinay

On Thu, 28 Nov 2019, 10:55 pm Vinayakumar B, 
wrote:

> Hi all,
>
> As a starter..
> Created a simple mvn based job (not yetus and docker as current qbt on
> trunk on x86) in
>
> https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/7/console
>
> Right now using the manual tools installed and the previous workarounds
> mentioned related to various third-party dependencies related ARM
> architecture. This will be triggered daily once automatically.
>
> Going forward we can make sure that yetus with docker works fine in this
> node as well and configure similar to x86 qbt run.
>
> -Vinay
>
> On Thu, 28 Nov 2019, 7:30 am Zhenyu Zheng, 
> wrote:
>
>> Thanks for the reply Chris, And really appriaciated about all the things
>> you have done to made our node work. I'm sending this ML to send out info
>> about the node is ready. And hope someone from Hadoop project could help
>> us
>> set some new jobs/builds, I totally understand your role and opinion, I'm
>> not asking you to add jobs for Hadoop, I'm just trying to make clear about
>> what we are looking for.
>>
>> As Chris mentioned in previous email interactions, there are 3 kinds of CI
>> nodes available in the CI system, the 1st and 2nd type have to use the
>> current infra management tools to install tools and software required for
>> the system, which the infra management tool is currently not ready for ARM
>> platform. And the 3rd kind of CI nodes is what we are ready now - we
>> manually install all the required tools and software and maintain them
>> according to infra's other nodes.  And we will try to make the infra
>> management tools usable for ARM platform to make the nodes type2 or type1.
>>
>> As for jobs/builds, seems a periodic job/builds like
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/
>> seems
>> to be the most suitable for what we are looking for the current step.
>> Since
>> we are still having some errors and failures(15 erros in Hadoop-YARN, 4
>> Failures and 2 Errors in Hadoop-HDFS, 23 Failures in Hadoop-MapReduce,
>> which is a quite small number comparing to the total number of tests, and
>> failures/errors in same sub-project seems to be caused by same problem)
>> that our team will work on, so we want to propose 4 different jobs similar
>> to the mechanism used in Hadoop-trunk-Commit, a SCM triggered periodic job
>> test out building and UT for each sub-project:
>> Hadoop-YARN-trunk-Commit-Aarch64, Hadoop-HDFS-trunk-Commit-Aarch64,
>> Hadoop-MapReducer-trunk-Commit-Aarch64 and Hadoop-Common-trunk-Commit
>> Aarch64 to be more tracked for each project. We can also start one by one,
>> of cause.
>>
>> Hope this could clear all the misunderstanding.
>>
>> BR,
>>
>> On Wed, Nov 27, 2019 at 10:28 PM Chris Thistlethwaite 
>> wrote:
>>
>> > If anyone would like to follow along in JIRA, here's the ticket
>> > https://issues.apache.org/jira/browse/INFRA-19369. I've been updating
>> > that ticket with any issues. arm-poc has been moved to a node in
>> Singapore
>> > and will need to be tested again with builds.
>> >
>> > I'm going to mention again that someone from Hadoop should be changing
>> > these builds in order to run against arm-poc. In my reply below, I
>> thought
>> > that the project knew about the ARM nodes and was involved with setting
>> up
>> > new builds, which is why I said I'd be willing to make simple changes
>> for
>> > testing. However I don't want to change things without the knowledge of
>> the
>> > project. The builds themselves are created by the project, not Infra,
>> which
>> > means I have no idea which build should run against ARM vs any other
>> CPU.
>> >
>> > -Chris T.
>> > #asfinfra
>> >
>> > On 11/22/19 9:28 AM, Chris Thistlethwaite wrote:
>> >
>> > In order to run builds against arm-poc, someone (me included) will need
>> to
>> > change a build config to only use that label. The node itself isn't
>> fully
>> > built out like our other ASF nodes, due to the fact that it's ARM and we
>> > don't have all the packaged tools built for that architecture, it will
>> > likely take some time to fix issues.
>> >
>> >
>> > -Chris T.
>> > #asfinfra
>> >
>> > On 11/22/19 3:46 AM, bo zhaobo wrote:
>> >
>> > Thanks. That would be great if a project can use the ARM test worker to
>> do
>> > the specific testing on ARM.
>> >
>> > Also I think it's better to make @Chris Thistlethwaite <
>> chr...@apache.org> know
>> > this email.  Could you please give some kind advices? Thank you.
>> >
>> > BR
>> >
>> > ZhaoBo
>> >
>> >
>> >
>> > [image: Mailtrack]
>> > <
>> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&;>
>> Sender
>> > notified by
>> > Mailtrack
>> > <
>> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&;>
>> 19/11/22
>> > 下午04:42:30
>> >
>> > Zhenyu Zheng  于2019年11月

Re: Some updates for Hadoop on ARM and next steps

[Vinayakumar B]

Hi all,

As a starter..
Created a simple mvn based job (not yetus and docker as current qbt on
trunk on x86) in

https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/7/console

Right now using the manual tools installed and the previous workarounds
mentioned related to various third-party dependencies related ARM
architecture. This will be triggered daily once automatically.

Going forward we can make sure that yetus with docker works fine in this
node as well and configure similar to x86 qbt run.

-Vinay

On Thu, 28 Nov 2019, 7:30 am Zhenyu Zheng, 
wrote:

> Thanks for the reply Chris, And really appriaciated about all the things
> you have done to made our node work. I'm sending this ML to send out info
> about the node is ready. And hope someone from Hadoop project could help us
> set some new jobs/builds, I totally understand your role and opinion, I'm
> not asking you to add jobs for Hadoop, I'm just trying to make clear about
> what we are looking for.
>
> As Chris mentioned in previous email interactions, there are 3 kinds of CI
> nodes available in the CI system, the 1st and 2nd type have to use the
> current infra management tools to install tools and software required for
> the system, which the infra management tool is currently not ready for ARM
> platform. And the 3rd kind of CI nodes is what we are ready now - we
> manually install all the required tools and software and maintain them
> according to infra's other nodes.  And we will try to make the infra
> management tools usable for ARM platform to make the nodes type2 or type1.
>
> As for jobs/builds, seems a periodic job/builds like
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/
> seems
> to be the most suitable for what we are looking for the current step. Since
> we are still having some errors and failures(15 erros in Hadoop-YARN, 4
> Failures and 2 Errors in Hadoop-HDFS, 23 Failures in Hadoop-MapReduce,
> which is a quite small number comparing to the total number of tests, and
> failures/errors in same sub-project seems to be caused by same problem)
> that our team will work on, so we want to propose 4 different jobs similar
> to the mechanism used in Hadoop-trunk-Commit, a SCM triggered periodic job
> test out building and UT for each sub-project:
> Hadoop-YARN-trunk-Commit-Aarch64, Hadoop-HDFS-trunk-Commit-Aarch64,
> Hadoop-MapReducer-trunk-Commit-Aarch64 and Hadoop-Common-trunk-Commit
> Aarch64 to be more tracked for each project. We can also start one by one,
> of cause.
>
> Hope this could clear all the misunderstanding.
>
> BR,
>
> On Wed, Nov 27, 2019 at 10:28 PM Chris Thistlethwaite 
> wrote:
>
> > If anyone would like to follow along in JIRA, here's the ticket
> > https://issues.apache.org/jira/browse/INFRA-19369. I've been updating
> > that ticket with any issues. arm-poc has been moved to a node in
> Singapore
> > and will need to be tested again with builds.
> >
> > I'm going to mention again that someone from Hadoop should be changing
> > these builds in order to run against arm-poc. In my reply below, I
> thought
> > that the project knew about the ARM nodes and was involved with setting
> up
> > new builds, which is why I said I'd be willing to make simple changes for
> > testing. However I don't want to change things without the knowledge of
> the
> > project. The builds themselves are created by the project, not Infra,
> which
> > means I have no idea which build should run against ARM vs any other CPU.
> >
> > -Chris T.
> > #asfinfra
> >
> > On 11/22/19 9:28 AM, Chris Thistlethwaite wrote:
> >
> > In order to run builds against arm-poc, someone (me included) will need
> to
> > change a build config to only use that label. The node itself isn't fully
> > built out like our other ASF nodes, due to the fact that it's ARM and we
> > don't have all the packaged tools built for that architecture, it will
> > likely take some time to fix issues.
> >
> >
> > -Chris T.
> > #asfinfra
> >
> > On 11/22/19 3:46 AM, bo zhaobo wrote:
> >
> > Thanks. That would be great if a project can use the ARM test worker to
> do
> > the specific testing on ARM.
> >
> > Also I think it's better to make @Chris Thistlethwaite <
> chr...@apache.org> know
> > this email.  Could you please give some kind advices? Thank you.
> >
> > BR
> >
> > ZhaoBo
> >
> >
> >
> > [image: Mailtrack]
> > <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&;>
> Sender
> > notified by
> > Mailtrack
> > <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&;>
> 19/11/22
> > 下午04:42:30
> >
> > Zhenyu Zheng  于2019年11月22日周五 下午4:32写道：
> >
> >> Hi Hadoop,
> >>
> >>
> >>
> >> First off, I want to thanks to Wei-Chiu for having me on the last week's
> >> Hadoop community sync to introduce our ideas of ARM support on Hadoop.
> And
> >> also for all the attendees for listening and providing suggestions.
> >>
> >>
> >>
> >> I want to provide some update on the status

Re: Some updates for Hadoop on ARM and next steps

[Zhenyu Zheng]

Thanks for the reply Chris, And really appriaciated about all the things
you have done to made our node work. I'm sending this ML to send out info
about the node is ready. And hope someone from Hadoop project could help us
set some new jobs/builds, I totally understand your role and opinion, I'm
not asking you to add jobs for Hadoop, I'm just trying to make clear about
what we are looking for.

As Chris mentioned in previous email interactions, there are 3 kinds of CI
nodes available in the CI system, the 1st and 2nd type have to use the
current infra management tools to install tools and software required for
the system, which the infra management tool is currently not ready for ARM
platform. And the 3rd kind of CI nodes is what we are ready now - we
manually install all the required tools and software and maintain them
according to infra's other nodes.  And we will try to make the infra
management tools usable for ARM platform to make the nodes type2 or type1.

As for jobs/builds, seems a periodic job/builds like
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/ seems
to be the most suitable for what we are looking for the current step. Since
we are still having some errors and failures(15 erros in Hadoop-YARN, 4
Failures and 2 Errors in Hadoop-HDFS, 23 Failures in Hadoop-MapReduce,
which is a quite small number comparing to the total number of tests, and
failures/errors in same sub-project seems to be caused by same problem)
that our team will work on, so we want to propose 4 different jobs similar
to the mechanism used in Hadoop-trunk-Commit, a SCM triggered periodic job
test out building and UT for each sub-project:
Hadoop-YARN-trunk-Commit-Aarch64, Hadoop-HDFS-trunk-Commit-Aarch64,
Hadoop-MapReducer-trunk-Commit-Aarch64 and Hadoop-Common-trunk-Commit
Aarch64 to be more tracked for each project. We can also start one by one,
of cause.

Hope this could clear all the misunderstanding.

BR,

On Wed, Nov 27, 2019 at 10:28 PM Chris Thistlethwaite 
wrote:

> If anyone would like to follow along in JIRA, here's the ticket
> https://issues.apache.org/jira/browse/INFRA-19369. I've been updating
> that ticket with any issues. arm-poc has been moved to a node in Singapore
> and will need to be tested again with builds.
>
> I'm going to mention again that someone from Hadoop should be changing
> these builds in order to run against arm-poc. In my reply below, I thought
> that the project knew about the ARM nodes and was involved with setting up
> new builds, which is why I said I'd be willing to make simple changes for
> testing. However I don't want to change things without the knowledge of the
> project. The builds themselves are created by the project, not Infra, which
> means I have no idea which build should run against ARM vs any other CPU.
>
> -Chris T.
> #asfinfra
>
> On 11/22/19 9:28 AM, Chris Thistlethwaite wrote:
>
> In order to run builds against arm-poc, someone (me included) will need to
> change a build config to only use that label. The node itself isn't fully
> built out like our other ASF nodes, due to the fact that it's ARM and we
> don't have all the packaged tools built for that architecture, it will
> likely take some time to fix issues.
>
>
> -Chris T.
> #asfinfra
>
> On 11/22/19 3:46 AM, bo zhaobo wrote:
>
> Thanks. That would be great if a project can use the ARM test worker to do
> the specific testing on ARM.
>
> Also I think it's better to make @Chris Thistlethwaite  
> know
> this email.  Could you please give some kind advices? Thank you.
>
> BR
>
> ZhaoBo
>
>
>
> [image: Mailtrack]
> 
>  Sender
> notified by
> Mailtrack
> 
>  19/11/22
> 下午04:42:30
>
> Zhenyu Zheng  于2019年11月22日周五 下午4:32写道：
>
>> Hi Hadoop,
>>
>>
>>
>> First off, I want to thanks to Wei-Chiu for having me on the last week's
>> Hadoop community sync to introduce our ideas of ARM support on Hadoop. And
>> also for all the attendees for listening and providing suggestions.
>>
>>
>>
>> I want to provide some update on the status:
>>
>> 1. Our teammate has successfully donated an ARM machine to the
>> ApacheInfra team, and it is setup for running:
>> https://builds.apache.org/computer/arm-poc/ it might be a good idea to
>> make use of it, like running some periodic jobs for some experiment, and it
>> will also benifit us for discussions and asking for help on identified
>> problems.
>>
>>
>>
>> 2. I've been keep try to test and debug sub-project by sub-project, and
>> here is the current status for YARN:
>>
>> When running the whole test suits, some of the test suit will be skipped
>> due to the rules of if some previous test fails, then skip this suit. So I
>> manually run those test suits again to see if they can pass, the full test
>> result is that:
>>
>> Total: 5688; Failure: 0; Error 15; Skipped 60
>>
>> Among 

Re: Some updates for Hadoop on ARM and next steps

[Chris Thistlethwaite]

If anyone would like to follow along in JIRA, here's the ticket 
https://issues.apache.org/jira/browse/INFRA-19369. I've been updating 
that ticket with any issues. arm-poc has been moved to a node in 
Singapore and will need to be tested again with builds.


I'm going to mention again that someone from Hadoop should be changing 
these builds in order to run against arm-poc. In my reply below, I 
thought that the project knew about the ARM nodes and was involved with 
setting up new builds, which is why I said I'd be willing to make simple 
changes for testing. However I don't want to change things without the 
knowledge of the project. The builds themselves are created by the 
project, not Infra, which means I have no idea which build should run 
against ARM vs any other CPU.


-Chris T.
#asfinfra

On 11/22/19 9:28 AM, Chris Thistlethwaite wrote:
In order to run builds against arm-poc, someone (me included) will 
need to change a build config to only use that label. The node itself 
isn't fully built out like our other ASF nodes, due to the fact that 
it's ARM and we don't have all the packaged tools built for that 
architecture, it will likely take some time to fix issues.



-Chris T.
#asfinfra

On 11/22/19 3:46 AM, bo zhaobo wrote:
Thanks. That would be great if a project can use the ARM test worker 
to do the specific testing on ARM.


Also I think it's better to make @Chris Thistlethwaite 
 know this email.  Could you please give 
some kind advices? Thank you.


BR

ZhaoBo



Mailtrack 
 
	Sender notified by
Mailtrack 
 
19/11/22 下午04:42:30 	



Zhenyu Zheng > 于2019年11月22日周五 下午4:32写道：


Hi Hadoop,

First off, I want to thanks to Wei-Chiu for having me on the last
week's Hadoop community sync to introduce our ideas of ARM
support on Hadoop. And also for all the attendees for listening
and providing suggestions.

I want to provide some update on the status:

1. Our teammate has successfully donated an ARM machine to the
ApacheInfra team, and it is setup for running:
https://builds.apache.org/computer/arm-poc/it might be a good
idea to make use of it, like running some periodic jobs for some
experiment, and it will also benifit us for discussions and
asking for help on identified problems.

2. I've been keep try to test and debug sub-project by
sub-project, and here is the current status for YARN:

When running the whole test suits, some of the test suit will be
skipped due to the rules of if some previous test fails, then
skip this suit. So I manually run those test suits again to see
if they can pass, the full test result is that:

Total: 5688; Failure: 0; Error 15; Skipped 60

Among the 15 errors, 13 of them came from the ``Apache Hadoop
YARN TimelineService HBase tests`` test suit. The other 2 came
from ``Apache Hadoop YARN DistributedShell`` suit.

3. Some walk-arounds:

1) The only walk-arounds for build Hadoop on ARM is to pre-build
grpc-java, which my teammates are working with the community to
release a newer version with ARM support:
github.com/grpc/grpc-java/issues/6364


2) For YARN tests, the TimelineService HBase suit need either
HBase 1.4.8 or 2.0.2 which can only be built under protocbuf
2.5.0(HBase 1.4.8, HBase 2.0.2 external) and protocbuf
3.5.1(HBase 2.0.2 internal), so we have to pre-build them. And
the new cause of the error is still under debugging.

3) The rest of the know issue and possible walk-arounds are
reported to Hadoop Jira and are now under Wei-Chiu's tent jira
report: https://issues.apache.org/jira/browse/HADOOP-16723

I have put all the test logs in the attachment and error related
surefire reports in my github
https://github.com/ZhengZhenyu/HadoopTestLogs/issues/1(the
attachment size is limited for sending mailling list), please
have a check if you are interested.

So, how should we move a little bit forward and make use of the
ARM resources in ApacheInfra?

Best Regards,

Zhenyu

Re: Some updates for Hadoop on ARM and next steps

[Chris Thistlethwaite]

In order to run builds against arm-poc, someone (me included) will need 
to change a build config to only use that label. The node itself isn't 
fully built out like our other ASF nodes, due to the fact that it's ARM 
and we don't have all the packaged tools built for that architecture, it 
will likely take some time to fix issues.



-Chris T.
#asfinfra

On 11/22/19 3:46 AM, bo zhaobo wrote:
Thanks. That would be great if a project can use the ARM test worker 
to do the specific testing on ARM.


Also I think it's better to make @Chris Thistlethwaite 
 know this email.  Could you please give 
some kind advices? Thank you.


BR

ZhaoBo



Mailtrack 
 
	Sender notified by
Mailtrack 
 
19/11/22 下午04:42:30 	



Zhenyu Zheng > 于2019年11月22日周五 下午4:32写道：


Hi Hadoop,

First off, I want to thanks to Wei-Chiu for having me on the last
week's Hadoop community sync to introduce our ideas of ARM support
on Hadoop. And also for all the attendees for listening and
providing suggestions.

I want to provide some update on the status:

1. Our teammate has successfully donated an ARM machine to the
ApacheInfra team, and it is setup for running:
https://builds.apache.org/computer/arm-poc/it might be a good idea
to make use of it, like running some periodic jobs for some
experiment, and it will also benifit us for discussions and asking
for help on identified problems.

2. I've been keep try to test and debug sub-project by
sub-project, and here is the current status for YARN:

When running the whole test suits, some of the test suit will be
skipped due to the rules of if some previous test fails, then skip
this suit. So I manually run those test suits again to see if they
can pass, the full test result is that:

Total: 5688; Failure: 0; Error 15; Skipped 60

Among the 15 errors, 13 of them came from the ``Apache Hadoop YARN
TimelineService HBase tests`` test suit. The other 2 came from
``Apache Hadoop YARN DistributedShell`` suit.

3. Some walk-arounds:

1) The only walk-arounds for build Hadoop on ARM is to pre-build
grpc-java, which my teammates are working with the community to
release a newer version with ARM support:
github.com/grpc/grpc-java/issues/6364


2) For YARN tests, the TimelineService HBase suit need either
HBase 1.4.8 or 2.0.2 which can only be built under protocbuf
2.5.0(HBase 1.4.8, HBase 2.0.2 external) and protocbuf 3.5.1(HBase
2.0.2 internal), so we have to pre-build them. And the new cause
of the error is still under debugging.

3) The rest of the know issue and possible walk-arounds are
reported to Hadoop Jira and are now under Wei-Chiu's tent jira
report: https://issues.apache.org/jira/browse/HADOOP-16723

I have put all the test logs in the attachment and error related
surefire reports in my github
https://github.com/ZhengZhenyu/HadoopTestLogs/issues/1(the
attachment size is limited for sending mailling list), please have
a check if you are interested.

So, how should we move a little bit forward and make use of the
ARM resources in ApacheInfra?

Best Regards,

Zhenyu

Re: Some updates for Hadoop on ARM and next steps

[bo zhaobo]

Thanks. That would be great if a project can use the ARM test worker to do
the specific testing on ARM.

Also I think it's better to make @Chris Thistlethwaite  know
this email.  Could you please give some kind advices? Thank you.

BR

ZhaoBo



[image: Mailtrack]

Sender
notified by
Mailtrack

19/11/22
下午04:42:30

Zhenyu Zheng  于2019年11月22日周五 下午4:32写道：

> Hi Hadoop,
>
>
>
> First off, I want to thanks to Wei-Chiu for having me on the last week's
> Hadoop community sync to introduce our ideas of ARM support on Hadoop. And
> also for all the attendees for listening and providing suggestions.
>
>
>
> I want to provide some update on the status:
>
> 1. Our teammate has successfully donated an ARM machine to the ApacheInfra
> team, and it is setup for running:
> https://builds.apache.org/computer/arm-poc/ it might be a good idea to
> make use of it, like running some periodic jobs for some experiment, and it
> will also benifit us for discussions and asking for help on identified
> problems.
>
>
>
> 2. I've been keep try to test and debug sub-project by sub-project, and
> here is the current status for YARN:
>
> When running the whole test suits, some of the test suit will be skipped
> due to the rules of if some previous test fails, then skip this suit. So I
> manually run those test suits again to see if they can pass, the full test
> result is that:
>
> Total: 5688; Failure: 0; Error 15; Skipped 60
>
> Among the 15 errors, 13 of them came from the ``Apache Hadoop YARN
> TimelineService HBase tests`` test suit. The other 2 came from ``Apache
> Hadoop YARN DistributedShell`` suit.
>
>
>
> 3. Some walk-arounds:
>
>
>
> 1) The only walk-arounds for build Hadoop on ARM is to pre-build
> grpc-java, which my teammates are working with the community to release a
> newer version with ARM support: github.com/grpc/grpc-java/issues/6364
>
> 2) For YARN tests, the TimelineService HBase suit need either HBase 1.4.8
> or 2.0.2 which can only be built under protocbuf 2.5.0(HBase 1.4.8, HBase
> 2.0.2 external) and protocbuf 3.5.1(HBase 2.0.2 internal), so we have to
> pre-build them. And the new cause of the error is still under debugging.
>
> 3) The rest of the know issue and possible walk-arounds are reported to
> Hadoop Jira and are now under Wei-Chiu's tent jira report:
> https://issues.apache.org/jira/browse/HADOOP-16723
>
>
>
> I have put all the test logs in the attachment and error related surefire
> reports in my github
> https://github.com/ZhengZhenyu/HadoopTestLogs/issues/1 (the attachment
> size is limited for sending mailling list), please have a check if you are
> interested.
>
>
>
> So, how should we move a little bit forward and make use of the ARM
> resources in ApacheInfra?
>
>
>
> Best Regards,
>
>
>
> Zhenyu
>

Some updates for Hadoop on ARM and next steps

[Zhenyu Zheng]

Hi Hadoop,



First off, I want to thanks to Wei-Chiu for having me on the last week's
Hadoop community sync to introduce our ideas of ARM support on Hadoop. And
also for all the attendees for listening and providing suggestions.



I want to provide some update on the status:

1. Our teammate has successfully donated an ARM machine to the ApacheInfra
team, and it is setup for running:
https://builds.apache.org/computer/arm-poc/ it might be a good idea to make
use of it, like running some periodic jobs for some experiment, and it will
also benifit us for discussions and asking for help on identified problems.



2. I've been keep try to test and debug sub-project by sub-project, and
here is the current status for YARN:

When running the whole test suits, some of the test suit will be skipped
due to the rules of if some previous test fails, then skip this suit. So I
manually run those test suits again to see if they can pass, the full test
result is that:

Total: 5688; Failure: 0; Error 15; Skipped 60

Among the 15 errors, 13 of them came from the ``Apache Hadoop YARN
TimelineService HBase tests`` test suit. The other 2 came from ``Apache
Hadoop YARN DistributedShell`` suit.



3. Some walk-arounds:



1) The only walk-arounds for build Hadoop on ARM is to pre-build grpc-java,
which my teammates are working with the community to release a newer
version with ARM support: github.com/grpc/grpc-java/issues/6364

2) For YARN tests, the TimelineService HBase suit need either HBase 1.4.8
or 2.0.2 which can only be built under protocbuf 2.5.0(HBase 1.4.8, HBase
2.0.2 external) and protocbuf 3.5.1(HBase 2.0.2 internal), so we have to
pre-build them. And the new cause of the error is still under debugging.

3) The rest of the know issue and possible walk-arounds are reported to
Hadoop Jira and are now under Wei-Chiu's tent jira report:
https://issues.apache.org/jira/browse/HADOOP-16723



I have put all the test logs in the attachment and error related surefire
reports in my github
https://github.com/ZhengZhenyu/HadoopTestLogs/issues/1 (the
attachment size is limited for sending mailling list), please have a check
if you are interested.



So, how should we move a little bit forward and make use of the ARM
resources in ApacheInfra?



Best Regards,



Zhenyu