Re: Code coverage report on github PRs

[Wilfred Spiegelenburg]

Ayush,

For YuniKorn we have codecov working. We did have to make some minor
changes to get that done.
It is all github action driven [1]. The action is approved for Apache repos.

The first thing is that we asked INFRA to get the codecov app added to our
repos [2].
Beside that to get codecov working for our go code it was pretty straight
forward. In the build for the repositories we generate code coverage data
using the standard go tool and write it to a file. I am not sure how to
generate the test reports from the java, but codecov has some examples for
that.

The codecov action picks up the file, uploads it and gets it processed. To
see the PR code coverage changes we run this for PRs and commit to our
master branch [3]. Codecov automatically generates the reports based on
those two points.

Wilfred

[1] https://github.com/codecov/codecov-action
[2] https://issues.apache.org/jira/browse/INFRA-20641
[3]
https://github.com/apache/yunikorn-core/blob/master/.github/workflows/main.yml


On Fri, 25 Nov 2022 at 00:44, Ayush Saxena  wrote:

> I tried to explore a bit more about codecov and tried to set that up in my
> local fork by allowing access and adding the Github Action in the yaml
> file(& lots of follow up fixes), I think it needs the tests as well to be
> executed as part of the github workflow, which we don't do, or there is
> some catch which I missed, need to further explore.
>
> Reading the doc[1], if not via github action it requires some token(Step:
> 2), I quickly went through the archives and found a ticket regarding the
> same by Spark folks in the past[2], guess they couldn't get that sorted.
>
> Need to explore a bit more, or get pointers from folks who have more
> experience around this.
>
> -Ayush
>
> [1] https://docs.codecov.com/docs
> [2] https://issues.apache.org/jira/browse/INFRA-12640
>
> On Thu, 24 Nov 2022 at 02:07, Wei-Chiu Chuang 
> wrote:
>
> > I believe most of them can be added by us using GitHub Workflow. There's
> a
> > marketplace for these tools and most of them are free for open source
> > projects.
> >
> > On Wed, Nov 23, 2022 at 11:43 AM Ayush Saxena 
> wrote:
> >
> >> A simple Infra ticket I suppose should get it done for us, eg.
> >> https://issues.apache.org/jira/browse/INFRA-23561
> >>
> >> -Ayush
> >>
> >> On Thu, 24 Nov 2022 at 01:00, Iñigo Goiri  wrote:
> >>
> >> > Now that we are using mostly GitHub PRs for the reviews and we have
> >> decent
> >> > integration for the builds etc there, I was wondering about code
> >> coverage
> >> > and reporting.
> >> > Is code coverage setup at all?
> >> > Does this come from the INFRA team?
> >> > What would it take to enable it otherwise?
> >> >
> >>
> >
>

Re: Reverted YARN-10063 from branch-3.2

[Wilfred Spiegelenburg]

Sorry for that, I had reverted it I thought but either I did not push or the 
push failed.
I just checked and my local branch-3.2 had the revert in it dated about 2 hours 
after the original commit.

Wilfred

> On 24 Apr 2020, at 05:52, Wei-Chiu Chuang  
> wrote:
> 
> It broke the build (see here )
> so revert the commit. Looks like it was unintentional.


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] EOL Hadoop branch-2.8

[Wilfred Spiegelenburg]

+1

Wilfred

> On 3 Mar 2020, at 05:48, Wei-Chiu Chuang  wrote:
> 
> I am sorry I forgot to start a VOTE thread.
> 
> This is the "official" vote thread to mark branch-2.8 End of Life. This is
> based on the following thread and the tracking jira (HADOOP-16880
> <https://issues.apache.org/jira/browse/HADOOP-16880>).
> 
> This vote will run for 7 days and conclude on March 9th (Mon) 11am PST.
> 
> Please feel free to share your thoughts.
> 
> Thanks,
> Weichiu
> 
> On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang 
> wrote:
> 
>> Looking at the EOL policy wiki:
>> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
>> 
>> The Hadoop community can still elect to make security update for EOL'ed
>> releases.
>> 
>> I think the EOL is to give more clarity to downstream applications (such
>> as HBase) the guidance of which Hadoop release lines are still active.
>> Additionally, I don't think it is sustainable to maintain 6 concurrent
>> release lines in this big project, which is why I wanted to start this
>> discussion.
>> 
>> Thoughts?
>> 
>> On Mon, Feb 24, 2020 at 10:22 AM Sunil Govindan  wrote:
>> 
>>> Hi Wei-Chiu
>>> 
>>> Extremely sorry for the late reply here.
>>> Cud u pls help to add more clarity on defining what will happen for
>>> branch-2.8 when we call EOL.
>>> Does this mean that, no more release coming out from this branch, or some
>>> more additional guidelines?
>>> 
>>> - Sunil
>>> 
>>> 
>>> On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang
>>>  wrote:
>>> 
>>>> This thread has been running for 7 days and no -1.
>>>> 
>>>> Don't think we've established a formal EOL process, but to publicize the
>>>> EOL, I am going to file a jira, update the wiki and post the
>>> announcement
>>>> to general@ and user@
>>>> 
>>>> On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia <
>>> dineshc@gmail.com>
>>>> wrote:
>>>> 
>>>>> Thanks Wei-Chiu for initiating this.
>>>>> 
>>>>> +1 for 2.8 EOL.
>>>>> 
>>>>> On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka 
>>>>> wrote:
>>>>> 
>>>>>> Thanks Wei-Chiu for starting the discussion,
>>>>>> 
>>>>>> +1 for the EoL.
>>>>>> 
>>>>>> -Akira
>>>>>> 
>>>>>> On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena 
>>>> wrote:
>>>>>> 
>>>>>>> Thanx Wei-Chiu for initiating this
>>>>>>> +1 for marking 2.8 EOL
>>>>>>> 
>>>>>>> -Ayush
>>>>>>> 
>>>>>>>> On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang <
>>> weic...@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th
>>>> 2018.
>>>>>>>> 
>>>>>>>> It's been 17 months since the release and the community by and
>>>> large
>>>>>> have
>>>>>>>> moved up to 2.9/2.10/3.x.
>>>>>>>> 
>>>>>>>> With Hadoop 3.3.0 over the horizon, is it time to start the EOL
>>>>>>> discussion
>>>>>>>> and reduce the number of active branches?
>>>>>>> 
>>>>>>> 
>>> -
>>>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>>>>>> For additional commands, e-mail:
>>> common-dev-h...@hadoop.apache.org
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Wilfred Spiegelenburg
Staff Software Engineer
 <https://www.cloudera.com/>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Wilfred Spiegelenburg]

+1 (non binding)

- build from source on MacOSX 10.14.2, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- deployed a single node cluster  (openjdk 1.8u191 / centos 7.5)
- uploaded the MR framework
- configured YARN with the FS
- ran multiple MR jobs

> On 29 Jan 2019, at 17:19, Sunil G  wrote:
> 
> Hi Folks,
> 
> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> 
> The artifacts are available here:
> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> 
> The RC tag in git is release-3.1.2-RC1:
> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> 
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> 
> This vote will run 5 days from now.
> 
> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> 
> We have done testing with a pseudo cluster and distributed shell job.
> 
> My +1 to start.
> 
> Best,
> Wangda Tan and Sunil Govindan
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
> ORDER BY priority DESC


Wilfred Spiegelenburg | Software Engineer
cloudera.com <https://www.cloudera.com/>

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Wilfred Spiegelenburg]

+1 (non binding)

- build from source on MacOSX 10.14.2, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- confirmed the checksum and signature
- deployed a single node cluster  (openjdk 1.8u191 / centos 7.5)
- uploaded the MR framework
- configured YARN with the FS
- ran multiple MR jobs

> On 8 Jan 2019, at 22:42, Sunil G  wrote:
> 
> Hi folks,
> 
> 
> Thanks to all of you who helped in this release [1] and for helping to vote
> for RC0. I have created second release candidate (RC1) for Apache Hadoop
> 3.2.0.
> 
> 
> Artifacts for this RC are available here:
> 
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
> 
> 
> RC tag in git is release-3.2.0-RC1.
> 
> 
> 
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
> 
> 
> This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.
> 
> 
> 
> 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
> 
> are the highlights of this release.
> 
> 1. Node Attributes Support in YARN
> 
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> 
> 3. Support service upgrade via YARN Service API and CLI
> 
> 4. HDFS Storage Policy Satisfier
> 
> 5. Support Windows Azure Storage - Blob file system in Hadoop
> 
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> 
> 7. Improvements in Router-based HDFS federation
> 
> 
> 
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> 
> I have done few testing with my pseudo cluster. My +1 to start.
> 
> 
> 
> Regards,
> 
> Sunil
> 
> 
> 
> [1]
> 
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> 
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC


Wilfred Spiegelenburg | Software Engineer
cloudera.com <https://www.cloudera.com/>

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Wilfred Spiegelenburg]

+1 (non binding)

- build from source on MacOSX 10.14.1, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- confirmed the checksum and signature
- deployed a single node cluster  (jdk 1.8u191 / centos 7.5)

Wilfred

> On 23 Nov 2018, at 23:06, Sunil G  wrote:
> 
> Hi folks,
> 
> 
> 
> Thanks to all contributors who helped in this release [1]. I have created
> 
> first release candidate (RC0) for Apache Hadoop 3.2.0.
> 
> 
> Artifacts for this RC are available here:
> 
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> 
> 
> 
> RC tag in git is release-3.2.0-RC0.
> 
> 
> 
> The maven artifacts are available via repository.apache.org at
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> 
> 
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> 
> 
> 
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
> 
> are the highlights of this release.
> 
> 1. Node Attributes Support in YARN
> 
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> 
> 3. Support service upgrade via YARN Service API and CLI
> 
> 4. HDFS Storage Policy Satisfier
> 
> 5. Support Windows Azure Storage - Blob file system in Hadoop
> 
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> 
> 7. Improvements in Router-based HDFS federation
> 
> 
> 
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> 
> I have done few testing with my pseudo cluster. My +1 to start.
> 
> 
> 
> Regards,
> 
> Sunil
> 
> 
> 
> [1]
> 
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> 
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC


Wilfred Spiegelenburg | Software Engineer
cloudera.com <https://www.cloudera.com/>

[jira] [Created] (HADOOP-11323) WritableComparator: default implementation of compare keeps reference to byte array

[Wilfred Spiegelenburg (JIRA)]

Wilfred Spiegelenburg created HADOOP-11323:
--

 Summary: WritableComparator: default implementation of compare 
keeps reference to byte array
 Key: HADOOP-11323
 URL: https://issues.apache.org/jira/browse/HADOOP-11323
 Project: Hadoop Common
  Issue Type: Improvement
  Components: performance
Affects Versions: 2.5.0
Reporter: Wilfred Spiegelenburg


When the default compare is used on a WritableComparator a reference to the 
second passed in byte array is kept in the buffer. Since WritableComparator 
keeps a reference to the buffer the byte will never be garbage collected. This 
can lead to a higher heap use than needed.

The buffer should drop the reference to the byte array passed in. We can null 
out the byte array reference since the buffer is a private variable for the 
class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)