Re: Client side testing

[Vincent Privat]

If someone's paid at Oracle to work on Jenkins plugins, this bug is a good
candidate :)
(given the Hudson/Jenkins history I don't think so, but well, sometimes we
have good surprises, as I can see Jenkins is used to power
https://ci.adoptopenjdk.net/)

2017-11-07 22:18 GMT+01:00 Alexandre (Shura) Iline <
alexandre.il...@oracle.com>:

> The issue is pretty generic and also pretty old. :(
>
> Thank you anyway.
>
> Let me know if there is anything we could help with on our side.
>
> Shuta
>
>
>
> On Nov 7, 2017, at 1:11 PM, Vincent Privat 
> wrote:
>
> OK I tried to setup the JDK coverage, and this is NOT easy.
> Right now I'm blocked by a bug in the Jenkins Jacoco plugin:
> https://issues.jenkins-ci.org/browse/JENKINS-36168
> Need to solve it first.
> If I find how to setup JDK coverage for an Ant project, it will deserve
> serious documentation. I will probably update JOSM wiki, but maybe it could
> land somewhere on https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach for other projects?
>
> 2017-11-06 23:51 GMT+01:00 Alexandre (Shura) Iline <
> alexandre.il...@oracle.com>:
>
>>
>> On Nov 6, 2017, at 2:48 PM, Vincent Privat 
>> wrote:
>>
>> Sure! Currently coverage is restricted to JOSM classes, for obvious
>> performance reasons, as we run a full build after each commit.
>> But I can add a parameter to change the monitored packages in our EA
>> builds.
>> Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
>> restrict only to packages from java.desktop module?
>>
>>
>> Thank you!
>>
>> Frankly, java.desktop is what I am more interested myself right now.
>>
>> java, javax, sun and com.sun would also cover what I am asking for, of
>> course.
>>
>> Shura
>>
>>
>>
>>
>> 2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
>> alexandre.il...@oracle.com>:
>>
>>> This is great that you are touching the new builds, thank you, Vincent.
>>>
>>> I am not familiar with JOSM code, but the app is great! I have played
>>> with it in the past.
>>>
>>> Do you think it would be possible, in addition to what you are doing, to
>>> collect code coverage for the  JDK code? Specifically for the now existing
>>> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
>>> you can imagine - so I do not know how easy it is. If it is easy, can you
>>> flip the switch?
>>>
>>> Shura
>>> > On Nov 3, 2017, at 8:33 AM, Vincent Privat 
>>> wrote:
>>> >
>>> > Hello Rory,
>>> > Thank you :)
>>> >
>>> > Here is how we test various OpenJDK builds:
>>> > 1.1: we use only EA builds from http://jdk.java.net/
>>> > 2.1.1: no
>>> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
>>> > 2.1.4: yes, using jacoco
>>> > 2.2: all tests are automated
>>> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series
>>> for each weekly build, but rather once every two or three builds, unless we
>>> were waiting for a specific correction from OpenJDK or another tool. At the
>>> end of development, we were not waiting for any correction from OpenJDK but
>>> were eagerly waiting for the updates of ASM and everything that as blocked
>>> by the non-availability of ASM (in particular various Maven plugins)
>>> > 2.4: Yes, they are visible here:
>>> > - JOSM core: https://josm.openstreetmap.de/
>>> jenkins/job/Java-EarlyAccess-JOSM/
>>> > - JOSM community plugins: https://josm.openstreetmap.de/
>>> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
>>> >
>>> > I just have initiated JDK10 tests today, and we have 53 failures
>>> caused by Byte Buddy not recognizing Java version 10:
>>> >
>>> > java.lang.IllegalArgumentException: Unknown Java version: 10
>>> >   at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersi
>>> on.ofJavaVersion(ClassFileVersion.java:135)
>>> >
>>> >
>>> > I'm sure Rafael is aware of it. I will see how to update our tools
>>> (equalsverifier here).
>>> >
>>> > Cheers,
>>> > Vincent
>>> >
>>> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell :
>>> >
>>> > Hi Vincent,
>>> >
>>> > I noticed that JOSM have been commended in the last two Quality
>>> reports [1] [2] for the number and
>>> > indeed the quality of bugs you have found and logged against various
>>> early access releases.
>>> > Thanks again for your contribution to the quality of our releases!
>>> >
>>> > The big question is how do you do it ?
>>> >
>>> > We would love to hear more details about your test environment, what
>>> tests you run etc. Below is a
>>> > series of questions that come to mind, but there may be other details
>>> that you would like to share.
>>> >   • Builds
>>> >   • do you build your own OpenJDK builds, use Oracle's or
>>> other EA builds ?
>>> >   • Tests
>>> >   • What tests do you run
>>> >   • do you run the JCK ?
>>> >   • open regression tests ?
>>> >   • other tests ?
>>> >

Re: Client side testing

[Vincent Privat]

OK I tried to setup the JDK coverage, and this is NOT easy.
Right now I'm blocked by a bug in the Jenkins Jacoco plugin:
https://issues.jenkins-ci.org/browse/JENKINS-36168
Need to solve it first.
If I find how to setup JDK coverage for an Ant project, it will deserve
serious documentation. I will probably update JOSM wiki, but maybe it could
land somewhere on
https://wiki.openjdk.java.net/display/quality/Quality+Outreach for other
projects?

2017-11-06 23:51 GMT+01:00 Alexandre (Shura) Iline <
alexandre.il...@oracle.com>:

>
> On Nov 6, 2017, at 2:48 PM, Vincent Privat 
> wrote:
>
> Sure! Currently coverage is restricted to JOSM classes, for obvious
> performance reasons, as we run a full build after each commit.
> But I can add a parameter to change the monitored packages in our EA
> builds.
> Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
> restrict only to packages from java.desktop module?
>
>
> Thank you!
>
> Frankly, java.desktop is what I am more interested myself right now.
>
> java, javax, sun and com.sun would also cover what I am asking for, of
> course.
>
> Shura
>
>
>
>
> 2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
> alexandre.il...@oracle.com>:
>
>> This is great that you are touching the new builds, thank you, Vincent.
>>
>> I am not familiar with JOSM code, but the app is great! I have played
>> with it in the past.
>>
>> Do you think it would be possible, in addition to what you are doing, to
>> collect code coverage for the  JDK code? Specifically for the now existing
>> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
>> you can imagine - so I do not know how easy it is. If it is easy, can you
>> flip the switch?
>>
>> Shura
>> > On Nov 3, 2017, at 8:33 AM, Vincent Privat 
>> wrote:
>> >
>> > Hello Rory,
>> > Thank you :)
>> >
>> > Here is how we test various OpenJDK builds:
>> > 1.1: we use only EA builds from http://jdk.java.net/
>> > 2.1.1: no
>> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
>> > 2.1.4: yes, using jacoco
>> > 2.2: all tests are automated
>> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series for
>> each weekly build, but rather once every two or three builds, unless we
>> were waiting for a specific correction from OpenJDK or another tool. At the
>> end of development, we were not waiting for any correction from OpenJDK but
>> were eagerly waiting for the updates of ASM and everything that as blocked
>> by the non-availability of ASM (in particular various Maven plugins)
>> > 2.4: Yes, they are visible here:
>> > - JOSM core: https://josm.openstreetmap.de/
>> jenkins/job/Java-EarlyAccess-JOSM/
>> > - JOSM community plugins: https://josm.openstreetmap.de/
>> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
>> >
>> > I just have initiated JDK10 tests today, and we have 53 failures caused
>> by Byte Buddy not recognizing Java version 10:
>> >
>> > java.lang.IllegalArgumentException: Unknown Java version: 10
>> >   at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersi
>> on.ofJavaVersion(ClassFileVersion.java:135)
>> >
>> >
>> > I'm sure Rafael is aware of it. I will see how to update our tools
>> (equalsverifier here).
>> >
>> > Cheers,
>> > Vincent
>> >
>> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell :
>> >
>> > Hi Vincent,
>> >
>> > I noticed that JOSM have been commended in the last two Quality reports
>> [1] [2] for the number and
>> > indeed the quality of bugs you have found and logged against various
>> early access releases.
>> > Thanks again for your contribution to the quality of our releases!
>> >
>> > The big question is how do you do it ?
>> >
>> > We would love to hear more details about your test environment, what
>> tests you run etc. Below is a
>> > series of questions that come to mind, but there may be other details
>> that you would like to share.
>> >   • Builds
>> >   • do you build your own OpenJDK builds, use Oracle's or
>> other EA builds ?
>> >   • Tests
>> >   • What tests do you run
>> >   • do you run the JCK ?
>> >   • open regression tests ?
>> >   • other tests ?
>> >   • do you measure code coverage ?
>> >   • Are the tests Automated or Manual ?
>> >   • How frequently to do execute the tests
>> >   • Are the results visible via an external link
>> > Rgds, Rory
>> >
>> > [1] https://wiki.openjdk.java.net/display/quality/Quality+Outrea
>> ch+report+September+2017
>> > [2] https://wiki.openjdk.java.net/display/quality/Quality+Outrea
>> ch+report+March+2017
>> > --
>> > Rgds,Rory O'Donnell
>> > Quality Engineering Manager
>> > Oracle EMEA , Dublin, Ireland
>> >
>> >
>>
>>
>
>

Re: Client side testing

[Vincent Privat]

Sure! Currently coverage is restricted to JOSM classes, for obvious
performance reasons, as we run a full build after each commit.
But I can add a parameter to change the monitored packages in our EA
builds.
Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
restrict only to packages from java.desktop module?

2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
alexandre.il...@oracle.com>:

> This is great that you are touching the new builds, thank you, Vincent.
>
> I am not familiar with JOSM code, but the app is great! I have played with
> it in the past.
>
> Do you think it would be possible, in addition to what you are doing, to
> collect code coverage for the  JDK code? Specifically for the now existing
> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
> you can imagine - so I do not know how easy it is. If it is easy, can you
> flip the switch?
>
> Shura
> > On Nov 3, 2017, at 8:33 AM, Vincent Privat 
> wrote:
> >
> > Hello Rory,
> > Thank you :)
> >
> > Here is how we test various OpenJDK builds:
> > 1.1: we use only EA builds from http://jdk.java.net/
> > 2.1.1: no
> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
> > 2.1.4: yes, using jacoco
> > 2.2: all tests are automated
> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series for
> each weekly build, but rather once every two or three builds, unless we
> were waiting for a specific correction from OpenJDK or another tool. At the
> end of development, we were not waiting for any correction from OpenJDK but
> were eagerly waiting for the updates of ASM and everything that as blocked
> by the non-availability of ASM (in particular various Maven plugins)
> > 2.4: Yes, they are visible here:
> > - JOSM core: https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-
> JOSM/
> > - JOSM community plugins: https://josm.openstreetmap.de/
> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
> >
> > I just have initiated JDK10 tests today, and we have 53 failures caused
> by Byte Buddy not recognizing Java version 10:
> >
> > java.lang.IllegalArgumentException: Unknown Java version: 10
> >   at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.
> ofJavaVersion(ClassFileVersion.java:135)
> >
> >
> > I'm sure Rafael is aware of it. I will see how to update our tools
> (equalsverifier here).
> >
> > Cheers,
> > Vincent
> >
> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell :
> >
> > Hi Vincent,
> >
> > I noticed that JOSM have been commended in the last two Quality reports
> [1] [2] for the number and
> > indeed the quality of bugs you have found and logged against various
> early access releases.
> > Thanks again for your contribution to the quality of our releases!
> >
> > The big question is how do you do it ?
> >
> > We would love to hear more details about your test environment, what
> tests you run etc. Below is a
> > series of questions that come to mind, but there may be other details
> that you would like to share.
> >   • Builds
> >   • do you build your own OpenJDK builds, use Oracle's or
> other EA builds ?
> >   • Tests
> >   • What tests do you run
> >   • do you run the JCK ?
> >   • open regression tests ?
> >   • other tests ?
> >   • do you measure code coverage ?
> >   • Are the tests Automated or Manual ?
> >   • How frequently to do execute the tests
> >   • Are the results visible via an external link
> > Rgds, Rory
> >
> > [1] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+September+2017
> > [2] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+March+2017
> > --
> > Rgds,Rory O'Donnell
> > Quality Engineering Manager
> > Oracle EMEA , Dublin, Ireland
> >
> >
>
>

Re: Client side testing

[Rory O'Donnell]

Hi Vincent,

Thanks for the update, plenty of info for us to review. We may comeback 
with follow on questions.


Rgds,Rory
On 03/11/2017 15:33, Vincent Privat wrote:

Hello Rory,
Thank you :)

Here is how we test various OpenJDK builds:
1.1: we use only EA builds from http://jdk.java.net/
2.1.1: no
2.1.2/2.1.3: we run all unit and integration tests of JOSM
2.1.4: yes, using jacoco
2.2: all tests are automated
2.3: it depends. During OpenJDK9 cycle we did not run a test series 
for each weekly build, but rather once every two or three builds, 
unless we were waiting for a specific correction from OpenJDK or 
another tool. At the end of development, we were not waiting for any 
correction from OpenJDK but were eagerly waiting for the updates of 
ASM and everything that as blocked by the non-availability of ASM (in 
particular various Maven plugins)

2.4: Yes, they are visible here:
- JOSM core: 
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM/
- JOSM community plugins: 
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM-Plugins/


I just have initiated JDK10 tests today, and we have 53 failures 
caused by Byte Buddy not recognizing Java version 10:


java.lang.IllegalArgumentException: Unknown Java version: 10
at 
nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.ofJavaVersion(ClassFileVersion.java:135)

I'm sure Rafael is aware of it. I will see how to update our tools 
(equalsverifier here).


Cheers,
Vincent

2017-11-03 14:05 GMT+01:00 Rory O'Donnell >:



Hi Vincent,

I noticed that JOSM have been commended in the last two Quality
reports [1] [2] for the number and
indeed the quality of bugs you have found and logged against
various early access releases.
Thanks again for your contribution to the quality of our releases!

The big question is how do you do it ?

We would love to hear more details about your test environment,
what tests you run etc. Below is a
series of questions that come to mind, but there may be other
details that you would like to share.

 1. Builds
 1. do you build your own OpenJDK builds, use Oracle's or
other EA builds ?
 2. Tests
 1. What tests do you run
 1. do you run the JCK ?
 2. open regression tests ?
 3. other tests ?
 4. do you measure code coverage ?
 2. Are the tests Automated or Manual ?
 3. How frequently to do execute the tests
 4. Are the results visible via an external link

Rgds, Rory

[1]

https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017


[2]

https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017



-- 
Rgds,Rory O'Donnell

Quality Engineering Manager
Oracle EMEA , Dublin, Ireland




--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

Client side testing

[Rory O'Donnell]

Hi Vincent,

I noticed that JOSM have been commended in the last two Quality reports 
[1] [2] for the number and
indeed the quality of bugs you have found and logged against various 
early access releases.

Thanks again for your contribution to the quality of our releases!

The big question is how do you do it ?

We would love to hear more details about your test environment, what 
tests you run etc. Below is a
series of questions that come to mind, but there may be other details 
that you would like to share.


1. Builds
1. do you build your own OpenJDK builds, use Oracle's or other EA
   builds ?
2. Tests
1. What tests do you run
1. do you run the JCK ?
2. open regression tests ?
3. other tests ?
4. do you measure code coverage ?
2. Are the tests Automated or Manual ?
3. How frequently to do execute the tests
4. Are the results visible via an external link

Rgds, Rory

[1] 
https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017
[2] 
https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017


--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland