Re: Client side testing

2017-11-06 Thread Vincent Privat 
Sure! Currently coverage is restricted to JOSM classes, for obvious
performance reasons, as we run a full build after each commit.
But I can add a parameter to change the monitored packages in our EA
builds.
Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
restrict only to packages from java.desktop module?

2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
alexandre.il...@oracle.com>:

> This is great that you are touching the new builds, thank you, Vincent.
>
> I am not familiar with JOSM code, but the app is great! I have played with
> it in the past.
>
> Do you think it would be possible, in addition to what you are doing, to
> collect code coverage for the  JDK code? Specifically for the now existing
> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
> you can imagine - so I do not know how easy it is. If it is easy, can you
> flip the switch?
>
> Shura
> > On Nov 3, 2017, at 8:33 AM, Vincent Privat 
> wrote:
> >
> > Hello Rory,
> > Thank you :)
> >
> > Here is how we test various OpenJDK builds:
> > 1.1: we use only EA builds from http://jdk.java.net/
> > 2.1.1: no
> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
> > 2.1.4: yes, using jacoco
> > 2.2: all tests are automated
> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series for
> each weekly build, but rather once every two or three builds, unless we
> were waiting for a specific correction from OpenJDK or another tool. At the
> end of development, we were not waiting for any correction from OpenJDK but
> were eagerly waiting for the updates of ASM and everything that as blocked
> by the non-availability of ASM (in particular various Maven plugins)
> > 2.4: Yes, they are visible here:
> > - JOSM core: https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-
> JOSM/
> > - JOSM community plugins: https://josm.openstreetmap.de/
> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
> >
> > I just have initiated JDK10 tests today, and we have 53 failures caused
> by Byte Buddy not recognizing Java version 10:
> >
> > java.lang.IllegalArgumentException: Unknown Java version: 10
> >   at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.
> ofJavaVersion(ClassFileVersion.java:135)
> >
> >
> > I'm sure Rafael is aware of it. I will see how to update our tools
> (equalsverifier here).
> >
> > Cheers,
> > Vincent
> >
> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell :
> >
> > Hi Vincent,
> >
> > I noticed that JOSM have been commended in the last two Quality reports
> [1] [2] for the number and
> > indeed the quality of bugs you have found and logged against various
> early access releases.
> > Thanks again for your contribution to the quality of our releases!
> >
> > The big question is how do you do it ?
> >
> > We would love to hear more details about your test environment, what
> tests you run etc. Below is a
> > series of questions that come to mind, but there may be other details
> that you would like to share.
> >   • Builds
> >   • do you build your own OpenJDK builds, use Oracle's or
> other EA builds ?
> >   • Tests
> >   • What tests do you run
> >   • do you run the JCK ?
> >   • open regression tests ?
> >   • other tests ?
> >   • do you measure code coverage ?
> >   • Are the tests Automated or Manual ?
> >   • How frequently to do execute the tests
> >   • Are the results visible via an external link
> > Rgds, Rory
> >
> > [1] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+September+2017
> > [2] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+March+2017
> > --
> > Rgds,Rory O'Donnell
> > Quality Engineering Manager
> > Oracle EMEA , Dublin, Ireland
> >
> >
>
>

Re: Client side testing

2017-11-06 Thread Rory O'Donnell 

Hi Vincent,

Thanks for the update, plenty of info for us to review. We may comeback 
with follow on questions.


Rgds,Rory
On 03/11/2017 15:33, Vincent Privat wrote:

Hello Rory,
Thank you :)

Here is how we test various OpenJDK builds:
1.1: we use only EA builds from http://jdk.java.net/
2.1.1: no
2.1.2/2.1.3: we run all unit and integration tests of JOSM
2.1.4: yes, using jacoco
2.2: all tests are automated
2.3: it depends. During OpenJDK9 cycle we did not run a test series 
for each weekly build, but rather once every two or three builds, 
unless we were waiting for a specific correction from OpenJDK or 
another tool. At the end of development, we were not waiting for any 
correction from OpenJDK but were eagerly waiting for the updates of 
ASM and everything that as blocked by the non-availability of ASM (in 
particular various Maven plugins)

2.4: Yes, they are visible here:
- JOSM core: 
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM/
- JOSM community plugins: 
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM-Plugins/


I just have initiated JDK10 tests today, and we have 53 failures 
caused by Byte Buddy not recognizing Java version 10:


java.lang.IllegalArgumentException: Unknown Java version: 10
at 
nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.ofJavaVersion(ClassFileVersion.java:135)

I'm sure Rafael is aware of it. I will see how to update our tools 
(equalsverifier here).


Cheers,
Vincent

2017-11-03 14:05 GMT+01:00 Rory O'Donnell >:



Hi Vincent,

I noticed that JOSM have been commended in the last two Quality
reports [1] [2] for the number and
indeed the quality of bugs you have found and logged against
various early access releases.
Thanks again for your contribution to the quality of our releases!

The big question is how do you do it ?

We would love to hear more details about your test environment,
what tests you run etc. Below is a
series of questions that come to mind, but there may be other
details that you would like to share.

 1. Builds
 1. do you build your own OpenJDK builds, use Oracle's or
other EA builds ?
 2. Tests
 1. What tests do you run
 1. do you run the JCK ?
 2. open regression tests ?
 3. other tests ?
 4. do you measure code coverage ?
 2. Are the tests Automated or Manual ?
 3. How frequently to do execute the tests
 4. Are the results visible via an external link

Rgds, Rory

[1]

https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017


[2]

https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017



-- 
Rgds,Rory O'Donnell

Quality Engineering Manager
Oracle EMEA , Dublin, Ireland




--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland