Re: Project ideas for student projects in the area of testing

[Jesse Glick]

On Tue, Feb 21, 2017 at 10:22 AM, Robert Sandell  wrote:
> I think Jesse added some in the mercurial plugin when he extracted
> docker-fixtures from the ATH.

A work in progress but yes.

> And the ATH has plenty of examples as well as docker-fixtures itself

Right, this is where most of the examples would be.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr27AZbcxYnyqxuptvXz2f1ErqB4M4yB3jJH0mWksPPYfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Robert Sandell]

I just wrote one for my experimentation with the  ldap plugin here
https://github.com/rsandell/ldap-plugin/commit/2c85eb0d3f258529562b111a75604e0af7a3486c

I think Jesse added some in the mercurial plugin when he extracted
docker-fixtures from the ATH.
And the ATH has plenty of examples as well as docker-fixtures itself that
has some tests that I used for inspiration.

/B

On Tue, Feb 21, 2017 at 2:48 PM, Mark Waite 
wrote:

> Thanks for the suggestion.  Any pointers to examples of cases which use
> docker-fixtures as a test dependency?
>
> Mark Waite
>
> On Tue, Feb 21, 2017 at 6:45 AM Jesse Glick  wrote:
>
>> If Selenium testing is overkill, then just reproduce the bug in
>> `git-plugin/src/test/`. You can add a `docker-fixtures` test dependency if
>> it is helpful.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to jenkinsci-dev+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/jenkinsci-dev/CANfRfr0EL9okVMftzt_BSdZVntpcs9ZgkAcwpEaHibV1W8hXQ
>> g%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jenkinsci-dev/CAO49JtHvoThDzZdgAucJV1OG6jPjj
> sUFkEx2SGT33om-cf_b9A%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Robert Sandell
*Software Engineer*
*CloudBees Inc.*

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CALzHZS1t21ZvVKFbn95Q6-XEh6GXC%2BoMErbxd6ROqicKD_CcYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Mark Waite]

Thanks for the suggestion.  Any pointers to examples of cases which use
docker-fixtures as a test dependency?

Mark Waite

On Tue, Feb 21, 2017 at 6:45 AM Jesse Glick  wrote:

> If Selenium testing is overkill, then just reproduce the bug in
> `git-plugin/src/test/`. You can add a `docker-fixtures` test dependency if
> it is helpful.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0EL9okVMftzt_BSdZVntpcs9ZgkAcwpEaHibV1W8hXQg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtHvoThDzZdgAucJV1OG6jPjjsUFkEx2SGT33om-cf_b9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Jesse Glick]

If Selenium testing is overkill, then just reproduce the bug in
`git-plugin/src/test/`. You can add a `docker-fixtures` test dependency if
it is helpful.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0EL9okVMftzt_BSdZVntpcs9ZgkAcwpEaHibV1W8hXQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Ullrich Hafner]

I’ll ask them, but from the past courses I can say that they will probably have 
not enough time: most of them are doing their master in parallel to their 
normal job. 
(I am already very happy if they can spend a day per week for the project at 
the least...)

> Am 16.02.2017 um 10:43 schrieb Oleg Nenashev :
> 
> Hi Ulli,
> 
> Do you consider aligning these activities with Google Summer of Code? 
> Currently we are looking for project ideas, and actually test framework 
> improvements + test improvements could be a good project. 
> 
> I understand that GSoC implies a significant time dedication by students, 
> which likely goes far beyond a single course, but it may it can be an option 
> if the students are interested.
> 
> Thanks in advance,
> Oleg
> 
> понедельник, 13 февраля 2017 г., 22:40:00 UTC+3 пользователь Mark Waite 
> написал:
> I'd love to have students help create Jenkins jobs which illustrate specific 
> bugs in the git plugin, for example.  Sample jobs which illustrate different 
> bugs as Jenkins jobs are available from the lts-with-plugins branch of my 
> docker repository.  I've found it very powerful to have a Jenkins job per bug 
> report, either as a pipeline job (branch in a git repository like my 
> jenkins-bugs repository) or as a freestyle job which shows the bug and marks 
> itself unstable when the bug is detected.
> 
> I hope to propose a blog posting describing the technique of using one or 
> more Jenkins job definitions for each bug.  I've found it very helpful to 
> verify bugs remain fixed while detecting problems on multiple platforms.
> 
> Mark Waite
> 
> On Mon, Feb 13, 2017 at 11:44 AM Victor Martinez  > wrote:
> Maybe it is a bit out of the scope and I don't know whether there is already 
> a ongoing testing framework for the pipeline. But, is it something feasible? 
> It might be a really good contribution to the community
> Cheers
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-de...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/75a29c62-c428-4ffe-8b59-a74053281829%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/73BFA5C4-1114-4055-ACA1-A68B32502F6C%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Mark Waite]

I'm think that a Jenkins job which asserts the conditions related to a bug
report is a regression test.  I admit that the Jenkins jobs I create are
not Selenium tests, and they do not require a web browser in order to run.
They are definitely not able to exercise JavaScript, and they do not
evaluate behavior across multiple browsers.  If the goal is to check
JavaScript or to check browser compatibility, then Jenkins jobs are a poor
choice.

If the goal is to duplicate a bug report, and reuse the steps that
duplicated the bug report for later regression testing, then I think a
Jenkins job is simpler to create, easier to maintain, and easier to
diagnose than a Selenium test.

Ulrich, if you are interested in some "test drive" bug reports from the git
plugin or the git client plugin, you can review how I use the technique to
report and duplicate bugs in:

   - JENKINS-42204  -
   NPE due to recent GitSCMSource change
   - JENKINS-41948  -
   NoSuchMethodError scanning multi-branch project
   - JENKINS-41867  -
   IllegalStateException scanning multi-branch project
   - JENKINS-41785  -
   Pipeline status icon no longer shows worst status

If your students would like to try to duplicate bugs that I think are
interesting, I would suggest starting with:

   - JENKINS-38860  -
   renamed submodules fail checkout
   - JENKINS-41553  -
   submodule defs read from wrong location
   - JENKINS-42004  -
   jobs triggered unnecessarily (scenario 1)
   - JENKINS-41906  -
   jobs triggered unnecessarily (scenario 2)
   - JENKINS-40733  -
   notifyCommit provides wrong parameters
   - JENKINS-40352  -
   duplicate changesets in pipeline jobs (and other jobs)
   - JENKINS-39621  -
   infinite loop of job rebuilds

Those bug reports are likely well suited to creating one or more jobs to
show that the bug exists.  If they want to use the infrastructure already
provided in the lts-with-plugins branch
 of my docker
repository , they're welcome to do
that.

If you decide that the technique works well enough for the students, then
you can consider attempting other bugs from the git-plugin and
git-client-plugin projects, most recent bugs first.  I suspect other SCM
system bugs (Subversion, etc.) are reasonably well suited to the same
technique.

I've also found that it is quite powerful to create a pipeline based branch
per bug verification job so that Jenkins will automatically create the job
and execute it.  You can see examples of that technique in my jenkins-bugs
repository  (which is
referenced in several different ways from the docker instance).  That has
the benefit that I don't have to commit anything to the docker repository,
and the Jenkins pipeline code still creates and runs a job per bug.

I'm also OK if you decide it is better to submit pull requests to the
acceptance test harness.  The acceptance test harness has the compelling
benefit that there are many more people creating Selenium tests with the
acceptance test harness, and there are many more locations where those
tests can be executed.

In my case, I don't need the flexibility of a Selenium test, and I want the
easier diagnose and trivial parallel execution that comes from running a
Jenkins instance with multiple slaves and with tests defined as Jenkins
jobs.

Thanks!
Mark Waite

P.S. I have pipeline utility functions assertLogContains and
assertLogDoesNotContain to search for specific strings in the log files.
When one of those functions finds something unexpected, it marks the job as
"Unstable".  I can then search for all failed jobs and all unstable jobs in
my Jenkins instance to locate test failures.  Because it is a docker
instance, I can spread the execution load trivially across multiple agents
by adding or deleting agents.  I get interesting (and useful) test
randomization by choosing a mix of operating systems (Windows, CentOS 6.
CentOS 7, Debian 7, Debian 8, Ubuntu 14, Ubuntu 16, etc.) and git versions.



On Fri, Feb 17, 2017 at 5:47 AM Jesse Glick  wrote:

> I would also much rather see a genuine regression test. If some aspect of
> ATH is too weak to reproduce the bug properly, then that is what needs
> fixing.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> 

Re: Project ideas for student projects in the area of testing

[Jesse Glick]

I would also much rather see a genuine regression test. If some aspect of
ATH is too weak to reproduce the bug properly, then that is what needs
fixing.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2eGwB-hbFXtJa%2BJE72HbTqE6t8iTqYQO_iRL-SBarbBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Ullrich Hafner]

> Am 13.02.2017 um 20:39 schrieb Mark Waite :
> 
> I'd love to have students help create Jenkins jobs which illustrate specific 
> bugs in the git plugin, for example.  Sample jobs which illustrate different 
> bugs as Jenkins jobs are available from the lts-with-plugins branch of my 
> docker repository.  I've found it very powerful to have a Jenkins job per bug 
> report, either as a pipeline job (branch in a git repository like my 
> jenkins-bugs repository) or as a freestyle job which shows the bug and marks 
> itself unstable when the bug is detected.
> 
> I hope to propose a blog posting describing the technique of using one or 
> more Jenkins job definitions for each bug.  I've found it very helpful to 
> verify bugs remain fixed while detecting problems on multiple platforms.

That sounds interesting and would be a good basis. Do you think you can give 
some more pointers until the course starts (15. March)?

A job per bug report is also a good working unit so we can start with a simple 
one and then move to more difficult setups. If there would be an example to 
start with it would be even better.

What is not yet clear: these jobs you are referring to sound like a test setup 
for the ATH. So if you have such a setup it would be not much harder to add a 
verification step as an ATH test case. Then you will also have regression 
tests. Would that be also useful?

In which order should we pick up issues? Priority in Jira? Or Newest first?

> 
> Mark Waite
> 
> On Mon, Feb 13, 2017 at 11:44 AM Victor Martinez 
> > wrote:
> Maybe it is a bit out of the scope and I don't know whether there is already 
> a ongoing testing framework for the pipeline. But, is it something feasible? 
> It might be a really good contribution to the community
> Cheers
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtFfBbcs16yghxjQHB58EF3%2B%3D3mUGOck1WK-EpQ_eaD17w%40mail.gmail.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/4F6F3DB6-1D43-4085-B54A-F4A6CCEE19E9%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Ullrich Hafner]

This looks somewhat too complicated for a course, would better fit for a 
thesis… 

> Am 13.02.2017 um 19:44 schrieb Victor Martinez 
> :
> 
> Maybe it is a bit out of the scope and I don't know whether there is already 
> a ongoing testing framework for the pipeline. But, is it something feasible? 
> It might be a really good contribution to the community
> Cheers 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/117D0D22-E8F9-4022-BD0F-37466DF4595B%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Oleg Nenashev]

Hi Ulli,

Do you consider aligning these activities with Google Summer of Code? 
Currently we are looking for project ideas, and actually test framework 
improvements + test improvements could be a good project. 

I understand that GSoC implies a significant time dedication by students, 
which likely goes far beyond a single course, but it may it can be an 
option if the students are interested.

Thanks in advance,
Oleg

понедельник, 13 февраля 2017 г., 22:40:00 UTC+3 пользователь Mark Waite 
написал:
>
> I'd love to have students help create Jenkins jobs which illustrate 
> specific bugs in the git plugin, for example.  Sample jobs which illustrate 
> different bugs as Jenkins jobs are available from the lts-with-plugins 
> branch of my docker repository.  I've found it very powerful to have a 
> Jenkins job per bug report, either as a pipeline job (branch in a git 
> repository like my jenkins-bugs repository) or as a freestyle job which 
> shows the bug and marks itself unstable when the bug is detected.
>
> I hope to propose a blog posting describing the technique of using one or 
> more Jenkins job definitions for each bug.  I've found it very helpful to 
> verify bugs remain fixed while detecting problems on multiple platforms.
>
> Mark Waite
>
> On Mon, Feb 13, 2017 at 11:44 AM Victor Martinez  > wrote:
>
>> Maybe it is a bit out of the scope and I don't know whether there is 
>> already a ongoing testing framework for the pipeline. But, is it something 
>> feasible? It might be a really good contribution to the community
>> Cheers
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkinsci-de...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/75a29c62-c428-4ffe-8b59-a74053281829%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project ideas for student projects in the area of testing

[Mark Waite]

I'd love to have students help create Jenkins jobs which illustrate
specific bugs in the git plugin, for example.  Sample jobs which illustrate
different bugs as Jenkins jobs are available from the lts-with-plugins
branch of my docker repository.  I've found it very powerful to have a
Jenkins job per bug report, either as a pipeline job (branch in a git
repository like my jenkins-bugs repository) or as a freestyle job which
shows the bug and marks itself unstable when the bug is detected.

I hope to propose a blog posting describing the technique of using one or
more Jenkins job definitions for each bug.  I've found it very helpful to
verify bugs remain fixed while detecting problems on multiple platforms.

Mark Waite

On Mon, Feb 13, 2017 at 11:44 AM Victor Martinez <
victormartinezru...@gmail.com> wrote:

> Maybe it is a bit out of the scope and I don't know whether there is
> already a ongoing testing framework for the pipeline. But, is it something
> feasible? It might be a really good contribution to the community
> Cheers
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtFfBbcs16yghxjQHB58EF3%2B%3D3mUGOck1WK-EpQ_eaD17w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Project ideas for student projects in the area of testing

[Victor Martinez]

Maybe it is a bit out of the scope and I don't know whether there is already a 
ongoing testing framework for the pipeline. But, is it something feasible? It 
might be a really good contribution to the community
Cheers 

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Project ideas for student projects in the area of testing

[Ullrich Hafner]

In the summer semester I will again give a testing course here at the 
University of Applied Sciences in Munich. 
Several master students have the chance to improve their testing skills on a 
real project: Jenkins.
The scope is integration and system testing (maybe also some unit testing if 
there is some work required, too).

Do you have some ideas what needs to be done in the area of testing in Jenkins? 
What are our pain points, etc.?

Currently I’m thinking about the following topics:
- Extend the existing acceptance test cases (ATH) for 
a) core (i.e., create tests for Jira issues)
b) plugins (i.e., create tests for the most important and not yet 
covered plugins)
- Improve the speed of the ATH execution
- Make the ATH work on all operation systems (Windows, Mac, Linux) 
- Set up some performance tests for Jenkins 
- Participate in LTS testing 

Are there any other things where we could help? 

Ulli

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/5AC51D02-6DDA-4B66-ACF3-875B619B3ACB%40gmail.com.
For more options, visit https://groups.google.com/d/optout.