Re: [Pkg-javascript-devel] testing during build is useless now with autopkgtests
On 2016, നവംബർ 26 6:34:26 PM IST, "Jérémy Lal" wrote: >Hi, > >i've been doing in all my latest uploads this setup: >- autopkgtest runs the package test suite >- build does not run tests > >Because: >1) it feels wrong to add Depends to Build-Depends, >and debian/tests/control already have all the info to properly run the >tests >2) when an autopkgtest fails, archive-wide testing reports the failure >as a package bug, >rendering tests during build useless anyway Are you sure this is automatic? I don't think autopkgtest failure is considered serious bug where as FTBFS is serious. tests during build helps find problems in archive wide rebuilds, which is very common. >I know that in some cases, one really need a built tree to run the >tests, >but it's not the majority. Usually a tree with patches applied is >enough. > >So please Team, don't re-add to all my packages "run tests during >build". >(and, of course, do whatever you see best when you're packaging it >yourself). I don't think that is the right way to approach team maintained packages. If we are looking at "my package" and "your package" inside the team, that defeats the purpose of team maintenance. Instead we should be setting standards for the whole team and document it as team policy. If you think this is better way of enabling tests, lets do it across all team maintained packages. >Jérémy -- Pkg-javascript-devel mailing list Pkg-javascript-devel@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel
Re: [Pkg-javascript-devel] testing during build is useless now with autopkgtests
On Saturday 26 November 2016 11:23 PM, Jérémy Lal wrote: > Side question - Do you know why we're not there yet ? I guess its because autopkgtest is new service and we don't want to disrupt this release with more rc bugs. It may change in next release cycle and autopkgtest failure can be rc. > Having an RC bug because of a test failure is wrong too ! Sometimes > the failure comes from the test itself, so it's not so bad as it looks. We are adding tests to detect problems early and many times it is really a bug because some dependencies are changed and functionality that worked before is no longer working. If a bug is wrongly tagged rc, we can just reduce the severity, but missing a real bug is worse. > Also some tests are meant to run in a build tree - in which case of course, > they need to run during build, and some tests are meant to run after > the package is installed. Some tests are "in-between" they can run > in the source tree without being built. > > What i want to say is that defining a "standard" choice for tests is not > as obvious as it seems. We can agree upon some strategy though. Yes, lets build consensus inside team about that strategy and document it. > Maybe tests that need source tree (be it built or not) must run during build, > and autopkgtests should only be for testing if the nodejs module can > be required (which is already a default test from autopkgtest) ? > What do you think ? Having tests in autopkgtest helps us find if reverse dependencies are broken when updating. Tools like build-and-upload from pkg-ruby-extras repository (which I use) depend on autopkgtest to verify updates don't break reverse dependencies. I suggest we do both until autopkgtest can report failures as bugs automatically. > All right, but between mistakes and wrong-doings, when it "just > works", please don't > change what has been done before - especially without testing it > before uploading. > Also discussing it or at least notifying it, would be a good idea. This specific case of node-rimraf was caused by a clear bug and I did not think enabling tests during build was done purposefully. none-rimraf did not declare dependency on node-glob, autopkgtest worked because node-tap pulled it in, when testing node-cpr, which used mocha, it failed. I did run build-and-upload (script from pkg-ruby-extras) before uploading. I wonder why pristine-tar was failing to generate correct orig.tar.gz. signature.asc Description: OpenPGP digital signature -- Pkg-javascript-devel mailing list Pkg-javascript-devel@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel
Re: [Pkg-javascript-devel] testing during build is useless now with autopkgtests
2016-11-26 18:41 GMT+01:00 Jérémy Lal : > 2016-11-26 17:15 GMT+01:00 Pirate Praveen : >> >> >> On 2016, നവംബർ 26 6:34:26 PM IST, "Jérémy Lal" wrote: >>>Hi, >>> >>>i've been doing in all my latest uploads this setup: >>>- autopkgtest runs the package test suite >>>- build does not run tests >>> >>>Because: >>>1) it feels wrong to add Depends to Build-Depends, >>>and debian/tests/control already have all the info to properly run the >>>tests >>>2) when an autopkgtest fails, archive-wide testing reports the failure >>>as a package bug, >>>rendering tests during build useless anyway >> >> No, that is not correct. If tests during build fails, it is rc and it is >> found in archive wide rebuilds. But autopkgtest failures are not rc and the >> bug reporting is not automatic. I know this for certain because autopkgtest >> for gitlab has been failing for some time. So at least in the current >> situation, autopkgtest is not a replacement for tests during build, it may >> reach that point in future, but we are not there yet. Side question - Do you know why we're not there yet ? Having an RC bug because of a test failure is wrong too ! Sometimes the failure comes from the test itself, so it's not so bad as it looks. Also some tests are meant to run in a build tree - in which case of course, they need to run during build, and some tests are meant to run after the package is installed. Some tests are "in-between" they can run in the source tree without being built. What i want to say is that defining a "standard" choice for tests is not as obvious as it seems. We can agree upon some strategy though. Maybe tests that need source tree (be it built or not) must run during build, and autopkgtests should only be for testing if the nodejs module can be required (which is already a default test from autopkgtest) ? What do you think ? >>>I know that in some cases, one really need a built tree to run the >>>tests, >>>but it's not the majority. Usually a tree with patches applied is >>>enough. >>> >>>So please Team, don't re-add to all my packages "run tests during >>>build". >>>(and, of course, do whatever you see best when you're packaging it >>>yourself). >> >> This is not the right way to approach team maintained packages. If we end up >> differentiating "my package" and "your package", we defeat the purpose of >> team maintenance. Instead, we should be setting standards for the whole >> team, document it as team policy and implement it across all team maintained >> packages. All right, but between mistakes and wrong-doings, when it "just works", please don't change what has been done before - especially without testing it before uploading. Also discussing it or at least notifying it, would be a good idea. Jérémy -- Pkg-javascript-devel mailing list Pkg-javascript-devel@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel
Re: [Pkg-javascript-devel] testing during build is useless now with autopkgtests
On 2016, നവംബർ 26 6:34:26 PM IST, "Jérémy Lal" wrote: >Hi, > >i've been doing in all my latest uploads this setup: >- autopkgtest runs the package test suite >- build does not run tests > >Because: >1) it feels wrong to add Depends to Build-Depends, >and debian/tests/control already have all the info to properly run the >tests >2) when an autopkgtest fails, archive-wide testing reports the failure >as a package bug, >rendering tests during build useless anyway No, that is not correct. If tests during build fails, it is rc and it is found in archive wide rebuilds. But autopkgtest failures are not rc and the bug reporting is not automatic. I know this for certain because autopkgtest for gitlab has been failing for some time. So at least in the current situation, autopkgtest is not a replacement for tests during build, it may reach that point in future, but we are not there yet. >I know that in some cases, one really need a built tree to run the >tests, >but it's not the majority. Usually a tree with patches applied is >enough. > >So please Team, don't re-add to all my packages "run tests during >build". >(and, of course, do whatever you see best when you're packaging it >yourself). This is not the right way to approach team maintained packages. If we end up differentiating "my package" and "your package", we defeat the purpose of team maintenance. Instead, we should be setting standards for the whole team, document it as team policy and implement it across all team maintained packages. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Pkg-javascript-devel mailing list Pkg-javascript-devel@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel
