Re: [gentoo-dev] tests
Am Mittwoch, 2. Mai 2007 schrieb Rémi Cardona: > Piotr Jaroszyński a écrit : > > On Tuesday 01 of May 2007 21:53:36 Maurice van der Pot wrote: > >> I'm not sure why this is a reply to my message instead of the > >> message I replied to. They both provide more or less the same > >> choice to the user. > > > > Err I wasn't providing any choices for users yet, I only thought > > about the below as things that can be wanted by users/devs and > > asked whether I missed something. How we will end up distinguishing > > them is another story... > > > >> - run all tests > >> - run only reasonable tests > >> - run only necessary tests > >> - don't run tests at all > > Philosophical question : > > What's "reasonable"? Where do you draw the line? Same question for > "necessary". Re 'necessary': Tests for scientific packages are surely necessary, i'd even claim they're mandatory. Nothing is as bad as having a program that yields unreproducable (read: wrong) results. Been there, seen that, had the primordial urge to kill things. Danny -- Danny van Dyk <[EMAIL PROTECTED]> Gentoo/AMD64 Project, Gentoo Scientific Project -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
Hi Daniel, Am Mittwoch, 2. Mai 2007 schrieb Daniel Gryniewicz: > Honestly, tests are nice, but too many of them are broken upstream, > and we are not (and should not be, IMO) in the position of fixing > them all. If a developer wants to work with her upstream to fix the > tests in her packages, great and more power to her. Most of us are > swamped just supporting them, let alone fixing test cases. You > really need an upstream who cares a lot about tests for the tests to > be meaningful and work. Lots of upstreams don't currently care, and > have inherited obsolete and (now) broken tests from previous > maintainers. When you read Piotr's original mail carefully, you will see that he lists 'non-functional' as category, and nobody keeps you from declaring your packages' test-suites as such. However, keep in mind that several other maintainers don't have so many problems with their test-suites. > I think this thread in general overestimates the value of tests in > packages. I think we will find, if we go through the effort, that > more of them are useless and/or broken than are useful. My 2 cents. As a member of the sci team I have to say I completely disagree with you here. sci-* packages mostly have reasonable test suites, the importance to run them is very high (you do want reproducable and correct results, don't you?). However, sometimes you cannot run those tests from an ebuild's environment, for example when you need a running x-server. Danny -- Danny van Dyk <[EMAIL PROTECTED]> Gentoo/AMD64 Project, Gentoo Scientific Project -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
Piotr Jaroszyński a écrit : On Tuesday 01 of May 2007 21:53:36 Maurice van der Pot wrote: I'm not sure why this is a reply to my message instead of the message I replied to. They both provide more or less the same choice to the user. Err I wasn't providing any choices for users yet, I only thought about the below as things that can be wanted by users/devs and asked whether I missed something. How we will end up distinguishing them is another story... - run all tests - run only reasonable tests - run only necessary tests - don't run tests at all Philosophical question : What's "reasonable"? Where do you draw the line? Same question for "necessary". These two notions are very much subjective, both in terms of required time, dependencies, but also in terms of perceived usefulness. Cheers, Rémi -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Wed, 2007-05-02 at 01:12 +0100, Stephen Bennett wrote: > On Tue, 01 May 2007 19:46:56 -0400 > Daniel Gryniewicz <[EMAIL PROTECTED]> wrote: > > > There is one serious problem with this: Who's going to do the work to > > figure all this out for the 11,000 odd packages in the tree? This > > seems like a *huge* amount of work, work that I have no plan on doing > > for the 100-odd packages I (help) maintain, let alone the 4-10 > > different versions of each package. I highly doubt other maintainers > > want to do this kind of work either. > > Last I heard the intention was to tie it to the EAPI=1 bump, so that > packages can be updated one by one as they move to the newer eapi. > Current (ie EAPI=0) ebuilds will continue to function as they have done. Sure, but now you're requiring me to go through all that extra work if I want any of the benefits of EAPI=1. Or alternatively, dooming us to support EAPI=0 forever, since I don't want to do that work. Or, third option, is that everyone marks their packages as "low priority tests, don't run them" just to switch to EAPI=1, and we have no gain over what we have now. Honestly, tests are nice, but too many of them are broken upstream, and we are not (and should not be, IMO) in the position of fixing them all. If a developer wants to work with her upstream to fix the tests in her packages, great and more power to her. Most of us are swamped just supporting them, let alone fixing test cases. You really need an upstream who cares a lot about tests for the tests to be meaningful and work. Lots of upstreams don't currently care, and have inherited obsolete and (now) broken tests from previous maintainers. I think this thread in general overestimates the value of tests in packages. I think we will find, if we go through the effort, that more of them are useless and/or broken than are useful. My 2 cents. Daniel -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Wed, May 02, 2007 at 12:55:05AM +0100, Ciaran McCreesh wrote: > You're talking implementation details. This isn't the time for that! > No-one has worked out what, if anything, is to be done, so you can't > know how much work whatever it is is. > > Having said that, there's no need to figure it out for the whole tree > in one go if it's an EAPI change. Please stop stating that fallacy; binding it into an EAPI version means you *have* to plan for the whole tree (meaning figure it out up front). Skipping the whole "plan the fall out of it" bit of EAPI versioning means you wind up pushing multiple versions out (fragmenting the format), or forcing changes on folks while claiming "yeah, won't affect ya". ~harring pgp1p83xto4d5.pgp Description: PGP signature
Re: [gentoo-dev] tests
On Tue, 01 May 2007 19:46:56 -0400 Daniel Gryniewicz <[EMAIL PROTECTED]> wrote: > There is one serious problem with this: Who's going to do the work to > figure all this out for the 11,000 odd packages in the tree? This > seems like a *huge* amount of work, work that I have no plan on doing > for the 100-odd packages I (help) maintain, let alone the 4-10 > different versions of each package. I highly doubt other maintainers > want to do this kind of work either. You're talking implementation details. This isn't the time for that! No-one has worked out what, if anything, is to be done, so you can't know how much work whatever it is is. Having said that, there's no need to figure it out for the whole tree in one go if it's an EAPI change. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Tue, May 01, 2007 at 07:46:56PM -0400, Daniel Gryniewicz wrote: > On Wed, 2007-05-02 at 01:32 +0200, Marius Mauch wrote: > > I'd approach it a bit different: Before creating fixed classification > > groups I'd first identify the attributes of tests that should be used > > for those classifications. > > a) cost (in terms of runtime, resource usage, additional deps) > > b) effectiveness (does a failing/working test mean the package is > > broken/working?) > > c) importance (is there a realistic chance for the test to be useful?) > > d) correctness (does the test match the implementation? overlaps a bit > > with effectiveness) > > e) others? > There is one serious problem with this: Who's going to do the work to > figure all this out for the 11,000 odd packages in the tree? This seems > like a *huge* amount of work, work that I have no plan on doing for the > 100-odd packages I (help) maintain, let alone the 4-10 different > versions of each package. I highly doubt other maintainers want to do > this kind of work either. This wouldn't be an instant transition, and a lot of packages would be covered under the 'importance' side. Using crypto packages as an example, the costs are low (compare known input+output pairs), the effectiveness is high, the importance is high (witness the checksum problems caused in the tree some months ago), and the correctness is very high. The mysql testcases on the other hand have a low effectiveness, there have been lots of cases where they break due to userpriv or sandbox and high cost. For the packages I maintain, I'd definitely implement test stuff for the crypto and system-admin packages where feasible, but for a lot of others I wouldn't bother - the cost/benefit ratio is not high enough. -- Robin Hugh Johnson Gentoo Linux Developer & Council Member E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgprsk8CFF5tt.pgp Description: PGP signature
Re: [gentoo-dev] tests
On Tue, 01 May 2007 19:46:56 -0400 Daniel Gryniewicz <[EMAIL PROTECTED]> wrote: > There is one serious problem with this: Who's going to do the work to > figure all this out for the 11,000 odd packages in the tree? This > seems like a *huge* amount of work, work that I have no plan on doing > for the 100-odd packages I (help) maintain, let alone the 4-10 > different versions of each package. I highly doubt other maintainers > want to do this kind of work either. Last I heard the intention was to tie it to the EAPI=1 bump, so that packages can be updated one by one as they move to the newer eapi. Current (ie EAPI=0) ebuilds will continue to function as they have done. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Wed, 2 May 2007 01:32:20 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: > (btw, could someone give some real examples for packages with > "necessary" tests?) There're two groups of packages with necessary tests that come to mind: those that are very compiler / system sensitive (certain scientific apps), and those where upstream says that 'make check' is a vital part of the build process. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Wed, 2007-05-02 at 01:32 +0200, Marius Mauch wrote: > > I'd approach it a bit different: Before creating fixed classification > groups I'd first identify the attributes of tests that should be used > for those classifications. > a) cost (in terms of runtime, resource usage, additional deps) > b) effectiveness (does a failing/working test mean the package is > broken/working?) > c) importance (is there a realistic chance for the test to be useful?) > d) correctness (does the test match the implementation? overlaps a bit > with effectiveness) > e) others? There is one serious problem with this: Who's going to do the work to figure all this out for the 11,000 odd packages in the tree? This seems like a *huge* amount of work, work that I have no plan on doing for the 100-odd packages I (help) maintain, let alone the 4-10 different versions of each package. I highly doubt other maintainers want to do this kind of work either. Daniel -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tue, 1 May 2007 15:08:56 +0200 Piotr Jaroszyński <[EMAIL PROTECTED]> wrote: > Hello, > > There was some discussion about forcing/not forcing tests in EAPI-1, > but there was clearly no compromise. Imho, tests are very important > and thus I want to discuss them a little more, but in more sensible > fashion. > > Firstly each test can be(not all categories are mutually exclusive): > - not existant > - non-functional > - not runnable from ebuild > - useful but unreasonable resource-wise > - useful and reasonable resource-wise > - necessary > - known to partially fail but with a way of skipping failing tests > - known to partially fail but with no easy way of skipping failing > tests Is that list comprehensive? I'd approach it a bit different: Before creating fixed classification groups I'd first identify the attributes of tests that should be used for those classifications. a) cost (in terms of runtime, resource usage, additional deps) b) effectiveness (does a failing/working test mean the package is broken/working?) c) importance (is there a realistic chance for the test to be useful?) d) correctness (does the test match the implementation? overlaps a bit with effectiveness) e) others? Each of these needs to be considered if we want to find a good compromise of which tests to run and which not. A test with high cost can still be worth running if effectiveness, correctness and importance are also high, on the other hand a test with little effectiveness, correctness and/or importance probably isn't worth running even with zero cost. Now the tricky question is how to actually measure those attributes. > Secondly we must answer the question how precisely we want to > distinguish them, so users/dev can choose which categories of tests > they want to run. What comes to mind is: > - run all tests > - run only necessary tests > - run only reasonable tests > - don't run tests at all > Again, is that list comprehensive? Problem is that terms like "reasonable" or "necessary" are quite subjective (regarding both humans and machines), and in this special context even "all" could be interpreted in different ways (btw, could someone give some real examples for packages with "necessary" tests?). So I think a more fine grained classification is needed that can be adopted for specific use cases (e.g. the mips+embedded profiles might want different defaults than the amd64+desktop profiles). Marius -- Public Key at http://www.genone.de/info/gpg-key.pub In the beginning, there was nothing. And God said, 'Let there be Light.' And there was still nothing, but you could see a bit better. signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Tue, 01 May 2007 14:52:30 -0700 Josh Saddler <[EMAIL PROTECTED]> wrote: > anyway, on the subject of tests...as others have covered the *first* > time this was discussed on the lists, mandatory tests being run every > time the user installs a package? no. oh hell no. we don't seem to do > that much with the packages in our tree now, do we? The first time this was discussed didn't come up with an ideal solution, and the existing situation is severely broken. Hence why this thread is going back to first principles -- once we've established what the problem really is, *then* we can start discussing solutions again. If you don't have anything constructive to say about the former, please keep quiet about the latter. That *you* cannot see an adequate solution does not mean that others won't after proper discussion. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Tue, 1 May 2007 21:53:36 +0200 Maurice van der Pot <[EMAIL PROTECTED]> wrote: > > Too complicated. Bombarding the user with pointless alternatives is > > not the same as giving the user choice. > > I'm not sure why this is a reply to my message instead of the message > I replied to. They both provide more or less the same choice to the > user. This thread is not about what's to be presented to the user. This thread is about the tests. Discussing what's to be presented to the user at this stage is premature. > > I'm also highly sceptical that the properties you listed are > > boolean. Resource hungry on an IP22 could be a walk in the park for > > an X16... > > I suppose that's possible, but if you look at it like that probably > everything can be called resource hungry on some machines. And if you > own a Blue Gene, you probably don't worry too much and enable > everything. > > Or do you mean that for instance tests involving lots of floating > point calculations are a big deal for cpus that use FP emulation? > Isn't that peanuts compared to the tests that would be called > resource hungry here? The point is, on some archs it is reasonable to expect that many users will have sixteen plus logical CPUs with frequencies measured in at least hundreds of MHz and memory measured in gigabytes, and many other users will have a single sub-hundred MHz CPU and maybe 128MBytes RAM if we're lucky. Test suites like, say, Boost's, are trivial to run on the former and impossible on the latter. > We wouldn't have to prove to anyone that a test is resource hungry. We > would just have to put each set of tests into one of two groups. If > you're not sure in which group it belongs, it probably doesn't matter > that much anyway. > > Look at merge times... everybody agrees openoffice, mozilla firefox, > gcc and qt take quite some time to emerge and that vim, bash and > iptables do not. That's the kind of distinction that would be useful. It's not that simple. You're forgetting that many archs routinely deal with systems with eight or sixteen way CPUs. If a package parallelises, it's fast on such systems. If it doesn't, it's immensely slow. > > > fex: > > Please don't abuse the English language in that manner. > > Since you took the time to highlight this apparently grave injustice > to the English language, would you please explain it to me so I can do > better next time? 'fex' isn't English, and it comes across as extremely annoying and unprofessional to many native speakers. It's worse than AOL style 'r u 2' things because 'e.g' is a similarly short and entirely correct alternative. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Wednesday 02 of May 2007 00:28:42 Josh Saddler wrote: > Not a knee jerk reaction, just a strong one. One of the key reasons why > mandatory tests were not desired was the fact that sometimes much more > stuff will be installed than what you'd normally get. Exhibit A: > robbat2's message just sent on diradm that normally just needs openldap > with USE=minimal, but building for tests requires all of openldap, > samba, etc. Where did you read that this thread is about forcing tests? That was the black and white approach and we all know how it failed... The purpose of this discussion is to figure out a compromise between the current state and force all, because neither of them is good. -- Best Regards, Piotr Jaroszyński -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
Stephen Bennett wrote: > On Tue, 01 May 2007 14:52:30 -0700 > Josh Saddler <[EMAIL PROTECTED]> wrote: > >> anyway, on the subject of tests...as others have covered the *first* >> time this was discussed on the lists, mandatory tests being run every >> time the user installs a package? no. oh hell no. we don't seem to do >> that much with the packages in our tree now, do we? > > Care to turn that into a reasoned argument rather than what appears to > be a knee-jerk reaction? Not a knee jerk reaction, just a strong one. One of the key reasons why mandatory tests were not desired was the fact that sometimes much more stuff will be installed than what you'd normally get. Exhibit A: robbat2's message just sent on diradm that normally just needs openldap with USE=minimal, but building for tests requires all of openldap, samba, etc. I'd like to think we aren't in the practice of forcing users to install cruft on their systems. If you need more examples from the last thread, assuming you don't still have local archives, I could scrounge 'em from gmane I suppose, though we're both equally capable of typing in search phrases. The tests subject wasn't brought up that long ago, either. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] tests
On Tue, 01 May 2007 14:52:30 -0700 Josh Saddler <[EMAIL PROTECTED]> wrote: > anyway, on the subject of tests...as others have covered the *first* > time this was discussed on the lists, mandatory tests being run every > time the user installs a package? no. oh hell no. we don't seem to do > that much with the packages in our tree now, do we? Care to turn that into a reasoned argument rather than what appears to be a knee-jerk reaction? -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tue, May 01, 2007 at 10:10:28PM +0200, Jure Varlec wrote: > On Tuesday 01 of May 2007 21:24:17 R??mi Cardona wrote: > > - require other and bigger deps than what the actual package requires > Hm, perhaps this one should be split into: > -- additional deps are already installed > -- additional deps are not yet installed No, it's more complicated than that. One of my packages, app-admin/diradm, needs deps built a certain way. Building it for normal usage just requires openldap built with USE=minimal. Building it for tests requires a full OpenLDAP, and additionally requires all of samba. -- Robin Hugh Johnson Gentoo Linux Developer & Council Member E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgp0NQ0Q5DnFA.pgp Description: PGP signature
Re: [gentoo-dev] tests
Maurice van der Pot wrote: >>> fex: >> Please don't abuse the English language in that manner. > > Since you took the time to highlight this apparently grave injustice to > the English language, would you please explain it to me so I can do > better next time? he just doesn't like it because it's ferringb-speak and it makes him cranky. anyway, on the subject of tests...as others have covered the *first* time this was discussed on the lists, mandatory tests being run every time the user installs a package? no. oh hell no. we don't seem to do that much with the packages in our tree now, do we? signature.asc Description: OpenPGP digital signature
[gentoo-dev] Re: tests
> Firstly each test can be(not all categories are mutually exclusive): > (...) How many of these we can find is not really that important. I mentioned the different categories just to show that tests are not black and white and we need more then boolean choice to make good use of them. What we need to figure out is the categories we want to distinguish between in ebuilds and *then* how to implement that sanely. -- Best Regards, Piotr Jaroszyński -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tuesday 01 of May 2007 21:24:17 Rémi Cardona wrote: > - require other and bigger deps than what the actual package requires Hm, perhaps this one should be split into: -- additional deps are already installed -- additional deps are not yet installed Regards signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] tests
On Tuesday 01 of May 2007 21:53:36 Maurice van der Pot wrote: > I'm not sure why this is a reply to my message instead of the message I > replied to. They both provide more or less the same choice to the user. Err I wasn't providing any choices for users yet, I only thought about the below as things that can be wanted by users/devs and asked whether I missed something. How we will end up distinguishing them is another story... > - run all tests > - run only reasonable tests > - run only necessary tests > - don't run tests at all -- Best Regards, Piotr Jaroszyński -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tue, May 01, 2007 at 06:35:22PM +0100, Ciaran McCreesh wrote: > On Tue, 1 May 2007 19:18:28 +0200 > Maurice van der Pot <[EMAIL PROTECTED]> wrote: > > I'd say, let the user decide based on the properties > > Too complicated. Bombarding the user with pointless alternatives is not > the same as giving the user choice. I'm not sure why this is a reply to my message instead of the message I replied to. They both provide more or less the same choice to the user. > I'm also highly sceptical that the properties you listed are boolean. > Resource hungry on an IP22 could be a walk in the park for an X16... I suppose that's possible, but if you look at it like that probably everything can be called resource hungry on some machines. And if you own a Blue Gene, you probably don't worry too much and enable everything. Or do you mean that for instance tests involving lots of floating point calculations are a big deal for cpus that use FP emulation? Isn't that peanuts compared to the tests that would be called resource hungry here? We wouldn't have to prove to anyone that a test is resource hungry. We would just have to put each set of tests into one of two groups. If you're not sure in which group it belongs, it probably doesn't matter that much anyway. Look at merge times... everybody agrees openoffice, mozilla firefox, gcc and qt take quite some time to emerge and that vim, bash and iptables do not. That's the kind of distinction that would be useful. > > fex: > Please don't abuse the English language in that manner. Since you took the time to highlight this apparently grave injustice to the English language, would you please explain it to me so I can do better next time? Maurice. -- Maurice van der Pot Gentoo Linux Developer [EMAIL PROTECTED] http://www.gentoo.org Creator of BiteMe! [EMAIL PROTECTED] http://www.kfk4ever.com pgpHou3xcVjwv.pgp Description: PGP signature
Re: [gentoo-dev] tests
Piotr Jaroszyński wrote: > Hello, > > There was some discussion about forcing/not forcing tests in EAPI-1, but > there > was clearly no compromise. Imho, tests are very important and thus I want to > discuss them a little more, but in more sensible fashion. > > Firstly each test can be(not all categories are mutually exclusive): > - not existant > - non-functional > - not runnable from ebuild > - useful but unreasonable resource-wise > - useful and reasonable resource-wise > - necessary > - known to partially fail but with a way of skipping failing tests > - known to partially fail but with no easy way of skipping failing tests - require other and bigger deps than what the actual package requires Rémi -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] Last rites media-sound/jsynthlib
federico ha scritto: > William L. Thomson Jr. ha scritto: > >> Last rites for media-sound/jsynthlib >> >> Last upstream release was 0.20-beta, released March, 2005. >> >> The package has been masked, and will be moved to Java junkyard overlay >> after 30 days pass. Unless someone cares about this package, needs it, >> uses it, and/or is willing to updated it. >> >> > I need/use it! > shall I provide an updated ebuild or try to maintain it? > > http://bugs.gentoo.org/show_bug.cgi?id=137688 -- Federico -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tuesday 01 of May 2007 19:18:28 Maurice van der Pot wrote: > Isn't it easier to list a set of boolean properties of _individual_ > tests? It was just a list of different test classes, which came to mind. The question, which still persist, was how precisely we want to divide them into groups as current boolean choice seems to be not enough. > I'd say, let the user decide based on the properties, fex: It seems to be too early for that. Firstly we should figure out the "properties" and then we can think how to deliver them for end-users. -- Best Regards, Piotr Jaroszyński -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tue, 1 May 2007 19:18:28 +0200 Maurice van der Pot <[EMAIL PROTECTED]> wrote: > I'd say, let the user decide based on the properties Too complicated. Bombarding the user with pointless alternatives is not the same as giving the user choice. I'm also highly sceptical that the properties you listed are boolean. Resource hungry on an IP22 could be a walk in the park for an X16... > fex: Please don't abuse the English language in that manner. > I'm sorry if this counts as a solution, but I wasn't sure the list you > gave was the kind of list we need and I didn't know another way of > explaining the use of the properties I listed. Well, every solution offered so far is at least moderately icky... I believe the idea is to look at the problem in more detail before going off in what's probably the wrong direction again. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Tue, May 01, 2007 at 03:08:56PM +0200, Piotr Jaroszyński wrote:
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?
Isn't it easier to list a set of boolean properties of _individual_
tests?
- We don't need "non existent".
- Non-functional and known to partially fail come down to "known to
fail" for individual tests.
- I have no idea what "not runnable from ebuild" is.
- Unreasonable could be "resource hungry" or "needs additional deps"
(those are two different things).
- If a test is "necessary" I don't see why we should allow it to be
skipped.
- And about skipping failing tests. There is always a way to skip
failing tests: not running any of them. It's just the granularity that
is different, but the user doesn't care about that.
> Secondly we must answer the question how precisely we want to distinguish
> them, so users/dev can choose which categories of tests they want to run.
> What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?
I'd say, let the user decide based on the properties, fex:
run known to fail : no
run resource hungry: yes
run additional deps: no
if ( (known to fail == false || run known to fail == true) &&
(resource hungry == false || run resource hungry == true) &&
(additional deps == false || run additional deps == true) )
{
run the test
}
So for each test/set of tests and for each possible reason not to run
it, either the reason does not apply to this test or the user explicitly
said to run it anyway.
You don't see the "way of skipping failing tests" in this last part.
This is because if the decision is made not to run a test, the smallest
set that can be skipped (possibly all tests for an ebuild) are skipped.
> Please don't post solutions unless we figure out which options we really want
> to deliver.
I'm sorry if this counts as a solution, but I wasn't sure the list you
gave was the kind of list we need and I didn't know another way of
explaining the use of the properties I listed.
Regards,
Maurice.
--
Maurice van der Pot
Gentoo Linux Developer [EMAIL PROTECTED] http://www.gentoo.org
Creator of BiteMe! [EMAIL PROTECTED] http://www.kfk4ever.com
pgpZvWXoOPeIK.pgp
Description: PGP signature
Re: [gentoo-dev] tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Daniel Gryniewicz wrote: > On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote: >> Hello, >> >> There was some discussion about forcing/not forcing tests in EAPI-1, but >> there >> was clearly no compromise. Imho, tests are very important and thus I want to >> discuss them a little more, but in more sensible fashion. >> >> Firstly each test can be(not all categories are mutually exclusive): >> - not existant >> - non-functional >> - not runnable from ebuild >> - useful but unreasonable resource-wise >> - useful and reasonable resource-wise >> - necessary >> - known to partially fail but with a way of skipping failing tests >> - known to partially fail but with no easy way of skipping failing tests >> Is that list comprehensive? >> >> Secondly we must answer the question how precisely we want to distinguish >> them, so users/dev can choose which categories of tests they want to run. >> What comes to mind is: >> - run all tests >> - run only necessary tests >> - run only reasonable tests >> - don't run tests at all >> Again, is that list comprehensive? >> > > Don't forget tests that have heavy requirements to run. Many gnome > tests, for example, need a virtual X to run, which puts a new set of > DEPENDS requirements on your system. > > Daniel > And sometimes these extra deps can result in circular deps. - -- Vlastimil Babka (Caster) Gentoo/Java -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGN2mKtbrAj05h3oQRAsIPAJ40sOQV97jBCUFAIcKZFHJ1mRiu4QCggfz6 Toh/ZYYpU7lJfmVOqQclaWo= =ULIL -END PGP SIGNATURE- -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote: > Hello, > > There was some discussion about forcing/not forcing tests in EAPI-1, but > there > was clearly no compromise. Imho, tests are very important and thus I want to > discuss them a little more, but in more sensible fashion. > > Firstly each test can be(not all categories are mutually exclusive): > - not existant > - non-functional > - not runnable from ebuild > - useful but unreasonable resource-wise > - useful and reasonable resource-wise > - necessary > - known to partially fail but with a way of skipping failing tests > - known to partially fail but with no easy way of skipping failing tests > Is that list comprehensive? > > Secondly we must answer the question how precisely we want to distinguish > them, so users/dev can choose which categories of tests they want to run. > What comes to mind is: > - run all tests > - run only necessary tests > - run only reasonable tests > - don't run tests at all > Again, is that list comprehensive? > Don't forget tests that have heavy requirements to run. Many gnome tests, for example, need a virtual X to run, which puts a new set of DEPENDS requirements on your system. Daniel -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] Re: That time again...
Michael Cummings wrote: > On Thu, Apr 26, 2007 at 08:29:43PM +0200, Christian Faulhammer wrote: >> You are declared official Project Status Report Gathering Manager. > > I have to concede, I saw this last thursday and sat on it all weekend, mulling > it over. I realize you (most likely) meant it as a joke, but for what its > worth, I think I'd suck at it if I had to do it consistently. > > That said...expect more until someone does take up the role ;) Now figuring > out > how to get this stuff rolled into the gwn (i smell an email to gwn-feedback in > my future, I know) so that it could get better coverage (since the mudrakers > only look to -dev for the flame wars, not the fun stuff). > > ~mcummings > My project status glep (and requisite software) is about 50% done, I just need to get the ruby -> GuideXML -> xml part done and polish off the glep (and get it approved). -Alec -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
Piotr Jaroszyński wrote: > Hello, > > There was some discussion about forcing/not forcing tests in EAPI-1, but > there > was clearly no compromise. Imho, tests are very important and thus I want to > discuss them a little more, but in more sensible fashion. > > Firstly each test can be(not all categories are mutually exclusive): > - not existant > - non-functional > - not runnable from ebuild > - useful but unreasonable resource-wise > - useful and reasonable resource-wise > - necessary > - known to partially fail but with a way of skipping failing tests > - known to partially fail but with no easy way of skipping failing tests > Is that list comprehensive? There are some tests that require root, and thus only fails with userpriv. Kernel modules that attempt to load themselves to make sure there are no problems are an example. I forget the exact package offhand. Otherwise I like where this discussion is going ;) -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] Monthly Gentoo Council Reminder for May
Petteri Räty wrote: Mike Frysinger kirjoitti: If you have something you'd wish for us to chat about, maybe even vote on, let us know ! Simply reply to this e-mail for the whole Gentoo dev list to see. I would like the council to remind everyone that this is not appropriate for any team: https://bugs.gentoo.org/show_bug.cgi?id=176256#c4 The alpha arch team is *fully* agree with sparc in this case (as many others) and hope the council make a clear statement against this kind of keywording, which we think can create many more problems than solutions. We *really* appreciate the great job doing by our games herd (one of the best out there) but, in this case, this is not the way to follow. Thanks. -- Jose Luis Rivero [EMAIL PROTECTED] Gentoo/Doc Gentoo/Alpha -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] tests
On Tue, 01 May 2007 09:24:34 -0400 Josh Sled <[EMAIL PROTECTED]> wrote: > On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote: > > Firstly each test can be(not all categories are mutually exclusive): > [...] > > - necessary > > Could you qualify, please? Is this "necessary for the (non-test) > build artifact"? There are packages where upstream says that running 'make check' is an essential part of the build process, and that you shouldn't install without having done so. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] tests
On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote: > Firstly each test can be(not all categories are mutually exclusive): [...] > - necessary Could you qualify, please? Is this "necessary for the (non-test) build artifact"? If so, I'd not call it a test, just part of the build that's invoked via `make check`. ;) -- ...jsled http://asynchronous.org/ - a=jsled;b=asynchronous.org; echo [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
[gentoo-dev] tests
Hello, There was some discussion about forcing/not forcing tests in EAPI-1, but there was clearly no compromise. Imho, tests are very important and thus I want to discuss them a little more, but in more sensible fashion. Firstly each test can be(not all categories are mutually exclusive): - not existant - non-functional - not runnable from ebuild - useful but unreasonable resource-wise - useful and reasonable resource-wise - necessary - known to partially fail but with a way of skipping failing tests - known to partially fail but with no easy way of skipping failing tests Is that list comprehensive? Secondly we must answer the question how precisely we want to distinguish them, so users/dev can choose which categories of tests they want to run. What comes to mind is: - run all tests - run only necessary tests - run only reasonable tests - don't run tests at all Again, is that list comprehensive? Please don't post solutions unless we figure out which options we really want to deliver. -- Best Regards, Piotr Jaroszyński -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] Monthly Gentoo Council Reminder for May
Mike Frysinger kirjoitti: > > If you have something you'd wish for us to chat about, maybe even > vote on, let us know ! Simply reply to this e-mail for the whole > Gentoo dev list to see. > I would like the council to remind everyone that this is not appropriate for any team: https://bugs.gentoo.org/show_bug.cgi?id=176256#c4 Regards, Petteri signature.asc Description: OpenPGP digital signature
[gentoo-dev] Monthly Gentoo Council Reminder for May
This is your monthly friendly reminder ! Same bat time (typically the 2nd Thursday at 2000 UTC / 1600 EST), same bat channel (#gentoo-council @ irc.freenode.net) ! If you have something you'd wish for us to chat about, maybe even vote on, let us know ! Simply reply to this e-mail for the whole Gentoo dev list to see. Keep in mind that every GLEP *re*submission to the council for review must first be sent to the gentoo-dev mailing list 7 days (minimum) before being submitted as an agenda item which itself occurs 7 days before the meeting. Simply put, the gentoo-dev mailing list must be notified at least 14 days before the meeting itself. For more info on the Gentoo Council, feel free to browse our homepage: http://www.gentoo.org/proj/en/council/ -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] gentoo: static/dynamic linking libraries
On 01/05/07, Peter Gordon <[EMAIL PROTECTED]> wrote: > Does anyone care about static libs except for maybe really really low > level stuff? They are useful for rescue operations and whatnot, when a LiveCD or similar is not handy; or perhaps when the computer cannot boot from an alternative medium. That's the only major benefit I see of them. They are also very often used in chrooted environments. -- radoslaw. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] gentoo: static/dynamic linking libraries
Am Montag 30 April 2007 21:00 schrieb Kevin F. Quinn: > The thing about static libraries, is that the ebuild that creates them > doesn't know whether anything else will want to use them. It may be > that in practice, most libraries are never used in their static form - > but the point is that the ebuild doesn't know enough information to > make the decision. That's true for now, but it won't anymore when use dependencies are implemented. Then a corresponding useflag could be used to opt-in for static builds. > However, with INSTALL_MASK, the user makes the decision never to have > static binaries, and thus gets a system free of static libraries. Except the little detail that INSTALL_MASK definately breaks things. I tried it yesterday. The reason is that packages that build static libs (though --disable-static is set) will depend on other static libs. With INSTALL_MASK in place those static libs are never installed. Hence the build fails. So it is not a working option. pgpSSbjojvOsK.pgp Description: PGP signature
