[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
** Changed in: zorba Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: Fix Released Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
** Changed in: zorba Status: Confirmed = In Progress -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: In Progress Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
I have revamped the known-crashing test list arguments to ADD_TEST_DIRECTORY(). It now defers to EXPECTED_FAILURE() on non-Win32 platforms. It also checks that all tests passed to it actually exist. I extended the macro documentation accordingly. The only module currently making use of this functionality is the image module; I have modified it accordingly to pass bug numbers and name the tests correctly. ** Branch linked: lp:~zorba-coders/zorba/bug-874679-add-test-directory ** Branch linked: lp:~zorba-coders/zorba/bug-874679-image-module -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: In Progress Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
** Changed in: zorba Status: In Progress = Fix Committed -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: Fix Committed Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
** Changed in: zorba Status: New = Confirmed -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: Confirmed Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
** Changed in: zorba Milestone: 2.2 = 2.5 -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: New Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
I didn't intend for this solution to *replace* EXPECTED_FAILURE(). The list of bugs sent to this function should only be those which segfault on Windows, and hence they should be skipped entirely on Windows and marked as EXPECTED_FAILURE() on other platforms. Tests which fail for non-segfault reasons should continue to use the existing EXPECTED_FAILURE() mechanism. Basically, it's an ugly solution to an ugly problem. The KNOWN_SEGFAULT() proposal is a slightly less ugly solution, but more work to implement. -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: New Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
** Changed in: zorba Assignee: Gabriel Petrovay (gabipetrovay) = Chris Hillery (ceejatec) -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: New Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
Correction: This would only *work* for tests that failing uniformly on all platforms. -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: New Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
I was about to do this but I still have some question: You say: In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). With this approach you cannot determine INSIDE the macro which of the following things to do: 1. ADD_TEST 2. EXPECTED_FAILURE on all platforms 3. EXPECTED_FAILURE on Windows For the seg-faulting on Windows, EXPECTED_FAILURE can not be the solution. For the tests failing only on Windows, there is no way to determine the correct logic inside ADD_TEST_DIRECTORY based only on a test name and a bug number. This would only only for tests that failing uniformly on all platforms. So probably a KNOWN_SEGFAULT solution would be the only solution (at least from the ones you mentioned). Please give detail/feedback on this. -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: New Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Bug 874679] Re: Clean up second arg to ADD_TEST_DIRECTORY()
You say: ADD_TEST_DIRECTORY() needs to [...] accept a list of pairs [...] so it can pass the bug number to EXPECTED_FAILURE(). And how is this disabling tests on Windows? (because EXPECTED_FAILURE is not good enough on Windows) -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/874679 Title: Clean up second arg to ADD_TEST_DIRECTORY() Status in Zorba - The XQuery Processor: New Bug description: The second (optional) argument to ADD_TEST_DIRECTORY() contains a list of test cases that should be skipped (ie, not created at all). This is extremely dangerous. By bypassing them entirely, there is no way of detecting when the bug is fixed. We have already had serious problems caused by test removal hiding the introduction of new bugs in the past. I understand that removing a test that segfaults is necessary on Windows in order for the tests to run unattended. However, it should never be done on other platforms. So ADD_TEST_DIRECTORY() should instead mark the bugs as EXPECTED_FAILURE()s on other platforms. (I realize that *currently* all places that pass a list of segfaulting tests to ADD_TEST_DIRECTORY() are inside an IF(WIN32), but this is not sufficient - ADD_TEST_DIRECTORY() itself must enforce that the tests are skipped only on Windows, or else sooner or later someone will accidentally do it wrong. There's currently no documentation either, making this much more likely.) In order to do this, ADD_TEST_DIRECTORY() needs to either accept a list of pairs of the form (testname, bug ID), or else accept two parallel lists (testnames and bug IDs), so it can pass the bug number to EXPECTED_FAILURE(). A slightly more difficult approach would be to introduce a new KNOWN_SEGFAULT() macro that handled this in conjunction with ADD_TEST_DIRECTORY(). I'm not totally sure this is a good idea, though, because it would (silently) not do anything for tests added by any means other than ADD_TEST_DIRECTORY(). To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/874679/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
