On 05/06/2016 03:03 PM, Oleg Fayans wrote:



On 05/06/2016 12:08 PM, Martin Babinsky wrote:
On 05/06/2016 11:14 AM, Oleg Fayans wrote:


On 05/06/2016 09:48 AM, Martin Basti wrote:


On 06.05.2016 09:36, Oleg Fayans wrote:
Tests are finally stable:

============================= test session starts
==============================
platform linux2 -- Python 2.7.11 -- py-1.4.30 -- pytest-2.7.3
rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile: pytest.ini
plugins: multihost, sourceorder
collected 8 items

test_integration/test_dnssec.py ........

========================= 8 passed in 5561.48 seconds
==========================





PATCH 38 LGTM

PATCH 37 IIRC I refused to accept workaround for this issue when you
send this (almost the same) patch for first time, are you sure that we
want to hide real issues in tests, to just have green color there?


The underlying issue is 7 months old. Latest update in the issue from
Peter Spacek is: "I do not have time to investigate this issue now",
which means, that it will stay there for unpredictable amount of time
more. If we want to have a "green" jenkins that actually tests existing
features, we have to accept workarounds for such long-term issues

Martin

I have never been a big fan of "having a green jenkins whatever it
takes" but I understand that there are all kinds of pressure on your
team to deliver 100% stable test results.

If the test fails, let it fail or, even better, use 'xfail' markers so
that we know that this test fails and we should investigate.
Then all 8 existing cases would have to be marked as xfailed.

Sorry I lost context, does the whole test suite fail because of this?


I fit fails for such a long time, we should really invest some time to
look for the root cause of failure(s). If the appointed person does not
have time for this, he/she should be able to negotiate some time
allocated for the investigation. If you feel that the test failures are
not getting enough attention from us then you perhaps need to be more
proactive in the reporting.

I am quite OK if Peter Spacek receives some more time for investigation
of the root cause of the problem. What I am not OK with is having a
perfectly functional testsuite for otherwise working feature, that is
being deferred for months just because we do not approve of issue
workarounds.

Well if the test suite is perfectly functional then I don't see why it is deferred. If all the failures are caused by buggy zone signing then that is problem for us developers to fix, not yours.

I would also argue that since we tests for the feature are not passing in upstream QE environment then we can not consider the feature as "working".

(I do not want to play down the enormous amount work Petr and Martin put into implementing DNSSec zone signing, I am just stating how I see things)


We really should be fixing issues, not adding workarounds so that tests
pass.




--
Martin^3 Babinsky

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to