On 06.05.2016 16:38, Oleg Fayans wrote:

On 05/06/2016 03:25 PM, Petr Spacek wrote:
On 6.5.2016 15:03, Oleg Fayans wrote:

On 05/06/2016 12:08 PM, Martin Babinsky wrote:
On 05/06/2016 11:14 AM, Oleg Fayans wrote:

On 05/06/2016 09:48 AM, Martin Basti wrote:

On 06.05.2016 09:36, Oleg Fayans wrote:
Tests are finally stable:

============================= test session starts
==============================
platform linux2 -- Python 2.7.11 -- py-1.4.30 -- pytest-2.7.3
rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile: pytest.ini
plugins: multihost, sourceorder
collected 8 items

test_integration/test_dnssec.py ........

========================= 8 passed in 5561.48 seconds
==========================





PATCH 38 LGTM

PATCH 37 IIRC I refused to accept workaround for this issue when you
send this (almost the same) patch for first time, are you sure that we
want to hide real issues in tests, to just have green color there?

The underlying issue is 7 months old. Latest update in the issue from
Peter Spacek is: "I do not have time to investigate this issue now",
which means, that it will stay there for unpredictable amount of time
more. If we want to have a "green" jenkins that actually tests existing
features, we have to accept workarounds for such long-term issues

Martin
I have never been a big fan of "having a green jenkins whatever it
takes" but I understand that there are all kinds of pressure on your
team to deliver 100% stable test results.

If the test fails, let it fail or, even better, use 'xfail' markers so
that we know that this test fails and we should investigate.
Then all 8 existing cases would have to be marked as xfailed.
That is perfectly fine - the test simply found a bug and we have to fix it.
There is no point in having "green" tests just to have them.

Let me clarify my comment in the ticket that this is not a test blocker:
Red Hat Bugzilla is using this definition of TestBlocker:
        A test blocker is a bug that prevents at least one test (test case) from
being executed.

As far as I can tell this issue does not block anything. The tests execute and
correctly detect a bug.

It would be a TestBlocker if it was e.g. a bug in installer which prevented
the install and thus prevented some other test cases from being executed.

AFAIK this is not the case. Or am I wrong?

I fit fails for such a long time, we should really invest some time to
look for the root cause of failure(s). If the appointed person does not
have time for this, he/she should be able to negotiate some time
allocated for the investigation. If you feel that the test failures are
not getting enough attention from us then you perhaps need to be more
proactive in the reporting.
I am quite OK if Peter Spacek receives some more time for investigation
of the root cause of the problem. What I am not OK with is having a
perfectly functional testsuite for otherwise working feature, that is
being deferred for months just because we do not approve of issue
workarounds.
Sorry, I do not understand this. What do you mean?
What I mean is:
Do we support adding dnssec-enabled zones?
   - Yes
Does a dnssec-enabled zone display signature?
   - Yes, but after restarting of named-pkcs11.
Should this feature in it's current state be covered with automated tests
   - YES, definitely!
But I understand that most of the team have the opposite opinion on the
subject

We could probably add a separate test that checks exactly this bug: add
a dnssec-enabled zone and query it WITHOUT restarting named-pkcs11. And
mark it as xfail. This way we hit 2 goals simultaneously: test the
feature itself and have a constant reminder for you guys, that we still
have this problem that needs your attention.


Yes why not, we can duplicate code for the every issue.
We will have nice statics on github :)


We really should be fixing issues, not adding workarounds so that tests
pass.
+1, it is just a matter of priority


--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to