Rick Hillegas wrote:
On 3/8/11 4:03 PM, Mike Matrigali wrote:
Rick Hillegas wrote:
On 3/8/11 11:00 AM, Myrna van Lunteren wrote:
Hi,
It seems to me we have a relatively unstable test situation; it
appears to me there are many intermittent test failures...
There are a lot of open test issues; 69 per this query:
https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+derby+and+%22Bug+behavior+facts%22+%3D+%22Regression+Test+Failure%22+and+resolution+%3D+Unresolved&tempMax=1000
21 of these were opened since October 1, 2010:
https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+derby+and+%22Bug+behavior+facts%22+%3D+%22Regression+Test+Failure%22+and+resolution+%3D+Unresolved+and+createdDate++%3E%222010-10-01%22&tempMax=1000
Now that we're waiting with the release for a reply from legal, would
it make sense for the members of the community to dedicate some time
to sort through these issues and perhaps resolve some?
Myrna
+1
+1
I think it would be worthwhile to do a focused effort on the
outstanding intermittent errors. It seems like trunk has gotten much
worse in this
area than it has been. I often get 2-3 "known" issues when doing a
trunk run - this should not be the case as we go into a release.
I would be willing to devote time to this issue over the next few
weeks. I do think we need some sort of list of
issues to concentrate on, maybe making those on the list a blocker for
a release until someone does work to either fix or argue why they
should not be a blocker.
I would support a concerted bug fixing effort in 10.8.2, with a strong
emphasis on stabilizing our noisy tests. Perhaps someone will volunteer
to manage such a release and publish its criteria. Extra credit if we
wrap this up before everyone disappears for the summer.
I am particularly concerned about the statistics and interrupt based
errors we are seeing. These are the results of new features being
introduced and due to the intermittent nature it is hard to say if the
issue is the feature itself or just new testing showing up existing
issues.
I agree that there has been a lot of noise in these areas. The bugs I'm
aware of indicate that these features may not be delivering their
promised benefits in edge cases. So far I do not see any evidence that
these features have caused regressions, data corruptions, or wrong
results. My sense is that both of these features provide significant
improvements over Derby's behavior in previous releases--even with edge
case lapses. At this point I am in favor of releasing these features so
that customers can enjoy the incremental improvements and can give us
feedback on uncovered edge cases that we haven't found ourselves.
Thanks,
-Rick
I was wondering if we think that the "noisy" tests are just showing bugs
that we don't want to fix for the release, them maybe we should log or
upgrade the existing bugs and disable the tests so that individuals
don't have to reinvestigate the same issues over and over and over
again. The reason I bring it up now is that by making a release we are
going to again increase the number of times these errors show up again.
I know we have never had anything official, but it would be nice if we
could get to something like no nightly regression test errors in the
public reports across all platforms in the last 7 days.
As you pointed out I believe some of the problem is that we have added
testing that has uncovered bugs. Usually when we add test cases that
uncover bugs we log the JIRA and disable the test until the bug is
fixed. Maybe we should do so now also. I am fine leaving tests running
if the errors are hard to reproduce and someone is trying to gather more
information by enhancing the test. For instance DERBY-5018 is a good
example where debugging in continuing and the noise is helping to
understand the issue.
