It appears that Knut is working on fixing the issues which he logged
against the JDK 7 test runs. I expect we'll see a sharp drop in that noise.
I have created DERBY-5122 to help us keep track of tests which we
disable. The idea is to add a comment including the tag "DERBY-5122"
when you disable a test. That way we can find these problems again when
we have cycles to fix them properly. I'm ok with disabling tests which
fail consistently. Hopefully people will only disable tests on the
platforms where they actually fail.
I agree with Myrna's list of problem tests below. It would be great to
give all of these tests some attention in 10.8.2. Note that some of
these problems are heisenbugs. In my experience, it is hard to commit to
fixing heisenbugs in a bounded timeframe.
I think that the laudable goal of error-free tests for 7 consecutive
nights may prove to be overly aggressive for the 10.8.2 release manager.
I couldn't testify that we have ever seen more than a couple consecutive
nights of clean runs.
Thanks,
-Rick
On 3/9/11 7:16 PM, Myrna van Lunteren wrote:
On Wed, Mar 9, 2011 at 1:47 PM, Mike Matrigali<[email protected]> wrote:
Rick Hillegas wrote:
On 3/8/11 4:03 PM, Mike Matrigali wrote:
Rick Hillegas wrote:
On 3/8/11 11:00 AM, Myrna van Lunteren wrote:
Hi,
It seems to me we have a relatively unstable test situation; it
appears to me there are many intermittent test failures...
There are a lot of open test issues; 69 per this query:
https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+derby+and+%22Bug+behavior+facts%22+%3D+%22Regression+Test+Failure%22+and+resolution+%3D+Unresolved&tempMax=1000
21 of these were opened since October 1, 2010:
https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+derby+and+%22Bug+behavior+facts%22+%3D+%22Regression+Test+Failure%22+and+resolution+%3D+Unresolved+and+createdDate++%3E%222010-10-01%22&tempMax=1000
Now that we're waiting with the release for a reply from legal, would
it make sense for the members of the community to dedicate some time
to sort through these issues and perhaps resolve some?
Myrna
+1
+1
I think it would be worthwhile to do a focused effort on the outstanding
intermittent errors. It seems like trunk has gotten much worse in this
area than it has been. I often get 2-3 "known" issues when doing a trunk
run - this should not be the case as we go into a release.
I would be willing to devote time to this issue over the next few weeks.
I do think we need some sort of list of
issues to concentrate on, maybe making those on the list a blocker for a
release until someone does work to either fix or argue why they should not
be a blocker.
I would support a concerted bug fixing effort in 10.8.2, with a strong
emphasis on stabilizing our noisy tests. Perhaps someone will volunteer to
manage such a release and publish its criteria. Extra credit if we wrap this
up before everyone disappears for the summer.
I am particularly concerned about the statistics and interrupt based
errors we are seeing. These are the results of new features being
introduced and due to the intermittent nature it is hard to say if the issue
is the feature itself or just new testing showing up existing issues.
I agree that there has been a lot of noise in these areas. The bugs I'm
aware of indicate that these features may not be delivering their promised
benefits in edge cases. So far I do not see any evidence that these features
have caused regressions, data corruptions, or wrong results. My sense is
that both of these features provide significant improvements over Derby's
behavior in previous releases--even with edge case lapses. At this point I
am in favor of releasing these features so that customers can enjoy the
incremental improvements and can give us feedback on uncovered edge cases
that we haven't found ourselves.
Thanks,
-Rick
I was wondering if we think that the "noisy" tests are just showing bugs
that we don't want to fix for the release, them maybe we should log or
upgrade the existing bugs and disable the tests so that individuals don't
have to reinvestigate the same issues over and over and over again. The
reason I bring it up now is that by making a release we are going to again
increase the number of times these errors show up again.
I know we have never had anything official, but it would be nice if we
could get to something like no nightly regression test errors in the public
reports across all platforms in the last 7 days.
As you pointed out I believe some of the problem is that we have added
testing that has uncovered bugs. Usually when we add test cases that
uncover bugs we log the JIRA and disable the test until the bug is fixed.
Maybe we should do so now also. I am fine leaving tests running if the
errors are hard to reproduce and someone is trying to gather more
information by enhancing the test. For instance DERBY-5018 is a good
example where debugging in continuing and the noise is helping to understand
the issue.
I think obviously the first wish is to have the bugs causing test
failures get fixed.
But I'm ok with disabling a test case (preferably not an entire test)
until a bug is fixed, although I think it's always a little risky
because we are not doing some specific testing, and thus, may run into
a bug in that area. But having a test fail always also has that same
effect, and having tests fail for known reasons doesn't help.
A passing test doesn't mean there are no bugs...
However, there are currently a number of highly irritating bugs for
which it would be great to get a resolution.
I've pulled a list of the bugs that have shown up in the nightlies (at
sun and ibm) in trunk over the last two weeks, added with a couple of
my favorites.
If anyone has spare cycles and would be willing to look either fixing
these, or working around the problem situations...
DERBY-5119 - AccessTest.testQualifiers
DERBY-5109 - InterruptResilienceTest.testRAFWriteInterrupted
DERBY-5108 - AutomaticIndexStatisticsTest.testShutdownWhileScanningThenDelete
DERBY-5045 - UpdateStatisticsTest.testUpdateStatistics
DERBY-5081 - InterruptResilienceTest.testRAFReadWriteMultipleThreads
DERBY-5003 - ReplicationRun_Local_3_p5.testReplication_Local_3_p5_DERBY_3878
DERBY-5110 - testInvalidLDAPServerConnectionError.
DERBY-5026 - testStatsCreatedOnGrowthThenDeleteDb (or, fixed now?)
DERBY-5105 - NoSuchMethodError in upgrade tests
DERBY-4922 - DboPowersTest.testReEncrypt
DERBY-4905/4916 - tearDown/removeDir unable to delete files (on
windows), often seen in upgrade tests
DERBY-4762 - derbynet.NetworkServerControlClientCommandTest.testPing
DERBY-3993 - unit/T_RawStoreFactory.junit (perhaps the check for the
observers can be removed).
If anyone has any other favorites, please let me know...
If anyone is volunteering to work on any of these, let me know...
Myrna
