I tried to run smoke tests for 9.9, and they failed several times in a row
for me because of flaky logging tests. Since we also hit this failure
frequently in our CI, I spent some time looking at test execution.
I figured out that TestLogWatcher systematically fails when it is executed
after PackageManagerCLITest *in the same JVM*. By default, gradle is likely
to spawn several JVMs, and unless you are unlucky, these two tests will run
in different ones and you won't see any issue. (I can't say whether running
tests from the python script when validating the release has any impact on
test ordering and number of JVMs).
On branch_9x, I have a good repro rate (50% ?) locally by running the
following command. It's still not 100% because gradle can run tests in any
order. I'm not sure how to force that.
./gradlew test --tests PackageManagerCLITest --tests TestLogWatcher
-Ptests.jvms=1
(note that's necessary to force the number of forked JVMs to 1 so tests are
executed in a row in the same process).
Now, the root cause seems pretty obvious. Class PackageTool invokes
"Configurator.setRootLevel(Level.OFF)" and never reverts that. Any test
that later looks at was logged is very likely to fail.
Not sure what is the best path for a fix. Shutting down all logging (even
in a CLI tool) seems to be a bad practice to me...
Le mar. 15 juil. 2025 à 23:53, Houston Putman <hous...@apache.org> a écrit :
> Hey everyone,
>
> Update on the 9.9 release.
>
> There are two issues that popped up after fixing the
> ParallelHttpShardHandler bug:
>
> - https://github.com/apache/solr/pull/3428 (9.x specific, unrelated to
> the bug, but found while beasting tests)
> - https://github.com/apache/solr/pull/3429 (Fixes bug introduced into
> HttpShardHandler)
>
> Both should be good to go after quick reviews.
>
> The other big issue we are facing is regarding the smoke tester and
> buildAndPushRelease scripts. Both of these have started having issues
> regarding running tests that watch logs. (Like TestLogWatcher and
> RankFieldTest). Both of these watch for certain log events, and both of
> them fail when running the tests via the python scripts. We can see this
> has been happening since May 9th on the Apache Jenkins instances, but there
> is nothing introduced in Solr at the time that would explain it.
> https://ci-builds.apache.org/job/Solr/job/Solr-Smoketest-9.x/
>
> I am having a very hard time replicating this (outside of doing the full
> release process), but I'll hopefully have it solved by the time those 2 PRs
> are approved, merged, and backported.
>
> - Houston
>
> On Mon, Jul 7, 2025 at 10:14 AM Houston Putman <hous...@apache.org> wrote:
>
> > Fair enough, I was waiting to cut the branch since we weren't ready to do
> > the release yet. But I'll start that process now.
> >
> > - Houston
> >
> > On Mon, Jul 7, 2025 at 6:39 AM David Smiley <dsmi...@apache.org> wrote:
> >
> >> I noticed recently there seems to be no release branch separation, which
> >> is
> >> designed to bring about more stability.
> >>
> >> I've been waiting on merging
> >> https://issues.apache.org/jira/browse/SOLR-5707
> >> (a 4 digit JIRA from 2014), "Lucene Expressions for Solr" because the
> >> Lucene 9.12.2 bug fix includes a bug fix I worked on that significantly
> >> improves the usefulness of SOLR-5707. Notwithstanding a couple ignored
> >> tests (waiting for 9.12.2), that PR is ready for review. Even has a new
> >> ref guide page just for it.
> >>
> >>
> >> On Mon, Jul 7, 2025 at 12:07 AM Houston Putman <hous...@apache.org>
> >> wrote:
> >>
> >> > This is a big blocker: https://github.com/apache/solr/pull/3398
> >> >
> >> > And it needs some more eyes, but hopefully we can finish it out early
> >> this
> >> > week.
> >> >
> >> > We also need to actually do the lucene upgrade which I can do tomorrow
> >> > morning.
> >> >
> >> > After those two are done, I'll create an RC.
> >> >
> >> > - Houston
> >> >
> >> > On Thu, Jul 3, 2025 at 4:26 PM Anshum Gupta <ans...@apache.org>
> wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > Now that the Lucene release is done, is there something that is
> >> stopping
> >> > us
> >> > > from moving forward with this release? If not, let's build an RC
> early
> >> > next
> >> > > week.
> >> > >
> >> > > -Anshum
> >> > >
> >> > > On Wed, Jun 11, 2025 at 4:12 PM David Smiley <dsmi...@apache.org>
> >> wrote:
> >> > >
> >> > > > Yes, I _just_ back ported a Lucene fix to 9.12.2 that would make
> >> > > > finishing/committing
> >> https://issues.apache.org/jira/browse/SOLR-5707
> >> > > > (using
> >> > > > Lucene Expressions module in Solr) way more useful.
> >> > > >
> >> > > > On Wed, Jun 11, 2025 at 5:54 PM Houston Putman <
> hous...@apache.org>
> >> > > wrote:
> >> > > >
> >> > > > > We might also want to wait for the next Lucene 9.12.2 release,
> >> which
> >> > > > should
> >> > > > > hopefully happen soon?
> >> > > > >
> >> > > > > - Houston
> >> > > > >
> >> > > > > On Wed, Jun 11, 2025 at 8:51 AM Bruno Roustant <
> >> > > bruno.roust...@gmail.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1 Thanks Houston!
> >> > > > > >
> >> > > > > > Le mer. 11 juin 2025 à 00:29, Jan Høydahl <
> >> jan....@cominvent.com>
> >> > a
> >> > > > > écrit
> >> > > > > > :
> >> > > > > >
> >> > > > > > > Great plan. +1
> >> > > > > > >
> >> > > > > > > > 9. juni 2025 kl. 23:14 skrev Houston Putman <
> >> > hous...@apache.org
> >> > > >:
> >> > > > > > > >
> >> > > > > > > > Hey everyone,
> >> > > > > > > >
> >> > > > > > > > It's been a while since the last minor release and the
> >> > changelog
> >> > > > > looks
> >> > > > > > > > pretty good to get the next one in.
> >> > > > > > > >
> >> > > > > > > > I volunteer to do it probably sometime next week unless
> >> anyone
> >> > > > > objects.
> >> > > > > > > >
> >> > > > > > > > - Houston
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > >
> ---------------------------------------------------------------------
> >> > > > > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> >> > > > > > > For additional commands, e-mail: dev-h...@solr.apache.org
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
