Some nice work! I have only used the `npc_taxies` workload so far. Glad to hear `geonames` also works!
I followed up a with OSB slack channel about your fixes, as that team would have the most expertise in evaluating your fixes, thanks for opening the issues and PR fixes…. Let’s give it a week to see if they weigh in, and what they say might then flow downstream to the PRs that you opened for Solr-orbit…. ;-). Then, if we don’t see any comments etc, I’ll move forward with merging your PRs in Solr-orbit. I agree one hundred percent we need a better workload that demonstrates the use of function queries etc. Many of the issues we find aren’t in simple straight up queries, but when caches and complex queries are involved. I have been meaning to see if any of the workloads in https://github.com/opensearch-project/opensearch-benchmark-workloads would be a good inspiration…. Let’s start another thread about that! Eric > On Jul 25, 2026, at 12:56 PM, Serhiy Bzhezytskyy > <[email protected]> wrote: > > Hi Eric and all, > > Eric - I took the solr-orbit and ran it locally. Below is what came out, > including a correction to a number of > > my own. > > Short version: I have Solr-10.4-vs-Lucene-11 benchmark data now, and it > shows no measurable throughput difference. I could not get there without > fixing three defects(or i did something wrong) in solr-orbit's "aggregate" > command; those are now three issues and three pull requests, and the same > three upstream in OpenSearch Benchmark. > > What I ran: > > Three configurations, full geonames corpus (11M docs), 6g heap, 19 > operations, five clean runs each - > > fifteen runs, about 38 minutes per run: > > - official solr:10.0.0 from Docker Hub, as a control > > - a locally built Solr on Lucene 10.4 (current main) > > - a locally built Solr on Lucene 11-SNAPSHOT (my port branch) > > > The result: > > Aggregated over five runs per configuration, with relative standard > deviation per metric: > > operation thr 10.4 RSD% thr L11 RSD% delta% > > -------------------------- -------- ----- -------- ----- ------ > > index-append (unthrottled) 62954.86 9.32 63171.35 7.40 +0.34 > > check-cluster-health 71.93 26.64 81.39 23.07 +13.15 > > refresh-after-index 5.04 79.24 2.34 37.70 -53.55 > > 14 throttled query ops (varies) <=1.6 (varies) <=0.3 <=0.6 > > optimize none NA none NA n/a > > Not one of the nineteen operations shows a throughput delta that clears its > own noise. > > So: no evidence of a Lucene 11 throughput regression in this workload. > > Two honest caveats on reading that table. > > refresh-after-index has an RSD of 79 percent, so its -53 percent is not > information - that operation should be excluded from comparisons at this > sample size rather than reported with a delta beside it. > > The fourteen throttled query operations have tiny RSDs because their > throughput is pinned to their target rate by construction; those numbers > describe the scheduler, not Solr, and it's a mistake to read them as > stability. > > The gap, and what I did about it > > To get the table above I needed "solr-orbit aggregate", which is what > computes the weighted means and the RSDs. > > It could not run. Three separate defects, each hidden behind the previous > one: > > 1. Any campaign containing an operation that reports null metrics crashes > during metric reduction. min() and max() are reduced with value.get(field, > 0), and that default only applies when the key is absent, not when it's > present with a null value. The geonames workload's optimize operation > reports error_rate 1.0 and fully null throughput on every run, so every > geonames aggregation hits this. There is a second, independent occurrence > in calculate_rsd that only shows up on real data. > > 2. With that fixed, aggregate still fails every time, for any input: > update_config_object reads test_run.cluster_config_instance_params, and > TestRun defines cluster_config_params. The name being read occurs nowhere > else in the tree - it looks like a half-applied rename (2c85d505, and > e10342d1 later corrected the config key on that same line but left the > attribute). This one does not depend on the data, so as far as I can tell > aggregate cannot succeed for anyone on current main. That is the claim I > would most like checked, and the PR carries a test that is red without the > fix so it can be checked on the diff. > > 3. aggregate has no --workload-path, though run does. A workload developed > locally can be benchmarked and then not aggregated. That one is a parity > gap rather than a bug. > > > The reason the test suite doesn't catch the second one is worth flagging > for the project generally: every test in tests/aggregator_test.py passes a > bare Mock as the test run, and a Mock returns a happy new Mock for any > attribute name asked of it - so the wrong name reads fine. The suite is > green over a command that cannot run. > > All three are reproduced and fixed, one commit and one PR each, with tests, > and aggregate now completes end-to-end on the real fifteen-run data. > > In apache/solr-orbit: > > defect issue PR > > -------------------------- ----- --- > > null metrics crash 55 58 > > wrong TestRun attribute 56 59 > > no --workload-path 57 60 > > > https://github.com/apache/solr-orbit/pull/58 > <https://github.com/apache/solr-orbit/pull/58> > > https://github.com/apache/solr-orbit/pull/59 > <https://github.com/apache/solr-orbit/pull/59> > > https://github.com/apache/solr-orbit/pull/60 > <https://github.com/apache/solr-orbit/pull/60> > > The same three defects are present verbatim in OpenSearch Benchmark, which > solr-orbit is a port of - the two aggregator.py files are byte-identical > apart from the import module name - so they are inherited rather than > introduced by the port. Reported and fixed there too, each > cross-referencing its solr-orbit counterpart: issues 1093, 1094, 1095 and > PRs 1096, 1097, 1098 in opensearch-project/opensearch-benchmark. > > On the richer workloads you mentioned: agreed that's the bigger gap - there > are only three workloads today, and none of them targets the areas a Lucene > major touches hardest. I have opinions there (sort and DocValues, faceting, > vectors) but that's a separate thread; getting aggregate working seemed > like the prerequisite, since without it a multi-run campaign produces files > nobody can average. > > > Thanks, > > Serhiy > > On Wed, 22 Jul 2026 at 18:17, Eric Pugh <[email protected] > <mailto:[email protected]>> > wrote: > > > I hope that we have a shorter gap between the release of Lucene 11 and the > > first release of Solr 11 this time around. Especially as it feels like the > > pace of Lucene major version bumps is moving closer to a 12 to 18 month > > cycle… So this is exciting. > > > > I was going to ask if we wanted to revisit the past pattern of bumping > > Solr major version when we move to major version bumps of Lucene, but I > > guess because of the Java 25 requirement that is moot for this cycle. > > Since as long as we bump major versions of Java as a base requirement then > > we need to bump major version of Solr… > > > > One area that would help us would be to get the rest of the tooling in > > Solr-orbit over the finish line so we can build more confidence in the > > performance characteristics of Solr on newer Lucene versions. It would be > > great to start getting some comparative runs of Solr 10/Java 21 and Solr > > 11/Java 25….. Which needs more sophisticated Solr Orbit workloads. > > > > https://github.com/apache/solr-orbit <https://github.com/apache/solr-orbit> > > https://github.com/apache/solr-orbit-workloads > > <https://github.com/apache/solr-orbit-workloads> > > > > > > > > > > Eric > > > > > On Jul 22, 2026, at 10:58 AM, Serhiy Bzhezytskyy < > > [email protected]> wrote: > > > > > > Thanks David! > > > > > > Created SOLR-18313 (https://issues.apache.org/jira/browse/SOLR-18313 > > > <https://issues.apache.org/jira/browse/SOLR-18313> < > > https://issues.apache.org/jira/browse/SOLR-18313 > > <https://issues.apache.org/jira/browse/SOLR-18313>>) and > > > opened https://github.com/apache/solr/pull/4651 > > > <https://github.com/apache/solr/pull/4651> < > > https://github.com/apache/solr/pull/4651 > > <https://github.com/apache/solr/pull/4651>> for the two > > > forward-compatible changes — the SortField missing-value constructor and > > > Operations.concatenate(List). No behavior change; gradlew check is green > > on > > > current main. > > > > > > Agreed on holding the Java 25 bump until it serves a purpose — no point > > > paying the backport cost early. For what it's worth, there's no > > branch_11x > > > cut on the Lucene side yet and 10.6 is still in progress, so 11 doesn't > > > look imminent regardless. > > > > > > > > > On Wed, 22 Jul 2026 at 16:13, David Smiley <[email protected] > > > <mailto:[email protected]>> wrote: > > > > > > > Thanks for looking ahead at what's coming! > > > > > > > > On Wed, Jul 22, 2026 at 5:28 AM Serhiy Bzhezytskyy < > > > > [email protected] <mailto:[email protected]>> wrote: > > > > > > > > > Decision 1 — the Java 25 baseline. > > > > > > > > > > > > Yup; Solr 11 will be compelled to adopt Lucene's Java baseline. When we > > > > upgrade, anyway. This would be a good preparatory first step for Solr > > > > *prior* to adopting Lucene 11. > > > > > > > > However, I prefer we delay this until Lucene 11 is actually released. > > > > Divergences across Solr branches contribute to backport pain without > > *yet* > > > > serving a purpose. > > > > > > > > > > > > > Decision 2 — compound-file config. > > > > > > > > > > Lucene 11 removed the merge-policy maxMergeAtOnce and noCFSRatio > > > > settings, > > > > > and moved compound-file selection onto the codec's CompoundFormat. > > > > > > > > > > Two user-facing consequences: > > > > > > > > > > (a) a solrconfig.xml still setting those merge-policy options would > > fail > > > > > core init — in my branch I made Solr ignore-with-warning instead, so > > > > > existing configs keep loading. Is warn-and-ignore the behavior we > > want, > > > > or > > > > > should it hard-fail with a migration message? > > > > > > > > > > (b) useCompoundFile no longer reaches merged segments on its own > > (only > > > > > flushed ones), so I bridged it to the codec and added a > > > > maxCFSSegmentSizeMB > > > > > option as the noCFSRatio replacement. Sanity-check welcome on that > > > > > approach. > > > > > > > > > > > > > Sounds reasonable but I haven't looked. > > > > ... > > > > > > > > > > > > > Most of it is 11-only by design, but a couple compile against current > > > > main > > > > > today: > > > > > > > > > > change 10.4 10.5 10.6 > > > > > 11-only > > > > > > > > > > --------------------------------------------------------- ---- ---- > > > > > ------- > > > > > > > > > > SortField.setMissingValue -> constructor arg yes yes yes no > > > > > > > > > > Operations.concatenate(a,b) -> concatenate(List) yes yes yes no > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > So the first two rows are adoptable now — I verified they build > > against > > > > > current Solr main (on Lucene 10.4). > > > > > > > > > > > > > Fine idea as a preparatory step. > > > > > > > > > > > > > If there's interest I'm happy to split them into a small "prep for > > Lucene > > > > > 11" PR separate from the version bump. > > > > > > > > > > > > Please; thank you! > > > > > > > > ~ David > > > > > > > > Disclaimer > > > > The information contained in this communication from the sender is > > confidential. It is intended solely for use by the recipient and others > > authorized to receive it. If you are not the recipient, you are hereby > > notified that any disclosure, copying, distribution or taking action in > > relation of the contents of this information is strictly prohibited and may > > be unlawful. > > > > This email has been scanned for viruses and malware, and may have been > > automatically archived by Mimecast, a leader in email security and cyber > > resilience. Mimecast integrates email defenses with brand protection, > > security awareness training, web security, compliance and other essential > > capabilities. Mimecast helps protect large and small organizations from > > malicious activity, human error and technology failure; and to lead the > > movement toward building a more resilient world. To find out more, visit > > our website. > > Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
