janhoy opened a new pull request, #22:
URL: https://github.com/apache/solr-orbit-workloads/pull/22
### Description
`common_operations/optimize.json` always emitted `"max-segments": 1`, so
every workload including it force-merged the whole index down to a single
segment. Modern Solr honours the merge policy's `maxSegmentSizeMB` when
`maxSegments` is not given, and that is the better default: a single enormous
segment is a shape no production index would ever hold, and reaching it costs a
full rewrite of the index.
On the full `nyc_taxis` corpus that rewrite is the longest single operation
in a benchmark run by a wide margin — it sits between indexing and the query
tasks, so an hour and a half of work has to survive it before any query is
measured. It is also unrepresentative of what the query tasks would meet on a
real cluster.
Passing `max_segments` still forces the old behaviour for a workload that
genuinely wants one segment:
```
solr-orbit run --workload-params max_segments:1 ...
```
This pairs with the matching change in Solr Orbit's `SolrOptimize` runner,
which previously defaulted `max-segments` to `1` itself. Either default alone
is not enough: with the template passing `1` explicitly, the runner never sees
an unset value.
### Also: dropping the dead `timeout` parameter
`optimize_timeout` is read by nothing. The runner looks only at
`max-segments`, and pysolr's `optimize()` takes no timeout argument — the limit
that actually applies is the client's HTTP timeout from `--client-options`.
Leaving a parameter that silently does nothing is a trap: when `optimize` times
out, this is exactly the knob someone reaches for, and it would have no effect.
### Testing
Rendered the template through Jinja2 and parsed the result as JSON for the
unset, `max_segments: 1` and `max_segments: 5` cases, confirming the
conditional comma keeps it valid JSON in each:
```
{} -> {"operation-type": "optimize"}
{"max_segments": 1} -> {"operation-type": "optimize", "max-segments": 1}
{"max_segments": 5} -> {"operation-type": "optimize", "max-segments": 5}
```
Found while running the first full-corpus `nyc_taxis` benchmark on a
two-node EC2 cluster, where `optimize` on 165,346,692 documents was the
operation that exposed it.
🤖 Assisted by Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
