serhiy-bzhezytskyy opened a new issue, #55:
URL: https://github.com/apache/solr-orbit/issues/55
### Describe the bug
`solr-orbit aggregate` crashes for any set of test runs in which an
operation has `null` values in its `min`/`max` metric fields:
```
[ERROR] ❌ Cannot aggregate. '<' not supported between instances of
'NoneType' and 'NoneType'.
```
`calculate_weighted_average` in `solrorbit/aggregator.py` reduces
`min`/`max` with `value.get(metric_field, 0)`. The `0` default only applies
when the key is **absent** — it does not help when the key is **present with
value `None`**, which is what the stored results contain. `min()` then compares
`None` to `None` and raises.
This is not an edge case in practice: the `optimize` operation in the
geonames workload reports `error_rate: 1.0` with all-null throughput on every
run I have, and `optimize` is part of the default workload — so **every**
aggregation of a geonames campaign fails. The `max` branch has the same
problem, and the percentile/median branch below it (`value * iterations` after
a `.get(..., 0)`) would raise on `None` for the same reason.
There is a second, independent occurrence in `calculate_rsd`, on the path
from `build_aggregated_results_dict` that passes the per-run mean values. I
only found that one by running on real data after unit tests for the first site
were already green.
Aggregating N runs is the documented way to get a trustworthy result out of
a noisy environment, so in practice this blocks multi-run campaigns on the
default workload.
### To reproduce
1. Run the geonames workload more than once (repeat with a different
`--user-tag`):
```
solr-orbit run --pipeline=docker --distribution-version=10.0.0 \
--workload-path=<path>/solr-orbit-workloads/geonames \
--cluster-config-params="heap_size:6g" -k
```
2. Aggregate the resulting test runs:
```
solr-orbit aggregate --test-runs=<id1>,<id2>,<id3>,<id4>,<id5>
```
3. It fails immediately — `❌ FAILURE (took 0 seconds)`, nothing written.
The stored results contain, for every run:
```json
{ "task": "optimize",
"error_rate": 1.0,
"throughput": { "min": null, "mean": null, "median": null, "max": null,
"unit": "ops/s" } }
```
Minimal reproduction against the real method, if it helps — the code under
test is not mocked, only the cfg/store `__init__` is bypassed:
```python
from solrorbit.aggregator import Aggregator
agg = Aggregator.__new__(Aggregator)
agg.test_runs = {"run1": None, "run2": None}
agg.accumulated_iterations = {"run1": {"optimize": 1}, "run2": {"optimize":
1}}
bad = {"min": None, "mean": None, "median": None, "max": None, "unit":
"ops/s"}
agg.calculate_weighted_average({"throughput": [bad, bad]}, "optimize") #
TypeError
```
A healthy operation with numeric values aggregates fine, which is what
points at `None` as the trigger rather than anything about the harness.
### Expected behavior
`aggregate` completes and produces aggregated results. For an operation with
no valid samples I would expect the aggregate to carry `null` through for that
metric — honest, it says "not measured" — rather than substituting `0`, which
would read as a real measurement of zero throughput.
Note that `aggregate_json_by_key` in the same file already skips nulls
deliberately (`next((obj for obj in json_elements if obj is not None), None)`),
so handling null metrics is an established pattern here;
`calculate_weighted_average` just doesn't do it.
### Host / Environment
- solr-orbit `edb3d03e` (main, up to date at time of report), installed from
source, Python 3.12
- Solr 10.0.0 (official Docker image) via `--pipeline=docker`
- Workload: geonames, full corpus, 6g heap, 5 runs per configuration
- macOS 15 (arm64), Docker Desktop
### Additional context
Found while running a 5-runs-per-configuration geonames campaign to compare
Solr on Lucene 10.4 against Lucene 11 — `aggregate` is what computes the
weighted means and the per-metric RSD that make a multi-run result readable, so
this was a hard stop.
`git log -L225,225:solrorbit/aggregator.py` traces these lines to the
OpenSearch Benchmark commits the port inherited (`c6ec0a11` "Add aggregate
command" and `2d14ed40` "Change min/max to overall_min/overall_max"), and the
same code is present upstream, so this looks inherited rather than introduced
by the Solr port. I couldn't find an existing report in either project.
Separately (can open a dedicated issue if that's useful): `optimize`
reporting `error_rate: 1.0` and null throughput on every run suggests the
force-merge step isn't being measured at all. It's equal on both sides of a
comparison so it doesn't skew `compare`, but it is the direct cause of this
crash. I haven't root-caused that one.
### Relevant log output
```shell
Traceback (most recent call last):
File ".../solrorbit/benchmark.py", line 1218, in dispatch_sub_command
aggregator_instance.aggregate()
File ".../solrorbit/aggregator.py", line 284, in aggregate
aggregated_results = self.build_aggregated_results()
File ".../solrorbit/aggregator.py", line 176, in build_aggregated_results
aggregated_results = self.build_aggregated_results_dict()
File ".../solrorbit/aggregator.py", line 129, in
build_aggregated_results_dict
aggregated_task_metrics = self.calculate_weighted_average(task_metrics,
task)
File ".../solrorbit/aggregator.py", line 225, in calculate_weighted_average
weighted_metrics[metric]['overall_min'] = min(value.get(metric_field, 0)
for value in values)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
