EugeneYushin opened a new issue, #2665:
URL: https://github.com/apache/iceberg-go/issues/2665
### Feature Request / Improvement
## Summary
`metrics/otel.Reporter` records the commit `*.total` fields as monotonic
`Int64Counter`s. These fields are the table totals *after* a commit (taken
from
the snapshot summary `total-*` entries), not per-commit deltas. Summing them
over commits, or taking `rate()`/`increase()` over them, gives wrong numbers.
They should be gauges.
## Affected instruments
All are created in `metrics/otel/reporter.go` via `newCounter` and written
with `Add`:
- `iceberg.commit.records.total`
- `iceberg.commit.data_files.total`
- `iceberg.commit.delete_files.total`
- `iceberg.commit.file_size.total_bytes`
- `iceberg.commit.positional_deletes.total`
- `iceberg.commit.equality_deletes.total`
Their values come from `table/commit_metrics.go` (`buildCommitReport`), which
reads `total-records`, `total-data-files`, etc. from the new snapshot's
summary.
`updateSnapshotSummaries` rolls each of those forward as
previous total + added - removed, so each one describes the whole table at
that snapshot.
## Reproduction
Two appends to the same table, each adding 150 records. Snapshot summaries
give
`total-records` = 100, then 250.
```go
rep.Report(ctx, metrics.CommitReport{TableName: "db.t", Operation: "append",
Metrics: metrics.CommitMetricsResult{AddedRecords: c(150), TotalRecords:
c(100)}})
rep.Report(ctx, metrics.CommitReport{TableName: "db.t", Operation: "append",
Metrics: metrics.CommitMetricsResult{AddedRecords: c(150), TotalRecords:
c(250)}})
```
- Actual: `iceberg.commit.records.total` = **350** (a cumulative sum)
- Expected: **250** (the table's current total)
`iceberg.commit.records.added` = 300 is correct, because it is a delta.
With DELTA temporality (e.g.
`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta`)
a backend that sums the points shows the same inflated number.
## Why this is a divergence from Java
The package doc says names, units and attributes mirror Java's
`OtelMetricsReporter` (apache/iceberg#16250). That PR (still open) records
exactly
these six metrics as `LongGauge` and everything else as counters/histograms.
The
instrument *names* and units were aligned in #1767, but the instrument *type*
was not: #1767 describes every non-duration field as a monotonic counter.
## Out of scope
Out-of-order reports overwriting a newer gauge value. `CommitReport` carries
a
`SequenceNumber`, so comparing it before recording could be a follow-up if
needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]