EugeneYushin opened a new issue, #2665:
URL: https://github.com/apache/iceberg-go/issues/2665

   ### Feature Request / Improvement
   
   ## Summary
   
   `metrics/otel.Reporter` records the commit `*.total` fields as monotonic
   `Int64Counter`s. These fields are the table totals *after* a commit (taken 
from
   the snapshot summary `total-*` entries), not per-commit deltas. Summing them
   over commits, or taking `rate()`/`increase()` over them, gives wrong numbers.
   They should be gauges.
   
   ## Affected instruments
   
   All are created in `metrics/otel/reporter.go` via `newCounter` and written 
with `Add`:
   
   - `iceberg.commit.records.total`
   - `iceberg.commit.data_files.total`
   - `iceberg.commit.delete_files.total`
   - `iceberg.commit.file_size.total_bytes`
   - `iceberg.commit.positional_deletes.total`
   - `iceberg.commit.equality_deletes.total`
   
   Their values come from `table/commit_metrics.go` (`buildCommitReport`), which
   reads `total-records`, `total-data-files`, etc. from the new snapshot's 
summary.
   `updateSnapshotSummaries` rolls each of those forward as
   previous total + added - removed, so each one describes the whole table at 
that snapshot.
   
   ## Reproduction
   
   Two appends to the same table, each adding 150 records. Snapshot summaries 
give
   `total-records` = 100, then 250.
   
   ```go
   rep.Report(ctx, metrics.CommitReport{TableName: "db.t", Operation: "append",
       Metrics: metrics.CommitMetricsResult{AddedRecords: c(150), TotalRecords: 
c(100)}})
   rep.Report(ctx, metrics.CommitReport{TableName: "db.t", Operation: "append",
       Metrics: metrics.CommitMetricsResult{AddedRecords: c(150), TotalRecords: 
c(250)}})
   ```
   
   - Actual: `iceberg.commit.records.total` = **350** (a cumulative sum)
   - Expected: **250** (the table's current total)
   
   `iceberg.commit.records.added` = 300 is correct, because it is a delta.
   
   With DELTA temporality (e.g. 
`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta`)
   a backend that sums the points shows the same inflated number.
   
   ## Why this is a divergence from Java
   
   The package doc says names, units and attributes mirror Java's
   `OtelMetricsReporter` (apache/iceberg#16250). That PR (still open) records 
exactly
   these six metrics as `LongGauge` and everything else as counters/histograms. 
The
   instrument *names* and units were aligned in #1767, but the instrument *type*
   was not: #1767 describes every non-duration field as a monotonic counter.
   
   ## Out of scope
   
   Out-of-order reports overwriting a newer gauge value. `CommitReport` carries 
a
   `SequenceNumber`, so comparing it before recording could be a follow-up if 
needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to