pratapaditya04 commented on code in PR #4206:
URL: https://github.com/apache/gobblin/pull/4206#discussion_r3676561205


##########
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSJobObservabilityEvent.avsc:
##########
@@ -264,6 +264,18 @@
                 "type": ["null","string"],
                 "doc": "Whether the dataset passed the overall data quality 
check",
                 "default": null
+              },
+              {
+                "name": "snapshotsCommitted",
+                "type": "long",
+                "doc": "Number of table snapshots committed for the dataset in 
this run (e.g. Iceberg snapshot-replication commits), -1 if unsupported/unknown 
by the writer",
+                "default": -1
+              },
+              {
+                "name": "partitionsCommitted",
+                "type": "long",
+                "doc": "Number of partitions committed for the dataset in this 
run, -1 if unsupported/unknown by the writer",

Review Comment:
   Done — `partitionsCommitted` is now `partitionCommitted` of type 
`["null","string"]` with `default: null` (a single partition value, not a 
count).



##########
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSJobObservabilityEvent.avsc:
##########
@@ -264,6 +264,18 @@
                 "type": ["null","string"],
                 "doc": "Whether the dataset passed the overall data quality 
check",
                 "default": null
+              },
+              {
+                "name": "snapshotsCommitted",
+                "type": "long",
+                "doc": "Number of table snapshots committed for the dataset in 
this run (e.g. Iceberg snapshot-replication commits), -1 if unsupported/unknown 
by the writer",
+                "default": -1
+              },
+              {
+                "name": "partitionsCommitted",
+                "type": "long",
+                "doc": "Number of partitions committed for the dataset in this 
run, -1 if unsupported/unknown by the writer",

Review Comment:
   Renamed to `partitionCommitted` (singular). Threaded through 
`DatasetTaskSummary` (`String partitionCommitted`), `JobStatus`, 
`JobStatusRetriever`, and `JobStatistics.pdl` (`optional string`).



##########
gobblin-runtime/src/main/java/org/apache/gobblin/service/monitoring/JobStatusRetriever.java:
##########
@@ -60,6 +60,15 @@ public abstract class JobStatusRetriever implements 
LatestFlowExecutionIdTracker
   public static final String EVENT_NAME_FIELD = "eventName";
   public static final String NA_KEY = "NA";
 
+  // DDM (and other file/blob movement) writers report per-dataset copy 
metrics on the JobSummary
+  // GTE via these flat metadata keys; the job-status monitor persists them 
into the job state, and
+  // they are surfaced here on the JobStatistics REST surface. -1 = 
absent/unsupported.
+  public static final String BYTES_WRITTEN_FIELD = "ddm.bytesCopied";
+  public static final String RECORDS_WRITTEN_FIELD = "ddm.rowsCopied";
+  public static final String FILES_COMMITTED_FIELD = "ddm.filesCommitted";
+  public static final String SNAPSHOTS_COMMITTED_FIELD = 
"ddm.snapshotsCommitted";
+  public static final String PARTITIONS_COMMITTED_FIELD = 
"ddm.partitionsCommitted";
+

Review Comment:
   Done — dropped the `ddm.` prefix; the keys are now `bytesCopied`, 
`rowsCopied`, `filesCommitted`, `snapshotsCommitted`, `partitionCommitted`, 
matching the bare camelCase convention used by the other job-state keys.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to