phet commented on code in PR #3667:
URL: https://github.com/apache/gobblin/pull/3667#discussion_r1163069107
##########
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSObservabilityEventExperimental.avsc:
##########
@@ -208,12 +208,17 @@
{
"name": "bytesWritten",
"type": "long",
- "doc": "Number of bytes written for the dataset"
+ "doc": "Number of bytes written for the dataset, can be -1 if
unsupported by the writer (e.g. jdbc writer)"
},
{
- "name": "recordsWritten",
+ "name": "entitiesWritten",
"type": "long",
- "doc": "Number of records written for the dataset"
+ "doc": "Number of entities written for the dataset by the
Gobblin writer"
Review Comment:
"(e.g. files or records)"
##########
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSObservabilityEventExperimental.avsc:
##########
@@ -208,12 +208,17 @@
{
"name": "bytesWritten",
"type": "long",
- "doc": "Number of bytes written for the dataset"
+ "doc": "Number of bytes written for the dataset, can be -1 if
unsupported by the writer (e.g. jdbc writer)"
},
{
- "name": "recordsWritten",
+ "name": "entitiesWritten",
"type": "long",
- "doc": "Number of records written for the dataset"
+ "doc": "Number of entities written for the dataset by the
Gobblin writer"
+ },
+ {
+ "name": "datasetCommitSucceeded",
Review Comment:
nit: do we really want to repeat 'dataset', as in
`datasetsWritten[*].datasetCommitSucceeded` (e.g. we don't name it
`datasetBytesWritten`)? maybe `wasCommitted`(?)
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/DatasetTaskSummary.java:
##########
@@ -17,30 +17,26 @@
package org.apache.gobblin.runtime;
+import lombok.Data;
+
+import org.apache.gobblin.metrics.DatasetMetric;
+
+
/**
* A class returned by {@link org.apache.gobblin.runtime.SafeDatasetCommit} to
provide metrics for the dataset
* that can be reported as a single event in the commit phase.
*/
+@Data
public class DatasetTaskSummary {
private final String datasetUrn;
private final long recordsWritten;
private final long bytesWritten;
+ private final boolean datasetCommitSucceeded;
- public DatasetTaskSummary(String datasetUrn, long recordsWritten, long
bytesWritten) {
- this.datasetUrn = datasetUrn;
- this.recordsWritten = recordsWritten;
- this.bytesWritten = bytesWritten;
- }
-
- public String getDatasetUrn() {
- return datasetUrn;
- }
-
- public long getRecordsWritten() {
- return recordsWritten;
- }
-
- public long getBytesWritten() {
- return bytesWritten;
+ /**
+ * Convert a {@link DatasetTaskSummary} to a {@link DatasetMetric}.
+ */
+ public static DatasetMetric toDatasetMetric(DatasetTaskSummary
datasetTaskSummary) {
Review Comment:
NBD, but why not an instance method? (invocation syntax, inside `.map()`
would remain unchanged)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
