phet commented on code in PR #3667:
URL: https://github.com/apache/gobblin/pull/3667#discussion_r1160861533
##########
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSObservabilityEventExperimental.avsc:
##########
@@ -188,6 +188,38 @@
}
}
]
- }]
+ },
+ {
+ "name": "datasetsWritten",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": {
+ "type": "record",
+ "name": "DatasetMetric",
+ "doc": "DatasetMetric contains bytes and records written by
Gobblin writers for the dataset URN.",
+ "fields": [
+ {
+ "name": "datasetUrn",
+ "type": "string",
+ "doc": "URN of the dataset"
+ },
+ {
+ "name": "bytesWritten",
+ "type": "long",
+ "doc": "Number of bytes written for the dataset"
Review Comment:
which jobs is this applicable to? e.g. could it work for retention-release?
what about for pulling record-by-record from a CRM system and writing a subset
of the records fields to a relational DB?
how to measure the number of bytes "written" to the DB... maybe approximate
from the char count of the stringified SQL statement?
(the questions I'm unclear on are ones to anticipate from users seeking
clarity from these "doc" strings)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
