ajamato commented on a change in pull request #11184: [BEAM-4374] Update protos
related to MonitoringInfo.
URL: https://github.com/apache/beam/pull/11184#discussion_r398916506
##########
File path: model/pipeline/src/main/proto/metrics.proto
##########
@@ -52,38 +61,160 @@ message Annotation {
string value = 2;
}
-// Populated MonitoringInfoSpecs for specific URNs.
-// Indicating the required fields to be set.
-// SDKs and RunnerHarnesses can load these instances into memory and write a
-// validator or code generator to assist with populating and validating
-// MonitoringInfo protos.
+// A set of well known MonitoringInfo specifications.
message MonitoringInfoSpecs {
enum Enum {
- // TODO(BEAM-6926): Add the PTRANSFORM name as a required label after
- // upgrading the python SDK.
- USER_COUNTER = 0 [(monitoring_info_spec) = {
- urn: "beam:metric:user",
- type_urn: "beam:metrics:sum_int_64",
+ // Represents an integer counter where values are summed across bundles.
+ USER_SUM_INT64 = 0 [(monitoring_info_spec) = {
+ urn: "beam:metric:user:v1",
+ type: "beam:metrics:sum_int64:v1",
+ required_labels: ["PTRANSFORM", "NAMESPACE", "NAME"],
+ annotations: [{
+ key: "description",
+ value: "URN utilized to report user metric."
+ }]
+ }];
+
+ // Represents a double counter where values are summed across bundles.
+ USER_SUM_DOUBLE = 1 [(monitoring_info_spec) = {
+ urn: "beam:metric:user:v1",
+ type: "beam:metrics:sum_double:v1",
+ required_labels: ["PTRANSFORM", "NAMESPACE", "NAME"],
+ annotations: [{
+ key: "description",
+ value: "URN utilized to report user metric."
+ }]
+ }];
+
+ // Represents a distribution of an integer value where:
+ // - count: represents the number of values seen across all bundles
Review comment:
Now it seems like there technically aren't any fields named "count", "sum",
"min", "max". Just 4 encoded varints in that specific order. There is no longer
a proto or anything which defines this format.
If we are going to keep type urns, I think that there should be somewhere in
this file where you could a "TypeSpec", which describes how to encode each
opaque bytes payload. i.e. the coders used for each value, the order they must
be encoded. Or a proto that should be serialized into that bytes field, etc. A
description that will work for all languages. Right now you can only know that
from looking at your encoding code.
I think it would be best if SDK implemented could look at a reference file
like this and know how to populate the MonitoringInfo. That was the original
intention behind MonitoringInfoSpec, and I believe that is a bit lost now with
this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
