zentol commented on a change in pull request #6850: [FLINK-10252] Handle
oversized metric messges
URL: https://github.com/apache/flink/pull/6850#discussion_r225484559
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricQueryService.java
##########
@@ -109,7 +122,17 @@ public void onReceive(Object message) {
}
} else if (message instanceof CreateDump) {
MetricDumpSerialization.MetricSerializationResult dump =
serializer.serialize(counters, gauges, histograms, meters);
- getSender().tell(dump, getSelf());
+
+ int realMsgSize = dump.serializedMetrics.length;
+
+ if (realMsgSize > maximumFramesize) {
+ String overSizeErrorMsg = "The metric
dump message size : " + realMsgSize
+ + " exceeds the maximum akka
framesize : " + maximumFramesize + ".";
+ LOG.error(overSizeErrorMsg);
+ getSender().tell(new Status.Failure(new
IOException(overSizeErrorMsg)), getSelf());
Review comment:
alright, this isn't so easy to implement. I would suggest as a simple
counter-measure to retry the serialization without histograms (including
unregistering all histograms and rejecting future registrations), and if that
fails as well to fail with an exception. Making this more robust can be handled
in a follow-up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
