RocMarshal commented on code in PR #23589:
URL: https://github.com/apache/flink/pull/23589#discussion_r1376018673
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -795,6 +795,20 @@ public boolean isGraphContainingLoops() {
return config.getBoolean(GRAPH_CONTAINING_LOOPS, false);
}
+ /**
+ * In general, we don't clear any configuration. However, the {@link
#SERIALIZED_UDF} may be
+ * very large when operator includes some large objects, the
SERIALIZED_UDF is used to create a
+ * StreamOperator and usually only needs to be called once. {@link
#CHAINED_TASK_CONFIG} may be
+ * large as well due to the StreamConfig of all non-head operators in
OperatorChain will be
+ * serialized and stored in CHAINED_TASK_CONFIG. They can be cleared to
reduce the memory after
+ * StreamTask is initialized. If so, TM will have more memory during
running. See FLINK-33315
+ * and FLINK-33317 for more information.
+ */
+ public void clearInitialConfigs() {
+ config.removeKey(SERIALIZED_UDF);
+ config.removeKey(CHAINED_TASK_CONFIG);
+ }
Review Comment:
Thanks @1996fanrui for the reply.
Just to reiterate this comment,
As you know, there is a hypothetical premise and a consensus premise:
1. Assuming that this part of the memory is large (sorry, I am not familiar
with this part)
2. The JVM will perform GC at a secure point; GC needs to be at a safe
point, but it may not necessarily trigger GC when there is a safe point. So
even if the JVM reaches the safe point, it only means that there are more
conditions for reaching GC, and the probability of GC triggering is higher.
Based on the above two points:
Assuming the code runs here, but the JVM has not reached the safe point, GC
cannot be performed temporarily. Continuing to run will also take up some
space. When GC is performed, more memory needs to be reclaimed and the latency
will be longer. If we can make the JVM reach a safe point early, the
probability of early GC will increase, which can minimize GC latency or
allocate the latency to multiple GCs.
However, this type of case is extreme and the conditions are also unstable.
Let's turn our attention to the merger of PR :)
Thank you again for your patience~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
