JingsongLi commented on a change in pull request #10452:
[FLINK-15035][table-planner-blink] Introduce unknown memory setting to table in
blink planner
URL: https://github.com/apache/flink/pull/10452#discussion_r355097496
##########
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/exec/BatchExecNode.scala
##########
@@ -32,4 +33,13 @@ trait BatchExecNode[T] extends ExecNode[BatchPlanner, T]
with Logging {
*/
def getDamBehavior: DamBehavior
+ def setManagedMemoryWeight[X](
+ transformation: Transformation[X], memoryBytes: Long): Transformation[X]
= {
+ // Using Bytes can easily overflow
+ // Using MebiBytes to cast to int
+ // Careful about zero
+ val memoryMB = Math.max(1, (memoryBytes >> 20).toInt)
Review comment:
Hi @zhuzhurk , we can not control all transformations, because some of them
come from user's Source/Sink.
Hi @KurtYoung
- 1. Current Transformation not have an unknown value to indicate that the
weight not be set.
The thing is that we can not control the default value of Transformation. If
runtime let the default value of Transformation's weight is `null`, in this
way, we can set all null to zero, we can control.
- 2. The another way is just as @zhuzhurk said, we can consider default 1 to
be unknown value, in this way, we set all 1 to 0. But that is a little hacky.
- 3. The third way is just let it go. By default, 1 can't steal much memory
(we set sort/join/agg to 128). We can even set KibiBytes, then the
sort/join/agg is 128 * 1024.
I prefer option 3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
