[
https://issues.apache.org/jira/browse/FLINK-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-13752:
----------------------------
Description:
When comparing the 1.8 and 1.9.0-rc2 on a test streaming job, we found that the
performance on 1.9.0-rc2 is much lower than that of 1.8. By comparing the two
versions, we found that the count of Full GC on 1.9.0-rc2 is much more than
that on 1.8.
A further analysis found that the difference is due to in
_TaskExecutor#setupResultPartitionBookkeeping_, the anonymous function in
_taskTermimationWithResourceCleanFuture_ has referenced the
_TaskDeploymentDescriptor_, since this function will be kept till the task is
terminated, _TaskDeploymentDescriptor_ will also be kept referenced in the
closure and cannot be recycled by GC. In this job, _TaskDeploymentDescriptor_
of some tasks are as large as 10M, and the total heap is about 113M, thus the
kept _TaskDeploymentDescriptors_ will cause relatively large impact on GC and
performance.
was:
When comparing the 1.8 and 1.9.0-rc2 on a test streaming job, we found that the
performance on 1.9.0-rc2 is much lower than that of 1.8. By comparing the two
versions, we found that the count of Full GC on 1.9.0-rc2 is much more than
that on 1.8.
A further analysis found that the difference is due to in
TaskExecutor#setupResultPartitionBookkeeping, the anonymous function in
taskTermimationWithResourceCleanFuture has referenced the
TaskDeploymentDescriptor, since this function will be kept till the task is
terminated, TaskDeploymentDescriptor will also be kept referenced in the
closure and cannot be recycled by GC. In this job, TaskDeploymentDescriptor of
some tasks are as large as 10M, and the total heap is about 113M, thus the kept
TaskDeploymentDescriptors will cause relatively large impact on GC and
performance.
> TaskDeploymentDescriptor cannot be recycled by GC due to referenced by an
> anonymous function
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-13752
> URL: https://issues.apache.org/jira/browse/FLINK-13752
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.9.0
> Reporter: Yun Gao
> Priority: Major
>
> When comparing the 1.8 and 1.9.0-rc2 on a test streaming job, we found that
> the performance on 1.9.0-rc2 is much lower than that of 1.8. By comparing the
> two versions, we found that the count of Full GC on 1.9.0-rc2 is much more
> than that on 1.8.
> A further analysis found that the difference is due to in
> _TaskExecutor#setupResultPartitionBookkeeping_, the anonymous function in
> _taskTermimationWithResourceCleanFuture_ has referenced the
> _TaskDeploymentDescriptor_, since this function will be kept till the task is
> terminated, _TaskDeploymentDescriptor_ will also be kept referenced in the
> closure and cannot be recycled by GC. In this job, _TaskDeploymentDescriptor_
> of some tasks are as large as 10M, and the total heap is about 113M, thus the
> kept _TaskDeploymentDescriptors_ will cause relatively large impact on GC and
> performance.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
