[
https://issues.apache.org/jira/browse/FLINK-26593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ren shangtao updated FLINK-26593:
---------------------------------
Description:
I have A batch task that inserts 100 million data from table A into table B
after the primary key hash. I use Flink standalone mode to execute this task.
After several times of executing this task, I find that the Flink TaskManager
process occupies more and more memory. Finally, the Flink TaskManager process
is killed when the system runs out of memory.
{color:#FF0000}*Here is the message*{color}
!image-2022-03-11-10-15-00-682.png!
{color:#FF0000}*Before run the job*{color}
!image-2022-03-11-10-16-00-438.png!
*{color:#FF0000}After serval times{color}*
!image-2022-03-11-10-41-46-256.png!
My configuration
!image-2022-03-11-10-18-32-940.png!
!image-2022-03-11-10-19-31-014.png!
was:
I have A batch task that inserts 100 million data from table A into table B
after the primary key hash. I use Flink standalone mode to execute this task.
After several times of executing this task, I find that the Flink TaskManager
process occupies more and more memory. Finally, the Flink TaskManager process
is killed when the system runs out of memory.
Here is the message
!image-2022-03-11-10-15-00-682.png!
before run the job
!image-2022-03-11-10-16-00-438.png!
after serval times
!image-2022-03-11-10-16-53-678.png!
My configuration
!image-2022-03-11-10-18-32-940.png!
!image-2022-03-11-10-19-31-014.png!
> task manager can not free memory when jobs are finished caused the linux out
> of memory
> --------------------------------------------------------------------------------------
>
> Key: FLINK-26593
> URL: https://issues.apache.org/jira/browse/FLINK-26593
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Affects Versions: 1.14.3
> Environment: The flink cluster environment with five nodes
> The iceberg is 0.13.1 for flink 1.14
> The hadoop is 3.3.1
> The hive is 3.1.2
>
>
> !image-2022-03-11-10-20-41-739.png!
> Reporter: ren shangtao
> Priority: Blocker
> Labels: batch, flinksql, oom
> Attachments: image-2022-03-11-10-15-00-682.png,
> image-2022-03-11-10-16-00-438.png, image-2022-03-11-10-16-53-678.png,
> image-2022-03-11-10-18-32-940.png, image-2022-03-11-10-19-31-014.png,
> image-2022-03-11-10-20-41-739.png, image-2022-03-11-10-41-46-256.png
>
>
> I have A batch task that inserts 100 million data from table A into table B
> after the primary key hash. I use Flink standalone mode to execute this task.
> After several times of executing this task, I find that the Flink TaskManager
> process occupies more and more memory. Finally, the Flink TaskManager process
> is killed when the system runs out of memory.
>
> {color:#FF0000}*Here is the message*{color}
> !image-2022-03-11-10-15-00-682.png!
>
> {color:#FF0000}*Before run the job*{color}
> !image-2022-03-11-10-16-00-438.png!
>
> *{color:#FF0000}After serval times{color}*
> !image-2022-03-11-10-41-46-256.png!
>
> My configuration
> !image-2022-03-11-10-18-32-940.png!
> !image-2022-03-11-10-19-31-014.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
