[
https://issues.apache.org/jira/browse/FLINK-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-28024:
----------------------------------
Description:
-We observed several situations already where log files reached a file size of
over 120G. This caused the worker's disk usage to reach 100% resulting in the
worker machine to go "offline", i.e. not being available to pick up new tasks.-
The initially observed excessive log spilling is due to a TaskManager failing
fatally which results in the requested number of slots never becoming available
and the test job ending up in an infinite failover/restart loop. See further
details in the comment section.
was:We observed several situations already where log files reached a file
size of over 120G. This caused the worker's disk usage to reach 100% resulting
in the worker machine to go "offline", i.e. not being available to pick up new
tasks.
> KeyedStateCheckpointingITCase.KeyedStateCheckpointingITCase ends up in
> infinite failover loop
> ---------------------------------------------------------------------------------------------
>
> Key: FLINK-28024
> URL: https://issues.apache.org/jira/browse/FLINK-28024
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
> Affects Versions: 1.16.0
> Reporter: Matthias Pohl
> Assignee: Matthias Pohl
> Priority: Blocker
> Labels: test-stability
> Attachments: testWithRocksDbBackendIncremental.log.gz
>
>
> -We observed several situations already where log files reached a file size
> of over 120G. This caused the worker's disk usage to reach 100% resulting in
> the worker machine to go "offline", i.e. not being available to pick up new
> tasks.-
> The initially observed excessive log spilling is due to a TaskManager failing
> fatally which results in the requested number of slots never becoming
> available and the test job ending up in an infinite failover/restart loop.
> See further details in the comment section.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
