[
https://issues.apache.org/jira/browse/FLINK-17928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116939#comment-17116939
]
Piotr Nowojski commented on FLINK-17928:
----------------------------------------
Maybe, but the problem is still there:
{noformat}
[hadoop@ip-172-31-35-50 logs]$ zcat
master17-hdfs-unaligned-throughput2-application_1589453804748_0116.gz | grep
INFO | grep -i complet
2020-05-17 09:33:08,854 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 1 for job 7f2de1d00cc862534746985f4c021098 (25766077793 bytes in
38316 ms).
2020-05-17 09:34:23,546 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 2 for job 7f2de1d00cc862534746985f4c021098 (42803408024 bytes in
35183 ms).
2020-05-17 09:35:33,974 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 3 for job 7f2de1d00cc862534746985f4c021098 (42037807324 bytes in
33832 ms).
2020-05-17 09:36:47,447 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 4 for job 7f2de1d00cc862534746985f4c021098 (38890649497 bytes in
34755 ms).
2020-05-17 09:38:00,182 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 5 for job 7f2de1d00cc862534746985f4c021098 (41471661012 bytes in
34806 ms).
2020-05-17 09:39:09,803 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 6 for job 7f2de1d00cc862534746985f4c021098 (40503930228 bytes in
34256 ms).
2020-05-17 09:40:23,162 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 7 for job 7f2de1d00cc862534746985f4c021098 (41961537883 bytes in
35310 ms).
2020-05-17 09:41:35,483 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 8 for job 7f2de1d00cc862534746985f4c021098 (41942565096 bytes in
35262 ms).
{noformat}
It's reported as tens of GBs.
> Incorrect state size reported when using unaligned checkpoints
> ---------------------------------------------------------------
>
> Key: FLINK-17928
> URL: https://issues.apache.org/jira/browse/FLINK-17928
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.11.0
> Reporter: Piotr Nowojski
> Priority: Critical
> Fix For: 1.11.0
>
>
> Even when checkpoints on HDFS are between 100-300MBs, the reported state size
> is in orders of magnitude larger with values like:
> {noformat}
> 1GiB 1.5TiB 2.0TiB 2.1TiB 2.1TiB
> 148GiB 148GiB 148GiB 148GiB 148GiB 148GiB
> {noformat}
> it's probably because we have multiple
> {{Collection<InputChannelStateHandle>}}, and each of the individual handle
> returns the same value from {{AbstractChannelStateHandle#getStateSize}} - the
> full size of the spilled data, ignoring that only small portion of those data
> belong to a single input channel/result subpartition. In other words {{
> org.apache.flink.runtime.state.AbstractChannelStateHandle#getStateSize}}
> should be taking the offsets into account and return only the size of the
> data that belong exclusively to this handle.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
