[
https://issues.apache.org/jira/browse/FLINK-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-6525.
--------------------------------
Resolution: Invalid
No longer valid.
> Transferred TM log/stdout files are never removed from BlobStore
> ----------------------------------------------------------------
>
> Key: FLINK-6525
> URL: https://issues.apache.org/jira/browse/FLINK-6525
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.3.0, 1.4.0
> Reporter: Till Rohrmann
> Priority: Major
>
> The {{TaskManager}} uses the {{BlobClient}} to upload its stdout/log file to
> the {{BlobServer}}. If HA mode is enabled, then these files will also be
> uploaded to the {{BlobStore}}. Since the {{TaskManagerLogHandler}} only
> cleans up files from a TM in case it has already received another file from
> this TM and additionally does this in a non thread safe manner, it can easily
> happen that files won't get cleaned up from the {{BlobStore}}.
> I think we should not upload these kind of files to the persistent/HA
> {{BlobStore}}. We could do this by introducing a storage mode when uploading
> files to the {{BlobServer}} (e.g. {{HA_STORAGE}} vs. {{LOCAL_STORAGE}}).
> Additionally, we should also register a timeout for only locally stored files
> or at least store them under its {{JobID}} such that these files are also
> cleaned up once the job is being cleaned up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
