[ 
https://issues.apache.org/jira/browse/FLINK-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-6525.
--------------------------------
    Resolution: Invalid

No longer valid.

> Transferred TM log/stdout files are never removed from BlobStore
> ----------------------------------------------------------------
>
>                 Key: FLINK-6525
>                 URL: https://issues.apache.org/jira/browse/FLINK-6525
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Till Rohrmann
>            Priority: Major
>
> The {{TaskManager}} uses the {{BlobClient}} to upload its stdout/log file to 
> the {{BlobServer}}. If HA mode is enabled, then these files will also be 
> uploaded to the {{BlobStore}}. Since the {{TaskManagerLogHandler}} only 
> cleans up files from a TM in case it has already received another file from 
> this TM and additionally does this in a non thread safe manner, it can easily 
> happen that files won't get cleaned up from the {{BlobStore}}.
> I think we should not upload these kind of files to the persistent/HA 
> {{BlobStore}}. We could do this by introducing a storage mode when uploading 
> files to the {{BlobServer}} (e.g. {{HA_STORAGE}} vs. {{LOCAL_STORAGE}}). 
> Additionally, we should also register a timeout for only locally stored files 
> or at least store them under its {{JobID}} such that these files are also 
> cleaned up once the job is being cleaned up.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to