[
https://issues.apache.org/jira/browse/SPARK-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477679#comment-15477679
]
Apache Spark commented on SPARK-17475:
--------------------------------------
User 'frreiss' has created a pull request for this issue:
https://github.com/apache/spark/pull/15027
> HDFSMetadataLog should not leak CRC files
> -----------------------------------------
>
> Key: SPARK-17475
> URL: https://issues.apache.org/jira/browse/SPARK-17475
> Project: Spark
> Issue Type: Sub-task
> Components: Streaming
> Reporter: Frederick Reiss
>
> When HDFSMetadataLog uses a log directory on a filesystem other than HDFS
> (i.e. NFS or the driver node's local filesystem), the class leaves orphan
> checksum (CRC) files in the log directory. The files have names that follow
> the pattern "..[long UUID hex string].tmp.crc". These files exist
> HDFSMetaDataLog renames other temporary files without renaming the
> corresponding checksum files. There is one CRC file per batch, so the
> directory fills up quite quickly.
> I'm not certain, but this problem might also occur on certain versions of the
> HDFS APIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
