[ 
https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-17604:
--------------------------------
    Description: 
Currently with SPARK-15698, FileStreamSource metadata log will be compacted 
periodically (10 batches by default), this means compacted batch file will 
contain whole file entries been processed. With the time passed, the compacted 
batch file will be accumulated to a relative large file. 

With SPARK-17165, now {{FileStreamSource}} doesn't track the aged file entry, 
but in the log we still keep the full records,  this is not necessary and quite 
time-consuming during recovery. So here propose to also add file entry purging 
ability to {{FileStreamSource}} metadata log.

This is pending on SPARK-15698.

  was:
Currently with SPARK-15698, FileStreamSource metadata log will be compacted 
periodically (10 batches by default), this means compacted batch file will 
contain whole file entries been processed. With the time passed, the compacted 
batch file will be accumulated to a relative large file. 

With SPARK-17165, now {{FileStreamSource}} doesn't track the aged file entry, 
but in the log we still keep the full records,  this is not necessary and quite 
time-consuming during recovery. So here propose to also add file entry purging 
ability to {{FileStreamSource}} metadata log.


> Support purging aged file entry for FileStreamSource metadata log
> -----------------------------------------------------------------
>
>                 Key: SPARK-17604
>                 URL: https://issues.apache.org/jira/browse/SPARK-17604
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL, Streaming
>            Reporter: Saisai Shao
>            Priority: Minor
>
> Currently with SPARK-15698, FileStreamSource metadata log will be compacted 
> periodically (10 batches by default), this means compacted batch file will 
> contain whole file entries been processed. With the time passed, the 
> compacted batch file will be accumulated to a relative large file. 
> With SPARK-17165, now {{FileStreamSource}} doesn't track the aged file entry, 
> but in the log we still keep the full records,  this is not necessary and 
> quite time-consuming during recovery. So here propose to also add file entry 
> purging ability to {{FileStreamSource}} metadata log.
> This is pending on SPARK-15698.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to