Ashu Pachauri commented on HBASE-18399:

[~ram_krish] Yes, this case is also covered in the tests for HBASE-18398. The 
patch for that ensures that compacted files for a store are not moved to the 
archive unless they are written to the region manifest on the file system. This 
ensures that the SnapshotFileCache (and in turn the HFileCleaner) never has a 
stale view of the files it's considering to delete.

> Files in a snapshot can go missing even after the snapshot is taken 
> successfully
> --------------------------------------------------------------------------------
>                 Key: HBASE-18399
>                 URL: https://issues.apache.org/jira/browse/HBASE-18399
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver, Scanners
>            Reporter: Ashu Pachauri
>             Fix For: 1.3.2
> Files missing after the snapshot is taken (only applicable when the TTL for 
> the TimeToLiveHFileCleaner is small, like the default 5 mins)
>     * SnapshotManifest#addRegion visits store_file_A, but is yet to write it 
> to the manifest.
>     * store_file_A is marked as compacted away and HFileArchiver moves the 
> file to archive.
>     * HFileCleaner comes in and sees the store_file_A in archive. It adds the 
> file to the list of files that might need to be cleaned up.
>     * HFileCleaner's SnapshotHFileCleaner plugin is kicked in.
>     * SnapshotFileCache#getUnreferencedFiles also says that store_file_A is 
> unreferenced and should be cleaned up (It has not yet been written to the 
> manifest).
>     * SnapshotHFileCleaner is still going through rest of the files in 
> archive.
>     * store_file_A reference is created and written to snapshot manifest.
>     * Snapshot verification runs and sees the store_file_A is present in 
> archive, and thus the verification passes.
>     * Now, the SnapshotHFileCleaner finishes and TimeToLiveHFileCleaner is 
> triggered. If TTL has passed since the store_file_A was moved to archive 
> (SnapshotHFileCleaner could take easily several minutes to go through rest of 
> the files), the TimeToLiveHFileCleaner also marks the file as deletable.
>     * Since all cleaner plugins marked file as deletable, the store_file_A is 
> deleted.

This message was sent by Atlassian JIRA

Reply via email to