[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clara Xiong updated HBASE-15400: -------------------------------- Summary: Date Tiered Multiple Output for Date Tiered Compaction (was: Multiple Output for Date Tiered Compaction) > Date Tiered Multiple Output for Date Tiered Compaction > ------------------------------------------------------ > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Improvement > Components: Compaction > Reporter: Clara Xiong > Assignee: Clara Xiong > Fix For: 2.0.0 > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > I am starting from a prototype of date tiered file writer from > https://issues.apache.org/jira/browse/HBASE-15389 and will upload a patch > soon. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)