Any update from anyone on this? We are invoking major compaction manually and we see too many skipping of major compaction since we have MIN_VERSION to 1.
Thanks On Tue, Aug 11, 2015 at 5:09 PM, mukund murrali <mukundmurra...@gmail.com> wrote: > Yes. We have min version to be 1, hence I suppose there wont be any sort > of time based compaction. After triggering major compaction we had only the > following logs > > compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store > files, 0 compacting, 3 eligible, 50 blocking > > Thanks > > On Tue, Aug 11, 2015 at 3:25 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> bq. oldestTime -9223370597787064221ms >> >> This is due to minTimestamp missing from store file: >> >> Long minTimestamp = sf.getMinimumTimestamp(); >> >> long oldest = (minTimestamp == null) >> >> ? Long.MIN_VALUE >> >> : now - minTimestamp.longValue(); >> >> Can you pastebin log from RatioBasedCompactionPolicy in region server log >> after the manual compaction ? >> >> Thanks >> >> On Tue, Aug 11, 2015 at 2:06 AM, mukund murrali <mukundmurra...@gmail.com >> > >> wrote: >> >> > We are using hbase-1.0.0. The following logs appears for all regions in >> the >> > regionserver >> > >> > 2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker] >> > compactions.RatioBasedCompactionPolicy: Skipping major compaction of >> > >> > >> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy@7bc4e8d8 >> > because one (major) compacted file only, oldestTime >> -9223370597787064221ms >> > is < ttl=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0) >> > >> > >> > Yes after manual triggering the deletes purged. But we don't want to >> have >> > it manual. Any other config to avoid such scenario? >> > >> > Thanks >> > >> > >> > >> > On Mon, Aug 10, 2015 at 8:01 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > >> > > What release of hbase are you using ? >> > > >> > > Can you pastebin region server log with DEBUG logging ? >> > > >> > > I guess you have tried issuing manual command. Did it work ? >> > > >> > > Thanks >> > > >> > > On Mon, Aug 10, 2015 at 7:02 AM, mukund murrali < >> > mukundmurra...@gmail.com> >> > > wrote: >> > > >> > > > Any one help us in this :( Are we missing somewhere in the use >> case? >> > > None >> > > > of the deleted cells are undergoing major compaction. >> > > > >> > > > Thanks >> > > > >> > > > On Wed, Aug 5, 2015 at 12:04 PM, mukund murrali < >> > > mukundmurra...@gmail.com> >> > > > wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > We wanted to have deleted data for a week. So we configured >> > > > > >> > > > > MIN_VERSIONS => 1 >> > > > > KEEP_DELETED_CELLS => TTL >> > > > > TTL => 1 week. >> > > > > >> > > > > As per our understanding, after 1 week the deleted data becomes >> > > available >> > > > > for major compaction and should be purged (correct if wrong). >> Since >> > we >> > > > have >> > > > > time series data, we don't have any write operations in those >> regions >> > > > after >> > > > > a week . But major compaction never took place for any regions and >> > our >> > > > > overall size grew drastically though we have deletes happening. >> After >> > > > > analyzing, we found that major compaction takes place if any one >> of >> > > the 2 >> > > > > condition is satisfied. >> > > > > >> > > > > 1. If the time interval between major compaction is greater than a >> > week >> > > > > (default config). >> > > > > 2. if the block locality index falls below a threshold. >> > > > > >> > > > > In our case, since we have min_versions to be 1, the first case >> > > condition >> > > > > fails. Time to verify is set to Long.Max value, if min versions is >> > not >> > > 0. >> > > > > >> > > > > Second is block locality. To check the block locality index we >> > enabled >> > > > > fine logs. And we found the block locality is always 1, and we >> got >> > > logs >> > > > > stating "Skipping major compaction......". >> > > > > >> > > > > So, in this case is manually triggering major compaction the only >> > > choice? >> > > > > >> > > > > Thanks >> > > > > >> > > > >> > > >> > >> > >
