jackylau created HBASE-23149:
--------------------------------
Summary: hbase shouldPerformMajorCompaction logic is not correct
Key: HBASE-23149
URL: https://issues.apache.org/jira/browse/HBASE-23149
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 1.4.9
Reporter: jackylau
Fix For: 1.4.11
we can know that the major compaction is skipping but it is compacting that
region from the below regionserver's log.
and read the code and find it is not correct and i add mark "/*** ***/" below
public boolean shouldPerformMajorCompaction(final Collection<StoreFile>
filesToCompact)
throws IOException {
if (lowTimestamp > 0L && lowTimestamp < (now - mcTime)) {
if (filesToCompact.size() == 1) {
if (sf.isMajorCompaction() && (cfTTL == Long.MAX_VALUE || oldest < cfTTL)) {
float blockLocalityIndex =
sf.getHDFSBlockDistribution().getBlockLocalityIndex(
RSRpcServices.getHostname(comConf.conf, false));
if (blockLocalityIndex < comConf.getMinLocalityToForceCompact()) {
result = true;
} else {
LOG.debug("Skipping major compaction of " + regionInfo
+ " because one (major) compacted file only, oldestTime " + oldest
+ "ms is < TTL=" + cfTTL + " and blockLocalityIndex is " + blockLocalityIndex
+ " (min " + comConf.getMinLocalityToForceCompact() + ")");
}
} else if (cfTTL != HConstants.FOREVER && oldest > cfTTL) {
LOG.debug("Major compaction triggered on store " + regionInfo
+ ", because keyvalues outdated; time since last major compaction "
+ (now - lowTimestamp) + "ms");
result = true;
}
} else {
LOG.debug("Major compaction triggered on store " + regionInfo
+ "; time since last major compaction " + (now - lowTimestamp) + "ms");
}
result = true; /******here it is not right, it should be move to the above
*****/
}
return result;
}
2019-09-27 09:09:35,960 DEBUG [st129,16020,1568236573216_ChoreService_1]
compactions.RatioBasedCompactionPolicy: Skipping major compaction of
100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
because one (major) compacted file only, oldestTime 3758085589ms is <
TTL=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1]
compactions.SortedCompactionPolicy: Selecting compaction from 1 store files, 0
compacting, 1 eligible, 100 blocking
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1]
regionserver.HStore: 413a563092544e8df480fd601b2de71b - d: Initiating major
compaction (all files)
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1]
regionserver.CompactSplitThread: Large Compaction requested:
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4b5582f1;
Because: CompactionChecker requests major compaction; use default priority;
compaction_queue=(1:0), split_queue=0, merge_queue=0
2019-09-27 09:09:35,961 INFO
[regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579]
regionserver.HRegion: Starting compaction on d in region
100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
2019-09-27 09:09:35,961 INFO
[regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579]
regionserver.HStore: Starting compaction of 1 file(s) in d of
100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
into
tmpdir=hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/.tmp,
totalSize=5.1 G
2019-09-27 09:09:35,961 DEBUG
[regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579]
compactions.Compactor: Compacting
hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/d/3b4080f9b6f149e1b0a476058c8564e6,
keycount=83914030, bloomtype=NONE, size=5.1 G, encoding=FAST_DIFF,
compression=SNAPPY, seqNum=2621061, earliestPutTs=1565788490371
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
