2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1057235274
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java:
##########
@@ -1780,6 +1781,29 @@ public OptionalDouble getAvgStoreFileAge() {
return getStoreFileAgeStream().average();
}
+ @Override
+ public Map<String, Pair<Long, Long>> getStoreFilesAccessTimeAndSize() {
+ Collection<HStoreFile> storeFiles =
this.storeEngine.getStoreFileManager().getStorefiles();
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = new HashMap<>();
+ for (HStoreFile sf : storeFiles) {
+ if (sf.getReader() == null) {
+ continue;
+ }
+ FileStatus fileStatus;
+ try {
+ fileStatus = sf.getFileInfo().getFileStatus();
Review Comment:
@Apache9 sir.
Yes, this is really a serious problem.
Can we solve this problem by periodically accessing NN randomly, for
example, accessing it every hour or more to get fileStatus, so as to reduce
the pressure on the NN?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
// accumulate the count
mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
+ for (Pair<Long, Long> pair : sfAccessTimeAndSizeMap.values()) {
+ Long accessTime = pair.getFirst();
+ Long size = pair.getSecond();
+ for (int threshold : storeFilesAccessedDaysThresholds) {
+ Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold);
+ if ((now - accessTime) >= threshold * ONE_DAY_MS) {
Review Comment:
@Apache9 sir.
Yes, this is by design. For instance, if an HFile has not been accessed for
90 days, it is certain that it has also not been accessed for 30 and 7 days.
And what is your suggestion?
##########
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java:
##########
@@ -56,6 +56,8 @@ public interface MetricsRegionSource extends
Comparable<MetricsRegionSource> {
String ROW_READS_ONLY_ON_MEMSTORE_DESC = "Row reads happening completely out
of memstore";
String MIXED_ROW_READS = "mixedRowReadsCount";
String MIXED_ROW_READS_ON_STORE_DESC = "Row reads happening out of files and
memstore on store";
+ String STOREFILES_ACCESSED_DAYS_AND_SIZE = "storeFilesAccessed%sDaysSize";
Review Comment:
OK, sir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
