sam-an-cloudera commented on a change in pull request #662: Hive-21787:
Metastore table cache enhancement
URL: https://github.com/apache/hive/pull/662#discussion_r292719733
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -425,28 +667,11 @@ public boolean
updateTableColStats(List<ColumnStatisticsObj> colStatsForTable) {
} else {
// No stats exist for this key; add a new object to the cache
// TODO: get rid of deepCopy after making sure callers don't use
references
- if (maxCacheSizeInBytes > 0) {
- ObjectEstimator tblColStatsSizeEstimator =
- getMemorySizeEstimator(ColumnStatisticsObj.class);
- long estimatedMemUsage =
- tblColStatsSizeEstimator.estimate(colStatObj,
sizeEstimators);
- LOG.trace("Memory needed to cache Table Column Statistics
Object: {} is {} bytes",
- colStatObj, estimatedMemUsage);
- if (isCacheMemoryFull(estimatedMemUsage)) {
- LOG.debug(
- "Cannot cache Table Column Statistics Object: {}. Memory
needed is {} bytes, "
- + "whereas the memory remaining is: {} bytes.",
- colStatObj, estimatedMemUsage,
- (0.8 * maxCacheSizeInBytes - currentCacheSizeInBytes));
- return false;
- } else {
- currentCacheSizeInBytes += estimatedMemUsage;
- }
- LOG.trace("Current cache size: {} bytes",
currentCacheSizeInBytes);
- }
tableColStatsCache.put(key, colStatObj.deepCopy());
+ statsSize += getObjectSize(ColumnStatisticsObj.class, colStatObj);
Review comment:
I recall initially I coded to have existing size subtracted, but then I
realized all 4 refresh methods in TableWrapper class are actually doing
"replace", instead of "refresh". It creates a new hashmap, and puts the new
partitions, colstats, etc into it. For example, in
public void refreshTableColStats(List<ColumnStatisticsObj> colStatsForTable)
{
Map<String, ColumnStatisticsObj> newTableColStatsCache = new
HashMap<String, ColumnStatisticsObj>();
...
then later on in the same method, it replaces the tableColStatsCache with
the new map.
tableColStatsCache = newTableColStatsCache;
updateMemberSize(MemberName.TABLE_COL_STATS_CACHE, statsSize,
SizeMode.Snapshot);
That's why I updated the member size with only the new size, in a snapshot
mode, not delta mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
