[
https://issues.apache.org/jira/browse/HIVE-22410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970494#comment-16970494
]
Sam An commented on HIVE-22410:
-------------------------------
Finally got a change to review the patch. Here are my comments.
* We need unit test
* The logic will add the missing estimator. However, it might not be
addressing the real issue here. The estimator will be null only if the if
statement above it
if (lastClass != elementClass) {
...
}
is false, which means lastClass == elementClass. When that is the case,
lastEstimator should not have been null. But in this case, it is null, and that
in my opinion, points to some assumption in the code is wrong, and it is not
just in array estimation, but also in estimating Collection, Map sizes. This
highlights the importance of having the reproduction scenario and unit test,
otherwise, we don't know what we are actually fixing here.
> CachedStore Prewarm Failure NPE
> -------------------------------
>
> Key: HIVE-22410
> URL: https://issues.apache.org/jira/browse/HIVE-22410
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Affects Versions: 3.1.2, 2.3.7
> Environment: JDK 1.7.0_40
> Reporter: fanshilun
> Assignee: fanshilun
> Priority: Major
> Labels: cachedstore
> Fix For: 2.3.7, 3.1.3
>
> Attachments: HIVE-22410.1.patch, image-2019-11-01-11-14-42-114.png
>
>
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimateArrayElements(IncrementalObjectSizeEstimator.java:501)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:443)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:433)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:369)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.cached.SharedCache.populateTableInCache(SharedCache.java:1028)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.cached.CachedStore.prewarm(CachedStore.java:314)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.cached.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:492)
> [hive-metastore-2.3.5.jar:2.3.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [?:1.7.0_40]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [?:1.7.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [?:1.7.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [?:1.7.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [?:1.7.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [?:1.7.0_40]
> at java.lang.Thread.run(Thread.java:724) [?:1.7.0_40]
> *reason:*
> Methods estimateArrayElements throw NPE
> I found that the last class does not appear in the parent (HashMap<Class<?>,
> Object Estimator> ), and the parent does not contain the information of the
> ConcurrentHashMap Class.
> The solution is to initialize the class information before using it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
