[
https://issues.apache.org/jira/browse/HIVE-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611677#comment-15611677
]
Wataru Yukawa commented on HIVE-13374:
--------------------------------------
Thank you for comment.
By the way, I work at LINE.
I'm not sure if Yahoo Japan faces this problem.
> HiveServer2 hangs up if query to scan too many partition is submitted
> ---------------------------------------------------------------------
>
> Key: HIVE-13374
> URL: https://issues.apache.org/jira/browse/HIVE-13374
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 1.2.1
> Environment: HDP2.4.0
> JDK1.8
> Reporter: Wataru Yukawa
>
> If query to scan too many partition(for example, more than 40,000 partitions)
> is submitted, hiveserver2 hangs up
> hiveserver2 java heap status is the following.
> https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
> https://gyazo.com/65f12cd3875330995508ae11afd394a2
> OutOfMemoryError doesn't occur. but other queries fails.
> Maybe it is necessary to spill to disk...
> hiveserver2.log
> {code}
> ....
> 2016-03-28 13:44:28,923 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(517)) -
> Total number of paths: 47409, launching 50 threads to check non-combinable
> ones.
> 2016-03-28 13:44:29,444 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(439))
> - CombineHiveInputSplit creating pool for
> hdfs://.../aaa=.../yyyymmdd=20150721; using filter path
> hdfs://.../aaa=.../yyyymmdd=20150721
> ....
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444))
> - CombineHiveInputSplit: pool is already created for
> hdfs://.../aaa=.../yyyymmdd=20160326; using filter path
> hdfs://.../aaa=.../yyyymmdd=20160326
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444))
> - CombineHiveInputSplit: pool is already created for
> hdfs://.../aaa=.../yyyymmdd=20160327; using filter path
> hdfs://.../aaa=.../yyyymmdd=20160327
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444))
> - CombineHiveInputSplit: pool is already created for
> hdfs://.../aaa=.../yyyymmdd=20160328; using filter path
> hdfs://.../aaa=.../yyyymmdd=20160328
> 2016-03-28 13:45:00,285 INFO
> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]:
> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM
> or host machine (eg GC): pause of approximately 4587ms
> GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
> GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
> ....
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
