[ 
https://issues.apache.org/jira/browse/HIVE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069641#comment-16069641
 ] 

Chaozhong Yang commented on HIVE-16972:
---------------------------------------

[~wzheng] the root cause is "java.lang.ClassNotFoundException: 
org.apache.commons.collections4.Predicate" (weird error),  I will update and 
submit a new patch.

> FetchOperator: filter out inputSplits which length is zero
> ----------------------------------------------------------
>
>                 Key: HIVE-16972
>                 URL: https://issues.apache.org/jira/browse/HIVE-16972
>             Project: Hive
>          Issue Type: Improvement
>          Components: Physical Optimizer
>    Affects Versions: 2.1.0, 2.1.1
>            Reporter: Chaozhong Yang
>            Assignee: Chaozhong Yang
>         Attachments: HIVE-16972.2.patch, HIVE-16972.3.patch, 
> HIVE-16972.4.patch, HIVE-16972.patch
>
>
> * Background
>    We can describe the basic work flow of  common HQL query as follows:
>   1. compile and execute
>   2. fetch results
>   In many cases, we don't need to  worry about the issues fetching results 
> from HDFS(iff there are mapreduce jobs generated in planning step). However, 
> the number of results files on HDFS and data distribution will affect the 
> final status of HQL query, especially for HiveServer2. We have some map-only 
> queries, e.g: 
> {code:sql}
> select * from myTable where date > '20170101' and date <= '20170301' and id = 
> 88;
> {code}
>     This query will generate more than 20,000 files(look at screenshot image 
> uploaded) on HDFS and most of those files are empty. Of course, they are very 
> sparse. If we send TFetchResultsRequest from HiveServer2 client with  some 
> parameters(timeout:90s, maxRows:1024) , FetchOperator can not fetch 1024 rows 
> in 90 seconds and our HiveServer2 client will mark this TFetchResultsRequest 
> as timed out failure. Why? In fact, It's expensive to fetch results from 
> empty file. In our HDFS cluster( 5000+ DataNodes) , reading data from an 
> empty file will cost almost 100 ms (100ms * 1000 ==> 100s > 90s timeout). 
> Obviously, we can filter out those empty files or splits to speed up the 
> process of FetchResults. 
>   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to