Ferenc Erdelyi created HIVE-13613: ------------------------------------- Summary: Add computeSplitSize() to CombineHiveInputFormat and HiveInputFormat Key: HIVE-13613 URL: https://issues.apache.org/jira/browse/HIVE-13613 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 1.1.0 Reporter: Ferenc Erdelyi
The input formats that Hive uses (CombineHiveInputFormat and HiveInputFormat) do not use the computeSplitSize(). CombineHiveInputFormat and HiveInputFormat do not extend FileInputFormat so that functionality is not there. For tuning parquet file processing the computeSplitSize() could be used. Please add computeSplitSize() functionality to CombineHiveInputFormat and HiveInputFormat. Use case: It would be desirable for our Hive query to autoselect the right splitsize (and consequently number of mappers) based on the data's blocksize as this is providing us with significant performance gains (e.g. for processing parquet files). Looking in https://github.com/cloudera/hadoop-common/blob/cdh5-2.6.0_5.5.2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java this is the behaviour I would expect from computeSplitSize(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)