Also... mapreduce.input.fileinputformat.split.maxsize and, what is the size of your input files?
From: Ryan Harris Sent: Wednesday, September 30, 2015 2:37 PM To: 'user@hive.apache.org' Subject: RE: CombineHiveInputFormat not working what are your values for: mapred.min.split.size mapred.max.split.size hive.hadoop.supports.splittable.combineinputformat From: Pradeep Gollakota [mailto:pradeep...@gmail.com] Sent: Wednesday, September 30, 2015 2:20 PM To: user@hive.apache.org Subject: CombineHiveInputFormat not working Hi all, I have an external table of with the following DDL. ``` DROP TABLE IF EXISTS raw_events; CREATE EXTERNAL TABLE IF NOT EXISTS raw_events ( raw_event_string string) PARTITIONED BY (dc string, community string, dt string) STORED AS TEXTFILE LOCATION '/lithium/events/{dc}/{community}/events/{year}/{month}/{day}' ``` The files are loaded externally and are LZ4 compressed. When I run a query on this table for a single day, I'm getting 1 mapper per file even though the input format is set to CombineHiveInputFormat. Does anyone know if CombineHiveInputFormat does not work with LZ4 compressed files or have any idea why split combination is not working? Thanks! Pradeep ====================================================================== THIS ELECTRONIC MESSAGE, INCLUDING ANY ACCOMPANYING DOCUMENTS, IS CONFIDENTIAL and may contain information that is privileged and exempt from disclosure under applicable law. If you are neither the intended recipient nor responsible for delivering the message to the intended recipient, please note that any dissemination, distribution, copying or the taking of any action in reliance upon the message is strictly prohibited. If you have received this communication in error, please notify the sender immediately. Thank you.
