Lijia Liu created SPARK-22233:
---------------------------------
Summary: filter out empty InputSplit in HadoopRDD
Key: SPARK-22233
URL: https://issues.apache.org/jira/browse/SPARK-22233
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.2.0
Environment: spark version:Spark 2.2
master: yarn
deploy-mode: cluster
Reporter: Lijia Liu
Sometimes, Hive will create an empty table with many empty files, Spark use the
InputFormat stored in Hive Meta Store and will not combine the empty files and
therefore generate many tasks to handle this empty files.
Hive use CombineHiveInputFormat(hive.input.format) by default.
So, in this case, Spark will spends much more resources than hive.
3 suggestions:
1. add a configuration, filter out empty InputSplit in HadoopRDD.
2. add a configuration, user can customize the inputformatclass in
HadoopTableReader.
3. use the InputFormatClass configured in hive configuration(hive.input.format).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
