Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19464#discussion_r144030461
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -196,7 +196,10 @@ class HadoopRDD[K, V](
// add the credentials here as this can be called before SparkContext
initialized
SparkHadoopUtil.get.addCredentials(jobConf)
val inputFormat = getInputFormat(jobConf)
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ var inputSplits = inputFormat.getSplits(jobConf, minPartitions)
--- End diff --
How about:
```
val inputSplits = if (......) {
inputFormat.getSplits(jobConf, minPartitions).filter(_.getLength > 0)
} else {
inputFormat.getSplits(jobConf, minPartitions)
}
```
We should alway try to not use `var`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
