Github user cmccabe commented on the pull request:
https://github.com/apache/spark/pull/1486#issuecomment-55683469
I can see why you'd like to reduce visibility, but I don't think it's
possible here. In HadoopRDD, three new things are exposed with visibility
private [spark]. They are SPLIT_INFO_REFLECTIONS, convertSplitLocations, and
the SplitInfoReflections type. These things are exposed for the benefit of
NewHadoopRDD. As far as I can see, you can't just have "a single... function
call with a narrow interface" because HadoopRDD is dealing with different types
than NewHadoopRDD. HadoopRDD is dealing with HadoopPartition which needs to be
cast to org.apache.hadoop.mapred.InputSplitWithLocationInfo; NewHadoopRDD is
dealing with org.apache.hadoop.mapreduce.InputSplit. You will need two
separate code paths for this. Ultimately, the goal is to get an array of
org.apache.hadoop.mapred.SplitLocationInfo, at which point a common function
can be called, HadoopRDD#convertSplitLocationInfo. That common function
currently lives in HadoopRDD.scala.
We always have the freedom to refactor this in the future, since it's not
visible from outside Spark. I think it's very unlikely that any other code
inside Spark will start calling these methods, since they're obviously tied to
a very specific goal of extracting information from hadoop 2.x types. What do
you guys think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
