Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/962#issuecomment-47374032
Ok this is good to go now I think. Two changes:
(1) As @andrewor14 suggested, I added the read method to the UI as shown in
the image below
(2) I changed the DataReadMethod name from Hdfs to Hadoop, since @pwendell
pointed out that data won't necessarily have come from Hdfs
@pwendell also recommended checking the class name of the Hadoop input
split before trying to set the input metrics to ensure that the type of split
supports the getLength() method, because some split types (e.g., the HBase one)
just return 0 when you call getLength(). I looked into this a little bit and
there doesn't seem to be a good way to predict when an InputSplit subclass will
return an accurate value for getLength() (@pwendell's original suggestion of
checking to see if the class name ends with FileSplit is too restrictive
because CompositeInputSplit accurately returns the length). I think it's fine
to leave this as-as because if the InputSplit subclass used returns 0 from
getLength(), the total input size for the stage will be 0, so we won't show the
input size in the UI. As a result, I don't think this will be confusing to
users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
