Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20265190
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.isDefined) {
recordsSinceMetricsUpdate = 0
val bytesReadFn = bytesReadCallback.get
- inputMetrics.bytesRead = bytesReadFn()
+ inputMetrics.bytesRead = bytesReadFn() + bytesReadAtStart
--- End diff --
@sryza and I discussed this offline and Sandy suggested registering a
callback with task metrics to get the bytes read, as in his old commit:
https://github.com/sryza/spark/commit/9f1d77cc0eb0f5206ac52621fb85c99cae56f945
which seems to be the least-ugly solution here, given that Hadoop's methods
all provide the total bytes read across all splits read in the thread.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
