GitHub user guoxiaolongzte opened a pull request:
https://github.com/apache/spark/pull/20437
[SPARK-23270][Streaming][WEB-UI]FileInputDStream Streaming UI 's records
should not be set to the default value of 0, it should be the total number of
rows of new files.
## What changes were proposed in this pull request?
FileInputDStream Streaming UI 's records should not be set to the default
value of 0, it should be the total number of rows of new files.
------------------------------------------in FileInputDStream.scala
start------------------------------------
val inputInfo = StreamInputInfo(id, 0, metadata) // set to the default
value of 0
ssc.scheduler.inputInfoTracker.reportInfo(validTime, inputInfo)
case class StreamInputInfo(
inputStreamId: Int, numRecords: Long, metadata: Map[String, Any] =
Map.empty)
------------------------------------in FileInputDStream.scala
end--------------------------
------------------------------------------in DirectKafkaInputDStream.scala
start------------------------------------
val inputInfo = StreamInputInfo(id, rdd.count, metadata) //set to rdd count
as numRecords
ssc.scheduler.inputInfoTracker.reportInfo(validTime, inputInfo)
case class StreamInputInfo(
inputStreamId: Int, numRecords: Long, metadata: Map[String, Any] =
Map.empty)
------------------------------------in DirectKafkaInputDStream.scala
end----------------------
test methodï¼
./bin/spark-submit --class
org.apache.spark.examples.streaming.HdfsWordCount
examples/jars/spark-examples_2.11-2.4.0-SNAPSHOT.jar /spark/tmp/
fix after:
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guoxiaolongzte/spark SPARK-23270
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20437
----
commit 41148c605ddf48c155fc03611bca03af9d4e25a3
Author: guoxiaolong <guo.xiaolong1@...>
Date: 2018-01-30T11:30:49Z
[SPARK-23270][Streaming][WEB-UI]FileInputDStream Streaming UI 's records
should not be set to the default value of 0, it should be the total number of
rows of new files.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
