Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21721#discussion_r206374910
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala ---
@@ -163,7 +163,8 @@ class SourceProgress protected[sql](
val endOffset: String,
val numInputRows: Long,
val inputRowsPerSecond: Double,
- val processedRowsPerSecond: Double) extends Serializable {
+ val processedRowsPerSecond: Double,
+ val customMetrics: Option[JValue] = None) extends Serializable {
--- End diff --
Wait .. this is an exposed API, right? I guess this is exposed to Java API
too (for instance `query.lastProgress().sources()`)? In that case, we should
avoid Scala's Option and `org.json4s.*`. If this is supposed to be hidden here,
I think we should better find a way to hide this with package level access
modifier.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
