sunchao commented on code in PR #37205:
URL: https://github.com/apache/spark/pull/37205#discussion_r926282837
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:
##########
@@ -115,4 +116,14 @@ default ContinuousStream toContinuousStream(String
checkpointLocation) {
default CustomMetric[] supportedCustomMetrics() {
return new CustomMetric[]{};
}
+
+ /**
+ * Return an array of custom metrics which are collected with values at the
driver side only.
Review Comment:
nit: `Returns`
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:
##########
@@ -115,4 +116,14 @@ default ContinuousStream toContinuousStream(String
checkpointLocation) {
default CustomMetric[] supportedCustomMetrics() {
return new CustomMetric[]{};
}
+
+ /**
+ * Return an array of custom metrics which are collected with values at the
driver side only.
+ * Note that these metrics must be included in the supported custom metrics
reported by
+ * `supportedCustomMetrics`.
+ */
+ default CustomTaskMetric[] reportCustomDriverMetrics() {
+ CustomTaskMetric[] NO_METRICS = {};
+ return NO_METRICS;
Review Comment:
seems this is redundant?
```scala
return new CustomTaskMetric[]{};
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala:
##########
@@ -168,6 +168,18 @@ trait DataSourceV2ScanExecBase extends LeafExecNode {
}
}
+ protected def postDriverMetrics(): Unit = {
+ val driveSQLMetrics = scan.reportCustomDriverMetrics().map(
customTaskMetric => {
+ val metric = metrics(customTaskMetric.name())
Review Comment:
hmm what if the driver metric is not in `metrics`? this will throw
`NoSuchElementException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
