Re: [SQL] Why no numOutputRows metric for LocalTableScanExec in webUI?

Hi Ryan, That does make a lot of sense! Thanks for steering me in a right direction. Quoting SQLMetric [1]: > Updates on the driver side must be explicitly posted using SQLMetrics.postDriverMetricUpdates(). Why is LocalTableScanExec not following the "must" requirement? FileSourceScanExec does

Re: [SQL] Why no numOutputRows metric for LocalTableScanExec in webUI?

SQL metrics are collected using SparkListener. If there are no tasks, org.apache.spark.sql.execution.ui.SQLListener cannot collect any metrics. On Thu, Nov 16, 2017 at 1:53 AM, Jacek Laskowski wrote: > Hi, > > I seem to have figured out why the metric is not in the web UI for

Re: [SQL] Why no numOutputRows metric for LocalTableScanExec in webUI?

Hi, I seem to have figured out why the metric is not in the web UI for the query, but wish I knew how to explain it for any metric and operator. It seems that numOutputRows metric won't be displayed in web UI when a query uses no Spark jobs. val names = Seq("Jacek", "Agata").toDF("name") // no

[SQL] Why no numOutputRows metric for LocalTableScanExec in webUI?

Hi, I've been playing with LocalTableScanExec and noticed that it defines numOutputRows metric, but I couldn't find it in the diagram in web UI's Details for Query in SQL tab. Why? scala> spark.version res1: String = 2.3.0-SNAPSHOT scala> val hello = udf { s: String => s"Hello $s" } hello: