Hi,
I seem to have figured out why the metric is not in the web UI for the
query, but wish I knew how to explain it for any metric and operator.
It seems that numOutputRows metric won't be displayed in web UI when a
query uses no Spark jobs.
val names = Seq("Jacek", "Agata").toDF("name")
// no numOutputRows metric in web UI
names.show
// The query gives numOutputRows metric in web UI's Details for Query (SQL
tab)
scala> names.groupBy(length($"name")).count.show
That must be somewhat generic and I think has nothing to do with
LocalTableScanExec. Could anyone explain it in more detail? I'd appreciate.
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Wed, Nov 15, 2017 at 10:14 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> I've been playing with LocalTableScanExec and noticed that it
> defines numOutputRows metric, but I couldn't find it in the diagram in web
> UI's Details for Query in SQL tab. Why?
>
> scala> spark.version
> res1: String = 2.3.0-SNAPSHOT
>
> scala> val hello = udf { s: String => s"Hello $s" }
> hello: org.apache.spark.sql.expressions.UserDefinedFunction =
> UserDefinedFunction(<function1>,StringType,Some(List(StringType)))
>
> scala> Seq("Jacek").toDF("name").select(hello($"name")).show
> +-----------+
> | UDF(name)|
> +-----------+
> |Hello Jacek|
> +-----------+
>
> http://localhost:4040/SQL/execution/?id=0 shows no metrics for
> LocalTableScan. Is this intended?
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> Spark Structured Streaming https://bit.ly/spark-structured-streaming
> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
