[GitHub] [spark] maropu opened a new pull request #26623: [SPARK-29983][SQL] Add an independent config for an optional INTERVAL clause
maropu opened a new pull request #26623: [SPARK-29983][SQL] Add an independent config for an optional INTERVAL clause URL: https://github.com/apache/spark/pull/26623 ### What changes were proposed in this pull request? This pr is to add a new config only for an optional INTERVAL clause. In the master, this feature is enabled when `spark.sql.ansi.enabled`=true. This pr proposes to split off the optional interval flag from `spark.sql.ansi.enabled`. This comes from the @cloud-fan suggestion: https://github.com/apache/spark/pull/26584#discussion_r347939499 ### Why are the changes needed? An optional interval is defined in the ANSI/SQL standard, but some ANSI/SQL-compatible databases don't follow this feature, e.g., PosgreSQL. So, it is useful to have an independent config for the feature. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? No. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348937229
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
##
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.ui
+
+import java.{util => ju}
+import java.lang.{Long => JLong}
+import java.text.SimpleDateFormat
+import java.util.UUID
+import javax.servlet.http.HttpServletRequest
+
+import scala.collection.JavaConverters._
+import scala.xml.{Node, Unparsed}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.util.DateTimeUtils.getTimeZone
+import org.apache.spark.sql.execution.streaming.{QuerySummary,
StreamQueryStore}
+import org.apache.spark.sql.execution.ui.SQLTab
+import org.apache.spark.sql.streaming.StreamingQuery
+import org.apache.spark.sql.streaming.ui.UIUtils._
+import org.apache.spark.ui.{GraphUIData, JsCollector, UIUtils => SparkUIUtils,
WebUIPage}
+
+class StreamingQueryStatisticsPage(
+parent: SQLTab,
+store: Option[StreamQueryStore])
+ extends WebUIPage("streaming/statistics") with Logging {
+ val df = new SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'")
+ df.setTimeZone(getTimeZone("UTC"))
+
+ def generateLoadResources(request: HttpServletRequest): Seq[Node] = {
+// scalastyle:off
+
+
+
+// scalastyle:on
+ }
+
+ override def render(request: HttpServletRequest): Seq[Node] = {
+val parameterId = request.getParameter("id")
+require(parameterId != null && parameterId.nonEmpty, "Missing id
parameter")
+
+val (query, timeSinceStart) = if (store.nonEmpty) {
+ store.get.allStreamQueries.find { case (query, _) =>
+query.runId.equals(UUID.fromString(parameterId))
+ }.getOrElse(throw new Exception(s"Can not find streaming query
$parameterId"))
+} else {
+ throw new Exception(s"Can not find streaming query $parameterId")
+}
+
+val resources = generateLoadResources(request)
+val basicInfo = generateBasicInfo(query, timeSinceStart)
+val content =
+ store.synchronized { // make sure all parts in this page are consistent
+resources ++
+ basicInfo ++
+ generateStatTable(query)
+ }
+SparkUIUtils.headerSparkPage(request, "Streaming Query Statistics",
content, parent)
+ }
+
+ def generateTimeMap(times: Seq[Long]): Seq[Node] = {
+val js = "var timeFormat = {};\n" + times.map { time =>
+ val formattedTime =
+SparkUIUtils.formatBatchTime(time, 1, showMMSS = false)
+ s"timeFormat[$time] = '$formattedTime';"
+}.mkString("\n")
+
+{Unparsed(js)}
+ }
+
+ def generateVar(values: Array[(Long, ju.Map[String, JLong])]): Seq[Node] = {
+val js = "var timeToValues = {};\n" + values.map { case (x, y) =>
+ val s = y.asScala.toSeq.sortBy(_._1).map(e => s${e._2.toDouble})
+.mkString("[", ",", "]")
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348937202
##
File path: core/src/main/scala/org/apache/spark/ui/GraphUIData.scala
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import java.{util => ju}
+import java.lang.{Long => JLong}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+import scala.xml.{Node, Unparsed}
+
+/**
+ * A helper class to generate JavaScript and HTML for both timeline and
histogram graphs.
+ *
+ * @param timelineDivId the timeline `id` used in the html `div` tag
+ * @param histogramDivId the timeline `id` used in the html `div` tag
+ * @param data the data for the graph
+ * @param minX the min value of X axis
+ * @param maxX the max value of X axis
+ * @param minY the min value of Y axis
+ * @param maxY the max value of Y axis
+ * @param unitY the unit of Y axis
+ * @param batchInterval if `batchInterval` is not None, we will draw a line
for `batchInterval` in
+ * the graph
+ */
+private[spark] class GraphUIData(
+timelineDivId: String,
+histogramDivId: String,
+data: Seq[(Long, Double)],
+minX: Long,
+maxX: Long,
+minY: Double,
+maxY: Double,
+unitY: String,
+batchInterval: Option[Double] = None) {
+
+ private var dataJavaScriptName: String = _
+
+ def generateDataJs(jsCollector: JsCollector): Unit = {
+val jsForData = data.map { case (x, y) =>
+ s"""{"x": $x, "y": $y}"""
+}.mkString("[", ",", "]")
+dataJavaScriptName = jsCollector.nextVariableName
+jsCollector.addPreparedStatement(s"var $dataJavaScriptName = $jsForData;")
+ }
+
+ def generateTimelineHtml(jsCollector: JsCollector): Seq[Node] = {
+jsCollector.addPreparedStatement(s"registerTimeline($minY, $maxY);")
+if (batchInterval.isDefined) {
+ jsCollector.addStatement(
+"drawTimeline(" +
+ s"'#$timelineDivId', $dataJavaScriptName, $minX, $maxX, $minY,
$maxY, '$unitY'," +
+ s" ${batchInterval.get}" +
+ ");")
+} else {
+ jsCollector.addStatement(
+s"drawTimeline('#$timelineDivId', $dataJavaScriptName, $minX, $maxX,
$minY, $maxY," +
+ s" '$unitY');")
+}
+
+ }
+
+ def generateHistogramHtml(jsCollector: JsCollector): Seq[Node] = {
+val histogramData = s"$dataJavaScriptName.map(function(d) { return d.y; })"
+jsCollector.addPreparedStatement(s"registerHistogram($histogramData,
$minY, $maxY);")
+if (batchInterval.isDefined) {
+ jsCollector.addStatement(
+"drawHistogram(" +
+ s"'#$histogramDivId', $histogramData, $minY, $maxY, '$unitY',
${batchInterval.get}" +
+ ");")
+} else {
+ jsCollector.addStatement(
+s"drawHistogram('#$histogramDivId', $histogramData, $minY, $maxY,
'$unitY');")
+}
+
+ }
+
+ def generateAreaStackHtmlWithData(
+ jsCollector: JsCollector,
+ values: Array[(Long, ju.Map[String, JLong])],
+ operationLabels: Seq[String]): Seq[Node] = {
+val jsForData = values.map { case (x, y) =>
+ val s = y.asScala.toSeq.sortBy(_._1).map(e => s${e._1}":
"${e._2.toDouble})
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink
HeartSaVioR edited a comment on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink URL: https://github.com/apache/spark/pull/26590#issuecomment-556964599 @zsxwing Ah OK got it. That's a good point - reading files in FileStreamSink output directory without metadata information is unsafe anyway. Btw, actually I and @gaborgsomogyi considered about edge-cases which the query reads `sub-directory(-ies)` or `ancestor with recursive option` of FileStreamSink output directory, because the actual impact here is a kind of "side-effect" which "affects" other queries. It might be less problematic that the query will read the directory "incorrectly" and incorrect output will come up. The thing is, the query will also mess up the output directory as well since processed files will be cleaned up, which will also lead the files and metadata be out of sync and fail other queries as well. So I feel we still have to make a decision with consideration of possible side-effect; 1) try our best to prevent all known cases with (high?) costs, 2) consider these edge-cases as bad input and we don't care at all (maybe we could document it instead.) What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink
HeartSaVioR edited a comment on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink URL: https://github.com/apache/spark/pull/26590#issuecomment-556964599 @zsxwing Ah OK got it. That's a good point - reading files in FileStreamSink output directory without metadata information is unsafe anyway. Btw, actually I and @gaborgsomogyi considered about edge-cases which the query reads `sub-directory(-ies)` or `ancestor with recursive option` of FileStreamSink output directory, because the actual impact here is a kind of "side-effect" which "affects" other queries. It might be less problematic that the query will read the directory "incorrectly" and incorrect output will come up. The thing is, the query will also mess up the output directory as well since processed files will be cleaned up, which will also lead the files and metadata be out of sync and let other queries fail as well. So I feel we still have to make a decision with consideration of possible side-effect; 1) try our best to prevent all known cases with (high?) costs, 2) consider these edge-cases as bad input and we don't care at all (maybe we could document it instead.) What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink
HeartSaVioR commented on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink URL: https://github.com/apache/spark/pull/26590#issuecomment-556964599 @zsxwing Ah OK got it. That's a good point - reading files in FileStreamSink output directory without metadata information is unsafe anyway. Btw, actually I and @gaborgsomogyi considered about edge-cases which the query reads `sub-directory(-ies)` or `ancestor with recursive option` of FileStreamSink output directory, because the actual impact here is a kind of "side-effect" which "affects" other queries. It might be less problematic that the query will read the directory "incorrectly" and incorrect output will come up. The thing is, the query will also mess up the output directory as well since processed files will be cleaned up, which will also break other queries as well. So I feel we still have to make a decision with consideration of possible side-effect; 1) try our best to prevent all known cases with (high?) costs, 2) consider these edge-cases as bad input and we don't care at all (maybe we could document it instead.) What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
beliefer commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556962778 @HyukjinKwon @jiangxb1987 Thanks for your review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556961994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114202/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #26543: [SPARK-29911][SQL] Uncache cached tables when session closed
wangyum commented on a change in pull request #26543: [SPARK-29911][SQL]
Uncache cached tables when session closed
URL: https://github.com/apache/spark/pull/26543#discussion_r348934276
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
##
@@ -255,6 +254,29 @@ class ThriftServerQueryTestSuite extends
SQLQueryTestSuite {
}
}
+ test("SPARK-29911: Uncache cached tables when session closed") {
Review comment:
Could we move this test to a new test suite because it is used to test these
files:
https://github.com/apache/spark/tree/master/sql/core/src/test/resources/sql-tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully
AmplabJenkins removed a comment on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully URL: https://github.com/apache/spark/pull/26610#issuecomment-556961684 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114197/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556961989 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556961994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114202/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully
AmplabJenkins removed a comment on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully URL: https://github.com/apache/spark/pull/26610#issuecomment-556961679 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556961989 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
SparkQA removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556921021 **[Test build #114202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114202/testReport)** for PR 26619 at commit [`c9a05a5`](https://github.com/apache/spark/commit/c9a05a5f251a6e6bcc36da89e2b51bc608715ffe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
SparkQA commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556961641 **[Test build #114202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114202/testReport)** for PR 26619 at commit [`c9a05a5`](https://github.com/apache/spark/commit/c9a05a5f251a6e6bcc36da89e2b51bc608715ffe). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully
AmplabJenkins commented on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully URL: https://github.com/apache/spark/pull/26610#issuecomment-556961679 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully
AmplabJenkins commented on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully URL: https://github.com/apache/spark/pull/26610#issuecomment-556961684 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114197/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully
SparkQA removed a comment on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully URL: https://github.com/apache/spark/pull/26610#issuecomment-556911987 **[Test build #114197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114197/testReport)** for PR 26610 at commit [`50baee9`](https://github.com/apache/spark/commit/50baee913eb487e03155aa0cfa44b2870b847460). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully
SparkQA commented on issue #26610: [SPARK-29973][SS] Make `processedRowsPerSecond` calculated more accurately and meaningfully URL: https://github.com/apache/spark/pull/26610#issuecomment-556961261 **[Test build #114197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114197/testReport)** for PR 26610 at commit [`50baee9`](https://github.com/apache/spark/commit/50baee913eb487e03155aa0cfa44b2870b847460). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348932678
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
##
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.ui
+
+import java.{util => ju}
+import java.lang.{Long => JLong}
+import java.text.SimpleDateFormat
+import java.util.UUID
+import javax.servlet.http.HttpServletRequest
+
+import scala.collection.JavaConverters._
+import scala.xml.{Node, Unparsed}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.util.DateTimeUtils.getTimeZone
+import org.apache.spark.sql.execution.streaming.{QuerySummary,
StreamQueryStore}
+import org.apache.spark.sql.execution.ui.SQLTab
+import org.apache.spark.sql.streaming.StreamingQuery
+import org.apache.spark.sql.streaming.ui.UIUtils._
+import org.apache.spark.ui.{GraphUIData, JsCollector, UIUtils => SparkUIUtils,
WebUIPage}
+
+class StreamingQueryStatisticsPage(
+parent: SQLTab,
+store: Option[StreamQueryStore])
+ extends WebUIPage("streaming/statistics") with Logging {
+ val df = new SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'")
+ df.setTimeZone(getTimeZone("UTC"))
+
+ def generateLoadResources(request: HttpServletRequest): Seq[Node] = {
+// scalastyle:off
Review comment:
use `scalastyle:off` to disable `line length 100` check
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348932155
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
##
@@ -1102,6 +1102,84 @@ class StreamingQuerySuite extends StreamTest with
BeforeAndAfter with Logging wi
}
}
+ test("stream query summary with mixed batch and streaming V1 sources") {
+val streamingTriggerDF = spark.createDataset(1 to 10).toDF
+val streamingInputDF =
createSingleTriggerStreamingDF(streamingTriggerDF).toDF("value")
+val staticInputDF = spark.createDataFrame(Seq(1 -> "1", 2 ->
"2")).toDF("value", "anotherValue")
+
+// Trigger input has 10 rows, static input has 2 rows,
+// therefore after the first trigger, the calculated input rows should be
10
+val querySummary = getQuerySummary(streamingInputDF.join(staticInputDF,
"value"))
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556958747 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19065/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556958744 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556958747 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19065/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556958744 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348929717 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ordering.scala ## @@ -19,18 +19,24 @@ package org.apache.spark.sql.catalyst.expressions import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.BindReferences.bindReferences +import org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering import org.apache.spark.sql.types._ +/** + * A base class for generated/interpreted row ordering. + */ +abstract class BaseOrdering extends Ordering[InternalRow] Review comment: Ah, I see. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
SparkQA commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556958294 **[Test build #114214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114214/testReport)** for PR 26615 at commit [`fb0173e`](https://github.com/apache/spark/commit/fb0173e96f9e5cdb02f6ad915d7f02e9a2ba6bee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556957618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114196/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556958067 It seems we need the default impl. for compare in `BaseOrdering`. That seems to be a janino restriction. I'm not exactly sure about why though, IIRC I saw the similar behaviour before This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556957609 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556957618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114196/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556957609 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
SparkQA removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556911991 **[Test build #114196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114196/testReport)** for PR 26619 at commit [`1e0f23d`](https://github.com/apache/spark/commit/1e0f23de30984b26e8a911a560d25b9426300d47). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
SparkQA commented on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556957009 **[Test build #114196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114196/testReport)** for PR 26619 at commit [`1e0f23d`](https://github.com/apache/spark/commit/1e0f23de30984b26e8a911a560d25b9426300d47). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes
AmplabJenkins removed a comment on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes URL: https://github.com/apache/spark/pull/26618#issuecomment-556956197 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes
AmplabJenkins removed a comment on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes URL: https://github.com/apache/spark/pull/26618#issuecomment-556956205 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114191/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes
AmplabJenkins commented on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes URL: https://github.com/apache/spark/pull/26618#issuecomment-556956205 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114191/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes
AmplabJenkins commented on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes URL: https://github.com/apache/spark/pull/26618#issuecomment-556956197 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes
SparkQA removed a comment on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes URL: https://github.com/apache/spark/pull/26618#issuecomment-556901686 **[Test build #114191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114191/testReport)** for PR 26618 at commit [`dc95213`](https://github.com/apache/spark/commit/dc95213cc63dcab5f639a5efc5a555434d05f5cc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes
SparkQA commented on issue #26618: [SPARK-29980][SQL] Whitespaces handling for Cast and BinaryOperation between StringType and NumericTypes URL: https://github.com/apache/spark/pull/26618#issuecomment-556955627 **[Test build #114191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114191/testReport)** for PR 26618 at commit [`dc95213`](https://github.com/apache/spark/commit/dc95213cc63dcab5f639a5efc5a555434d05f5cc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive
gengliangwang commented on issue #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive URL: https://github.com/apache/spark/pull/26214#issuecomment-556954954 Looks reasonable to me. Please continue the work and fix the test failures This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556954064 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556954072 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19064/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556954072 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19064/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556954064 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
SparkQA commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556953585 **[Test build #114213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114213/testReport)** for PR 26622 at commit [`cce5a55`](https://github.com/apache/spark/commit/cce5a55579636cd57a1a8c8415af42ad9f91e241). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test]
Cheap UTF8String Trim
URL: https://github.com/apache/spark/pull/26622#discussion_r348924919
##
File path:
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##
@@ -553,6 +553,29 @@ public UTF8String trim() {
return copyUTF8String(s, e);
}
+ /**
+ * Trims space characters (ASCII 32) from both ends of this string.
+ *
+ * @return this string with no spaces at the start or end
+ */
+ public UTF8String nonCopyTrim() {
Review comment:
I will check this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348924861
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
##
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.ui
+
+import java.{util => ju}
+import java.lang.{Long => JLong}
+import java.text.SimpleDateFormat
+import java.util.UUID
+import javax.servlet.http.HttpServletRequest
+
+import scala.collection.JavaConverters._
+import scala.xml.{Node, Unparsed}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.util.DateTimeUtils.getTimeZone
+import org.apache.spark.sql.execution.streaming.{QuerySummary,
StreamQueryStore}
+import org.apache.spark.sql.execution.ui.SQLTab
+import org.apache.spark.sql.streaming.StreamingQuery
+import org.apache.spark.sql.streaming.ui.UIUtils._
+import org.apache.spark.ui.{GraphUIData, JsCollector, UIUtils => SparkUIUtils,
WebUIPage}
+
+class StreamingQueryStatisticsPage(
+parent: SQLTab,
+store: Option[StreamQueryStore])
+ extends WebUIPage("streaming/statistics") with Logging {
+ val df = new SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'")
+ df.setTimeZone(getTimeZone("UTC"))
+
+ def generateLoadResources(request: HttpServletRequest): Seq[Node] = {
+// scalastyle:off
+
+
+
+// scalastyle:on
+ }
+
+ override def render(request: HttpServletRequest): Seq[Node] = {
+val parameterId = request.getParameter("id")
+require(parameterId != null && parameterId.nonEmpty, "Missing id
parameter")
+
+val (query, timeSinceStart) = if (store.nonEmpty) {
+ store.get.allStreamQueries.find { case (query, _) =>
+query.runId.equals(UUID.fromString(parameterId))
+ }.getOrElse(throw new Exception(s"Can not find streaming query
$parameterId"))
+} else {
+ throw new Exception(s"Can not find streaming query $parameterId")
+}
+
+val resources = generateLoadResources(request)
+val basicInfo = generateBasicInfo(query, timeSinceStart)
+val content =
+ store.synchronized { // make sure all parts in this page are consistent
Review comment:
removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#discussion_r348924667 ## File path: sql/core/benchmarks/CastBenchmark-results.txt ## @@ -6,7 +6,7 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1 Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz Benchmark trim the string:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative -cast(str as int) as c_int 7105 8190 945 1.2 867.3 1.0X -cast(str as long) as c_long7520 8670 1629 1.1 918.0 0.9X +cast(str as int) as c_int 6263 8132 NaN 1.3 764.6 1.0X +cast(str as long) as c_long8199 9737 NaN 1.01000.9 0.8X Review comment: cardinality * 2, (-) is trim (+) is non-copy trim, the cost for `copyMemory` an int or long value is trial, I guess. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #26500: [SPARK-29874][SQL]Optimize Dataset.isEmpty()
AngersZh commented on issue #26500: [SPARK-29874][SQL]Optimize
Dataset.isEmpty()
URL: https://github.com/apache/spark/pull/26500#issuecomment-556952943
```
test("benchmark of empty") {
var start = System.currentTimeMillis()
var isEmpty = spark.range(1000)
.repartition(100)
.limit(1)
.groupBy()
.count()
.queryExecution.executedPlan.executeCollect().head.getLong(0) == 0
println(isEmpty)
var end = System.currentTimeMillis()
// scalastyle:off
println(s"duration = ${end - start}")
start = System.currentTimeMillis()
isEmpty = spark.range(1000)
.repartition(100)
.select()
.queryExecution.executedPlan.executeTake(1) == 0
println(isEmpty)
end = System.currentTimeMillis()
// scalastyle:off
println(s"duration = ${end - start}")
}
Result
false
duration = 7248
false
duration = 1449
```
@cloud-fan @maropu @srowen
The test case is simple but can mimic the behavior before and after the API
change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
AmplabJenkins commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556952278 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114199/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
HyukjinKwon commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556952411 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
AmplabJenkins commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556952271 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
AmplabJenkins removed a comment on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556952278 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114199/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
HyukjinKwon closed pull request #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
AmplabJenkins removed a comment on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556952271 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
SparkQA removed a comment on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556913586 **[Test build #114199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114199/testReport)** for PR 26510 at commit [`ba63574`](https://github.com/apache/spark/commit/ba635743c72f1ea3155f52ccd362c8873763efb1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port
SparkQA commented on issue #26510: [SPARK-29885][PYTHON][CORE] Improve the exception message when reading the daemon port URL: https://github.com/apache/spark/pull/26510#issuecomment-556951664 **[Test build #114199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114199/testReport)** for PR 26510 at commit [`ba63574`](https://github.com/apache/spark/commit/ba635743c72f1ea3155f52ccd362c8873763efb1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#discussion_r348923389 ## File path: sql/core/benchmarks/CastBenchmark-results.txt ## @@ -6,7 +6,7 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1 Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz Benchmark trim the string:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative -cast(str as int) as c_int 2478 3669 1046 1.7 604.9 1.0X -cast(str as long) as c_long1439 1548 94 2.8 351.4 1.7X +cast(str as int) as c_int 3169 3530 610 1.3 773.6 1.0X +cast(str as long) as c_long1812 1881 60 2.3 442.4 1.7X Review comment: @cloud-fan this is master branch to original trim result. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
cloud-fan commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348922696 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala ## @@ -93,7 +93,7 @@ case class SortAggregateExec( initialInputBufferOffset, resultExpressions, (expressions, inputSchema) => -newMutableProjection(expressions, inputSchema, subexpressionEliminationEnabled), +MutableProjection.create(expressions, inputSchema), Review comment: ah i see, thanks for confirmation! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
cloud-fan commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348922480 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ordering.scala ## @@ -19,18 +19,24 @@ package org.apache.spark.sql.catalyst.expressions import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.BindReferences.bindReferences +import org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering import org.apache.spark.sql.types._ +/** + * A base class for generated/interpreted row ordering. + */ +abstract class BaseOrdering extends Ordering[InternalRow] Review comment: ah i see, to work around type erasure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556950586 oh, I got the weird failures and that seems to come from the change https://github.com/apache/spark/pull/26615#discussion_r348907633. I'll check now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-55695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114208/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki commented on issue #26616: [SPARK-25392][Webui]Prevent error page when accessing pools page from history server
shahidki commented on issue #26616: [SPARK-25392][Webui]Prevent error page when accessing pools page from history server URL: https://github.com/apache/spark/pull/26616#issuecomment-556950382 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
cloud-fan commented on a change in pull request #26622:
[WIP][SPARK-28023][Test] Cheap UTF8String Trim
URL: https://github.com/apache/spark/pull/26622#discussion_r348922146
##
File path:
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##
@@ -553,6 +553,29 @@ public UTF8String trim() {
return copyUTF8String(s, e);
}
+ /**
+ * Trims space characters (ASCII 32) from both ends of this string.
+ *
+ * @return this string with no spaces at the start or end
+ */
+ public UTF8String nonCopyTrim() {
Review comment:
another way is to embed the trim logic into `toInt`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556949989 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
SparkQA removed a comment on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556940730 **[Test build #114208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114208/testReport)** for PR 26615 at commit [`036ff87`](https://github.com/apache/spark/commit/036ff87cc13af1b21dc046921c06de3d9f150a72). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink
zsxwing commented on issue #26590: [SPARK-29953][SS] Don't clean up source files for FileStreamSource if the files belong to the output of FileStreamSink URL: https://github.com/apache/spark/pull/26590#issuecomment-556950146 @HeartSaVioR I think we can simply detect whether we are using `MetadataLogFileIndex` here: https://github.com/apache/spark/blob/ba2bc4b0e0eea0c1b6732a18cb20e61e4f693156/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala#L205 We don't need to do such complicated check because for cases you are checking, we won't go through `MetadataLogFileIndex` so the result is not correct anyway and the user should not use such path. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
SparkQA commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556949970 **[Test build #114208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114208/testReport)** for PR 26615 at commit [`036ff87`](https://github.com/apache/spark/commit/036ff87cc13af1b21dc046921c06de3d9f150a72). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-55695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114208/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348922091
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.ui
+
+import java.text.SimpleDateFormat
+import java.util.TimeZone
+import javax.servlet.http.HttpServletRequest
+
+import scala.xml.Node
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.{QuerySummary,
StreamQueryStore}
+import org.apache.spark.sql.execution.ui.SQLTab
+import org.apache.spark.sql.streaming.StreamingQuery
+import org.apache.spark.sql.streaming.ui.UIUtils._
+import org.apache.spark.ui.{UIUtils => SparkUIUtils, WebUIPage}
+
+class StreamingQueryPage(parent: SQLTab, store: Option[StreamQueryStore])
+ extends WebUIPage("streaming") with Logging {
+ val df = new SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'")
+ df.setTimeZone(TimeZone.getDefault)
+
+ override def render(request: HttpServletRequest): Seq[Node] = {
+val content = store.synchronized {
+ generateStreamingQueryTable(request)
+}
+SparkUIUtils.headerSparkPage(request, "Streaming Query", content, parent)
+ }
+
+ def generateDataRow(request: HttpServletRequest, isActive: Boolean)
+(streamQuery: (StreamingQuery, Long)): Seq[Node] = {
+
+val (query, timeSinceStart) = streamQuery
+def details(detail: Any): Seq[Node] = {
+ val s = detail.asInstanceOf[String]
+ val isMultiline = s.indexOf('\n') >= 0
+ val summary = StringEscapeUtils.escapeHtml4(
+if (isMultiline) {
+ s.substring(0, s.indexOf('\n'))
+} else {
+ s
+})
+ val details = if (isMultiline) {
+// scalastyle:off
+
+ +details
+ ++
+
+{s}
+
+// scalastyle:on
+ } else {
+""
+ }
+ {summary}{details}
+}
+
+val statisticsLink = "%s/%s/streaming/statistics?id=%s"
+ .format(SparkUIUtils.prependBaseUri(request, parent.basePath),
parent.prefix, query.runId)
+
+val name = if (query.name == null || query.name.isEmpty) {
+ "null"
+} else {
+ query.name
+}
+
+val status = if (isActive) {
+ "RUNNING"
+} else {
+ query.exception.map(_.message) match {
Review comment:
nice
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
AmplabJenkins commented on issue #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#issuecomment-556949989 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus
AmplabJenkins removed a comment on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus URL: https://github.com/apache/spark/pull/26611#issuecomment-556949508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19063/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus
AmplabJenkins removed a comment on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus URL: https://github.com/apache/spark/pull/26611#issuecomment-556949497 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
cloud-fan commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#discussion_r348921668 ## File path: sql/core/benchmarks/CastBenchmark-results.txt ## @@ -0,0 +1,12 @@ + +Benchmark trim the string + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1 +Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz +Benchmark trim the string:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative + +cast(str as int) as c_int 2208 4341 1848 1.9 539.0 1.0X Review comment: what's the result of master branch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus
AmplabJenkins commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus URL: https://github.com/apache/spark/pull/26611#issuecomment-556949497 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus
AmplabJenkins commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus URL: https://github.com/apache/spark/pull/26611#issuecomment-556949508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19063/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
SparkQA commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556949073 **[Test build #114211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114211/testReport)** for PR 26622 at commit [`4fa535b`](https://github.com/apache/spark/commit/4fa535ba2e65bc514c1a89ddd600d7cfc1a54791). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus
SparkQA commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus URL: https://github.com/apache/spark/pull/26611#issuecomment-556949044 **[Test build #114212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114212/testReport)** for PR 26611 at commit [`6920dfc`](https://github.com/apache/spark/commit/6920dfcc2e26f6ed53e8e3afa024f7aee28b6847). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
yaooqinn commented on a change in pull request #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#discussion_r348920168 ## File path: sql/core/benchmarks/CastBenchmark-results.txt ## @@ -6,7 +6,7 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1 Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz Benchmark trim the string:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative -cast(str as int) as c_int 3169 3530 610 1.3 773.6 1.0X -cast(str as long) as c_long1812 1881 60 2.3 442.4 1.7X +cast(str as int) as c_int 2208 4341 1848 1.9 539.0 1.0X +cast(str as long) as c_long2039 3450 2146 2.0 497.8 1.1X Review comment: @cloud-fan the result seems not as we expected, I'd increase the cardinality and do another test round. Can you help me to see if I missed something? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348919850 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala ## @@ -93,7 +93,7 @@ case class SortAggregateExec( initialInputBufferOffset, resultExpressions, (expressions, inputSchema) => -newMutableProjection(expressions, inputSchema, subexpressionEliminationEnabled), +MutableProjection.create(expressions, inputSchema), Review comment: yea, I think so. https://github.com/apache/spark/blob/affaefe1f3e623e6ae2042648ab78ee7b89d1ed6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L84 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348919782
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.ui
+
+import java.text.SimpleDateFormat
+import java.util.TimeZone
+import javax.servlet.http.HttpServletRequest
+
+import scala.xml.Node
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.{QuerySummary,
StreamQueryStore}
+import org.apache.spark.sql.execution.ui.SQLTab
+import org.apache.spark.sql.streaming.StreamingQuery
+import org.apache.spark.sql.streaming.ui.UIUtils._
+import org.apache.spark.ui.{UIUtils => SparkUIUtils, WebUIPage}
+
+class StreamingQueryPage(parent: SQLTab, store: Option[StreamQueryStore])
+ extends WebUIPage("streaming") with Logging {
+ val df = new SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'")
+ df.setTimeZone(TimeZone.getDefault)
+
+ override def render(request: HttpServletRequest): Seq[Node] = {
+val content = store.synchronized {
+ generateStreamingQueryTable(request)
+}
+SparkUIUtils.headerSparkPage(request, "Streaming Query", content, parent)
+ }
+
+ def generateDataRow(request: HttpServletRequest, isActive: Boolean)
+(streamQuery: (StreamingQuery, Long)): Seq[Node] = {
+
+val (query, timeSinceStart) = streamQuery
+def details(detail: Any): Seq[Node] = {
+ val s = detail.asInstanceOf[String]
+ val isMultiline = s.indexOf('\n') >= 0
+ val summary = StringEscapeUtils.escapeHtml4(
+if (isMultiline) {
+ s.substring(0, s.indexOf('\n'))
+} else {
+ s
+})
+ val details = if (isMultiline) {
+// scalastyle:off
+
+ +details
+ ++
+
+{s}
+
+// scalastyle:on
+ } else {
+""
+ }
+ {summary}{details}
+}
+
+val statisticsLink = "%s/%s/streaming/statistics?id=%s"
+ .format(SparkUIUtils.prependBaseUri(request, parent.basePath),
parent.prefix, query.runId)
+
+val name = if (query.name == null || query.name.isEmpty) {
+ "null"
Review comment:
Make sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348919850 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala ## @@ -93,7 +93,7 @@ case class SortAggregateExec( initialInputBufferOffset, resultExpressions, (expressions, inputSchema) => -newMutableProjection(expressions, inputSchema, subexpressionEliminationEnabled), +MutableProjection.create(expressions, inputSchema), Review comment: yea, I think so. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556947361 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556947364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19062/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556947364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19062/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556947361 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
viirya commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348919072 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala ## @@ -93,7 +93,7 @@ case class SortAggregateExec( initialInputBufferOffset, resultExpressions, (expressions, inputSchema) => -newMutableProjection(expressions, inputSchema, subexpressionEliminationEnabled), +MutableProjection.create(expressions, inputSchema), Review comment: MutableProjection now directly gets the config from SQLConf.get.subexpressionEliminationEnabled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348918892
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.ui
+
+import java.text.SimpleDateFormat
+import java.util.TimeZone
+import javax.servlet.http.HttpServletRequest
+
+import scala.xml.Node
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.{QuerySummary,
StreamQueryStore}
+import org.apache.spark.sql.execution.ui.SQLTab
+import org.apache.spark.sql.streaming.StreamingQuery
+import org.apache.spark.sql.streaming.ui.UIUtils._
+import org.apache.spark.ui.{UIUtils => SparkUIUtils, WebUIPage}
+
+class StreamingQueryPage(parent: SQLTab, store: Option[StreamQueryStore])
+ extends WebUIPage("streaming") with Logging {
+ val df = new SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS'Z'")
+ df.setTimeZone(TimeZone.getDefault)
+
+ override def render(request: HttpServletRequest): Seq[Node] = {
+val content = store.synchronized {
Review comment:
Looks like redundant, removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI] Init structured streaming ui
uncleGen commented on a change in pull request #26201: [SPARK-29543][SS][UI]
Init structured streaming ui
URL: https://github.com/apache/spark/pull/26201#discussion_r348918326
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
##
@@ -393,7 +392,7 @@ class StreamingQueryManager private[sql] (sparkSession:
SparkSession) extends Lo
private def unregisterTerminatedStream(terminatedQueryId: UUID): Unit = {
activeQueriesLock.synchronized {
// remove from shared state only if the streaming query manager also
matches
-
sparkSession.sharedState.activeStreamingQueries.remove(terminatedQueryId, this)
+ sparkSession.sharedState.streamQueryStore.terminate(terminatedQueryId)
Review comment:
`StreamQueryStore` contained the logic of
`sparkSession.sharedState.activeStreamingQueries`, and guaranteed the
isolation. Besides, `StreamQueryStore` saves those inactive (completed or
failed) queries
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556945203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19061/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins removed a comment on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556945199 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556945203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19061/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348917715 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala ## @@ -93,7 +93,7 @@ case class SortAggregateExec( initialInputBufferOffset, resultExpressions, (expressions, inputSchema) => -newMutableProjection(expressions, inputSchema, subexpressionEliminationEnabled), +MutableProjection.create(expressions, inputSchema), Review comment: But, it seems `newMutableProjection` always ignores the parameter? https://github.com/apache/spark/pull/26615/files#diff-b9f96d092fb3fea76bcf75e016799678L468 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
AmplabJenkins commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556945199 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim
SparkQA commented on issue #26622: [WIP][SPARK-28023][Test] Cheap UTF8String Trim URL: https://github.com/apache/spark/pull/26622#issuecomment-556944801 **[Test build #114210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114210/testReport)** for PR 26622 at commit [`ee94f98`](https://github.com/apache/spark/commit/ee94f982de959457dff0d635f5ec7ba1cb29a83a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan
maropu commented on a change in pull request #26615: [SPARK-29977][SQL] Remove newMutableProjection/newOrdering/newNaturalAscendingOrdering from SparkPlan URL: https://github.com/apache/spark/pull/26615#discussion_r348917261 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ordering.scala ## @@ -19,18 +19,24 @@ package org.apache.spark.sql.catalyst.expressions import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.BindReferences.bindReferences +import org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering import org.apache.spark.sql.types._ +/** + * A base class for generated/interpreted row ordering. + */ +abstract class BaseOrdering extends Ordering[InternalRow] Review comment: Honestly, not sure. It seems this base class has been implemented for making it easily accessible from generated code? https://github.com/apache/spark/blob/d555f8fcc964e6e81f2d849de62d06877dca70c5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala#L158 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #26621: [MINOR][INFRA] Add `io` and `net` to GitHub Action Cache
HyukjinKwon closed pull request #26621: [MINOR][INFRA] Add `io` and `net` to GitHub Action Cache URL: https://github.com/apache/spark/pull/26621 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles
AmplabJenkins removed a comment on issue #26619: [WIP][SPARK-29981][BUILD][test-hadoop3.2] Add hive-1.2/2.3 profiles URL: https://github.com/apache/spark/pull/26619#issuecomment-556944198 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114192/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
