[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99539/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23193 **[Test build #99539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99539/testReport)** for PR 23193 at commit [`70c319b`](https://github.com/apache/spark/commit/70c319bdaaac4fc4b8b988a96be6f976a63b41bf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23192 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99536/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23192 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23192 **[Test build #99536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99536/testReport)** for PR 23192 at commit [`c037f4d`](https://github.com/apache/spark/commit/c037f4d2fa2c2844ac992d976b492e14ab9bed11). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23160: [SPARK-26196][WebUI] Total tasks title in the sta...
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23160#discussion_r238055413
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
---
@@ -661,8 +662,8 @@ $(document).ready(function () {
{data : "launchTime", name: "Launch Time", render:
formatDate},
{
data : function (row, type) {
-if (row.duration) {
-return type === 'display' ?
formatDuration(row.duration) : row.duration;
+if (row.taskMetrics &&
row.taskMetrics.executorRunTime) {
+return type === 'display' ?
formatDuration(row.taskMetrics.executorRunTime) :
row.taskMetrics.executorRunTime;
--- End diff --
Ok, that makes sense. However, you also need to update the search filter
method to not search on duration in StagesResource.scala here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala#L213.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23160: [SPARK-26196][WebUI] Total tasks title in the sta...
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23160#discussion_r238055402
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js
---
@@ -661,8 +662,8 @@ $(document).ready(function () {
{data : "launchTime", name: "Launch Time", render:
formatDate},
{
data : function (row, type) {
-if (row.duration) {
-return type === 'display' ?
formatDuration(row.duration) : row.duration;
+if (row.taskMetrics &&
row.taskMetrics.executorRunTime) {
+return type === 'display' ?
formatDuration(row.taskMetrics.executorRunTime) :
row.taskMetrics.executorRunTime;
--- End diff --
Ok, that makes sense. However, you also need to update the search filter
method to not search on duration in StagesResource.scala here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala#L213.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23161 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23161 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23161 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055143
--- Diff: R/pkg/R/functions.R ---
@@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x
= "character"),
column(jc)
})
+setClassUnion("characterOrstructTypeOrColumn", c("character",
"structType", "Column"))
--- End diff --
we should probably try to pull all the setClassUnion in one place. (to
avoid conflict or duplication)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055087
--- Diff: R/pkg/R/functions.R ---
@@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x
= "character"),
column(jc)
})
+setClassUnion("characterOrstructTypeOrColumn", c("character",
"structType", "Column"))
+
#' @details
#' \code{from_json}: Parses a column containing a JSON string into a
Column of \code{structType}
#' with the specified \code{schema} or array of \code{structType} if
\code{as.json.array} is set
#' to \code{TRUE}. If the string is unparseable, the Column will contain
the value NA.
#'
#' @rdname column_collection_functions
#' @param as.json.array indicating if input string is JSON array of
objects or a single object.
-#' @aliases from_json from_json,Column,characterOrstructType-method
+#' @aliases from_json from_json,Column,characterOrstructTypeOrColumn-method
#' @examples
#'
#' \dontrun{
#' df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as
d")
#' df2 <- mutate(df2, d2 = to_json(df2$d, dateFormat = 'dd/MM/'))
#' schema <- structType(structField("date", "string"))
#' head(select(df2, from_json(df2$d2, schema, dateFormat = 'dd/MM/')))
-
#' df2 <- sql("SELECT named_struct('name', 'Bob') as people")
#' df2 <- mutate(df2, people_json = to_json(df2$people))
#' schema <- structType(structField("name", "string"))
#' head(select(df2, from_json(df2$people_json, schema)))
-#' head(select(df2, from_json(df2$people_json, "name STRING")))}
+#' head(select(df2, from_json(df2$people_json, "name STRING")))
+#' head(select(df2, from_json(df2$people_json,
schema_of_json(head(df2)$people_json}
#' @note from_json since 2.2.0
-setMethod("from_json", signature(x = "Column", schema =
"characterOrstructType"),
+setMethod("from_json", signature(x = "Column", schema =
"characterOrstructTypeOrColumn"),
function(x, schema, as.json.array = FALSE, ...) {
if (is.character(schema)) {
- schema <- structType(schema)
+ jschema <- structType(schema)$jobj
+} else if (class(schema) == "structType") {
+ jschema <- schema$jobj
+} else {
+ jschema <- schema@jc
}
if (as.json.array) {
- jschema <-
callJStatic("org.apache.spark.sql.types.DataTypes",
- "createArrayType",
- schema$jobj)
-} else {
- jschema <- schema$jobj
+ # This case is R-specifically different. Unlike Scala and
Python side,
--- End diff --
what if `as.json.array` is TRUE but schema is also set?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055126
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
---
@@ -225,4 +225,10 @@ private[sql] object SQLUtils extends Logging {
}
sparkSession.sessionState.catalog.listTables(db).map(_.table).toArray
}
+
+ def createArrayType(elementType: DataType): ArrayType =
DataTypes.createArrayType(elementType)
--- End diff --
as mentioned before, I kinda have to disagree with you here... I'd prefer
less stuff in r/SQLUtils, and instead, call the scala/jvm method directly from
R...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055173
--- Diff: R/pkg/R/functions.R ---
@@ -202,8 +202,9 @@ NULL
#' \itemize{
#' \item \code{from_json}: a structType object to use as the
schema to use
#' when parsing the JSON string. Since Spark 2.3, the
DDL-formatted string is
-#' also supported for the schema.
-#' \item \code{from_csv}: a DDL-formatted string
+#' also supported for the schema. Since Spark 3.0,
\code{schema_of_json} or
+#' a string literal can also be accepted.
--- End diff --
that's true...? they are the same?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22683: [SPARK-25696] The storage memory displayed on spark Appl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22683 **[Test build #99544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99544/testReport)** for PR 22683 at commit [`5188c54`](https://github.com/apache/spark/commit/5188c54fcf33c24dac341c044f7ffa75c272bf52). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5605/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23160 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23160 **[Test build #99543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99543/testReport)** for PR 23160 at commit [`661566b`](https://github.com/apache/spark/commit/661566bd51efb40176327a58e5bed5dcfe7cacf0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23160 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5604/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23160 Hi @pgandhi999 , There is a small nit there. Recently we fixed a bug related to the "duration" metrics in the tasks table (see https://github.com/apache/spark/pull/23081), but that hasn't reflected here. I have updated the code --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23160 **[Test build #99542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99542/testReport)** for PR 23160 at commit [`71aff34`](https://github.com/apache/spark/commit/71aff3447b4c1a80e168a18202864191df189709). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23190 **[Test build #99541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99541/testReport)** for PR 23190 at commit [`3376524`](https://github.com/apache/spark/commit/33765248d2afdccf4e3cedf96200791ad48ef6be). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5603/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5602/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99540/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23190 **[Test build #99540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99540/testReport)** for PR 23190 at commit [`92eb086`](https://github.com/apache/spark/commit/92eb086c39d71db4dcf7f45e138401f56f709d54). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23190 **[Test build #99540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99540/testReport)** for PR 23190 at commit [`92eb086`](https://github.com/apache/spark/commit/92eb086c39d71db4dcf7f45e138401f56f709d54). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23037 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99530/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23037 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23037 **[Test build #99530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99530/testReport)** for PR 23037 at commit [`5f6aaf8`](https://github.com/apache/spark/commit/5f6aaf83ddaa6ff3da88a3aa5d78bef925f0bf80). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23177: [SPARK-26212][Build][test-maven] Upgrade maven version t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23177 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5601/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23177: [SPARK-26212][Build][test-maven] Upgrade maven version t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23177 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23193 **[Test build #99539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99539/testReport)** for PR 23193 at commit [`70c319b`](https://github.com/apache/spark/commit/70c319bdaaac4fc4b8b988a96be6f976a63b41bf). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5600/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23193: [SPARK-26226][SQL] Track optimization phase for s...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/23193 [SPARK-26226][SQL] Track optimization phase for streaming queries ## What changes were proposed in this pull request? In an earlier PR, we missed measuring the optimization phase time for streaming queries. This patch adds it. ## How was this patch tested? Given this is a debugging feature, and it is very convoluted to add tests to verify the phase is set properly, I am not introducing a streaming specific test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-26226-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23193.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23193 commit 70c319bdaaac4fc4b8b988a96be6f976a63b41bf Author: Reynold Xin Date: 2018-12-01T04:33:21Z SPARK-26226 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23193: [SPARK-26226][SQL] Track optimization phase for streamin...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/23193 cc @gatorsmile @jose-torres --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23177: [SPARK-26212][Build][test-maven] Upgrade maven version t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23177 **[Test build #99538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99538/testReport)** for PR 23177 at commit [`aa25833`](https://github.com/apache/spark/commit/aa258334170cc2ba009603b4547b4184b3736881). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23192 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23192 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5599/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23177: [SPARK-26212][Build][test-maven] Upgrade maven version t...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/23177 Sure, updated. Thanks for letting know them. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23154: [SPARK-26195][SQL] Correct exception messages in some cl...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/23154 LGTM cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23191 **[Test build #99537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99537/testReport)** for PR 23191 at commit [`8656936`](https://github.com/apache/spark/commit/8656936df9f5fd4f51c967aac3201c0988c05b5c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23192 **[Test build #99536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99536/testReport)** for PR 23192 at commit [`c037f4d`](https://github.com/apache/spark/commit/c037f4d2fa2c2844ac992d976b492e14ab9bed11). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5598/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23192: [SPARK-26221][SQL] Add queryId to IncrementalExecution
Github user rxin commented on the issue: https://github.com/apache/spark/pull/23192 cc @zsxwing @jose-torres --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23192: [SPARK-26221][SQL] Add queryId to IncrementalExec...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/23192 [SPARK-26221][SQL] Add queryId to IncrementalExecution ## What changes were proposed in this pull request? This is a small change for better debugging: to pass query uuid in IncrementalExecution, when we look at the QueryExecution in isolation to trace back the query. ## How was this patch tested? N/A - just add some field for better debugging. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-26241 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23192.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23192 commit c037f4d2fa2c2844ac992d976b492e14ab9bed11 Author: Reynold Xin Date: 2018-12-01T04:27:00Z [SPARK-26221] --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23191 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22612 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99529/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22612 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23191 **[Test build #99535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99535/testReport)** for PR 23191 at commit [`590d580`](https://github.com/apache/spark/commit/590d580bd7e6a1a9b1f7d5645d60671c0d93decc). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99535/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22612 **[Test build #99529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99529/testReport)** for PR 22612 at commit [`4c1f073`](https://github.com/apache/spark/commit/4c1f0732e49fbddc98cba2d9953491f36e3137fc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23191 **[Test build #99535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99535/testReport)** for PR 23191 at commit [`590d580`](https://github.com/apache/spark/commit/590d580bd7e6a1a9b1f7d5645d60671c0d93decc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5597/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23191 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99534/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23190 **[Test build #99534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99534/testReport)** for PR 23190 at commit [`8046944`](https://github.com/apache/spark/commit/8046944efdc66dc3072b07f5a426a006e46e4975). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23191 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23191: [SPARK-26219][CORE][branch-2.4] Executor summary ...
GitHub user shahidki31 opened a pull request: https://github.com/apache/spark/pull/23191 [SPARK-26219][CORE][branch-2.4] Executor summary should get updated for failure jobs in the history server UI ## What changes were proposed in this pull request? Back port the commit https://github.com/apache/spark/pull/23181 into Spark2.4 branch ## How was this patch tested? Added UT You can merge this pull request into a Git repository by running: $ git pull https://github.com/shahidki31/spark branch-2.4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23191.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23191 commit 590d580bd7e6a1a9b1f7d5645d60671c0d93decc Author: Shahid Date: 2018-12-01T04:05:39Z [SPARK-26219][CORE][branch-2.4] Executor summary should get updated for failure jobs in the history server UI --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23190 **[Test build #99534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99534/testReport)** for PR 23190 at commit [`8046944`](https://github.com/apache/spark/commit/8046944efdc66dc3072b07f5a426a006e46e4975). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5596/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23190 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkE...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/23190 cc @cloud-fan @gatorsmile @kiszk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23190: [MINOR][SQL]throw SparkOutOfMemoryError intead of...
GitHub user heary-cao opened a pull request: https://github.com/apache/spark/pull/23190 [MINOR][SQL]throw SparkOutOfMemoryError intead of SparkException in UnsafeHashedRelation ## What changes were proposed in this pull request? When build hash Map with one row of data and run out of memory, we should throw a SparkOutOfMemoryError exception, which is more accurate than SparkException. this PR fix it. ## How was this patch tested? N / A You can merge this pull request into a Git repository by running: $ git pull https://github.com/heary-cao/spark throwUnsafeHashedRelation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23190.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23190 commit 8046944efdc66dc3072b07f5a426a006e46e4975 Author: caoxuewen Date: 2018-12-01T03:58:32Z throw SparkOutOfMemoryError intead of SparkException in UnsafeHashedRelation --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5595/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23151 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23151 **[Test build #99533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99533/testReport)** for PR 23151 at commit [`6d2f2d3`](https://github.com/apache/spark/commit/6d2f2d36f0e8dcdbe54e82865f7bd1b10dd323bd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/23055 +1 with the latest changes. Thanks for taking care of this, @HyukjinKwon! Functionality is in two parts: changing the resource requests (which doesn't change) and limiting memory use in python. It is too bad that this broke, but I'm not sure how to deal with a platform that, as you say, has few contributors. I certainly wouldn't want to gate a feature like this on making sure someone tested it in Windows, unless we have CI set up for Windows builds. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23055 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99531/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23055 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23055 **[Test build #99531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99531/testReport)** for PR 23055 at commit [`5fe1c09`](https://github.com/apache/spark/commit/5fe1c0976acaaca444d98ce0dff82eb92d82076a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23055 @vanzin and @rdblue, I updated the doc because it sounds not wrong to me. But, for clarification, we shouldn't really document we support something that's not tested (in particular such case above that the failure case was found). Also, IMHO, it's better to make it simple when there's Windows issue in terms of maintenance since not so many Windows maintainers exist in Spark. The main functionality of that configuration is limiting resource if i'm not mistaken. The allocation ones in other modes is secondary if I am not mistaken. Technically someone should test it and fix the doc with showing how it works in another PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23151 **[Test build #99532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99532/testReport)** for PR 23151 at commit [`de0ef11`](https://github.com/apache/spark/commit/de0ef115ad6260b1919beaef06337e2867105d13). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23151 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99532/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23151 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23151 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5594/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir function to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23151 **[Test build #99532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99532/testReport)** for PR 23151 at commit [`de0ef11`](https://github.com/apache/spark/commit/de0ef115ad6260b1919beaef06337e2867105d13). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23151: [SPARK-26180][CORE][TEST] Reuse withTempDir funct...
Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/23151#discussion_r238049434
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -245,8 +245,7 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
}
test("cannot call addFile with different paths that have the same
filename") {
--- End diff --
ok, i have update it. please review it. thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hi...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20788 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23055 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23055 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5593/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23055 **[Test build #99531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99531/testReport)** for PR 23055 at commit [`5fe1c09`](https://github.com/apache/spark/commit/5fe1c0976acaaca444d98ce0dff82eb92d82076a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23055: [SPARK-26080][PYTHON] Skips Python resource limit...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23055#discussion_r238048542 --- Diff: docs/configuration.md --- @@ -190,6 +190,8 @@ of the most common options to set are: and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes. When PySpark is run in YARN or Kubernetes, this memory is added to executor resource requests. + +NOTE: This configuration is not supported on Windows. --- End diff -- > Python memory usage may not be limited on platforms that do not support resource limiting, such as Windows Let me change it to this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23055: [SPARK-26080][PYTHON] Skips Python resource limit...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23055#discussion_r238048453 --- Diff: docs/configuration.md --- @@ -190,6 +190,8 @@ of the most common options to set are: and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes. When PySpark is run in YARN or Kubernetes, this memory is added to executor resource requests. + +NOTE: This configuration is not supported on Windows. --- End diff -- I would say it's just supported or unsupported tho rather than making it complicated. I'm not 100% sure how it works on Windows with other modes. For instance, extra memory will probably allocated via Yarn but looks no one tested it before. Technically, it does not work on local mode as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23037 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5592/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23037 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/5592/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23037 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/5592/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
