[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91958/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21441 **[Test build #91958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91958/testReport)** for PR 21441 at commit [`66d2437`](https://github.com/apache/spark/commit/66d2437a369e4c58bc7b8d92a4c99211de2f79f4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21568: [SPARK-24562][TESTS] Support different configs fo...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21568#discussion_r195895110
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ---
@@ -104,11 +105,34 @@ class SQLQueryTestSuite extends QueryTest with
SharedSQLContext {
// We should ignore this file from processing.
)
+ private val configsAllJoinTypes =
Seq(Seq(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key ->
+SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.defaultValueString),
+Seq(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.PREFER_SORTMERGEJOIN.key -> "true"),
+Seq(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.PREFER_SORTMERGEJOIN.key -> "false")) -> true
+
+ /**
+ * Maps a test with the set of configurations it has to run with and a
flag indicating whether
+ * the output must be the same with different configs or it has to be
different.
+ */
+ private val testConfigs: Map[String, (Seq[Seq[(String, String)]],
Boolean)] = Map(
+"typeCoercion/native/decimalArithmeticOperations.sql" ->
+ (Seq(Seq(SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS.key -> "true"),
+Seq(SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS.key -> "false")) ->
false),
+"subquery/in-subquery/in-joins.sql" -> configsAllJoinTypes,
--- End diff --
Exactly, changing these configs should not change the output.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21568: [SPARK-24562][TESTS] Support different configs fo...
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21568#discussion_r195895102 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/decimalArithmeticOperations.sql_spark.sql.decimalOperations.allowPrecisionLoss-false.out --- @@ -0,0 +1,193 @@ +-- Automatically generated by SQLQueryTestSuite --- End diff -- Since we have two cases: 1 - Different configs produce different results (so different files) and in this case your suggestion is fine; 2 - Different configs produce the same results (so we have one golden file for all of them), how would you address this case? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20795: [SPARK-23486]cache the function name from the external c...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20795 kindly ping @kevinyu98 @dilipbiswal --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21565: [SPARK-24558][Core]wrong Idle Timeout value is us...
Github user sandeep-katta commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r195895035
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager(
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
executorsRemoved.foreach { removedExecutorId =>
+// If it is cachedBlcok timeout is configured using
+// spark.dynamicAllocation.cachedExecutorIdleTimeout
+val idleTimeout = if
(SparkEnv.get.blockManager.master.hasCachedBlocks(removedExecutorId)) {
--- End diff --
I have updated the code as per comments
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r195894982
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
How about `logWarning(s"Failed to look up a function: $funcName")`? I feel
printing `Utils.exceptionString(error)` is a little meaningless.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91955/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #91955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91955/testReport)** for PR 21389 at commit [`f78334c`](https://github.com/apache/spark/commit/f78334c8cd0e4438e1abf102fae72d090c4388be). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91957/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21565: [SPARK-24558][Core]wrong Idle Timeout value is us...
Github user sandeep-katta commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r195894845
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager(
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
executorsRemoved.foreach { removedExecutorId =>
+// If it is cachedBlcok timeout is configured using
+// spark.dynamicAllocation.cachedExecutorIdleTimeout
--- End diff --
yes, if it is cached block then IDLE time out taken from the
spark.dynamicAllocation.cachedExecutorIdleTimeout
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21441 **[Test build #91957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91957/testReport)** for PR 21441 at commit [`6c86d92`](https://github.com/apache/spark/commit/6c86d92c1f7ef1b0275e7f68be730c16a072554d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21565: [SPARK-24558][Core]wrong Idle Timeout value is us...
Github user sandeep-katta commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r195894823
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager(
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
executorsRemoved.foreach { removedExecutorId =>
+// If it is cachedBlcok timeout is configured using
+// spark.dynamicAllocation.cachedExecutorIdleTimeout
+val idleTimeout = if
(SparkEnv.get.blockManager.master.hasCachedBlocks(removedExecutorId)) {
--- End diff --
yes blockManagerMaster is referring to same
SparkEnv.get.blockManager.master. Will update the code.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21567 **[Test build #91964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91964/testReport)** for PR 21567 at commit [`a69df11`](https://github.com/apache/spark/commit/a69df1137f9918670523120a2a0516b855833e84). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91963/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20636 **[Test build #91963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91963/testReport)** for PR 20636 at commit [`a134091`](https://github.com/apache/spark/commit/a134091aad0c3f8e3674f6cd751c2b8d5d83e39e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21565: [SPARK-24558][Core]wrong Idle Timeout value is us...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r195894664
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager(
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
executorsRemoved.foreach { removedExecutorId =>
+// If it is cachedBlcok timeout is configured using
+// spark.dynamicAllocation.cachedExecutorIdleTimeout
--- End diff --
`// If it is a cached block, it uses cachedExecutorIdleTimeoutS for
timeout`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21565: [SPARK-24558][Core]wrong Idle Timeout value is us...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r195894599
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager(
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
executorsRemoved.foreach { removedExecutorId =>
+// If it is cachedBlcok timeout is configured using
+// spark.dynamicAllocation.cachedExecutorIdleTimeout
+val idleTimeout = if
(SparkEnv.get.blockManager.master.hasCachedBlocks(removedExecutorId)) {
--- End diff --
`blockManagerMaster.hasCachedBlocks(removedExecutorId)`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4128/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/233/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20636 **[Test build #91963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91963/testReport)** for PR 20636 at commit [`a134091`](https://github.com/apache/spark/commit/a134091aad0c3f8e3674f6cd751c2b8d5d83e39e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20636 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21575: [SPARK-24566][CORE] spark.storage.blockManagerSlaveTimeo...
Github user xueyumusic commented on the issue: https://github.com/apache/spark/pull/21575 I added the tests, thanks @maropu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20636 **[Test build #91962 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91962/testReport)** for PR 20636 at commit [`a134091`](https://github.com/apache/spark/commit/a134091aad0c3f8e3674f6cd751c2b8d5d83e39e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91962/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21577: [WIP] [SPARK-24552][core] Correctly identify tasks in ou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21577 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21577: [WIP] [SPARK-24552][core] Correctly identify tasks in ou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21577 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91950/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21577: [WIP] [SPARK-24552][core] Correctly identify tasks in ou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21577 **[Test build #91950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91950/testReport)** for PR 21577 at commit [`d471b74`](https://github.com/apache/spark/commit/d471b74d4562ca8aa8e68a7ff90d881a67de5e59). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4127/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/232/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20636 **[Test build #91962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91962/testReport)** for PR 20636 at commit [`a134091`](https://github.com/apache/spark/commit/a134091aad0c3f8e3674f6cd751c2b8d5d83e39e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21544: add one supported type missing from the javadoc
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21544 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21549 you can commwnt this line https://github.com/HyukjinKwon/spark/blob/master/dev/run-tests.py#L577 and add `pass` instead. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21544: add one supported type missing from the javadoc
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21544 Thanks. Merging in master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20636 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21578: [SPARK-24571][SQL] Support Char literals
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21578 **[Test build #91961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91961/testReport)** for PR 21578 at commit [`4210146`](https://github.com/apache/spark/commit/42101465583b11665317cba714a778ba5e8404fe). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21549 @cloud-fan, I will take a look tonight (singapore timezone). Please feel free to disable it for now to unblock other PRs if you think so. From a very quick look (it's mobile), I think it needs some time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21568: [SPARK-24562][TESTS] Support different configs for same ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21568 IIUC this pr modified code to run a single test file multiple times in `SQLQueryTestSuite` with different configurations. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21578: [SPARK-24571][SQL] Support Char literals
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21578 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21578: [SPARK-24571][SQL] Support Char literals
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21578 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21578: [SPARK-24571][SQL] Support Char literals
GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21578
[SPARK-24571][SQL] Support Char literals
## What changes were proposed in this pull request?
In the PR, I propose to automatically convert a `Literal` with `Char` type
to a `Literal` of `String` type. Currently, the following code:
```scala
val df = Seq("Amsterdam", "San Francisco", "London").toDF("city")
df.where($"city".contains('o')).show(false)
```
fails with the exception:
```
Unsupported literal type class java.lang.Character o
java.lang.RuntimeException: Unsupported literal type class
java.lang.Character o
at
org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:78)
```
The PR fixes this issue by converting `char` to `string` of length `1`. I
believe it makes sense to does not differentiate `char` and `string(1)` in _a
unified, multi-language data platform_ like Spark which supports languages like
Python.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 support-char-literals
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21578.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21578
commit 97097f58866a492e9bb450de53c5f51ec8ef78e6
Author: Maxim Gekk
Date: 2018-06-13T22:29:12Z
A test for converting Char to String
commit 87640c768f9c1c71defb2f476eabbfea67e0f9ed
Author: Maxim Gekk
Date: 2018-06-13T23:20:39Z
Support Char in StringConverter
commit 0fb4669f47a5abcfdf255d6fd99f070764343cfc
Author: Maxim Gekk
Date: 2018-06-13T23:21:43Z
Evaluate Char literal as String literal
commit 99dfbfe5fc299efe2ece9bea9493dcfe1d5b523d
Author: Maxim Gekk
Date: 2018-06-14T03:03:56Z
Added a test for filtering rows by using Char literal
commit 56fd59254d4483b7b9ff866cec7f0cf87c28bb14
Author: Maxim Gekk
Date: 2018-06-14T03:05:56Z
Cover the case of java.lang.Character
commit 657f7be117f6b96500c8e2d7d764bf96d94efa8f
Author: Maxim Gekk
Date: 2018-06-14T03:27:26Z
Improving of the test
commit 0f09ab2a1ea0584ae61e5f44fd604882208d956c
Author: Maxim Gekk
Date: 2018-06-16T03:31:05Z
Merge remote-tracking branch 'origin/master' into char-to-string
commit 42101465583b11665317cba714a778ba5e8404fe
Author: Maxim Gekk
Date: 2018-06-16T03:47:10Z
Adding ticket number to test's titles
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21568: [SPARK-24562][TESTS] Support different configs for same ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21568 I'm confused by the description. What does this PR actually do? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21568: [SPARK-24562][TESTS] Support different configs fo...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21568#discussion_r195893037
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ---
@@ -104,11 +105,34 @@ class SQLQueryTestSuite extends QueryTest with
SharedSQLContext {
// We should ignore this file from processing.
)
+ private val configsAllJoinTypes =
Seq(Seq(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key ->
+SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.defaultValueString),
+Seq(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.PREFER_SORTMERGEJOIN.key -> "true"),
+Seq(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.PREFER_SORTMERGEJOIN.key -> "false")) -> true
+
+ /**
+ * Maps a test with the set of configurations it has to run with and a
flag indicating whether
+ * the output must be the same with different configs or it has to be
different.
+ */
+ private val testConfigs: Map[String, (Seq[Seq[(String, String)]],
Boolean)] = Map(
+"typeCoercion/native/decimalArithmeticOperations.sql" ->
+ (Seq(Seq(SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS.key -> "true"),
+Seq(SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS.key -> "false")) ->
false),
+"subquery/in-subquery/in-joins.sql" -> configsAllJoinTypes,
--- End diff --
This entry and the others blow don't change existing test result output?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21574: [SPARK-24478][SQL][followup] Move projection and filter ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21574 Does this move actually make sense? It'd destroy stats estimation for partition pruning. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21568: [SPARK-24562][TESTS] Support different configs fo...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21568#discussion_r195892983 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/decimalArithmeticOperations.sql_spark.sql.decimalOperations.allowPrecisionLoss-false.out --- @@ -0,0 +1,193 @@ +-- Automatically generated by SQLQueryTestSuite --- End diff -- This new feature pretty looks to me. Btw, if we set multiple configurations, filenames get too long? How about adding the configuration info. in the head of files? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21549 @HyukjinKwon is it possible that the constant build failure is caused by the java style checker? Other PRs that don't touch java files are fine. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21276: [SPARK-24216][SQL] Spark TypedAggregateExpression uses g...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21276 this is a safe fix, I'm also merging it to 2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21549 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21549 **[Test build #91960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91960/testReport)** for PR 21549 at commit [`52e706b`](https://github.com/apache/spark/commit/52e706bf91a6fa8cbdeb54be36bd21a65065fe5c). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21549 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91960/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21549 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/231/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21549 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4126/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21549 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21549 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21549 **[Test build #91960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91960/testReport)** for PR 21549 at commit [`52e706b`](https://github.com/apache/spark/commit/52e706bf91a6fa8cbdeb54be36bd21a65065fe5c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21549: [SPARK-24542] [SQL] UDF series UDFXPathXXXX allow users ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21549 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...
Github user edwinalu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r195892287
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -251,6 +261,222 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
}
}
+ /**
+ * Test executor metrics update logging functionality. This checks that a
+ * SparkListenerExecutorMetricsUpdate event is added to the Spark history
+ * log if one of the executor metrics is larger than any previously
+ * recorded value for the metric, per executor per stage. The task
metrics
+ * should not be added.
+ */
+ private def testExecutorMetricsUpdateEventLogging() {
+val conf = getLoggingConf(testDirPath, None)
+val logName = "executorMetricsUpdated-test"
+val eventLogger = new EventLoggingListener(logName, None,
testDirPath.toUri(), conf)
+val listenerBus = new LiveListenerBus(conf)
+
+// expected ExecutorMetricsUpdate, for the given stage id and executor
id
+val expectedMetricsEvents: Map[(Int, String),
SparkListenerExecutorMetricsUpdate] =
--- End diff --
I'll add a check.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...
Github user edwinalu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r195892271
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -98,14 +101,53 @@ class ExecutorSummary private[spark](
val removeReason: Option[String],
val executorLogs: Map[String, String],
val memoryMetrics: Option[MemoryMetrics],
-val blacklistedInStages: Set[Int])
+val blacklistedInStages: Set[Int],
+@JsonSerialize(using = classOf[PeakMemoryMetricsSerializer])
+@JsonDeserialize(using = classOf[PeakMemoryMetricsDeserializer])
+val peakMemoryMetrics: Option[Array[Long]])
class MemoryMetrics private[spark](
val usedOnHeapStorageMemory: Long,
val usedOffHeapStorageMemory: Long,
val totalOnHeapStorageMemory: Long,
val totalOffHeapStorageMemory: Long)
+/** deserialzer for peakMemoryMetrics: convert to array ordered by metric
name */
+class PeakMemoryMetricsDeserializer extends
JsonDeserializer[Option[Array[Long]]] {
--- End diff --
Yes, changed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...
Github user edwinalu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r195892263
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -98,14 +101,53 @@ class ExecutorSummary private[spark](
val removeReason: Option[String],
val executorLogs: Map[String, String],
val memoryMetrics: Option[MemoryMetrics],
-val blacklistedInStages: Set[Int])
+val blacklistedInStages: Set[Int],
+@JsonSerialize(using = classOf[PeakMemoryMetricsSerializer])
+@JsonDeserialize(using = classOf[PeakMemoryMetricsDeserializer])
+val peakMemoryMetrics: Option[Array[Long]])
class MemoryMetrics private[spark](
val usedOnHeapStorageMemory: Long,
val usedOffHeapStorageMemory: Long,
val totalOnHeapStorageMemory: Long,
val totalOffHeapStorageMemory: Long)
+/** deserialzer for peakMemoryMetrics: convert to array ordered by metric
name */
+class PeakMemoryMetricsDeserializer extends
JsonDeserializer[Option[Array[Long]]] {
+ override def deserialize(
+ jsonParser: JsonParser,
+ deserializationContext: DeserializationContext): Option[Array[Long]]
= {
+val metricsMap = jsonParser.readValueAs(classOf[Option[Map[String,
Object]]])
--- End diff --
That works and is much cleaner, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...
Github user edwinalu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r195892278
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -251,6 +261,222 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
}
}
+ /**
+ * Test executor metrics update logging functionality. This checks that a
+ * SparkListenerExecutorMetricsUpdate event is added to the Spark history
+ * log if one of the executor metrics is larger than any previously
+ * recorded value for the metric, per executor per stage. The task
metrics
+ * should not be added.
+ */
+ private def testExecutorMetricsUpdateEventLogging() {
+val conf = getLoggingConf(testDirPath, None)
+val logName = "executorMetricsUpdated-test"
+val eventLogger = new EventLoggingListener(logName, None,
testDirPath.toUri(), conf)
+val listenerBus = new LiveListenerBus(conf)
+
+// expected ExecutorMetricsUpdate, for the given stage id and executor
id
+val expectedMetricsEvents: Map[(Int, String),
SparkListenerExecutorMetricsUpdate] =
+ Map(
+((0, "1"),
+ createExecutorMetricsUpdateEvent(1,
+new ExecutorMetrics(-1L,
+ Array(5000L, 50L, 50L, 20L, 50L, 10L, 100L, 30L, 70L,
20L,
+((0, "2"),
+ createExecutorMetricsUpdateEvent(2,
+new ExecutorMetrics(-1L,
+ Array(7000L, 70L, 50L, 20L, 10L, 10L, 50L, 30L, 80L, 40L,
+((1, "1"),
+ createExecutorMetricsUpdateEvent(1,
+new ExecutorMetrics(-1L,
+ Array(7000L, 70L, 50L, 30L, 60L, 30L, 80L, 55L, 50L, 0L,
+((1, "2"),
+ createExecutorMetricsUpdateEvent(2,
+new ExecutorMetrics(-1L,
+ Array(7000L, 70L, 50L, 40L, 10L, 30L, 50L, 60L, 40L, 40L)
+
+// Events to post.
+val events = Array(
+ SparkListenerApplicationStart("executionMetrics", None,
+1L, "update", None),
+ createExecutorAddedEvent(1),
+ createExecutorAddedEvent(2),
+ createStageSubmittedEvent(0),
+ createExecutorMetricsUpdateEvent(1,
--- End diff --
Adding comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21567 @xueyumusic Can you fix the error first? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21567 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21567 cc @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21567 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91959/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21567 **[Test build #91959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91959/testReport)** for PR 21567 at commit [`a8dc241`](https://github.com/apache/spark/commit/a8dc241d0d9a0c0a2fef1b09bad2ac52fe4f9dd9). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21567 **[Test build #91959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91959/testReport)** for PR 21567 at commit [`a8dc241`](https://github.com/apache/spark/commit/a8dc241d0d9a0c0a2fef1b09bad2ac52fe4f9dd9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21567 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21570: [SPARK-24564][TEST] Add test suite for RecordBina...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21570#discussion_r195891897
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package test.org.apache.spark.sql.execution.sort;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.memory.TaskMemoryManager;
+import org.apache.spark.memory.TestMemoryConsumer;
+import org.apache.spark.memory.TestMemoryManager;
+import org.apache.spark.sql.catalyst.expressions.UnsafeArrayData;
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow;
+import org.apache.spark.sql.execution.RecordBinaryComparator;
+import org.apache.spark.unsafe.Platform;
+import org.apache.spark.unsafe.UnsafeAlignedOffset;
+import org.apache.spark.unsafe.array.LongArray;
+import org.apache.spark.unsafe.memory.MemoryBlock;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.apache.spark.util.collection.unsafe.sort.*;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Test the RecordBinaryComparator, which compares two UnsafeRows by their
binary form.
+ */
+public class RecordBinaryComparatorSuite {
+
+ private final TaskMemoryManager memoryManager = new TaskMemoryManager(
--- End diff --
We need to use `TaskMemoryManager` for this test? Is not simple tests
(e.g., just comparing unsafe rows) enough?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21554: [SPARK-24546][SQL] InsertIntoDataSourceCommand ma...
Github user zheh12 closed the pull request at: https://github.com/apache/spark/pull/21554 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21288 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21288 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91946/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21288: [SPARK-24206][SQL] Improve FilterPushdownBenchmark bench...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21288 **[Test build #91946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91946/testReport)** for PR 21288 at commit [`4a9cec9`](https://github.com/apache/spark/commit/4a9cec91f9446161d4dde0cac20ccdccb9a112e7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21570: [SPARK-24564][TEST] Add test suite for RecordBina...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21570#discussion_r195891774
--- Diff:
core/src/test/java/org/apache/spark/memory/TestMemoryConsumer.java ---
@@ -43,6 +45,12 @@ void free(long size) {
used -= size;
taskMemoryManager.releaseExecutionMemory(size, this);
}
+
+ // Exposed for testing
--- End diff --
super nit: we need this comment? This class itself is used for test use
only?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21570: [SPARK-24564][TEST] Add test suite for RecordBina...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21570#discussion_r195891644 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java --- @@ -0,0 +1,255 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package test.org.apache.spark.sql.execution.sort; + +import org.apache.spark.SparkConf; +import org.apache.spark.memory.TaskMemoryManager; +import org.apache.spark.memory.TestMemoryConsumer; +import org.apache.spark.memory.TestMemoryManager; +import org.apache.spark.sql.catalyst.expressions.UnsafeArrayData; +import org.apache.spark.sql.catalyst.expressions.UnsafeRow; --- End diff -- super nit: better to merge imports into one. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21575: [SPARK-24566][CORE] spark.storage.blockManagerSlaveTimeo...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21575 btw, better to add tests and can you do? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4125/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/230/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21441 **[Test build #91958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91958/testReport)** for PR 21441 at commit [`66d2437`](https://github.com/apache/spark/commit/66d2437a369e4c58bc7b8d92a4c99211de2f79f4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to se...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21441#discussion_r195891280
--- Diff: project/SparkBuild.scala ---
@@ -464,7 +464,20 @@ object DockerIntegrationTests {
*/
object DependencyOverrides {
lazy val settings = Seq(
-dependencyOverrides += "com.google.guava" % "guava" % "14.0.1")
+dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" %
"2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-module-jaxb-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-databind" % "2.6.7")
+}
+
+/**
+ * Exclusions to work around sbt's dependency resolution being different
from Maven's.
+ */
+object ExcludeDependencies {
+ lazy val settings = Seq(
+excludeDependencies += "com.fasterxml.jackson.jaxrs" %
"jackson-jaxrs-json-provider",
+excludeDependencies += "javax.ws.rs" % "jsr311-api")
--- End diff --
Note to myself: this seems coming from Jersey which we exclude the
dependencies in Maven. Seems because different dependency resolution between
Maven and SBT IIUC.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to se...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21441#discussion_r195891263
--- Diff: project/SparkBuild.scala ---
@@ -464,7 +464,20 @@ object DockerIntegrationTests {
*/
object DependencyOverrides {
lazy val settings = Seq(
-dependencyOverrides += "com.google.guava" % "guava" % "14.0.1")
+dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" %
"2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-module-jaxb-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-databind" % "2.6.7")
+}
+
+/**
+ * Exclusions to work around sbt's dependency resolution being different
from Maven's.
+ */
+object ExcludeDependencies {
+ lazy val settings = Seq(
+excludeDependencies += "com.fasterxml.jackson.jaxrs" %
"jackson-jaxrs-json-provider",
+excludeDependencies += "javax.ws.rs" % "jsr311-api")
--- End diff --
Note to myself: this seems coming from Jersey which we exclude the
dependencies. Seems because different dependency resolution between Maven and
SBT IIUC.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to se...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21441#discussion_r195891199 --- Diff: pom.xml --- @@ -123,7 +123,7 @@ 1.6.0 3.4.6 2.6.0 -org.spark-project.hive +com.github.hyukjinkwon --- End diff -- I redistributed Spark's Hive fork that contains single line fix for Hadoop 3 support. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91956/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21441 **[Test build #91957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91957/testReport)** for PR 21441 at commit [`6c86d92`](https://github.com/apache/spark/commit/6c86d92c1f7ef1b0275e7f68be730c16a072554d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20636 **[Test build #91956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91956/testReport)** for PR 20636 at commit [`a134091`](https://github.com/apache/spark/commit/a134091aad0c3f8e3674f6cd751c2b8d5d83e39e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20636: [SPARK-23415][SQL][TEST] Make behavior of BufferHolderSp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20636 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/229/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4124/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21441: [DO-NOT-MERGE] Run tests against hadoop-3.1 to see the t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21441 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
