[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22790 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98036/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22790 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22790 **[Test build #98036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98036/testReport)** for PR 22790 at commit [`0decb23`](https://github.com/apache/spark/commit/0decb235cc31200cd7102daddf5c2f527c04e234). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22823#discussion_r228295120
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -910,12 +910,14 @@ class CodegenContext {
val blocks = new ArrayBuffer[String]()
val blockBuilder = new StringBuilder()
var length = 0
+val splitThreshold =
+ SQLConf.get.getConfString("spark.testing.codegen.splitThreshold",
"1024").toInt
--- End diff --
+1 for @wangyum 's comment.
@yucai . In this case, we can not call this `[TEST]` or `Refactor`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22837: [MINOR][TEST][BRANCH-2.4] Regenerate golden file `dateti...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22837 **[Test build #98043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98043/testReport)** for PR 22837 at commit [`f8685fa`](https://github.com/apache/spark/commit/f8685fa77914f80d6d1f2a3b3393b5d936378b05). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22827 @gatorsmile . Could you put `[BRANCH-2.4]` into PR title? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22820 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4496/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22820 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4496/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22837: [MINOR][TEST] Regenerate golden file `datetime.sq...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22837#discussion_r228293689
--- Diff: sql/core/src/test/resources/sql-tests/results/datetime.sql.out ---
@@ -82,9 +82,10 @@ struct
1 2
2 3
+
-- !query 9
select weekday('2007-02-03'), weekday('2009-07-30'),
weekday('2017-05-27'), weekday(null), weekday('1582-10-15 13:10:15')
--- !query 3 schema
+-- !query 9 schema
struct
--- !query 3 output
+-- !query 9 output
--- End diff --
@gatorsmile . This is very minor only in `branch-2.4`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22837: [MINOR][TEST] Fix datetime.sql.out
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/22837 [MINOR][TEST] Fix datetime.sql.out ## What changes were proposed in this pull request? `datetime.sql.out` is a generated golden file, but it's a little bit broken during manual [reverting](https://github.com/dongjoon-hyun/spark/commit/5d744499667fcd08825bca0ac6d5d90d6e110ebc#diff-79dd276be45ede6f34e24ad7005b0a7cR87). This doens't cause test failure because the difference is inside `comments` and blank lines. We had better fix this minor issue before RC5. ## How was this patch tested? Pass the Jenkins. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongjoon-hyun/spark fix_datetime_sql_out Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22837.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22837 commit f8685fa77914f80d6d1f2a3b3393b5d936378b05 Author: Dongjoon Hyun Date: 2018-10-25T17:57:59Z a --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4496/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21816: [SPARK-24794][CORE] Driver launched through rest should ...
Github user bsikander commented on the issue: https://github.com/apache/spark/pull/21816 Thank you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22817: [SPARK-25816][SQL] Fix attribute resolution in nested ex...
Github user peter-toth commented on the issue: https://github.com/apache/spark/pull/22817 @gatorsmile , I looked into this and it seems if we use `mapChildren` in `ResolveReferences` then `UnresolvedExtractValue` should define 2 children. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22829: [SPARK-25836][BUILD][K8S] For now disable kubernetes-int...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22829 **[Test build #98042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98042/testReport)** for PR 22829 at commit [`8ee6b30`](https://github.com/apache/spark/commit/8ee6b304d3f2c8376f2cc61a675e4c35234addb3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22817: [SPARK-25816][SQL] Fix attribute resolution in ne...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/spark/pull/22817#discussion_r228285647
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -407,7 +407,10 @@ case class ResolvedStar(expressions:
Seq[NamedExpression]) extends Star with Une
* can be key of Map, index of Array, field name of
Struct.
*/
case class UnresolvedExtractValue(child: Expression, extraction:
Expression)
- extends UnaryExpression with Unevaluable {
+ extends BinaryExpression with Unevaluable {
--- End diff --
if this change is not allowed then `children` we can override `children`
instead
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22830 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22830 **[Test build #98039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98039/testReport)** for PR 22830 at commit [`7f17c6e`](https://github.com/apache/spark/commit/7f17c6e1597849d812d041e5e8124448b8dc60c7). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22830 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98039/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22829: [SPARK-25836][BUILD][K8S] For now disable kuberne...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22829#discussion_r228285511 --- Diff: pom.xml --- @@ -2656,7 +2656,8 @@ kubernetes resource-managers/kubernetes/core -resource-managers/kubernetes/integration-tests + + --- End diff -- That's fine too. I even think this is a temporary measure, and that ideally these all live in the kubernetes module as test artifacts. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22830 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4495/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22830 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22820 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4494/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22820 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4494/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTable to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98022/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTable to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22823 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTable to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22823 **[Test build #98022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98022/testReport)** for PR 22823 at commit [`d2da3a0`](https://github.com/apache/spark/commit/d2da3a03363bb48a1debab492cd333d0f58ea4d4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 **[Test build #98041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98041/testReport)** for PR 22820 at commit [`c6b6723`](https://github.com/apache/spark/commit/c6b67230f3294cd4d8a6eeb0d8633c98c0808afc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22820 In reference to manifests that need to be updated in `dev/deps/spark-deps-hadoop-2.7` and `dev/deps/spark-deps-hadoop-3.1`. This most recent commit should fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22829: [SPARK-25836][BUILD][K8S] For now disable kuberne...
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22829#discussion_r228281269 --- Diff: pom.xml --- @@ -2656,7 +2656,8 @@ kubernetes resource-managers/kubernetes/core -resource-managers/kubernetes/integration-tests + + --- End diff -- I think this would mean it's not possible to run the integration tests at all via Maven, because the integration tests module will never be available. What about creating a separate profile, `kubernetes-integration-tests`, which includes both the `kubernetes` and the `kubernetes/integration-tests` module? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4494/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22817: [SPARK-25816][SQL] Fix attribute resolution in nested ex...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22817 **[Test build #98040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98040/testReport)** for PR 22817 at commit [`790812e`](https://github.com/apache/spark/commit/790812e174fb5cd712fbda63297e01ea9d495526). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22830 **[Test build #98039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98039/testReport)** for PR 22830 at commit [`7f17c6e`](https://github.com/apache/spark/commit/7f17c6e1597849d812d041e5e8124448b8dc60c7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r228278972 --- Diff: resource-managers/kubernetes/integration-tests/README.md --- @@ -41,12 +71,127 @@ The Spark code to test is handed to the integration test system via a tarball. H * `--spark-tgz ` - set `` to point to a tarball containing the Spark distribution to test. -TODO: Don't require the packaging of the built Spark artifacts into this tarball, just read them out of the current tree. +This Tarball should be created by first running `dev/make-distribution.sh` passing the `--tgz` flag and `-Pkubernetes` as one of the +options to ensure that Kubernetes support is included in the distribution. For more details on building a runnable distribution please +see the [Building Spark](https://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution) documentation. + +**TODO:** Don't require the packaging of the built Spark artifacts into this tarball, just read them out of the current tree. ## Customizing the Namespace and Service Account -* `--namespace ` - set `` to the namespace in which the tests should be run. -* `--service-account ` - set `` to the name of the Kubernetes service account to -use in the namespace specified by the `--namespace`. The service account is expected to have permissions to get, list, watch, +If no namespace is specified then a temporary namespace will be created and deleted during the test run. Similarly if no service +account is specified then the `default` service account for the namespace will be used. + +Using the `--namespace ` flag sets `` to the namespace in which the tests should be run. If this is supplied +then the tests assume this namespace exists in the K8S cluster and will not attempt to create it. Additionally this namespace must +have an appropriately authorized service account which can be customised via the `--service-account` flag. + +The `--service-account ` flag sets `` to the name of the Kubernetes service account to +use in the namespace specified by the `--namespace` flag. The service account is expected to have permissions to get, list, watch, and create pods. For clusters with RBAC turned on, it's important that the right permissions are granted to the service account in the namespace through an appropriate role and role binding. A reference RBAC configuration is provided in `dev/spark-rbac.yaml`. + +# Running the Test Directly + +If you prefer to run just the integration tests directly then you can customise the behaviour via properties passed to Maven using the +`-Dproperty=value` option e.g. --- End diff -- `If you prefer to run just the integration tests directly, then you can customize the behaviour via passing system properties to Maven. For example:` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/22805#discussion_r228280345
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/cloud/KubeConfigBackend.scala
---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.integrationtest.backend.cloud
+
+import java.nio.file.Paths
+
+import io.fabric8.kubernetes.client.{Config, DefaultKubernetesClient}
+import org.apache.spark.deploy.k8s.integrationtest.TestConstants
+import
org.apache.spark.deploy.k8s.integrationtest.backend.IntegrationTestBackend
+import org.apache.spark.internal.Logging
+
+private[spark] class KubeConfigBackend(var context: String)
+ extends IntegrationTestBackend with Logging {
+ // If no context supplied see if one was specified in the system
properties supplied
+ // to the tests
+ if (context == null) {
+context =
System.getProperty(TestConstants.CONFIG_KEY_KUBE_CONFIG_CONTEXT)
+ }
+ logInfo(s"K8S Integration tests will run against " +
+s"${if (context != null) s"context ${context}" else "default context"}
" +
+s" from users K8S config file")
+
+ private var defaultClient: DefaultKubernetesClient = _
+
+ override def initialize(): Unit = {
+// Auto-configure K8S client from K8S config file
+
System.setProperty(Config.KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY,
"true");
--- End diff --
Setting system properties is a little worrisome, for example what if
something else sets this system property as well? Would much prefer to treat
system properties as immutable after the JVM starts.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/22805#discussion_r228279841
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/cloud/KubeConfigBackend.scala
---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.integrationtest.backend.cloud
+
+import java.nio.file.Paths
+
+import io.fabric8.kubernetes.client.{Config, DefaultKubernetesClient}
+import org.apache.spark.deploy.k8s.integrationtest.TestConstants
+import
org.apache.spark.deploy.k8s.integrationtest.backend.IntegrationTestBackend
+import org.apache.spark.internal.Logging
+
+private[spark] class KubeConfigBackend(var context: String)
--- End diff --
Just use a default value for `context` instead of checking for null below.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22825: [SPARK-25772][SQL][FOLLOWUP] remove GetArrayFromMap
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22825 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98021/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22825: [SPARK-25772][SQL][FOLLOWUP] remove GetArrayFromMap
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22825 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22825: [SPARK-25772][SQL][FOLLOWUP] remove GetArrayFromMap
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22825 **[Test build #98021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98021/testReport)** for PR 22825 at commit [`a6c6faa`](https://github.com/apache/spark/commit/a6c6faa60d8846fb50845839905fc0b938046e02). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22790#discussion_r228279627
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala
---
@@ -109,7 +109,7 @@ class BisectingKMeansModel private[clustering] (
@Since("2.0.0")
override def save(sc: SparkContext, path: String): Unit = {
-BisectingKMeansModel.SaveLoadV1_0.save(sc, this, path)
+BisectingKMeansModel.SaveLoadV2_0.save(sc, this, path)
}
override protected def formatVersion: String = "1.0"
--- End diff --
I created #22830 for that, thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22830: [SPARK-25838][ML] Remove formatVersion from Savea...
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/22830 [SPARK-25838][ML] Remove formatVersion from Saveable ## What changes were proposed in this pull request? `Saveable` interface introduces `formatVersion` which is protected and it is used nowhere. So the PR proposes to remove it. ## How was this patch tested? existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/spark SPARK-25838 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22830.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22830 commit 7f17c6e1597849d812d041e5e8124448b8dc60c7 Author: Marco Gaido Date: 2018-10-25T18:07:22Z [SPARK-25838][ML] Remove formatVersion from Saveable --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22817: [SPARK-25816][SQL] ResolveReferences should work bottom-...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22817 **[Test build #98038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98038/testReport)** for PR 22817 at commit [`ac5c208`](https://github.com/apache/spark/commit/ac5c208b8670a472d85bf1e488a4ea89c4777534). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration testin...
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/22805 @rvesse I want to take back my comment on bumping the version of the fabric8 client in the same PR. Based on the discussion so far, it seems more appropriate to do that in a separate PR, like in #22820. Can you revert the changes to bump the version? Sorry about that. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22820 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22820 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98037/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 **[Test build #98037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98037/testReport)** for PR 22820 at commit [`a28ebcf`](https://github.com/apache/spark/commit/a28ebcf493343b45818c93524b879a7044c98022). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r228278587 --- Diff: resource-managers/kubernetes/integration-tests/README.md --- @@ -13,15 +13,45 @@ The simplest way to run the integration tests is to install and run Minikube, th dev/dev-run-integration-tests.sh The minimum tested version of Minikube is 0.23.0. The kube-dns addon must be enabled. Minikube should -run with a minimum of 3 CPUs and 4G of memory: +run with a minimum of 4 CPUs and 6G of memory: -minikube start --cpus 3 --memory 4096 +minikube start --cpus 4 --memory 6144 You can download Minikube [here](https://github.com/kubernetes/minikube/releases). # Integration test customization -Configuration of the integration test runtime is done through passing different arguments to the test script. The main useful options are outlined below. +Configuration of the integration test runtime is done through passing different arguments to the test script. +The main useful options are outlined below. + +## Using a different backend + +The integration test backend i.e. the K8S cluster used for testing is controlled by the `--deploy-mode` option. By default this +is set to `minikube`, the available backends are their perquisites are as follows. --- End diff -- `By default this is set to minikube. The available backends and their prerequisites are described below.` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22820 **[Test build #98037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98037/testReport)** for PR 22820 at commit [`a28ebcf`](https://github.com/apache/spark/commit/a28ebcf493343b45818c93524b879a7044c98022). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22818: [SPARK-25827][CORE] Allocate arrays smaller than Int.Max...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22818 Thanks, would it be also possible to double-check `Integer.MAX_VALUE` if you have not checked yet? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22790#discussion_r228276018
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala
---
@@ -109,7 +109,7 @@ class BisectingKMeansModel private[clustering] (
@Since("2.0.0")
override def save(sc: SparkContext, path: String): Unit = {
-BisectingKMeansModel.SaveLoadV1_0.save(sc, this, path)
+BisectingKMeansModel.SaveLoadV2_0.save(sc, this, path)
}
override protected def formatVersion: String = "1.0"
--- End diff --
I agree on that, I already have a patch for removing it (moreover this PR
can target 2.4, while removal should be done only on master I think). I am
submitting it. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTable to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22823 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTable to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98018/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22812: [SPARK-25817][SQL] Dataset encoder should support combin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22812 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22812: [SPARK-25817][SQL] Dataset encoder should support combin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22812 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98020/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22790#discussion_r228274470
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala
---
@@ -109,7 +109,7 @@ class BisectingKMeansModel private[clustering] (
@Since("2.0.0")
override def save(sc: SparkContext, path: String): Unit = {
-BisectingKMeansModel.SaveLoadV1_0.save(sc, this, path)
+BisectingKMeansModel.SaveLoadV2_0.save(sc, this, path)
}
override protected def formatVersion: String = "1.0"
--- End diff --
I changed the ```formatVersion``` to 2.0. There are quite a few files that
implement trait ```Saveable``` and have ```formatVersion```. I don't feel
comfortable to change other files for this PR. Maybe I will open a separate
jira to remove ```formatVersion``` from trait ```Saveable```?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTable to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22823 **[Test build #98018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98018/testReport)** for PR 22823 at commit [`b00396f`](https://github.com/apache/spark/commit/b00396f807aa4cd8c9e1a31fac232db85fb27f68). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22812: [SPARK-25817][SQL] Dataset encoder should support combin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22812 **[Test build #98020 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98020/testReport)** for PR 22812 at commit [`cf5a01e`](https://github.com/apache/spark/commit/cf5a01e7de4ece9e78c017d331027f722460ccac). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22790 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Clie...
Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/22820#discussion_r228269911
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
---
@@ -157,7 +157,10 @@ private[spark] object KubernetesUtils {
}.getOrElse(Seq(("container state", "N/A")))
}
- def formatTime(time: Time): String = {
-if (time != null) time.getTime else "N/A"
+ def formatTime(time: String): String = {
--- End diff --
The `time.getTime` is now a String. I wanted to keep the functionality, I
would remove if you deem it to be unnecessary.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228269628
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array('d', 'e',
'f'), (x, y) -> concat(x,
-- Zip with array coalesce
select zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x,
y)) as v;
-
-create or replace temporary view nested as values
- (1, map(1, 1, 2, 2, 3, 3)),
- (2, map(4, 4, 5, 5, 6, 6))
- as t(x, ys);
--- End diff --
Oops. Right!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22809: [SPARK-19851][SQL] Add support for EVERY and ANY (SOME) ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22809 **[Test build #98035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98035/testReport)** for PR 22809 at commit [`2bc9965`](https://github.com/apache/spark/commit/2bc996515ec1947b8a1b82f942bd6ebecc473277). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22790 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22790 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4493/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for classNameV2_...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22790 **[Test build #98036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98036/testReport)** for PR 22790 at commit [`0decb23`](https://github.com/apache/spark/commit/0decb235cc31200cd7102daddf5c2f527c04e234). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22809: [SPARK-19851][SQL] Add support for EVERY and ANY (SOME) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4492/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22809: [SPARK-19851][SQL] Add support for EVERY and ANY (SOME) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22809 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228267525
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array('d', 'e',
'f'), (x, y) -> concat(x,
-- Zip with array coalesce
select zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x,
y)) as v;
-
-create or replace temporary view nested as values
- (1, map(1, 1, 2, 2, 3, 3)),
- (2, map(4, 4, 5, 5, 6, 6))
- as t(x, ys);
--- End diff --
That has been removed already
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228267271
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array('d', 'e',
'f'), (x, y) -> concat(x,
-- Zip with array coalesce
select zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x,
y)) as v;
-
-create or replace temporary view nested as values
- (1, map(1, 1, 2, 2, 3, 3)),
- (2, map(4, 4, 5, 5, 6, 6))
- as t(x, ys);
--- End diff --
`higher-order-functions.sql.out`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22827 @dongjoon-hyun I think that can be kept. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22827 **[Test build #98034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98034/testReport)** for PR 22827 at commit [`8bdf02e`](https://github.com/apache/spark/commit/8bdf02eb71a30ece8dd5f17895697455218fbcfc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22827 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4491/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22827 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22827 Also, please remove `public static final int WORD_SIZE = 8;` in UnsafeRow.java. It's added by `map_entries`. - https://github.com/apache/spark/pull/22828/files?utf8=%E2%9C%93&diff=unified&w=1#diff-fa494816e13a1d53c1c55af01bc7a396L64 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22809: [SPARK-19851][SQL] Add support for EVERY and ANY (SOME) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22809 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22809: [SPARK-19851][SQL] Add support for EVERY and ANY (SOME) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4490/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22816: [SPARK-25822][PySpark]Fix a race condition when releasin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22816 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22816: [SPARK-25822][PySpark]Fix a race condition when releasin...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22816 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4489/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22829: [SPARK-25836][BUILD][K8S] For now disable kubernetes-int...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22829 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22829: [SPARK-25836][BUILD][K8S] For now disable kubernetes-int...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22829 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4488/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22829: [SPARK-25836][BUILD][K8S] For now disable kubernetes-int...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4488/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22829: [SPARK-25836][BUILD][K8S] For now disable kubernetes-int...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22829 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4488/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22809: [SPARK-19851][SQL] Add support for EVERY and ANY (SOME) ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22809 **[Test build #98033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98033/testReport)** for PR 22809 at commit [`08999f9`](https://github.com/apache/spark/commit/08999f98a3af6c7a30c545cec1c3657498fb39c0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22816: [SPARK-25822][PySpark]Fix a race condition when releasin...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22816 **[Test build #98032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98032/testReport)** for PR 22816 at commit [`2b0257e`](https://github.com/apache/spark/commit/2b0257ec79d7d45f0f1df24aab6a6ffa307a4bf2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22827#discussion_r228263430 --- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql --- @@ -65,21 +65,3 @@ create or replace temporary view nested as values (1, map(1, 1, 2, 2, 3, 3)), (2, map(4, 4, 5, 5, 6, 6)) as t(x, ys); --- End diff -- We should revert line 64~67 because they are introduced by [SPARK-23939 Add transform_keys function](https://github.com/apache/spark/commit/5b4a38d826807ea6733e4382c8f9b82a355a6eb4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22029: [SPARK-24395][SQL] IN operator should return NULL...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22029#discussion_r228262819
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
---
@@ -202,7 +209,11 @@ case class InSubquery(values: Seq[Expression], query:
ListQuery)
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = "expr1 _FUNC_(expr2, expr3, ...) - Returns true if `expr` equals
to any valN.",
+ usage = """
+expr1 _FUNC_(expr2, expr3, ...) - Returns true if `expr` equals to any
valN. Otherwise, if
+ spark.sql.legacy.inOperator.falseForNullField is false and any of
the elements or fields of
+ the elements is null it returns null, else it returns false.
--- End diff --
I think the point here is that only a string literal works, so it doesn't
work concat and/or interpolation. This just puts the string on different lines,
ie. the output is:
```
scala> sql("DESCRIBE FUNCTION IN").show(false)
+--+
|function_desc
|
+--+
|Function: in
|
|Class: org.apache.spark.sql.catalyst.expressions.In
|
|Usage:
expr1 in(expr2, expr3, ...) - Returns true if `expr` equals to any
valN. Otherwise, if
spark.sql.legacy.inOperator.falseForNullField is false and any of the
elements or fields of
the elements is null it returns null, else it returns false.
|
+--+
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22816: [SPARK-25822][PySpark]Fix a race condition when r...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/22816#discussion_r228262084
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -114,7 +114,7 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
context.addTaskCompletionListener[Unit] { _ =>
writerThread.shutdownOnTaskCompletion()
- if (!reuseWorker || !released.get) {
+ if (!reuseWorker || released.compareAndSet(false, true)) {
--- End diff --
Addressed
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] Remove newly added map related functi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22828 Oh, I'm closing this in favor of https://github.com/apache/spark/pull/22827. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] Remove newly added map related...
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/22828 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22821 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98017/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22821 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22821: [SPARK-25832][SQL] remove newly added map related functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22821 **[Test build #98017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98017/testReport)** for PR 22821 at commit [`726fc30`](https://github.com/apache/spark/commit/726fc3014bbfdc5e84f12ff42ec69b5e567c7af8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22829: [SPARK-25836][BUILD][K8S] For now disable kubernetes-int...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22829 **[Test build #98031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98031/testReport)** for PR 22829 at commit [`fea27cc`](https://github.com/apache/spark/commit/fea27cca1d015d02d9b78470250fa9a39d0e318d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22829: [SPARK-25836][BUILD][K8S] For now disable kuberne...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/22829 [SPARK-25836][BUILD][K8S] For now disable kubernetes-integration-tests ## What changes were proposed in this pull request? For now make building and running kubernetes-integration-tests manual. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-25836 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22829.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22829 commit fea27cca1d015d02d9b78470250fa9a39d0e318d Author: Sean Owen Date: 2018-10-25T17:10:09Z For now disable kubernetes-integration-tests --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22828 To run the Jenkins faster, I create a standalone PR instead of making a PR to #22821 . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22828#discussion_r228259220
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
---
@@ -61,8 +61,6 @@
*/
public final class UnsafeRow extends InternalRow implements
Externalizable, KryoSerializable {
- public static final int WORD_SIZE = 8;
-
--- End diff --
This is added by
[map_entries](https://github.com/apache/spark/commit/a6e883feb3b78232ad5cf636f7f7d5e825183041#diff-fa494816e13a1d53c1c55af01bc7a396R65)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22828 **[Test build #98030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98030/testReport)** for PR 22828 at commit [`71d3b3c`](https://github.com/apache/spark/commit/71d3b3ca87ec10616c5aab095652649fa48bb618). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22828 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22828 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4487/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r228256148 --- Diff: resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh --- @@ -71,19 +71,36 @@ if [[ $IMAGE_TAG == "N/A" ]]; then IMAGE_TAG=$(uuidgen); cd $UNPACKED_SPARK_TGZ - if [[ $DEPLOY_MODE == cloud ]] ; - then -$UNPACKED_SPARK_TGZ/bin/docker-image-tool.sh -r $IMAGE_REPO -t $IMAGE_TAG build -if [[ $IMAGE_REPO == gcr.io* ]] ; -then - gcloud docker -- push $IMAGE_REPO/spark:$IMAGE_TAG -else - $UNPACKED_SPARK_TGZ/bin/docker-image-tool.sh -r $IMAGE_REPO -t $IMAGE_TAG push -fi - else -# -m option for minikube. -$UNPACKED_SPARK_TGZ/bin/docker-image-tool.sh -m -r $IMAGE_REPO -t $IMAGE_TAG build - fi + + case $DEPLOY_MODE in +cloud|cloud-url) + # Build images + $UNPACKED_SPARK_TGZ/bin/docker-image-tool.sh -r $IMAGE_REPO -t $IMAGE_TAG build + + # Push images appropriately + if [[ $IMAGE_REPO == gcr.io* ]] ; + then +gcloud docker -- push $IMAGE_REPO/spark:$IMAGE_TAG + else +$UNPACKED_SPARK_TGZ/bin/docker-image-tool.sh -r $IMAGE_REPO -t $IMAGE_TAG push + fi + ;; + + docker-for-desktop) --- End diff -- The indention from this point seems to be off. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
