[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98905/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23049 **[Test build #98900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98900/testReport)** for PR 23049 at commit [`3269862`](https://github.com/apache/spark/commit/3269862c0b80bb7c546e9d45fd5fd4aa17aa1c7e). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #98902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98902/testReport)** for PR 23054 at commit [`42e32ad`](https://github.com/apache/spark/commit/42e32adda2da3717161fe5f8aa40febc1f32465e). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23056 **[Test build #98903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98903/testReport)** for PR 23056 at commit [`2759521`](https://github.com/apache/spark/commit/2759521df7f2dffc9ddb9379e0b1dac6721da366). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22138 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98904/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98902/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22138 **[Test build #98904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98904/testReport)** for PR 22138 at commit [`fd4ff83`](https://github.com/apache/spark/commit/fd4ff833b6c2b5889d55ee4053970b56ee2b273d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22138 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23056 **[Test build #98905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98905/testReport)** for PR 23056 at commit [`2759521`](https://github.com/apache/spark/commit/2759521df7f2dffc9ddb9379e0b1dac6721da366). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98903/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23049 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23056 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23049 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98900/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5074/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23056 **[Test build #98906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98906/testReport)** for PR 23056 at commit [`2759521`](https://github.com/apache/spark/commit/2759521df7f2dffc9ddb9379e0b1dac6721da366). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/22138 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/23049 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23054 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #98907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98907/testReport)** for PR 23054 at commit [`42e32ad`](https://github.com/apache/spark/commit/42e32adda2da3717161fe5f8aa40febc1f32465e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23049 **[Test build #98908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98908/testReport)** for PR 23049 at commit [`3269862`](https://github.com/apache/spark/commit/3269862c0b80bb7c546e9d45fd5fd4aa17aa1c7e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22138 **[Test build #98909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98909/testReport)** for PR 22138 at commit [`fd4ff83`](https://github.com/apache/spark/commit/fd4ff833b6c2b5889d55ee4053970b56ee2b273d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5075/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23049 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5076/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23044: [SPARK-26073][SQL][FOLLOW-UP] remove invalid comment as ...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/23044 thanks, @cloud-fan, @viirya, @kiszk. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23030 **[Test build #98910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98910/testReport)** for PR 23030 at commit [`8ab6c3b`](https://github.com/apache/spark/commit/8ab6c3bce2a88ec0efed9bc012e21d1cc997eb46). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5077/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98910/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23030 **[Test build #98910 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98910/testReport)** for PR 23030 at commit [`8ab6c3b`](https://github.com/apache/spark/commit/8ab6c3bce2a88ec0efed9bc012e21d1cc997eb46). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22995: [SPARK-25998] [CORE] Change TorrentBroadcast to hold wea...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/22995 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23017: [WIP][SPARK-26015][K8S] Set a default UID for Spa...
Github user rvesse commented on a diff in the pull request:
https://github.com/apache/spark/pull/23017#discussion_r234143917
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala
---
@@ -17,13 +17,13 @@
package org.apache.spark.deploy.k8s.integrationtest
import org.scalatest.concurrent.Eventually
-import scala.collection.JavaConverters._
import
org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.{k8sTestTag,
INTERVAL, TIMEOUT}
+import org.scalatest.Tag
--- End diff --
As I was just wanting to debug this specific failing test it was useful to
run it on its own repeatedly as I investigated the issue. I backed out other
changes I made to enable this (like not cleaning up the tests driver pod) so I
will back this out as well
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23013: [SPARK-25023] More detailed security guidance for...
Github user rvesse commented on a diff in the pull request: https://github.com/apache/spark/pull/23013#discussion_r234144540 --- Diff: docs/running-on-kubernetes.md --- @@ -15,7 +15,19 @@ container images and entrypoints.** # Security Security in Spark is OFF by default. This could mean you are vulnerable to attack by default. -Please see [Spark Security](security.html) and the specific security sections in this doc before running Spark. +Please see [Spark Security](security.html) and the specific advice below before running Spark. + +## User Identity + +Images built from the project provided Dockerfiles do not contain any [`USER`](https://docs.docker.com/engine/reference/builder/#user) directives. This means that the resulting images will be running the Spark processes as `root` inside the container. On unsecured clusters this may provide an attack vector for privilege escalation and container breakout. Therefore security conscious deployments should consider providing custom images with `USER` directives specifying an unprivileged UID and GID. --- End diff -- I would like this PR to go in first as this will also want back porting as-is to branch-2.4 Then in PR #23017 I will update the docs to explain the new defaults, how to customise etc. before that gets merged into master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23017: [SPARK-26015][K8S] Set a default UID for Spark on...
Github user rvesse commented on a diff in the pull request:
https://github.com/apache/spark/pull/23017#discussion_r234146162
--- Diff:
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile ---
@@ -53,5 +54,9 @@ COPY data /opt/spark/data
ENV SPARK_HOME /opt/spark
WORKDIR /opt/spark/work-dir
+RUN chmod g+w /opt/spark/work-dir
ENTRYPOINT [ "/opt/entrypoint.sh" ]
+
+# Specify the User that the actual main process will run as
+USER ${spark_uid}
--- End diff --
Done
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22598: [SPARK-25501][SS] Add kafka delegation token support.
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/22598 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22598: [SPARK-25501][SS] Add kafka delegation token support.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22598 **[Test build #98912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98912/testReport)** for PR 22598 at commit [`2a0cdb7`](https://github.com/apache/spark/commit/2a0cdb7f397abdc8ce411e2f5c08cf8029676e90). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [SPARK-26015][K8S] Set a default UID for Spark on K8S Im...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23017 **[Test build #98911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98911/testReport)** for PR 23017 at commit [`8f4fd19`](https://github.com/apache/spark/commit/8f4fd194e89a5062fc694cf917e1c2f744294495). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23042: [SPARK-26070][SQL] add rule for implicit type coe...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23042#discussion_r234155688
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -138,6 +138,11 @@ object TypeCoercion {
case (DateType, TimestampType)
=> if (conf.compareDateTimestampInTimestamp) Some(TimestampType)
else Some(StringType)
+// to support a popular use case of tables using Decimal(X, 0) for
long IDs instead of strings
+// see SPARK-26070 for more details
+case (n: DecimalType, s: StringType) if n.scale == 0 =>
Some(DecimalType(n.precision, n.scale))
--- End diff --
@cloud-fan I think we have seen many issues on this. I don't think there is
a standard for them, every RDBMS has different rules. The worst thing about the
current rules IMHO is that they are not even coherent in Spark (see #19635 for
instance).
The option I'd prefer is to follow Postgres behavior, ie. no implicit cast
at all. When there is a type mismatch the user has to choose how to cast the
things. It is a bit more effort on user side, but it is the safest option IMHO.
What do you think?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22154: [SPARK-23711][SPARK-25140][SQL] Catch correct exc...
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/22154#discussion_r234177079
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
---
@@ -17,17 +17,33 @@
package org.apache.spark.sql.catalyst.expressions
+import java.util.concurrent.ExecutionException
+
import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodeAndComment,
CodeGenerator}
import org.apache.spark.sql.catalyst.plans.PlanTestBase
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{IntegerType, LongType}
+import org.apache.spark.sql.types.IntegerType
class CodeGeneratorWithInterpretedFallbackSuite extends SparkFunSuite with
PlanTestBase {
- test("UnsafeProjection with codegen factory mode") {
-val input = Seq(LongType, IntegerType)
- .zipWithIndex.map(x => BoundReference(x._2, x._1, true))
+ object FailedCodegenProjection
+ extends CodeGeneratorWithInterpretedFallback[Seq[Expression],
UnsafeProjection] {
+
+override protected def createCodeGeneratedObject(in: Seq[Expression]):
UnsafeProjection = {
+ val invalidCode = new CodeAndComment("invalid code", Map.empty)
+ // We assume this compilation throws an exception
--- End diff --
I'd use this comment as part of an exception (say `IllegalStateException`
or similar) that should be thrown rather than returning `null`. I think that
would make the comment part of the code itself and can be checked in tests (by
catching the exception).
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #98907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98907/testReport)** for PR 23054 at commit [`42e32ad`](https://github.com/apache/spark/commit/42e32adda2da3717161fe5f8aa40febc1f32465e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98907/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of primitive type under...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23056 **[Test build #98906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98906/testReport)** for PR 23056 at commit [`2759521`](https://github.com/apache/spark/commit/2759521df7f2dffc9ddb9379e0b1dac6721da366). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23056: [SPARK-26034][PYTHON][TESTS] Break large mllib/tests.py ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23056 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98906/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22138 **[Test build #98909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98909/testReport)** for PR 22138 at commit [`fd4ff83`](https://github.com/apache/spark/commit/fd4ff833b6c2b5889d55ee4053970b56ee2b273d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22138 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98909/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/23057 cc @cloud-fan @viirya --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23057: [SPARK-26078][SQL] Dedup self-join attributes on ...
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/23057 [SPARK-26078][SQL] Dedup self-join attributes on IN subqueries ## What changes were proposed in this pull request? When there is a self-join as result of a IN subquery, the join condition may be invalid, resulting in trivially true predicates and return wrong results. The PR deduplicates the subquery output in order to avoid the issue. ## How was this patch tested? added UT You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/spark SPARK-26078 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23057.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23057 commit 2af656a6b8ddae00555b04ecdbc7873adc6fc0b6 Author: Marco Gaido Date: 2018-11-16T12:27:35Z [SPARK-26078][SQL] Dedup self-join attributes on subqueries --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23049 **[Test build #98908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98908/testReport)** for PR 23049 at commit [`3269862`](https://github.com/apache/spark/commit/3269862c0b80bb7c546e9d45fd5fd4aa17aa1c7e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5078/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23049 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23049: [SPARK-26076][Build][Minor] Revise ambiguous error messa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98908/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23057 **[Test build #98913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98913/testReport)** for PR 23057 at commit [`2af656a`](https://github.com/apache/spark/commit/2af656a6b8ddae00555b04ecdbc7873adc6fc0b6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98913/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23057 **[Test build #98913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98913/testReport)** for PR 23057 at commit [`2af656a`](https://github.com/apache/spark/commit/2af656a6b8ddae00555b04ecdbc7873adc6fc0b6). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5079/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23057 **[Test build #98914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98914/testReport)** for PR 23057 at commit [`a71b1c6`](https://github.com/apache/spark/commit/a71b1c6abd566e52063b3fb0343db5178ac67c8f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22598: [SPARK-25501][SS] Add kafka delegation token support.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22598 **[Test build #98912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98912/testReport)** for PR 22598 at commit [`2a0cdb7`](https://github.com/apache/spark/commit/2a0cdb7f397abdc8ce411e2f5c08cf8029676e90). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22598: [SPARK-25501][SS] Add kafka delegation token support.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22598 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98912/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22598: [SPARK-25501][SS] Add kafka delegation token support.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22598 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S
Github user nrchakradhar commented on the issue: https://github.com/apache/spark/pull/23013 The description is very useful and concise. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23039: [SPARK-26066][SQL] Move truncatedString to sql/ca...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/23039#discussion_r234202827
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1594,6 +1594,13 @@ object SQLConf {
"WHERE, which does not follow SQL standard.")
.booleanConf
.createWithDefault(false)
+
+ val MAX_TO_STRING_FIELDS = buildConf("spark.sql.debug.maxToStringFields")
+.doc("Maximum number of fields of sequence-like entries that can be
converted to strings " +
--- End diff --
nit: `that` is not necessary if I am correct.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22779: [SPARK-25786][CORE]If the ByteBuffer.hasArray is ...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22779#discussion_r234204540
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala ---
@@ -497,6 +498,17 @@ class KryoSerializerAutoResetDisabledSuite extends
SparkFunSuite with SharedSpar
deserializationStream.close()
assert(serInstance.deserialize[Any](helloHello) === ((hello, hello)))
}
+
+ test("ByteBuffer.array -- UnsupportedOperationException") {
--- End diff --
It would be good to add a prefix like "SPARK-25786: ...".
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23052: [SPARK-26081][SQL] Prevent empty files for empty partiti...
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/23052 > Similar changes were proposed in Parquet few years ago (by me) and reverted. What was the main reason to revert it? If it is possible could you give me a link to your PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22979: [SPARK-25977][SQL] Parsing decimals from CSV using local...
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22979 @HyukjinKwon Could you take a look at this one more time, please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23039: [SPARK-26066][SQL] Move truncatedString to sql/catalyst ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23039 **[Test build #98915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98915/testReport)** for PR 23039 at commit [`7180c2e`](https://github.com/apache/spark/commit/7180c2ea672783d903f29c0013fd5e1817949ed6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [SPARK-26015][K8S] Set a default UID for Spark on K8S Im...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23017 **[Test build #98911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98911/testReport)** for PR 23017 at commit [`8f4fd19`](https://github.com/apache/spark/commit/8f4fd194e89a5062fc694cf917e1c2f744294495). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23052: [SPARK-26081][SQL] Prevent empty files for empty ...
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/23052#discussion_r234211079
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
---
@@ -174,13 +174,18 @@ private[csv] class CsvOutputWriter(
context: TaskAttemptContext,
params: CSVOptions) extends OutputWriter with Logging {
- private val charset = Charset.forName(params.charset)
+ private var univocityGenerator: Option[UnivocityGenerator] = None
- private val writer = CodecStreams.createOutputStreamWriter(context, new
Path(path), charset)
-
- private val gen = new UnivocityGenerator(dataSchema, writer, params)
+ override def write(row: InternalRow): Unit = {
+val gen = univocityGenerator.getOrElse {
--- End diff --
I do think it is fine to write only headers if an user wants to have them.
Filtering the header out on this level could be slightly difficult.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [SPARK-26015][K8S] Set a default UID for Spark on K8S Im...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23017 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98911/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [SPARK-26015][K8S] Set a default UID for Spark on K8S Im...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23017 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 @imaxxs @rxin I think its a good time to remove this, I will update the PR if you are all ok. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S
Github user srowen commented on the issue: https://github.com/apache/spark/pull/23013 Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23030: [MINOR][YARN] Make memLimitExceededLogMessage mor...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/23030#discussion_r234232203
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -735,18 +744,8 @@ private[yarn] class YarnAllocator(
private object YarnAllocator {
val MEM_REGEX = "[0-9.]+ [KMG]B"
- val PMEM_EXCEEDED_PATTERN =
-Pattern.compile(s"$MEM_REGEX of $MEM_REGEX physical memory used")
- val VMEM_EXCEEDED_PATTERN =
-Pattern.compile(s"$MEM_REGEX of $MEM_REGEX virtual memory used")
+ val PMEM_EXCEEDED_PATTERN = raw"$MEM_REGEX of $MEM_REGEX physical memory
used".r
--- End diff --
We can still inline these patterns right?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23013: [SPARK-25023] More detailed security guidance for...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/23013 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23030: [MINOR][YARN] Make memLimitExceededLogMessage mor...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23030#discussion_r234233444
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -735,18 +744,8 @@ private[yarn] class YarnAllocator(
private object YarnAllocator {
val MEM_REGEX = "[0-9.]+ [KMG]B"
- val PMEM_EXCEEDED_PATTERN =
-Pattern.compile(s"$MEM_REGEX of $MEM_REGEX physical memory used")
- val VMEM_EXCEEDED_PATTERN =
-Pattern.compile(s"$MEM_REGEX of $MEM_REGEX virtual memory used")
+ val PMEM_EXCEEDED_PATTERN = raw"$MEM_REGEX of $MEM_REGEX physical memory
used".r
--- End diff --
Yes. I will do it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/23057 Thanks @mgaido91. I will review this tomorrow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23030 **[Test build #98916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98916/testReport)** for PR 23030 at commit [`66b3e7f`](https://github.com/apache/spark/commit/66b3e7f600acb4b630a1387d4e17c73fb11ffce5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5080/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22952: [SPARK-20568][SS] Rename files which are completed in pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22952 **[Test build #98917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98917/testReport)** for PR 22952 at commit [`3f6b5fb`](https://github.com/apache/spark/commit/3f6b5fbf01b2e78dfc9ecf7e3b45ef771fec74a7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23030 **[Test build #98916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98916/testReport)** for PR 23030 at commit [`66b3e7f`](https://github.com/apache/spark/commit/66b3e7f600acb4b630a1387d4e17c73fb11ffce5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98916/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23030: [MINOR][YARN] Make memLimitExceededLogMessage more clean
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23030 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23057 **[Test build #98914 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98914/testReport)** for PR 23057 at commit [`a71b1c6`](https://github.com/apache/spark/commit/a71b1c6abd566e52063b3fb0343db5178ac67c8f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22952: [SPARK-20568][SS] Rename files which are completed in pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22952 **[Test build #98918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98918/testReport)** for PR 22952 at commit [`33c5681`](https://github.com/apache/spark/commit/33c5681ab022116133576e4e27c50e346c1ffba9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23057 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98914/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22952: [SPARK-20568][SS] Rename files which are completed in pr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22952 **[Test build #98919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98919/testReport)** for PR 22952 at commit [`ca26b41`](https://github.com/apache/spark/commit/ca26b4136adc09fb9015c973953b50d894fc8779). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22952: [SPARK-20568][SS] Rename files which are completed in pr...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/22952 @zsxwing @dongjoon-hyun @steveloughran Thanks all for the valuable feedback! I applied review comments. While I covered the new feature with new UTs, I'm yet to test this manually with HDFS. I'll find the time to do manual test in next week. For cloud storages, TBH, it's not easy for me to do manual test against them, so I'd wish to lean on reviewers' eyes and experiences. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
