[GitHub] spark issue #23088: [SPARK-26119][CORE][WEBUI]Task summary table should cont...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23088 **[Test build #99635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99635/testReport)** for PR 23088 at commit [`f8cfb54`](https://github.com/apache/spark/commit/f8cfb544a805ecb5d1056f703dde4e7705ad1810). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23211 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99636/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23211 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23194: [MINOR][SQL] Combine the same codes in test cases
Github user CarolinePeng commented on a diff in the pull request:
https://github.com/apache/spark/pull/23194#discussion_r238523807
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -377,41 +377,42 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
}
- test("CTAS a managed table with the existing empty directory") {
-val tableLoc = new
File(spark.sessionState.catalog.defaultTablePath(TableIdentifier("tab1")))
-try {
+ protected def withTableLocation(tableNames: String)(f: File => Unit):
Unit = {
--- End diff --
Ok,thank you !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23211 **[Test build #99636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99636/testReport)** for PR 23211 at commit [`f4bb126`](https://github.com/apache/spark/commit/f4bb126472eb5a808a3ae94bcfb59e0674e01217). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23194: [MINOR][SQL] Combine the same codes in test cases
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23194 **[Test build #99644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99644/testReport)** for PR 23194 at commit [`867ee5e`](https://github.com/apache/spark/commit/867ee5e5ee73d00ea492655aabaa407089ea0f91). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23194: [MINOR][SQL] Combine the same codes in test cases
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23194: [MINOR][SQL] Combine the same codes in test cases
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5701/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23155: [MINOR][K8S] add missing docs for podTemplateCont...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/23155#discussion_r238521078 --- Diff: docs/running-on-kubernetes.md --- @@ -932,15 +935,31 @@ specific to Spark on Kubernetes. (none) Specify the local file that contains the driver [pod template](#pod-template). For example - spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml` + spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml + + + + spark.kubernetes.driver.podTemplateContainerName + (none) + + Specify the container name to be used as a basis for the driver in the given [pod template](#pod-template). + For example spark.kubernetes.driver.podTemplateContainerName=spark-driver spark.kubernetes.executor.podTemplateFile (none) Specify the local file that contains the executor [pod template](#pod-template). For example - spark.kubernetes.executor.podTemplateFile=/path/to/executor-pod-template.yaml` + spark.kubernetes.executor.podTemplateFile=/path/to/executor-pod-template.yaml + + + + spark.kubernetes.executor.podTemplateContainerName + (none) + + Specify the container name to be used as a basis for the executor in the given [pod template](#pod-template). --- End diff -- ditto. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23155: [MINOR][K8S] add missing docs for podTemplateCont...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/23155#discussion_r238521043 --- Diff: docs/running-on-kubernetes.md --- @@ -932,15 +935,31 @@ specific to Spark on Kubernetes. (none) Specify the local file that contains the driver [pod template](#pod-template). For example - spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml` + spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml + + + + spark.kubernetes.driver.podTemplateContainerName + (none) + + Specify the container name to be used as a basis for the driver in the given [pod template](#pod-template). + For example spark.kubernetes.driver.podTemplateContainerName=spark-driver spark.kubernetes.executor.podTemplateFile (none) Specify the local file that contains the executor [pod template](#pod-template). For example --- End diff -- ditto. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23155: [MINOR][K8S] add missing docs for podTemplateCont...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/23155#discussion_r238521015 --- Diff: docs/running-on-kubernetes.md --- @@ -932,15 +935,31 @@ specific to Spark on Kubernetes. (none) Specify the local file that contains the driver [pod template](#pod-template). For example --- End diff -- Please fix this, too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23155: [MINOR][K8S] add missing docs for podTemplateCont...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/23155#discussion_r238520981 --- Diff: docs/running-on-kubernetes.md --- @@ -932,15 +935,31 @@ specific to Spark on Kubernetes. (none) Specify the local file that contains the driver [pod template](#pod-template). For example - spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml` + spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml + + + + spark.kubernetes.driver.podTemplateContainerName + (none) + + Specify the container name to be used as a basis for the driver in the given [pod template](#pod-template). --- End diff -- Actually, `[pod template](#pod-template)` is not working. Could you try the following and check with `SKIP_API=1 jekyll build`? ``` pod template ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22468: [SPARK-25374][SQL] SafeProjection supports fallba...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22468#discussion_r238520267
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala
---
@@ -535,4 +535,100 @@ class UnsafeRowConverterSuite extends SparkFunSuite
with Matchers with PlanTestB
assert(unsafeRow.getSizeInBytes ==
8 + 8 * 2 + roundedSize(field1.getSizeInBytes) +
roundedSize(field2.getSizeInBytes))
}
+
+ testBothCodegenAndInterpreted("SPARK-25374 converts back into safe
representation") {
+def convertBackToInternalRow(inputRow: InternalRow, fields:
Array[DataType]): InternalRow = {
+ val unsafeProj = UnsafeProjection.create(fields)
+ val unsafeRow = unsafeProj(inputRow)
+ val safeProj = SafeProjection.create(fields)
+ safeProj(unsafeRow)
+}
+
+// Simple tests
+val inputRow = InternalRow.fromSeq(Seq(
+ false, 3.toByte, 15.toShort, -83, 129L, 1.0f, 8.0,
UTF8String.fromString("test"),
+ Decimal(255), CalendarInterval.fromString("interval 1 day"),
Array[Byte](1, 2)
+))
+val fields1 = Array(
+ BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType,
+ DoubleType, StringType, DecimalType.defaultConcreteType,
CalendarIntervalType,
+ BinaryType)
+
+assert(convertBackToInternalRow(inputRow, fields1) === inputRow)
+
+// Array tests
+val arrayRow = InternalRow.fromSeq(Seq(
+ createArray(1, 2, 3),
+ createArray(
+createArray(Seq("a", "b", "c").map(UTF8String.fromString): _*),
+createArray(Seq("d").map(UTF8String.fromString): _*))
+))
+val fields2 = Array[DataType](
+ ArrayType(IntegerType),
+ ArrayType(ArrayType(StringType)))
+
+assert(convertBackToInternalRow(arrayRow, fields2) === arrayRow)
+
+// Struct tests
+val structRow = InternalRow.fromSeq(Seq(
+ InternalRow.fromSeq(Seq[Any](1, 4.0)),
+ InternalRow.fromSeq(Seq(
+UTF8String.fromString("test"),
+InternalRow.fromSeq(Seq(
+ 1,
+ createArray(Seq("2", "3").map(UTF8String.fromString): _*)
+))
+ ))
+))
+val fields3 = Array[DataType](
+ StructType(
+StructField("c0", IntegerType) ::
+StructField("c1", DoubleType) ::
+Nil),
+ StructType(
+StructField("c2", StringType) ::
+StructField("c3", StructType(
+ StructField("c4", IntegerType) ::
+ StructField("c5", ArrayType(StringType)) ::
+ Nil)) ::
+Nil))
+
+assert(convertBackToInternalRow(structRow, fields3) === structRow)
+
+// Map tests
+val mapRow = InternalRow.fromSeq(Seq(
+ createMap(Seq("k1", "k2").map(UTF8String.fromString): _*)(1, 2),
+ createMap(
+createMap(3, 5)(Seq("v1", "v2").map(UTF8String.fromString): _*),
+createMap(7, 9)(Seq("v3", "v4").map(UTF8String.fromString): _*)
+ )(
+createMap(Seq("k3", "k4").map(UTF8String.fromString):
_*)(3.toShort, 4.toShort),
+createMap(Seq("k5", "k6").map(UTF8String.fromString):
_*)(5.toShort, 6.toShort)
+ )))
+val fields4 = Array[DataType](
+ MapType(StringType, IntegerType),
+ MapType(MapType(IntegerType, StringType), MapType(StringType,
ShortType)))
+
+// Since `ArrayBasedMapData` does not override `equals` and `hashCode`,
+// we need to take care of it to compare rows.
+def toComparable(d: Any): Any = d match {
--- End diff --
this does nothing, isn't it?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22952: [SPARK-20568][SS] Provide option to clean up completed f...
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/22952 @zsxwing @gaborgsomogyi What we were trying to do is enforcing archive path so that moved files will not make overlap with source path. There may be same file name with different directory so I'm also trying to persist its own path in final archived path, which means archive files will not be placed in same directory. Based on above, I thought enforcing archive path with checking glob path is not easy to do, because without knowing final archive path (per file) we can't check it matches with glob pattern. That's why I just would rather restrict all subdirectories instead of finding a way to check against glob pattern. Actually I'm a bit afraid that we might be putting too much complexity on enforcing archive path. If we are OK with not enforcing archive path and just verify the final archive path doesn't overlap source path per each source file, it would be simple to do. We can make Spark not moving file and log warning message to let end users specify other directory. Would like to hear everyone's thought and idea. Thanks in advance! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23174 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23174 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99634/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23174 **[Test build #99634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99634/testReport)** for PR 23174 at commit [`791b5ee`](https://github.com/apache/spark/commit/791b5ee88d2f3c856aad640f662472124e098c66). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23155: [MINOR][K8S] add missing docs for podTemplateContainerNa...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23155 Oh, I thought it could be handled by another Kubernetes committers. I'll take a look again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23174 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99629/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23174 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23169: [SPARK-26103][SQL] Limit the length of debug strings for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99631/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23169: [SPARK-26103][SQL] Limit the length of debug strings for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23169 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23169: [SPARK-26103][SQL] Limit the length of debug strings for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23169 **[Test build #99631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99631/testReport)** for PR 23169 at commit [`a4be985`](https://github.com/apache/spark/commit/a4be985b5e7eeed43409ba0b0cda2a45c44e5110). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23174 **[Test build #99629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99629/testReport)** for PR 23174 at commit [`791b5ee`](https://github.com/apache/spark/commit/791b5ee88d2f3c856aad640f662472124e098c66). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99641/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22468 **[Test build #99641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99641/testReport)** for PR 22468 at commit [`0b23adb`](https://github.com/apache/spark/commit/0b23adb78f6b0553829e11f113a897856d5afa41). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22468: [SPARK-25374][SQL] SafeProjection supports fallba...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22468#discussion_r238515544
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
---
@@ -106,4 +106,19 @@ class CodeGeneratorWithInterpretedFallbackSuite
extends SparkFunSuite with PlanT
assert(proj(input).toSeq(StructType.fromDDL("c0 int, c1 int")) ===
expected)
}
}
+
+ test("SPARK-25374 Correctly handles NoOp in SafeProjection") {
+val exprs = Seq(Add(BoundReference(0, IntegerType, nullable = true),
Literal.create(1)), NoOp)
+val input = InternalRow.fromSeq(1 :: 1 :: Nil)
+val expected = 2 :: null :: Nil
+withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenOnly) {
--- End diff --
nvm, this is the code style in this test suite
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22468: [SPARK-25374][SQL] SafeProjection supports fallba...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22468#discussion_r238515444
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallbackSuite.scala
---
@@ -106,4 +106,19 @@ class CodeGeneratorWithInterpretedFallbackSuite
extends SparkFunSuite with PlanT
assert(proj(input).toSeq(StructType.fromDDL("c0 int, c1 int")) ===
expected)
}
}
+
+ test("SPARK-25374 Correctly handles NoOp in SafeProjection") {
+val exprs = Seq(Add(BoundReference(0, IntegerType, nullable = true),
Literal.create(1)), NoOp)
+val input = InternalRow.fromSeq(1 :: 1 :: Nil)
+val expected = 2 :: null :: Nil
+withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenOnly) {
--- End diff --
can we use `testWithBothCodegenAndIntepreted`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22911: [SPARK-25815][k8s] Support kerberos in client mode, keyt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99630/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22468: [SPARK-25374][SQL] SafeProjection supports fallba...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22468#discussion_r238515227 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala --- @@ -166,29 +166,40 @@ object UnsafeProjection } } -/** - * A projection that could turn UnsafeRow into GenericInternalRow --- End diff -- can we keep this comment? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22911: [SPARK-25815][k8s] Support kerberos in client mode, keyt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22911 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22911: [SPARK-25815][k8s] Support kerberos in client mode, keyt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22911 **[Test build #99630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99630/testReport)** for PR 22911 at commit [`05333ea`](https://github.com/apache/spark/commit/05333ea11bc50ade8f3ba8e1d2116bad9d3d7c89). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23212: [SPARK-25498][SQL][FOLLOW-UP] Return an empty config set...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23212 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23207: [SPARK-26193][SQL] Implement shuffle write metrics in SQ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23207 **[Test build #99643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99643/testReport)** for PR 23207 at commit [`7c8e516`](https://github.com/apache/spark/commit/7c8e5161904f1fd0fa4d99e6c497ef1be3542bdb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23207: [SPARK-26193][SQL] Implement shuffle write metrics in SQ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23207 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5700/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23207: [SPARK-26193][SQL] Implement shuffle write metrics in SQ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23207 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23207: [SPARK-26193][SQL] Implement shuffle write metrics in SQ...
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/23207 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23092: [SPARK-26094][CORE][STREAMING] createNonEcFile creates p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99625/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23092: [SPARK-26094][CORE][STREAMING] createNonEcFile creates p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23092 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23092: [SPARK-26094][CORE][STREAMING] createNonEcFile creates p...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23092 **[Test build #99625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99625/testReport)** for PR 23092 at commit [`891e37c`](https://github.com/apache/spark/commit/891e37cd371350906035cfbf93c618035a4293b1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #6840: [SPARK-8333] [SQL] Spark failed to delete temp directory ...
Github user liceHuang commented on the issue: https://github.com/apache/spark/pull/6840 > Ignore this, for now. Did it fixed in spark 2.Xï¼I have the same promble ,on windows7 64bit --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22899: [SPARK-25573] Combine resolveExpression and resolve in t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22899 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22899: [SPARK-25573] Combine resolveExpression and resolve in t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22899 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5699/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22899: [SPARK-25573] Combine resolveExpression and resolve in t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22899 **[Test build #99642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99642/testReport)** for PR 22899 at commit [`45e314a`](https://github.com/apache/spark/commit/45e314aa33fb0c108f185519ff58db691c34d58c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23169: [SPARK-26103][SQL] Limit the length of debug strings for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23169 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23169: [SPARK-26103][SQL] Limit the length of debug strings for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99632/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23169: [SPARK-26103][SQL] Limit the length of debug strings for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23169 **[Test build #99632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99632/testReport)** for PR 23169 at commit [`f0f75c2`](https://github.com/apache/spark/commit/f0f75c25b95010d63ecdf83bb9f280687361d154). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22904: [SPARK-25887][K8S] Configurable K8S context suppo...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22904#discussion_r238502012
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
---
@@ -67,8 +66,16 @@ private[spark] object SparkKubernetesClientFactory {
val dispatcher = new Dispatcher(
ThreadUtils.newDaemonCachedThreadPool("kubernetes-dispatcher"))
-// TODO [SPARK-25887] Create builder in a way that respects
configurable context
-val config = new ConfigBuilder()
+// Allow for specifying a context used to auto-configure from the
users K8S config file
+val kubeContext = sparkConf.get(KUBERNETES_CONTEXT).filter(c =>
StringUtils.isNotBlank(c))
+logInfo(s"Auto-configuring K8S client using " +
+ s"${if (kubeContext.isEmpty) s"context ${kubeContext.get}" else
"current context"}" +
+ s" from users K8S config file")
+
+// Start from an auto-configured config with the desired context
+// Fabric 8 uses null to indicate that the users current context
should be used so if no
+// explicit setting pass null
+val config = new
ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
--- End diff --
(Just to clarify I'm referring to the Spark config. Too many config files
involved here...)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23080: [SPARK-26108][SQL] Support custom lineSep in CSV datasou...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23080 It's fixed in upcoming Spark. Spark 2.4 does not support it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23080: [SPARK-26108][SQL] Support custom lineSep in CSV datasou...
Github user pooja-murarka commented on the issue:
https://github.com/apache/spark/pull/23080
I am testing **lineSep** with spark 2.4
data.csv : "a",1 "c",2 "d",3
val schema : StructType =
StructType(
Seq(
StructField(name = "dteday", dataType = StringType),
StructField(name = "hr", dataType = IntegerType)
)
_val logData = spark.read.format("csv").schema(schema).option("lineSep",
"\t").load("data.csv")_
But can only see schema without any data.
scala> logData.show()
+--++
|dteday| hr|
+--++
| null|null|
+--++
Can you please suggest if i missed something or above fix has not been
merged with branch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5698/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23196#discussion_r238496050
--- Diff:
sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
---
@@ -49,8 +49,8 @@ class HiveCompatibilitySuite extends HiveQueryFileTest
with BeforeAndAfter {
override def beforeAll() {
super.beforeAll()
TestHive.setCacheTables(true)
-// Timezone is fixed to America/Los_Angeles for those timezone
sensitive tests (timestamp_*)
-TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"))
+// Timezone is fixed to GMT for those timezone sensitive tests
(timestamp_*)
--- End diff --
@MaxGekk, BTW, why does this have to be GMT?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22468 **[Test build #99641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99641/testReport)** for PR 22468 at commit [`0b23adb`](https://github.com/apache/spark/commit/0b23adb78f6b0553829e11f113a897856d5afa41). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing ...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23196#discussion_r238495344 --- Diff: docs/sql-migration-guide-upgrade.md --- @@ -33,6 +33,8 @@ displayTitle: Spark SQL Upgrading Guide - Spark applications which are built with Spark version 2.4 and prior, and call methods of `UserDefinedFunction`, need to be re-compiled with Spark 3.0, as they are not binary compatible with Spark 3.0. + - Since Spark 3.0, JSON datasource uses java.time API for parsing and generating JSON content. New formatting implementation supports date/timestamp patterns conformed to ISO 8601. To switch back to the implementation used in Spark 2.4 and earlier, set `spark.sql.legacy.timeParser.enabled` to `true`. --- End diff -- I think we can add an example that shows the diff. IIRC it has a difference about exact match or non-exact match. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23212: [SPARK-25498][SQL][FOLLOW-UP] Return an empty config set...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23212 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23212: [SPARK-25498][SQL][FOLLOW-UP] Return an empty config set...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23212 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5697/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22590: [SPARK-25574][SQL]Add an option `keepQuotes` for parsing...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22590 Hm, let's try to find a way to expose other parse options. I think we shouldn't allow every options available on Univocity ... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23212: [SPARK-25498][SQL][FOLLOW-UP] Return an empty config set...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23212 **[Test build #99640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99640/testReport)** for PR 23212 at commit [`ca09bb8`](https://github.com/apache/spark/commit/ca09bb82d49d29a5d6f088d2250e980bab64a8b7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23212: [SPARK-25498][SQL][FOLLOW-UP] Return an empty con...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/23212 [SPARK-25498][SQL][FOLLOW-UP] Return an empty config set when regenerating the golden files ## What changes were proposed in this pull request? This pr is to return an empty config set when regenerating the golden files in `SQLQueryTestSuite`. This is the follow-up of #22512. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark SPARK-25498-FOLLOWUP Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23212.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23212 commit ca09bb82d49d29a5d6f088d2250e980bab64a8b7 Author: Takeshi Yamamuro Date: 2018-12-04T00:39:46Z Fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23212: [SPARK-25498][SQL][FOLLOW-UP] Return an empty config set...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/23212 cc: @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99639/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22468 **[Test build #99639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99639/testReport)** for PR 22468 at commit [`cec8480`](https://github.com/apache/spark/commit/cec84806c49ff7e616fc11e8703f0880807f41d9). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23088: [SPARK-26119][CORE][WEBUI]Task summary table should cont...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23088 Hi @vanzin , I have opened a JIRA for disk store case https://issues.apache.org/jira/browse/SPARK-26260. I will try to work on the same. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...
Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23088#discussion_r238492653
--- Diff:
core/src/test/scala/org/apache/spark/status/AppStatusStoreSuite.scala ---
@@ -77,6 +77,34 @@ class AppStatusStoreSuite extends SparkFunSuite {
assert(store.count(classOf[CachedQuantile]) === 2)
}
+ test("only successfull task have taskSummary") {
+val store = new InMemoryStore()
+(0 until 5).foreach { i => store.write(newTaskData(i, "FAILED")) }
--- End diff --
Done.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23088: [SPARK-26119][CORE][WEBUI]Task summary table shou...
Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23088#discussion_r238492582
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -148,11 +148,20 @@ private[spark] class AppStatusStore(
// cheaper for disk stores (avoids deserialization).
val count = {
Utils.tryWithResource(
-store.view(classOf[TaskDataWrapper])
- .parent(stageKey)
- .index(TaskIndexNames.EXEC_RUN_TIME)
- .first(0L)
- .closeableIterator()
+if (store.isInstanceOf[LevelDB]) {
--- End diff --
Done. Thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23211 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5694/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23211 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22514 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22514 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5695/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22468 **[Test build #99639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99639/testReport)** for PR 22468 at commit [`cec8480`](https://github.com/apache/spark/commit/cec84806c49ff7e616fc11e8703f0880807f41d9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22468: [SPARK-25374][SQL] SafeProjection supports fallback to a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22468 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5696/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20433 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20433 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5693/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20433 **[Test build #99638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99638/testReport)** for PR 20433 at commit [`7cb8f87`](https://github.com/apache/spark/commit/7cb8f87ba83fb04e891e069b8339dcef6d835ed9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22514 **[Test build #99637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99637/testReport)** for PR 22514 at commit [`e04812d`](https://github.com/apache/spark/commit/e04812d6aff0270ab4c0e6c3b2e204a682739e23). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23211 **[Test build #99636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99636/testReport)** for PR 23211 at commit [`f4bb126`](https://github.com/apache/spark/commit/f4bb126472eb5a808a3ae94bcfb59e0674e01217). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23088: [SPARK-26119][CORE][WEBUI]Task summary table should cont...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23088 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5692/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23088: [SPARK-26119][CORE][WEBUI]Task summary table should cont...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23088 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/23211 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23088: [SPARK-26119][CORE][WEBUI]Task summary table should cont...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23088 **[Test build #99635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99635/testReport)** for PR 23088 at commit [`f8cfb54`](https://github.com/apache/spark/commit/f8cfb544a805ecb5d1056f703dde4e7705ad1810). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INT...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20433 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22468: [SPARK-25374][SQL] SafeProjection supports fallba...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22468#discussion_r238490121
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala ---
@@ -157,4 +157,22 @@ object InternalRow {
getValueNullSafe
}
}
+
+ /**
+ * Returns a writer for an `InternalRow` with given data type.
+ */
+ def getWriter(ordinal: Int, dt: DataType): (InternalRow, Any) => Unit =
dt match {
--- End diff --
ok
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22512: [SPARK-25498][SQL] InterpretedMutableProjection s...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22512#discussion_r238489997
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ---
@@ -148,12 +156,25 @@ class SQLQueryTestSuite extends QueryTest with
SharedSQLContext {
})
// When we are regenerating the golden files we don't need to run
all the configs as they
// all need to return the same result
- if (regenerateGoldenFiles && configs.nonEmpty) {
-configs.take(1)
+ if (regenerateGoldenFiles) {
+if (configs.nonEmpty) {
+ configs.take(1)
+} else {
+ Array.empty[Array[(String, String)]]
--- End diff --
ok
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22857#discussion_r238489445
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -31,14 +31,14 @@ import org.apache.spark.scheduler.{SparkListener,
SparkListenerJobEnd}
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.expressions.Uuid
import org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation
-import org.apache.spark.sql.catalyst.plans.logical.{Filter,
OneRowRelation, Union}
+import org.apache.spark.sql.catalyst.plans.logical.{OneRowRelation, Union}
--- End diff --
Yea, also it's unrelated import cleanup. It should be discouraged because
it might make backporting / reverting potentially difficult, and sometimes
those changes make readers confused.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23174 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23174 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/5691/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23174 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5691/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23211 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99628/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23211 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23211: [SPARK-19712][SQL] Move PullupCorrelatedPredicates and R...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23211 **[Test build #99628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99628/testReport)** for PR 23211 at commit [`f4bb126`](https://github.com/apache/spark/commit/f4bb126472eb5a808a3ae94bcfb59e0674e01217). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23055: [SPARK-26080][PYTHON] Skips Python resource limit on Win...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23055 Last comment was a minor comment for a doc - actually the whole point was a minor one. It does related with Windows. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23174: [SPARK-26194][k8s] Auto generate auth secret for k8s app...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23174 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/5691/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23200: [SPARK-26033][SPARK-26034][PYTHON][FOLLOW-UP] Small clea...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23200 Thank you @srowen and @BryanCutler --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23191: [SPARK-26219][CORE][branch-2.4] Executor summary ...
Github user shahidki31 closed the pull request at: https://github.com/apache/spark/pull/23191 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22904: [SPARK-25887][K8S] Configurable K8S context suppo...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22904#discussion_r238484698
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
---
@@ -67,8 +66,16 @@ private[spark] object SparkKubernetesClientFactory {
val dispatcher = new Dispatcher(
ThreadUtils.newDaemonCachedThreadPool("kubernetes-dispatcher"))
-// TODO [SPARK-25887] Create builder in a way that respects
configurable context
-val config = new ConfigBuilder()
+// Allow for specifying a context used to auto-configure from the
users K8S config file
+val kubeContext = sparkConf.get(KUBERNETES_CONTEXT).filter(c =>
StringUtils.isNotBlank(c))
+logInfo(s"Auto-configuring K8S client using " +
+ s"${if (kubeContext.isEmpty) s"context ${kubeContext.get}" else
"current context"}" +
+ s" from users K8S config file")
+
+// Start from an auto-configured config with the desired context
+// Fabric 8 uses null to indicate that the users current context
should be used so if no
+// explicit setting pass null
+val config = new
ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
--- End diff --
> we don't propagate the submission clients config file
We don't propagate the config *file* itself, but we do propagate all its
contents, as far as I remember. But given your explanation it should work fine,
unless there's a different config context with the same name inside the
container...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23191: [SPARK-26219][CORE][branch-2.4] Executor summary should ...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23191 Thanks a lot @vanzin @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
