[GitHub] [spark] ScrapCodes commented on a change in pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
ScrapCodes commented on a change in pull request #30472:
URL: https://github.com/apache/spark/pull/30472#discussion_r537298366
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
##
@@ -94,25 +120,41 @@ private[spark] object KubernetesClientUtils extends
Logging {
val confDir = Option(conf.getenv(ENV_SPARK_CONF_DIR)).orElse(
conf.getOption("spark.home").map(dir => s"$dir/conf"))
if (confDir.isDefined) {
- val confFiles = listConfFiles(confDir.get)
- logInfo(s"Spark configuration files loaded from $confDir :
${confFiles.mkString(",")}")
- confFiles.map { file =>
-val source = Source.fromFile(file)(Codec.UTF8)
-val mapping = (file.getName -> source.mkString)
-source.close()
-mapping
- }.toMap
+ val confFiles = listConfFiles(confDir.get,
conf.get(Config.CONFIG_MAP_MAXSIZE))
+ val filesSeq: Seq[Option[(String, String)]] = confFiles.map { file =>
+try {
+ val source = Source.fromFile(file)(Codec.UTF8)
+ val mapping = Some(file.getName -> source.mkString)
+ source.close()
+ mapping
+} catch {
+ case e: MalformedInputException =>
+logWarning(s"skipped a non UTF-8 encoded file
${file.getAbsolutePath}.", e)
+None
+}
+ }
+ val truncatedMap = truncateToSize(filesSeq.flatten,
conf.get(Config.CONFIG_MAP_MAXSIZE))
+ logInfo(s"Spark configuration files loaded from $confDir :" +
Review comment:
Challenge with trying to truncate before reading is, we do not know
which files will have `MalformedInputException` i.e. they are not encoded with
UTF-8.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes commented on a change in pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
ScrapCodes commented on a change in pull request #30472:
URL: https://github.com/apache/spark/pull/30472#discussion_r537297478
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
##
@@ -51,6 +54,29 @@ private[spark] object KubernetesClientUtils extends Logging {
propertiesWriter.toString
}
+ object StringLengthOrdering extends Ordering[(String, String)] {
+override def compare(x: (String, String), y: (String, String)): Int = {
+ // compare based on file length and break the tie with string comparison
of keys.
+ (x._1.length + x._2.length).compare(y._1.length + y._2.length) * 10 +
+x._1.compareTo(y._1)
+}
+ }
+
+ def truncateToSize(seq: Seq[(String, String)], maxSize: Long): Map[String,
String] = {
+// First order the entries in order of their size.
+// Skip entries if the resulting Map size exceeds maxSize.
+val ordering: Ordering[(String, String)] = StringLengthOrdering
+val sortedSet = SortedSet[(String, String)](seq : _*)(ordering)
+var i: Int = 0
+val map = mutable.HashMap[String, String]()
+for (item <- sortedSet) {
+ i += item._1.length + item._2.length
+ if ( i < maxSize) {
Review comment:
P.S. I have evaluated the approach of trying to extend SortedSet, and
some how make it size limited i.e. items cannot be added beyond a size is
reached. But that was more complex in terms of code, so here I have chosen a
piece of code that is more simpler to maintain and does not need an overhauling
every time we upgrade scala major version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
SparkQA commented on pull request #30637: URL: https://github.com/apache/spark/pull/30637#issuecomment-739742191 **[Test build #132342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132342/testReport)** for PR 30637 at commit [`c14185e`](https://github.com/apache/spark/commit/c14185e0d7ba3c73415aef8e2653641c1b33f0ed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
yaooqinn commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739742054 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
SparkQA removed a comment on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739675368 **[Test build #132329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132329/testReport)** for PR 30639 at commit [`11c0241`](https://github.com/apache/spark/commit/11c024165e7d0ccc5b5c9f23e9406f8921528d2b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
SparkQA commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739741669 **[Test build #132329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132329/testReport)** for PR 30639 at commit [`11c0241`](https://github.com/apache/spark/commit/11c024165e7d0ccc5b5c9f23e9406f8921528d2b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] linhongliu-db commented on pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
linhongliu-db commented on pull request #30637: URL: https://github.com/apache/spark/pull/30637#issuecomment-739740497 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] linhongliu-db closed pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
linhongliu-db closed pull request #30637: URL: https://github.com/apache/spark/pull/30637 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30638: [SPARK-31677][SS] Remove reduplicate cache of stream query progress info in StreamingExecution
AmplabJenkins removed a comment on pull request #30638: URL: https://github.com/apache/spark/pull/30638#issuecomment-739739612 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36932/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30638: [SPARK-31677][SS] Remove reduplicate cache of stream query progress info in StreamingExecution
SparkQA commented on pull request #30638: URL: https://github.com/apache/spark/pull/30638#issuecomment-739739586 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36932/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30624: [SPARK-33676][SQL] Require exact matching of partition spec to the schema in V2 `ALTER TABLE .. ADD/DROP PARTITION`
cloud-fan commented on a change in pull request #30624:
URL: https://github.com/apache/spark/pull/30624#discussion_r537294768
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolvePartitionSpec.scala
##
@@ -51,14 +61,16 @@ object ResolvePartitionSpec extends Rule[LogicalPlan] {
private def resolvePartitionSpecs(
tableName: String,
partSpecs: Seq[PartitionSpec],
- partSchema: StructType): Seq[ResolvedPartitionSpec] =
+ partSchema: StructType,
+ checkSpec: TablePartitionSpec => Unit = _ => ()):
Seq[ResolvedPartitionSpec] =
Review comment:
Why do we make it so general as a callback function? Shall we just call
`requireExactMatchedPartitionSpec` inside this method?
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolvePartitionSpec.scala
##
@@ -51,14 +61,16 @@ object ResolvePartitionSpec extends Rule[LogicalPlan] {
private def resolvePartitionSpecs(
tableName: String,
partSpecs: Seq[PartitionSpec],
- partSchema: StructType): Seq[ResolvedPartitionSpec] =
+ partSchema: StructType,
+ checkSpec: TablePartitionSpec => Unit = _ => ()):
Seq[ResolvedPartitionSpec] =
Review comment:
Why do we make it so general as a callback function? Shall we just call
`requireExactMatchedPartitionSpec` inside this method?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30638: [SPARK-31677][SS] Remove reduplicate cache of stream query progress info in StreamingExecution
AmplabJenkins commented on pull request #30638: URL: https://github.com/apache/spark/pull/30638#issuecomment-739739612 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36932/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
AmplabJenkins removed a comment on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-739737943 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36937/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
AmplabJenkins commented on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-739737943 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36937/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
SparkQA commented on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-739737919 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36937/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30573: [SPARK-26341][WEBUI]Expose executor memory metrics at the stage level, in the Stages tab
AmplabJenkins removed a comment on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-739736810 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36935/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30573: [SPARK-26341][WEBUI]Expose executor memory metrics at the stage level, in the Stages tab
AmplabJenkins commented on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-739736810 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36935/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30573: [SPARK-26341][WEBUI]Expose executor memory metrics at the stage level, in the Stages tab
SparkQA commented on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-739736782 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36935/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
AmplabJenkins removed a comment on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739735290 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36934/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
SparkQA commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739735771 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36939/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes commented on a change in pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
ScrapCodes commented on a change in pull request #30472:
URL: https://github.com/apache/spark/pull/30472#discussion_r537291078
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
##
@@ -51,6 +54,29 @@ private[spark] object KubernetesClientUtils extends Logging {
propertiesWriter.toString
}
+ object StringLengthOrdering extends Ordering[(String, String)] {
+override def compare(x: (String, String), y: (String, String)): Int = {
+ // compare based on file length and break the tie with string comparison
of keys.
+ (x._1.length + x._2.length).compare(y._1.length + y._2.length) * 10 +
+x._1.compareTo(y._1)
+}
Review comment:
Good questions, and hoping that I have understood correctly, attempting
to answer.
1) We need to compare the file sizes _including their names_, because that
is how they will occupy space in a config map.
2) We would like to give priority to as many config files as possible we can
mount (or store in the configMap), so we would like to sort them by size.
3) If the two files are equal in lengths, then we do not want to declare
them equal, so in the compare equation we add their string compare results as
well. This is done to break the tie. So `*10` is done to give more priority to
compare by their length but if the comparison result is equal, i.e. two files
are exactly equal in length then we get zero as the comparison result. By
adding string comparison result of their names, i.e. when two files have equal
lengths, we check if the two files have same name as well ? Then we say they
are truly equal - in principle this should not happen, because they are files
in the same directory i.e. `SPARK_CONF_DIR`. So, each file will get some
ordering value as a result of computing comparison equation, and no data will
be lost.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
SparkQA commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739735265 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36934/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
AmplabJenkins commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739735290 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36934/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30481: [SPARK-33526][SQL] Add config to control if cancel invoke interrupt task on thriftserver
SparkQA commented on pull request #30481: URL: https://github.com/apache/spark/pull/30481#issuecomment-739735394 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36936/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
AmplabJenkins removed a comment on pull request #30637: URL: https://github.com/apache/spark/pull/30637#issuecomment-739733537 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132318/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
AmplabJenkins commented on pull request #30637: URL: https://github.com/apache/spark/pull/30637#issuecomment-739733537 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132318/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
SparkQA removed a comment on pull request #30637: URL: https://github.com/apache/spark/pull/30637#issuecomment-739642602 **[Test build #132318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132318/testReport)** for PR 30637 at commit [`c14185e`](https://github.com/apache/spark/commit/c14185e0d7ba3c73415aef8e2653641c1b33f0ed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30637: [SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query
SparkQA commented on pull request #30637: URL: https://github.com/apache/spark/pull/30637#issuecomment-739732931 **[Test build #132318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132318/testReport)** for PR 30637 at commit [`c14185e`](https://github.com/apache/spark/commit/c14185e0d7ba3c73415aef8e2653641c1b33f0ed). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30625: [SPARK-33677][SQL] Skip LikeSimplification rule if pattern contains any escapeChar
SparkQA commented on pull request #30625: URL: https://github.com/apache/spark/pull/30625#issuecomment-739732826 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36933/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
SparkQA commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739731930 **[Test build #132341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132341/testReport)** for PR 30559 at commit [`0475866`](https://github.com/apache/spark/commit/04758662955e4e5f88404c128f627172a14e6d2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
SparkQA commented on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739731529 **[Test build #132340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132340/testReport)** for PR 30636 at commit [`c23f61f`](https://github.com/apache/spark/commit/c23f61fb18f2cc012942f1990177706124e9d075). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET
cloud-fan commented on pull request #30045: URL: https://github.com/apache/spark/pull/30045#issuecomment-739731557 I think you are right about reality. But it seems safer to handle the per-session initial configs as it's a property of the `SparkSession`. It's kind of a potential bug and we'd better fix it earlier. @yaooqinn what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30598: [SPARK-33654][SQL] Migrate CACHE TABLE to use UnresolvedRelation to resolve identifier
AmplabJenkins removed a comment on pull request #30598: URL: https://github.com/apache/spark/pull/30598#issuecomment-739729851 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132326/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
AmplabJenkins removed a comment on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739729852 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132325/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
AmplabJenkins removed a comment on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739729846 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132319/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
AmplabJenkins removed a comment on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739729850 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36931/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
AmplabJenkins commented on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739729845 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36930/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
AmplabJenkins commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739729850 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36931/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
AmplabJenkins removed a comment on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739729845 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36930/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30598: [SPARK-33654][SQL] Migrate CACHE TABLE to use UnresolvedRelation to resolve identifier
AmplabJenkins commented on pull request #30598: URL: https://github.com/apache/spark/pull/30598#issuecomment-739729851 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132326/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
AmplabJenkins commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739729846 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132319/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
AmplabJenkins commented on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739729852 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132325/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30638: [SPARK-31677][SS] Remove reduplicate cache of stream query progress info in StreamingExecution
SparkQA commented on pull request #30638: URL: https://github.com/apache/spark/pull/30638#issuecomment-739725829 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36932/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30620: [SPARK-33671][SQL] Remove VIEW checks from V1 table commands
dongjoon-hyun closed pull request #30620: URL: https://github.com/apache/spark/pull/30620 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
SparkQA removed a comment on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739642644 **[Test build #132319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132319/testReport)** for PR 30586 at commit [`1f47ce0`](https://github.com/apache/spark/commit/1f47ce06b6de63004c7c3b1dde5071711f5bfe70). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
SparkQA commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739725350 **[Test build #132319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132319/testReport)** for PR 30586 at commit [`1f47ce0`](https://github.com/apache/spark/commit/1f47ce06b6de63004c7c3b1dde5071711f5bfe70). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
SparkQA commented on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-739725020 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36937/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30598: [SPARK-33654][SQL] Migrate CACHE TABLE to use UnresolvedRelation to resolve identifier
SparkQA removed a comment on pull request #30598: URL: https://github.com/apache/spark/pull/30598#issuecomment-739657691 **[Test build #132326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132326/testReport)** for PR 30598 at commit [`3cf6e7d`](https://github.com/apache/spark/commit/3cf6e7dc931a966a456bc17df2bcbcaa000edd89). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30598: [SPARK-33654][SQL] Migrate CACHE TABLE to use UnresolvedRelation to resolve identifier
SparkQA commented on pull request #30598: URL: https://github.com/apache/spark/pull/30598#issuecomment-739724613 **[Test build #132326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132326/testReport)** for PR 30598 at commit [`3cf6e7d`](https://github.com/apache/spark/commit/3cf6e7dc931a966a456bc17df2bcbcaa000edd89). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
SparkQA commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739724025 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36931/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30573: [SPARK-26341][WEBUI]Expose executor memory metrics at the stage level, in the Stages tab
SparkQA commented on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-739723762 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36935/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
SparkQA commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739722926 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36934/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
wangyum commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739717971 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
SparkQA commented on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739716511 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36930/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
dongjoon-hyun closed pull request #30634: URL: https://github.com/apache/spark/pull/30634 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
dongjoon-hyun commented on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739715592 Merged to master for Apache Spark 3.2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
SparkQA removed a comment on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739657204 **[Test build #132325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132325/testReport)** for PR 30634 at commit [`d6cf1c3`](https://github.com/apache/spark/commit/d6cf1c37d5fc0a88cbb58cf33735c0a7f7580b5e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
SparkQA commented on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739714735 **[Test build #132325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132325/testReport)** for PR 30634 at commit [`d6cf1c3`](https://github.com/apache/spark/commit/d6cf1c37d5fc0a88cbb58cf33735c0a7f7580b5e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
AmplabJenkins removed a comment on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739709730 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36929/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
SparkQA commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739709848 **[Test build #132339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132339/testReport)** for PR 30639 at commit [`03c3611`](https://github.com/apache/spark/commit/03c3611835df0bc0b1396efb3304e0c74d675ef3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
AmplabJenkins commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739709730 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36929/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30639: [DO-NOT-MERGE] Investigate HiveThriftHttpServerSuite test failures
SparkQA commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739709711 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36929/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #30639: [DO-NOT-MERGE] Revert "[SPARK-31953][SS] Add Spark Structured Streaming History Server Support"
HyukjinKwon removed a comment on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739706672 Hm .. let me investigate a bit more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30639: [DO-NOT-MERGE] Revert "[SPARK-31953][SS] Add Spark Structured Streaming History Server Support"
HyukjinKwon commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739706672 Hm .. let me investigate a bit more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30639: [DO-NOT-MERGE] Revert "[SPARK-31953][SS] Add Spark Structured Streaming History Server Support"
HyukjinKwon closed pull request #30639: URL: https://github.com/apache/spark/pull/30639 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
AmplabJenkins removed a comment on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739704020 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132331/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
SparkQA removed a comment on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739675458 **[Test build #132331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132331/testReport)** for PR 30559 at commit [`0475866`](https://github.com/apache/spark/commit/04758662955e4e5f88404c128f627172a14e6d2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
AmplabJenkins commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739704020 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132331/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
SparkQA commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739703759 **[Test build #132331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132331/testReport)** for PR 30559 at commit [`0475866`](https://github.com/apache/spark/commit/04758662955e4e5f88404c128f627172a14e6d2c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
SparkQA commented on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-739702575 **[Test build #132337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132337/testReport)** for PR 29966 at commit [`afea73f`](https://github.com/apache/spark/commit/afea73fcadc1d6641c6feb1b07b7fc874b836b29). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29122: [SPARK-32320][PYSPARK] Remove mutable default arguments
SparkQA commented on pull request #29122: URL: https://github.com/apache/spark/pull/29122#issuecomment-739702405 **[Test build #132338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132338/testReport)** for PR 29122 at commit [`11f3790`](https://github.com/apache/spark/commit/11f3790ac3ccc15efd7944e0fc1a3a2e3e8e3247). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30573: [SPARK-26341][WEBUI]Expose executor memory metrics at the stage level, in the Stages tab
SparkQA commented on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-739702243 **[Test build #132335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132335/testReport)** for PR 30573 at commit [`c85fd16`](https://github.com/apache/spark/commit/c85fd16af6f35c46bc20cccac9d5d947b7c74af8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30481: [SPARK-33526][SQL] Add config to control if cancel invoke interrupt task on thriftserver
SparkQA commented on pull request #30481: URL: https://github.com/apache/spark/pull/30481#issuecomment-739702208 **[Test build #132336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132336/testReport)** for PR 30481 at commit [`d1ebbba`](https://github.com/apache/spark/commit/d1ebbba35946262cff8787ec95190bc5f6db318f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30625: [SPARK-33677][SQL] Skip LikeSimplification rule if pattern contains any escapeChar
SparkQA commented on pull request #30625: URL: https://github.com/apache/spark/pull/30625#issuecomment-739702161 **[Test build #132333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132333/testReport)** for PR 30625 at commit [`65e0d63`](https://github.com/apache/spark/commit/65e0d63ad8bb5db14791f79d8a4e2572be493d3b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30638: [SPARK-31677][SS] Remove reduplicate cache of stream query progress info in StreamingExecution
SparkQA commented on pull request #30638: URL: https://github.com/apache/spark/pull/30638#issuecomment-739701749 **[Test build #132332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132332/testReport)** for PR 30638 at commit [`f9fb037`](https://github.com/apache/spark/commit/f9fb037d244d14ef97583bf5f19fcb06b7983c78). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30586: [SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results
SparkQA commented on pull request #30586: URL: https://github.com/apache/spark/pull/30586#issuecomment-739701825 **[Test build #132334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132334/testReport)** for PR 30586 at commit [`10544a6`](https://github.com/apache/spark/commit/10544a6e8ad6b8a409e239cdb96aee893d6c08a3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30598: [SPARK-33654][SQL] Migrate CACHE TABLE to use UnresolvedRelation to resolve identifier
AmplabJenkins removed a comment on pull request #30598: URL: https://github.com/apache/spark/pull/30598#issuecomment-739700828 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36926/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30628: [SPARK-33679][SQL] Enable spark.sql.adaptive.enabled by default
AmplabJenkins removed a comment on pull request #30628: URL: https://github.com/apache/spark/pull/30628#issuecomment-739700814 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
AmplabJenkins removed a comment on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739700816 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132324/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
AmplabJenkins removed a comment on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739700817 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36925/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30600: [SPARK-33655][SQL] Improve performance of processing FETCH_PRIOR
AmplabJenkins removed a comment on pull request #30600: URL: https://github.com/apache/spark/pull/30600#issuecomment-739700818 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36928/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30598: [SPARK-33654][SQL] Migrate CACHE TABLE to use UnresolvedRelation to resolve identifier
AmplabJenkins commented on pull request #30598: URL: https://github.com/apache/spark/pull/30598#issuecomment-739700828 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36926/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30600: [SPARK-33655][SQL] Improve performance of processing FETCH_PRIOR
AmplabJenkins commented on pull request #30600: URL: https://github.com/apache/spark/pull/30600#issuecomment-739700818 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36928/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
AmplabJenkins commented on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739700816 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132324/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30628: [SPARK-33679][SQL] Enable spark.sql.adaptive.enabled by default
AmplabJenkins commented on pull request #30628: URL: https://github.com/apache/spark/pull/30628#issuecomment-739700814 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30634: [SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13
AmplabJenkins commented on pull request #30634: URL: https://github.com/apache/spark/pull/30634#issuecomment-739700817 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36925/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30636: [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier
SparkQA commented on pull request #30636: URL: https://github.com/apache/spark/pull/30636#issuecomment-739700269 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36930/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30517: [DO-NOT-MERGE][test-maven] Test compatibility for Parquet 1.11.1, Avro 1.10.1 and Hive 2.3.8
wangyum commented on pull request #30517: URL: https://github.com/apache/spark/pull/30517#issuecomment-739699628 cc @gszadovszky @shangxinli The problem is that DataSource v2 will push down the partition filter to [`SpecificParquetRecordReaderBase`](https://github.com/apache/spark/blob/26badc4cc4ea6d68c8c5d50cf2c83e4904aacc0d/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L138-L153) or [`InternalParquetRecordReader`](https://github.com/apache/parquet-mr/blob/86808425d5f67ddb2b4c80b1d1a06015bc92be10/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/InternalParquetRecordReader.java#L185). If we use `reader.getFilteredRecordCount()`, the result is empty. Do you think it should be fixed on the Parquet side or Spark side? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
AngersZh commented on a change in pull request #29966:
URL: https://github.com/apache/spark/pull/29966#discussion_r537260301
##
File path: core/src/test/scala/org/apache/spark/SparkContextSuite.scala
##
@@ -366,6 +366,29 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
}
}
+ test("SPARK-33084: add jar when path contains comma") {
+withTempDir { tmpDir =>
+ val tmpJar = File.createTempFile("Test,UDTF", ".jar", tmpDir)
+ sc = new SparkContext(new
SparkConf().setAppName("test").setMaster("local"))
+ sc.addJar(tmpJar.getAbsolutePath)
+ assert(sc.listJars().exists(_.contains("Test,UDTF")))
Review comment:
> [#29966
(comment)](https://github.com/apache/spark/pull/29966#issuecomment-739631644)
hm, I think it would be better to check jars are added correctly in executors,
too.
Yea， will update a test case later
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path
maropu commented on a change in pull request #29966:
URL: https://github.com/apache/spark/pull/29966#discussion_r537259934
##
File path: core/src/test/scala/org/apache/spark/SparkContextSuite.scala
##
@@ -366,6 +366,29 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
}
}
+ test("SPARK-33084: add jar when path contains comma") {
+withTempDir { tmpDir =>
+ val tmpJar = File.createTempFile("Test,UDTF", ".jar", tmpDir)
+ sc = new SparkContext(new
SparkConf().setAppName("test").setMaster("local"))
+ sc.addJar(tmpJar.getAbsolutePath)
+ assert(sc.listJars().exists(_.contains("Test,UDTF")))
Review comment:
https://github.com/apache/spark/pull/29966#issuecomment-739631644 hm, I
think it would be better to check jars are added correctly in executors, too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29950: [SPARK-32945][SQL] Avoid collapsing projects if reaching max allowed common exprs
viirya commented on pull request #29950: URL: https://github.com/apache/spark/pull/29950#issuecomment-739698601 I recently generalized subexpression elimination feature to interpreted project and predicate. So now both whole-stage codegen and interpreted execution support subexpression elimination that could avoid the performance issue caused by embedding common expressions from collapsing projects. That's said, I think this patch is less useful for now. I'm closing it now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya closed pull request #29950: [SPARK-32945][SQL] Avoid collapsing projects if reaching max allowed common exprs
viirya closed pull request #29950: URL: https://github.com/apache/spark/pull/29950 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30628: [SPARK-33679][SQL] Enable spark.sql.adaptive.enabled by default
SparkQA removed a comment on pull request #30628: URL: https://github.com/apache/spark/pull/30628#issuecomment-739658360 **[Test build #132327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132327/testReport)** for PR 30628 at commit [`682a2ec`](https://github.com/apache/spark/commit/682a2ec9d80c49c905b10ff6eb2a0bb5dccad81d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30628: [SPARK-33679][SQL] Enable spark.sql.adaptive.enabled by default
SparkQA commented on pull request #30628: URL: https://github.com/apache/spark/pull/30628#issuecomment-739697824 **[Test build #132327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132327/testReport)** for PR 30628 at commit [`682a2ec`](https://github.com/apache/spark/commit/682a2ec9d80c49c905b10ff6eb2a0bb5dccad81d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET
yaooqinn commented on pull request #30045: URL: https://github.com/apache/spark/pull/30045#issuecomment-739697379 > @yaooqinn active session is a thread local. What if we create a new session in another thread? w/o `SparkSession.clearDefaultSession()`, we will not create a new one  w/ `SparkSession.clearDefaultSession()`, the shared state is also not able to reach  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30639: [DO-NOT-MERGE] Revert "[SPARK-31953][SS] Add Spark Structured Streaming History Server Support"
SparkQA commented on pull request #30639: URL: https://github.com/apache/spark/pull/30639#issuecomment-739696401 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36929/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30559: [SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries
SparkQA commented on pull request #30559: URL: https://github.com/apache/spark/pull/30559#issuecomment-739696316 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36931/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn removed a comment on pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET
yaooqinn removed a comment on pull request #30045: URL: https://github.com/apache/spark/pull/30045#issuecomment-739694075 > @yaooqinn active session is a thread local. What if we create a new session in another thread? In this case, then we have to clean the default session ahead This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET
yaooqinn commented on pull request #30045: URL: https://github.com/apache/spark/pull/30045#issuecomment-739694075 > @yaooqinn active session is a thread local. What if we create a new session in another thread? In this case, then we have to clean the default session ahead This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #30624: [SPARK-33676][SQL] Require exact matching of partition spec to the schema in V2 `ALTER TABLE .. ADD/DROP PARTITION`
MaxGekk commented on pull request #30624: URL: https://github.com/apache/spark/pull/30624#issuecomment-739693369 @stczwd @cloud-fan Could you review this PR, please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
