[GitHub] [spark] SparkQA commented on issue #25389: [SPARK-28657][CORE] Fix currentContext Instance failed sometimes
SparkQA commented on issue #25389: [SPARK-28657][CORE] Fix currentContext Instance failed sometimes URL: https://github.com/apache/spark/pull/25389#issuecomment-527730765 **[Test build #110076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110076/testReport)** for PR 25389 at commit [`6dced2a`](https://github.com/apache/spark/commit/6dced2af9c6311bd9bd3390abc12ebe522b39213). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
HyukjinKwon commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527730590 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25661: [SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
SparkQA commented on issue #25661: [SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527729813 **[Test build #110086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110086/testReport)** for PR 25661 at commit [`9f6b4f0`](https://github.com/apache/spark/commit/9f6b4f0afc84351ce3df8e676591765c39945e4b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527729821 **[Test build #110087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110087/testReport)** for PR 20965 at commit [`c3bf1ac`](https://github.com/apache/spark/commit/c3bf1ac83a521fdb3b99037726f4c9ee8d8d7bef). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527729390 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15102/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527729384 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527729384 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527729390 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15102/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527728928 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527728932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110085/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527728928 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527728852 **[Test build #110085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110085/testReport)** for PR 25309 at commit [`bb29488`](https://github.com/apache/spark/commit/bb2948811e42eba2b99a1d118a5eb4559cae57bb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527728932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110085/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527725294 **[Test build #110085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110085/testReport)** for PR 25309 at commit [`bb29488`](https://github.com/apache/spark/commit/bb2948811e42eba2b99a1d118a5eb4559cae57bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-527725294 **[Test build #110085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110085/testReport)** for PR 25309 at commit [`bb29488`](https://github.com/apache/spark/commit/bb2948811e42eba2b99a1d118a5eb4559cae57bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527723395 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
SparkQA removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527721092 **[Test build #110084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110084/testReport)** for PR 25649 at commit [`3e72e66`](https://github.com/apache/spark/commit/3e72e662182e565c9b0b5e6fa34d1ea08e1affd0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527723404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110084/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527723404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110084/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527723395 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527723335 **[Test build #110084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110084/testReport)** for PR 25649 at commit [`3e72e66`](https://github.com/apache/spark/commit/3e72e662182e565c9b0b5e6fa34d1ea08e1affd0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split
aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r320557819
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -824,59 +944,158 @@ case class HashAggregateExec(
// generating input columns, we use `currentVars`.
ctx.currentVars = new Array[ExprCode](aggregateBufferAttributes.length) ++
input
+val aggNames = aggregateExpressions.map(_.aggregateFunction.prettyName)
+// Computes start offsets for each aggregation function code
+// in the underlying buffer row.
+val bufferStartOffsets = {
+ val offsets = mutable.ArrayBuffer[Int]()
+ var curOffset = 0
+ updateExprs.foreach { exprsForOneFunc =>
+offsets += curOffset
+curOffset += exprsForOneFunc.length
+ }
+ offsets.toArray
+}
+
val updateRowInRegularHashMap: String = {
ctx.INPUT_ROW = unsafeRowBuffer
- val boundUpdateExpr = bindReferences(updateExpr, inputAttr)
- val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val boundUpdateExprs = updateExprs.map { updateExprsForOneFunc =>
+bindReferences(updateExprsForOneFunc, inputAttr)
+ }
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExprs.flatten)
val effectiveCodes = subExprs.codes.mkString("\n")
- val unsafeRowBufferEvals =
ctx.withSubExprEliminationExprs(subExprs.states) {
-boundUpdateExpr.map(_.genCode(ctx))
+ val unsafeRowBufferEvals = boundUpdateExprs.map {
boundUpdateExprsForOneFunc =>
+ctx.withSubExprEliminationExprs(subExprs.states) {
+ boundUpdateExprsForOneFunc.map(_.genCode(ctx))
+}
}
- val updateUnsafeRowBuffer = unsafeRowBufferEvals.zipWithIndex.map { case
(ev, i) =>
-val dt = updateExpr(i).dataType
-CodeGenerator.updateColumn(unsafeRowBuffer, dt, i, ev,
updateExpr(i).nullable)
+
+ val aggCodeBlocks = updateExprs.indices.map { i =>
+val rowBufferEvalsForOneFunc = unsafeRowBufferEvals(i)
+val boundUpdateExprsForOneFunc = boundUpdateExprs(i)
+val bufferOffset = bufferStartOffsets(i)
+
+// All the update code for aggregation buffers should be placed in the
end
+// of each aggregation function code.
+val updateRowBuffers = rowBufferEvalsForOneFunc.zipWithIndex.map {
case (ev, j) =>
+ val updateExpr = boundUpdateExprsForOneFunc(j)
+ val dt = updateExpr.dataType
+ val nullable = updateExpr.nullable
+ CodeGenerator.updateColumn(unsafeRowBuffer, dt, bufferOffset + j,
ev, nullable)
+}
+code"""
+ |// evaluate aggregate function for ${aggNames(i)}
+ |${evaluateVariables(rowBufferEvalsForOneFunc)}
+ |// update unsafe row buffer
+ |${updateRowBuffers.mkString("\n").trim}
+ """.stripMargin
+ }
+
+ lazy val nonSplitAggCode = {
+s"""
+ |// common sub-expressions
+ |$effectiveCodes
+ |// evaluate aggregate functions and update aggregation buffers
+ |${aggCodeBlocks.fold(EmptyBlock)(_ + _)}
+ """.stripMargin
+ }
+
+ if (conf.codegenSplitAggregateFunc &&
+ aggCodeBlocks.map(_.length).sum > conf.methodSplitThreshold) {
+val maybeSplitCode = splitAggregateExpressions(
+ ctx, aggNames, boundUpdateExprs, aggCodeBlocks, subExprs.states)
+
+maybeSplitCode.map { updateAggCode =>
+ s"""
+ |// do aggregate
+ |// common sub-expressions
+ |$effectiveCodes
+ |// evaluate aggregate functions and update aggregation buffers
+ |$updateAggCode
+ """.stripMargin
+}.getOrElse {
+ nonSplitAggCode
+}
+ } else {
+nonSplitAggCode
}
- s"""
- |// common sub-expressions
- |$effectiveCodes
- |// evaluate aggregate function
- |${evaluateVariables(unsafeRowBufferEvals)}
- |// update unsafe row buffer
- |${updateUnsafeRowBuffer.mkString("\n").trim}
- """.stripMargin
}
val updateRowInHashMap: String = {
if (isFastHashMapEnabled) {
if (isVectorizedHashMapEnabled) {
ctx.INPUT_ROW = fastRowBuffer
- val boundUpdateExpr = bindReferences(updateExpr, inputAttr)
- val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val boundUpdateExprs = updateExprs.map { updateExprsForOneFunc =>
+bindReferences(updateExprsForOneFunc, inputAttr)
+ }
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExprs.flatten)
val effectiveCodes = subExprs.codes.mkString("\n")
- val fastRowEvals =
[GitHub] [spark] AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527722077 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15101/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527722075 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527722077 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15101/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527722075 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527721493 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110082/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
SparkQA removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527719707 **[Test build #110082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110082/testReport)** for PR 25671 at commit [`0b72f6d`](https://github.com/apache/spark/commit/0b72f6df16da8a19ae0d3cf26cccf8049002c4ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527721491 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25670: [SPARK-28869][CORE] Roll over event log files
AmplabJenkins removed a comment on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-527721550 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110073/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25670: [SPARK-28869][CORE] Roll over event log files
AmplabJenkins removed a comment on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-527721549 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
SparkQA commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527721483 **[Test build #110082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110082/testReport)** for PR 25671 at commit [`0b72f6d`](https://github.com/apache/spark/commit/0b72f6df16da8a19ae0d3cf26cccf8049002c4ff). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait HasNumFeatures extends Params ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527721493 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110082/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25670: [SPARK-28869][CORE] Roll over event log files
AmplabJenkins commented on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-527721549 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25670: [SPARK-28869][CORE] Roll over event log files
AmplabJenkins commented on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-527721550 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110073/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527721491 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25670: [SPARK-28869][CORE] Roll over event log files
SparkQA removed a comment on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-527695526 **[Test build #110073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110073/testReport)** for PR 25670 at commit [`4bb9de0`](https://github.com/apache/spark/commit/4bb9de012468e3602e5e0fc836c97923bf10efe0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25670: [SPARK-28869][CORE] Roll over event log files
SparkQA commented on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-527721211 **[Test build #110073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110073/testReport)** for PR 25670 at commit [`4bb9de0`](https://github.com/apache/spark/commit/4bb9de012468e3602e5e0fc836c97923bf10efe0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527721092 **[Test build #110084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110084/testReport)** for PR 25649 at commit [`3e72e66`](https://github.com/apache/spark/commit/3e72e662182e565c9b0b5e6fa34d1ea08e1affd0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
SparkQA commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527719707 **[Test build #110082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110082/testReport)** for PR 25671 at commit [`0b72f6d`](https://github.com/apache/spark/commit/0b72f6df16da8a19ae0d3cf26cccf8049002c4ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
SparkQA commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527719700 **[Test build #110083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110083/testReport)** for PR 25601 at commit [`9f738cf`](https://github.com/apache/spark/commit/9f738cf6708af6f4c5bf8bbaa981d7c69f3bdbd3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527719330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15099/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins removed a comment on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527719326 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527719372 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15100/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527719367 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
imback82 commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320554867
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowDatabasesExec.scala
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalog.v2.SupportsNamespaces
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing databases.
+ */
+case class ShowDatabasesExec(
+output: Seq[Attribute],
+catalog: SupportsNamespaces,
+pattern: Option[String])
+extends LeafExecNode {
+ override protected def doExecute(): RDD[InternalRow] = {
+val namespaces = catalog.listNamespaces().flatMap(getNamespaces(catalog,
_))
Review comment:
@cloud-fan @rdblue Thanks for your suggestions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527719372 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15100/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527719367 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527719326 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
AmplabJenkins commented on issue #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671#issuecomment-527719330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15099/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25570: [SPARK-23519][SQL] create view should work from query with duplicate output columns
gatorsmile commented on issue #25570: [SPARK-23519][SQL] create view should work from query with duplicate output columns URL: https://github.com/apache/spark/pull/25570#issuecomment-527718886 @hem1891 Could you open a PR against 2.4? We can backport it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng opened a new pull request #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side
zhengruifeng opened a new pull request #25671: [SPARK-28968][ML] Add HasNumFeatures in the scala side URL: https://github.com/apache/spark/pull/25671 ### What changes were proposed in this pull request? Add HasNumFeatures in the scala side, with `1<<18` as the default value ### Why are the changes needed? HasNumFeatures is already added in the py side, it is reasonable to keep them in sync. I don't find other similar place. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing testsuites This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
SparkQA commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527718215 **[Test build #110081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110081/testReport)** for PR 25601 at commit [`ba1e7f4`](https://github.com/apache/spark/commit/ba1e7f477a50408eac0e1795d21dd56ac5c4ed5d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zzcclp commented on issue #25511: [SPARK-22955][DSTREAMS] - graceful shutdown shouldn't lead to job gen…
zzcclp commented on issue #25511: [SPARK-22955][DSTREAMS] - graceful shutdown shouldn't lead to job gen… URL: https://github.com/apache/spark/pull/25511#issuecomment-527717908 Thank @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zzcclp commented on issue #25439: [SPARK-28709][DSTREAMS] Fix StreamingContext leak through Streaming
zzcclp commented on issue #25439: [SPARK-28709][DSTREAMS] Fix StreamingContext leak through Streaming URL: https://github.com/apache/spark/pull/25439#issuecomment-527718000 Thank @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527717754 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527717759 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15098/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527717754 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527717759 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15098/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
imback82 commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320552696
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowNamespacesExec.scala
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalog.v2.SupportsNamespaces
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing namespaces.
+ */
+case class ShowNamespacesExec(
+output: Seq[Attribute],
+catalog: SupportsNamespaces,
+namespace: Option[Seq[String]],
+pattern: Option[String])
+extends LeafExecNode {
+ override protected def doExecute(): RDD[InternalRow] = {
+val namespaces = namespace.map{ ns =>
+if (ns.nonEmpty) {
Review comment:
@rdblue @cloud-fan this is for handling the case `SHOW NAMESPACES IN
catalogname`. In this case, should we list the root namespaces or call
`listNamespaces` with an empty array?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#discussion_r320552073 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -1515,6 +1519,7 @@ MINUTES: 'MINUTES'; MONTH: 'MONTH'; MONTHS: 'MONTHS'; MSCK: 'MSCK'; +NAMESPACES: 'NAMESPACES'; Review comment: I need to update `docs/sql-keywords.md` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#discussion_r320551791 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -92,6 +92,8 @@ statement | DROP database (IF EXISTS)? db=errorCapturingIdentifier (RESTRICT | CASCADE)? #dropDatabase | SHOW DATABASES (LIKE? pattern=STRING)? #showDatabases +| SHOW NAMESPACES ((FROM | IN) multipartIdentifier)? Review comment: I put both `FROM` and `IN` similar to `SHOW TABLES`. Please let me know if `FROM` is not needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
imback82 commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320552362
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/InMemoryTableCatalog.scala
##
@@ -30,12 +30,9 @@ import org.apache.spark.sql.sources.v2.Table
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class InMemoryTableCatalog extends TableCatalog with SupportsNamespaces {
+class InMemoryTableCatalogBase extends TableCatalog {
Review comment:
I tried to have a catalog without `SupportsNamespace`. If you have a better
name, please let me know. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
SparkQA commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527716760 **[Test build #110080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110080/testReport)** for PR 25601 at commit [`9974a58`](https://github.com/apache/spark/commit/9974a58d65bce948e5ab6f27ec64aa7c587598f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527716401 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527716401 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins commented on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527716397 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25601#issuecomment-527716397 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527714715 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110075/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type
AmplabJenkins removed a comment on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type URL: https://github.com/apache/spark/pull/25277#issuecomment-527714660 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25601: [SPARK-28856][SQL] Implement SHOW DATABASES for Data Source V2 Tables
imback82 commented on a change in pull request #25601: [SPARK-28856][SQL]
Implement SHOW DATABASES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25601#discussion_r320550781
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -2296,6 +2296,13 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
}
}
+ /**
+ * Create a [[ShowDatabasesStatement]] command.
+ */
+ override def visitShowDatabases(ctx: ShowDatabasesContext): LogicalPlan =
withOrigin(ctx) {
Review comment:
Changed to Namespace*
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527698591 **[Test build #110075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110075/testReport)** for PR 20965 at commit [`14e9a0e`](https://github.com/apache/spark/commit/14e9a0e7e43c6badd39dfe7b7e4fa1b772324db2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type
SparkQA removed a comment on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type URL: https://github.com/apache/spark/pull/25277#issuecomment-527710687 **[Test build #110077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110077/testReport)** for PR 25277 at commit [`59b3598`](https://github.com/apache/spark/commit/59b35986d18360e00dc760e6b51f2098532185f1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527714710 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type
AmplabJenkins removed a comment on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type URL: https://github.com/apache/spark/pull/25277#issuecomment-527714664 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110077/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527714710 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527714715 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110075/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type
AmplabJenkins commented on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type URL: https://github.com/apache/spark/pull/25277#issuecomment-527714664 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110077/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type
AmplabJenkins commented on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type URL: https://github.com/apache/spark/pull/25277#issuecomment-527714660 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527714336 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110078/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type
SparkQA commented on issue #25277: [SPARK-28637][SQL] Thriftserver support interval type URL: https://github.com/apache/spark/pull/25277#issuecomment-527714601 **[Test build #110077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110077/testReport)** for PR 25277 at commit [`59b3598`](https://github.com/apache/spark/commit/59b35986d18360e00dc760e6b51f2098532185f1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-527714630 **[Test build #110075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110075/testReport)** for PR 20965 at commit [`14e9a0e`](https://github.com/apache/spark/commit/14e9a0e7e43c6badd39dfe7b7e4fa1b772324db2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
SparkQA removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527712250 **[Test build #110078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110078/testReport)** for PR 25661 at commit [`2d8ea81`](https://github.com/apache/spark/commit/2d8ea81802271dd3903ee915ab83a8c5a12958bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527714334 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527714336 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110078/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527714334 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
SparkQA commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527714323 **[Test build #110078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110078/testReport)** for PR 25661 at commit [`2d8ea81`](https://github.com/apache/spark/commit/2d8ea81802271dd3903ee915ab83a8c5a12958bb). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527713369 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins removed a comment on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527713371 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527713369 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
zhengruifeng commented on a change in pull request #21632:
[SPARK-19591][ML][MLlib] Add sample weights to decision trees
URL: https://github.com/apache/spark/pull/21632#discussion_r320549522
##
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala
##
@@ -37,7 +37,7 @@ import org.apache.spark.sql.types.{DataType, DoubleType,
StructType}
* Note: Marked as private and DeveloperApi since this may be made public in
the future.
*/
private[ml] trait DecisionTreeParams extends PredictorParams
- with HasCheckpointInterval with HasSeed {
+ with HasCheckpointInterval with HasSeed with HasWeightCol {
Review comment:
@imatiach-msft @srowen Here params `weightCol` and
`minWeightFractionPerNode` are introduced into `DecisionTreeParams` and also
exposed to RF and GBT.
But RF and GBT do not support sample weighting for now. Is there any plan to
support it? or we should put these params into `DecisionTreeRegressorParams`
and `DecisionTreeClassifierParams`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar"
AmplabJenkins commented on issue #25661: [WIP][SPARK-28957][SQL] Copy any "spark.hive.foo=bar" spark properties into hadoop conf as "hive.foo=bar" URL: https://github.com/apache/spark/pull/25661#issuecomment-527713371 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision trees
zhengruifeng commented on a change in pull request #21632:
[SPARK-19591][ML][MLlib] Add sample weights to decision trees
URL: https://github.com/apache/spark/pull/21632#discussion_r320549522
##
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala
##
@@ -37,7 +37,7 @@ import org.apache.spark.sql.types.{DataType, DoubleType,
StructType}
* Note: Marked as private and DeveloperApi since this may be made public in
the future.
*/
private[ml] trait DecisionTreeParams extends PredictorParams
- with HasCheckpointInterval with HasSeed {
+ with HasCheckpointInterval with HasSeed with HasWeightCol {
Review comment:
@imatiach-msft Params `weightCol` and `minWeightFractionPerNode` are
introduced `DecisionTreeParams` and also exposed to RF and GBT.
But RF and GBT do not support sample weighting for now. Is there any plan to
support it? or we should put these params into `DecisionTreeRegressorParams`
and `DecisionTreeClassifierParams`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712588 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110079/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712585 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
SparkQA removed a comment on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712265 **[Test build #110079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110079/testReport)** for PR 25649 at commit [`cfd19ef`](https://github.com/apache/spark/commit/cfd19ef5d2282bcd849121d466d2bbab69b70eb7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712588 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110079/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712574 **[Test build #110079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110079/testReport)** for PR 25649 at commit [`cfd19ef`](https://github.com/apache/spark/commit/cfd19ef5d2282bcd849121d466d2bbab69b70eb7). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public final class JavaStructuredKerberizedKafkaWordCount ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712585 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples
SparkQA commented on issue #25649: [SPARK-28694][EXAMPLES]Add Java/Scala StructuredKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25649#issuecomment-527712265 **[Test build #110079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110079/testReport)** for PR 25649 at commit [`cfd19ef`](https://github.com/apache/spark/commit/cfd19ef5d2282bcd849121d466d2bbab69b70eb7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
