[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361094602
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -218,6 +252,11 @@ object BinaryArithmetic {
""")
case class Add(left: Expression, right: Expression) extends BinaryArithmetic {
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
then let's be consistent and follow java overflow behavior when ansi is
false.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
AmplabJenkins removed a comment on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568684148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20501/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
AmplabJenkins removed a comment on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568684141 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26946: [SPARK-30036][SQL] Fix: REPARTITION hint does not work with order by
cloud-fan commented on a change in pull request #26946: [SPARK-30036][SQL] Fix:
REPARTITION hint does not work with order by
URL: https://github.com/apache/spark/pull/26946#discussion_r361093943
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
##
@@ -421,6 +421,52 @@ class PlannerSuite extends SharedSparkSession {
}
}
+ test("SPARK-30036: Romove unnecessary RoundRobinPartitioning " +
+ "if SortExec is followed by RoundRobinPartitioning") {
+val distribution = OrderedDistribution(SortOrder(Literal(1), Ascending) ::
Nil)
+val partitioning = RoundRobinPartitioning(5)
+assert(!partitioning.satisfies(distribution))
+
+val inputPlan = SortExec(SortOrder(Literal(1), Ascending) :: Nil,
+ global = true,
+ child = ShuffleExchangeExec(
+partitioning,
+DummySparkPlan(outputPartitioning = partitioning)))
+val outputPlan =
EnsureRequirements(spark.sessionState.conf).apply(inputPlan)
+assert(outputPlan.find {
+ case ShuffleExchangeExec(_: RoundRobinPartitioning, _, _) => true
+ case _ => false
+}.isEmpty,
+ "RoundRobinPartitioning should be changed to RangePartitioning")
+
+val query = testData.select('key, 'value).repartition(2).sort('key.asc)
+assert(query.rdd.getNumPartitions == 2)
+assert(query.rdd.collectPartitions()(0).map(_.get(0)).toSeq == (1 to 50))
+ }
+
+ test("SPARK-30036: Romove unnecessary HashPartitioning " +
+"if SortExec is followed by HashPartitioning") {
+val distribution = OrderedDistribution(SortOrder(Literal(1), Ascending) ::
Nil)
+val partitioning = HashPartitioning(Literal(1) :: Nil, 5)
+assert(!partitioning.satisfies(distribution))
+
+val inputPlan = SortExec(SortOrder(Literal(1), Ascending) :: Nil,
+ global = true,
+ child = ShuffleExchangeExec(
+partitioning,
+DummySparkPlan(outputPartitioning = partitioning)))
+val outputPlan =
EnsureRequirements(spark.sessionState.conf).apply(inputPlan)
+assert(outputPlan.find {
+ case ShuffleExchangeExec(_: HashPartitioning, _, _) => true
+ case _ => false
+}.isEmpty,
+ "HashPartitioning should be changed to RangePartitioning")
+
+val query = testData.select('key, 'value).repartition(5,
'key).sort('key.asc)
Review comment:
I'm not very sure about this. `df.repartition` is a low-level API that
allows users to hash-partition the data. There is also a
`df.repartitionByRange` to do range partitioning. We shouldn't break users'
expectations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
maropu commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568684923 In addition, can you add more examples for quoted paths of ADD FILE/JAR? https://github.com/apache/spark/pull/26779 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361093504
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -218,6 +252,11 @@ object BinaryArithmetic {
""")
case class Add(left: Expression, right: Expression) extends BinaryArithmetic {
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
Yes, we have.
we now have `+/-/unaray_-` in java overflow behavior(2.4 or maybe earlier)
and we have '*/-' in null for overflow behavior (3.0)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #22881: [SPARK-25855][CORE] Don't use erasure coding for event logs by default
gatorsmile commented on a change in pull request #22881: [SPARK-25855][CORE] Don't use erasure coding for event logs by default URL: https://github.com/apache/spark/pull/22881#discussion_r361093119 ## File path: docs/configuration.md ## @@ -761,6 +761,17 @@ Apart from these, the following properties are also available, and may be useful Compression will use spark.io.compression.codec. + + spark.eventLog.allowErasureCoding Review comment: cc @Ngone51 Could you submit a PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
AmplabJenkins commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568684148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20501/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
AmplabJenkins commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568684141 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #22881: [SPARK-25855][CORE] Don't use erasure coding for event logs by default
gatorsmile commented on a change in pull request #22881: [SPARK-25855][CORE] Don't use erasure coding for event logs by default URL: https://github.com/apache/spark/pull/22881#discussion_r361093055 ## File path: docs/configuration.md ## @@ -761,6 +761,17 @@ Apart from these, the following properties are also available, and may be useful Compression will use spark.io.compression.codec. + + spark.eventLog.allowErasureCoding Review comment: Change `spark.eventLog.allowErasureCoding` to `spark.eventLog.allowErasureCoding.enabled` ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
SparkQA commented on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch URL: https://github.com/apache/spark/pull/26930#issuecomment-568683847 **[Test build #115706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115706/testReport)** for PR 26930 at commit [`555c3a0`](https://github.com/apache/spark/commit/555c3a0a896d909df36e05a2588442685c1929e5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
SparkQA commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568683861 **[Test build #115705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115705/testReport)** for PR 26996 at commit [`3f5e19f`](https://github.com/apache/spark/commit/3f5e19f9a15803671cbad443141ca0faeef79f19). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
AmplabJenkins removed a comment on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568667892 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation
cloud-fan commented on issue #26996: [SPARK-30342][SQL][DOC]Update LIST FILE/JAR command Documentation URL: https://github.com/apache/spark/pull/26996#issuecomment-568683564 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever
cloud-fan commented on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever URL: https://github.com/apache/spark/pull/26975#issuecomment-568682889 I don't think it's safe to not reschedule it. Looking at the comment in `executorLost`, we want to reschedule because the shuffle files are all lost in this executor. The special case is `killedByOtherAttempt.contains(tid)`, which means a speculative task has finished on **another executor**. For the stage attempt, there is no guarantee that 2 tasks of the same partition will be run on different executors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
AmplabJenkins removed a comment on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch URL: https://github.com/apache/spark/pull/26930#issuecomment-568682363 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20500/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
AmplabJenkins removed a comment on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch URL: https://github.com/apache/spark/pull/26930#issuecomment-568682359 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568682318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20499/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568682314 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568682314 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568682318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20499/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
AmplabJenkins commented on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch URL: https://github.com/apache/spark/pull/26930#issuecomment-568682359 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
AmplabJenkins commented on issue #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch URL: https://github.com/apache/spark/pull/26930#issuecomment-568682363 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20500/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
SparkQA commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-568682028 **[Test build #115704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115704/testReport)** for PR 26973 at commit [`170944c`](https://github.com/apache/spark/commit/170944c488132f5f5e51d8e1a080f52f48c41df6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
viirya commented on a change in pull request #26930: [SPARK-30290][Core] Count
for merged block when fetching continuous blocks in batch
URL: https://github.com/apache/spark/pull/26930#discussion_r361090792
##
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##
@@ -337,14 +337,25 @@ final class ShuffleBlockFetcherIterator(
assertPositiveBlockSize(blockId, size)
curBlocks += FetchBlockInfo(blockId, size, mapIndex)
curRequestSize += size
- if (curRequestSize >= targetRemoteRequestSize ||
- curBlocks.size >= maxBlocksInFlightPerAddress) {
+ // For batch fetch, the actual block in flight should count for merged
block.
+ val exceedsMaxBlocksInFlightPerAddress = !doBatchFetch &&
+curBlocks.size >= maxBlocksInFlightPerAddress
+ if (curRequestSize >= targetRemoteRequestSize ||
exceedsMaxBlocksInFlightPerAddress) {
// Add this FetchRequest
val mergedBlocks = mergeContinuousShuffleBlockIdsIfNeeded(curBlocks)
-collectedRemoteRequests += new FetchRequest(address, mergedBlocks)
-logDebug(s"Creating fetch request of $curRequestSize at $address "
- + s"with ${mergedBlocks.size} blocks")
+ .grouped(maxBlocksInFlightPerAddress)
curBlocks = new ArrayBuffer[FetchBlockInfo]
+mergedBlocks.foreach { mergedBlock =>
+ if (mergedBlock.size == maxBlocksInFlightPerAddress) {
+collectedRemoteRequests += new FetchRequest(address, mergedBlock)
+logDebug(s"Creating fetch request of $curRequestSize at $address "
+ + s"with ${mergedBlock.size} blocks")
+ } else {
+// The last group might not exceed `maxBlocksInFlightPerAddress`.
Put it back
Review comment:
oh, yea. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
SparkQA commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568682043 **[Test build #115703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115703/testReport)** for PR 26995 at commit [`7293377`](https://github.com/apache/spark/commit/729337756261a26a2dc376ede9db05a52f238c76). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361089810
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -218,6 +252,11 @@ object BinaryArithmetic {
""")
case class Add(left: Expression, right: Expression) extends BinaryArithmetic {
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
in spark 2.4, do we have internal arithmetic operations? The non-ANSI
behavior should follow the old behavior.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361089663
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -75,12 +80,29 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
"""})
case _: CalendarIntervalType =>
val iu = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
- defineCodeGen(ctx, ev, c => s"$iu.negate($c)")
+ nullSafeCodeGen(ctx, ev, interval => s"""
+try {
+ ${ev.value} = $iu.negate($interval);
Review comment:
Seems a safeNegate need a patch for generated code, like
```java
s"""
| if (${ev.value} = null) {
|(${ev.isNull} = true;
|}
"""
```, which does not make thing better
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568680558 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20497/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
AmplabJenkins commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-568680540 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568680552 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568680552 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
AmplabJenkins commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-568680544 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20498/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
AmplabJenkins removed a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-568680544 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20498/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
AmplabJenkins removed a comment on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-568680540 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568680558 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20497/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
cloud-fan commented on a change in pull request #26930: [SPARK-30290][Core]
Count for merged block when fetching continuous blocks in batch
URL: https://github.com/apache/spark/pull/26930#discussion_r361089166
##
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##
@@ -337,14 +337,25 @@ final class ShuffleBlockFetcherIterator(
assertPositiveBlockSize(blockId, size)
curBlocks += FetchBlockInfo(blockId, size, mapIndex)
curRequestSize += size
- if (curRequestSize >= targetRemoteRequestSize ||
- curBlocks.size >= maxBlocksInFlightPerAddress) {
+ // For batch fetch, the actual block in flight should count for merged
block.
+ val exceedsMaxBlocksInFlightPerAddress = !doBatchFetch &&
+curBlocks.size >= maxBlocksInFlightPerAddress
+ if (curRequestSize >= targetRemoteRequestSize ||
exceedsMaxBlocksInFlightPerAddress) {
// Add this FetchRequest
val mergedBlocks = mergeContinuousShuffleBlockIdsIfNeeded(curBlocks)
-collectedRemoteRequests += new FetchRequest(address, mergedBlocks)
-logDebug(s"Creating fetch request of $curRequestSize at $address "
- + s"with ${mergedBlocks.size} blocks")
+ .grouped(maxBlocksInFlightPerAddress)
curBlocks = new ArrayBuffer[FetchBlockInfo]
+mergedBlocks.foreach { mergedBlock =>
+ if (mergedBlock.size == maxBlocksInFlightPerAddress) {
+collectedRemoteRequests += new FetchRequest(address, mergedBlock)
+logDebug(s"Creating fetch request of $curRequestSize at $address "
+ + s"with ${mergedBlock.size} blocks")
+ } else {
+// The last group might not exceed `maxBlocksInFlightPerAddress`.
Put it back
Review comment:
when we reach here, it's not `might not exceed`, it's `does not exceed`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource
MaxGekk commented on issue #26973: [SPARK-30323][SQL] Support filters pushdown in CSV datasource URL: https://github.com/apache/spark/pull/26973#issuecomment-568680164 jenkins, retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
SparkQA commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568680253 **[Test build #115702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115702/testReport)** for PR 26993 at commit [`3fb3391`](https://github.com/apache/spark/commit/3fb3391edcee6c467b0189de2dcdaf4dac610083). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361088588
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -218,6 +252,11 @@ object BinaryArithmetic {
""")
case class Add(left: Expression, right: Expression) extends BinaryArithmetic {
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
Yes, the current behavior of master is separating a) decimal(which is null
for overflow) from b) other types. This pr (so far)is just adding intervals to
group a). We may reach an agreement on which way to follow first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
HyukjinKwon commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568679812 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26678: [SPARK-30226][SQL] Remove withXXX functions in WriteBuilder
cloud-fan commented on issue #26678: [SPARK-30226][SQL] Remove withXXX functions in WriteBuilder URL: https://github.com/apache/spark/pull/26678#issuecomment-568679515 There is a proposal to do branch cut on Jan 31, so we still have more than a month to add new features. Since we all agree that this is an improvement(minor or significant), shall we get this in? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event URL: https://github.com/apache/spark/pull/26983#issuecomment-56868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115682/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event URL: https://github.com/apache/spark/pull/26983#issuecomment-56866 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup
AmplabJenkins removed a comment on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup URL: https://github.com/apache/spark/pull/26994#issuecomment-568677494 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115688/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
AmplabJenkins commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event URL: https://github.com/apache/spark/pull/26983#issuecomment-56868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115682/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
AmplabJenkins removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event URL: https://github.com/apache/spark/pull/26983#issuecomment-56866 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup
AmplabJenkins commented on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup URL: https://github.com/apache/spark/pull/26994#issuecomment-568677489 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup
AmplabJenkins commented on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup URL: https://github.com/apache/spark/pull/26994#issuecomment-568677494 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115688/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
SparkQA removed a comment on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event URL: https://github.com/apache/spark/pull/26983#issuecomment-568640215 **[Test build #115682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115682/testReport)** for PR 26983 at commit [`66b2ddc`](https://github.com/apache/spark/commit/66b2ddceef7871a5e726a26f4a03dea5cac03aed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup
AmplabJenkins removed a comment on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup URL: https://github.com/apache/spark/pull/26994#issuecomment-568677489 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
SparkQA commented on issue #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event URL: https://github.com/apache/spark/pull/26983#issuecomment-568677399 **[Test build #115682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115682/testReport)** for PR 26983 at commit [`66b2ddc`](https://github.com/apache/spark/commit/66b2ddceef7871a5e726a26f4a03dea5cac03aed). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
AmplabJenkins removed a comment on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-568677028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20495/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup
SparkQA commented on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup URL: https://github.com/apache/spark/pull/26994#issuecomment-568677258 **[Test build #115688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115688/testReport)** for PR 26994 at commit [`a471a9b`](https://github.com/apache/spark/commit/a471a9bf6b6ca8de42132f283ee8e54e2cc44bee). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend
AmplabJenkins removed a comment on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/26980#issuecomment-568677053 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20494/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568677052 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20496/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
cloud-fan commented on a change in pull request #26983: [SPARK-30331][SQL] Set
isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r361085709
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -39,6 +40,25 @@ class AdaptiveQueryExecSuite
setupTestData()
private def runAdaptiveAndVerifyResult(query: String): (SparkPlan,
SparkPlan) = {
+var finalPlanExistsOnce: Option[Boolean] = None
+val listener = new SparkListener {
+ override def onOtherEvent(event: SparkListenerEvent): Unit = {
+event match {
+ case SparkListenerSQLAdaptiveExecutionUpdate(_, _, sparkPlanInfo) =>
+if (sparkPlanInfo.simpleString.startsWith(
+ "AdaptiveSparkPlan(isFinalPlan=true)")) {
+ if (finalPlanExistsOnce.isDefined) {
+finalPlanExistsOnce = Some(false)
Review comment:
isn't it much simpler to use an int count?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup
SparkQA removed a comment on issue #26994: [SPARK-30339][SQL] Avoid to fail twice in function lookup URL: https://github.com/apache/spark/pull/26994#issuecomment-568652819 **[Test build #115688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115688/testReport)** for PR 26994 at commit [`a471a9b`](https://github.com/apache/spark/commit/a471a9bf6b6ca8de42132f283ee8e54e2cc44bee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568677052 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20496/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
AmplabJenkins commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-568677028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20495/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend
AmplabJenkins removed a comment on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/26980#issuecomment-568677050 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568677043 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
AmplabJenkins commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-568677023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568676688 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115680/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568677043 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend
AmplabJenkins commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/26980#issuecomment-568677050 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend
AmplabJenkins commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/26980#issuecomment-568677053 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20494/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
AmplabJenkins removed a comment on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-568677023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568676684 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361085337
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -218,6 +252,11 @@ object BinaryArithmetic {
""")
case class Add(left: Expression, right: Expression) extends BinaryArithmetic {
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
shall we make the overflow behavior consistent? e.g. other numeric types
follow the java overflow behavior and interval returns null for overflow, which
is inconsistent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend
SparkQA commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/26980#issuecomment-568676649 **[Test build #115699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115699/testReport)** for PR 26980 at commit [`b4e29e8`](https://github.com/apache/spark/commit/b4e29e86a73be3dfb657268cee480af95ff1995a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
maropu commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568676653 cuz the commit related to the failure has bee reverted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568676688 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115680/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568676693 **[Test build #115701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115701/testReport)** for PR 26656 at commit [`d98ea41`](https://github.com/apache/spark/commit/d98ea4139454e3346e220aa0e1df8a2e4da18cea). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568676684 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
yaooqinn commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361085310
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -75,12 +80,29 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
"""})
case _: CalendarIntervalType =>
val iu = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
- defineCodeGen(ctx, ev, c => s"$iu.negate($c)")
+ nullSafeCodeGen(ctx, ev, interval => s"""
+try {
+ ${ev.value} = $iu.negate($interval);
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
SparkQA commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-56867 **[Test build #115700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115700/testReport)** for PR 26924 at commit [`3d7f435`](https://github.com/apache/spark/commit/3d7f435f8452faff71b98a9163cd8e86e77c0a79). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26875: [SPARK-30245][SQL] Add cache for Like and RLike when pattern is not static
viirya commented on a change in pull request #26875: [SPARK-30245][SQL] Add
cache for Like and RLike when pattern is not static
URL: https://github.com/apache/spark/pull/26875#discussion_r361085053
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
##
@@ -240,11 +245,16 @@ case class RLike(left: Expression, right: Expression)
extends StringRegexExpress
}
} else {
val rightStr = ctx.freshName("rightStr")
- val pattern = ctx.freshName("pattern")
+ val pattern = ctx.addMutableState(patternClass, "pattern")
+ val lastRightStr = ctx.addMutableState(classOf[String].getName,
"lastRightStr")
+
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
s"""
String $rightStr = $eval2.toString();
- $patternClass $pattern = $patternClass.compile($rightStr);
+ if ($rightStr != $lastRightStr) {
Review comment:
Do you mean to use `equals()`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
SparkQA removed a comment on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568637494 **[Test build #115680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115680/testReport)** for PR 26993 at commit [`3fb3391`](https://github.com/apache/spark/commit/3fb3391edcee6c467b0189de2dcdaf4dac610083). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
SparkQA commented on issue #26993: [WIP][SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-568676401 **[Test build #115680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115680/testReport)** for PR 26993 at commit [`3fb3391`](https://github.com/apache/spark/commit/3fb3391edcee6c467b0189de2dcdaf4dac610083). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
maropu commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-568676443 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
Ngone51 commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-568676248 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError
Ngone51 commented on issue #26924: [SPARK-30285][CORE] Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError URL: https://github.com/apache/spark/pull/26924#issuecomment-568676209 PySpark failure introduced by a mistake merge, which has been reverted just now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361084982
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -75,12 +80,29 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
"""})
case _: CalendarIntervalType =>
val iu = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
- defineCodeGen(ctx, ev, c => s"$iu.negate($c)")
+ nullSafeCodeGen(ctx, ev, interval => s"""
+try {
+ ${ev.value} = $iu.negate($interval);
Review comment:
shall we have 2 methods `negate` and `safeNegate`? It's good to keep the
generated code small.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] seayoun edited a comment on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever
seayoun edited a comment on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever URL: https://github.com/apache/spark/pull/26975#issuecomment-568676054 @cloud-fan The task status is inconsistent indeed, however we need to avoid `Resubmitted` instead of `handleFailedTask`, > change task.running to !successful(task.index) && task.running in executorLost change this will cause `Resubmit` and rescheduler the task again, `handleFailedTask` won't resheculer it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] seayoun commented on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever
seayoun commented on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever URL: https://github.com/apache/spark/pull/26975#issuecomment-568676054 @cloud-fan The task status is inconsistent indeed, however we need to avoid `Resubmitted` instead of `handleFailedTask`, > change task.running to !successful(task.index) && task.running in executorLost change this will cause `Resubmit` and rescheduler the task again, `handleFailedTask` won't resheculer it, WDYT ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361084511
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -37,6 +37,11 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
with ExpectsInputTypes with NullIntolerant {
private val checkOverflow = SQLConf.get.ansiEnabled
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
+case _ => super.nullable
Review comment:
nit: `child.nullable` is more clear here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361084666
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -75,12 +80,29 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
"""})
case _: CalendarIntervalType =>
val iu = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
- defineCodeGen(ctx, ev, c => s"$iu.negate($c)")
+ nullSafeCodeGen(ctx, ev, interval => s"""
+try {
+ ${ev.value} = $iu.negate($interval);
+} catch (ArithmeticException e) {
+ if ($checkOverflow) {
+throw new ArithmeticException("-($interval) caused interval
overflow.");
+ } else {
+${ev.isNull} = true;
+ }
+}
+ """)
}
protected override def nullSafeEval(input: Any): Any = dataType match {
-case CalendarIntervalType =>
IntervalUtils.negate(input.asInstanceOf[CalendarInterval])
-case _ => numeric.negate(input)
+case CalendarIntervalType =>
+ try {
+ IntervalUtils.negate(input.asInstanceOf[CalendarInterval])
Review comment:
nit: indentation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361084406
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -37,6 +37,11 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
with ExpectsInputTypes with NullIntolerant {
private val checkOverflow = SQLConf.get.ansiEnabled
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
isn't it true for all types?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
cloud-fan commented on a change in pull request #26995: [SPARK-30341][SQL]
Overflow check for interval arithmetic operations
URL: https://github.com/apache/spark/pull/26995#discussion_r361084406
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -37,6 +37,11 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression
with ExpectsInputTypes with NullIntolerant {
private val checkOverflow = SQLConf.get.ansiEnabled
+ override def nullable: Boolean = dataType match {
+case CalendarIntervalType if !checkOverflow => true
Review comment:
isn't it true for all types?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend
Ngone51 commented on issue #26980: [SPARK-27348][Core] HeartbeatReceiver should remove lost executors from CoarseGrainedSchedulerBackend URL: https://github.com/apache/spark/pull/26980#issuecomment-568675032 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568673994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115692/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
maropu edited a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568669077 We cannot check it outside `BinaryArithmetic` like ~numeric~ decimal `CheckOverflow`? It seems this pr includes the same try-catch patterns. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
SparkQA removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568661213 **[Test build #115692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115692/testReport)** for PR 26995 at commit [`67767c0`](https://github.com/apache/spark/commit/67767c043671a2c0960494ab06e022317cec12d1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins removed a comment on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568673988 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568673994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115692/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
SparkQA commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568673923 **[Test build #115692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115692/testReport)** for PR 26995 at commit [`67767c0`](https://github.com/apache/spark/commit/67767c043671a2c0960494ab06e022317cec12d1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
AmplabJenkins commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-568673988 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26930: [SPARK-30290][Core] Count for merged block when fetching continuous blocks in batch
viirya commented on a change in pull request #26930: [SPARK-30290][Core] Count
for merged block when fetching continuous blocks in batch
URL: https://github.com/apache/spark/pull/26930#discussion_r361082828
##
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##
@@ -337,13 +337,17 @@ final class ShuffleBlockFetcherIterator(
assertPositiveBlockSize(blockId, size)
curBlocks += FetchBlockInfo(blockId, size, mapIndex)
curRequestSize += size
- if (curRequestSize >= targetRemoteRequestSize ||
- curBlocks.size >= maxBlocksInFlightPerAddress) {
+ // For batch fetch, the actual block in flight should count for merged
block.
+ val readyForCollectingBlocks = !doBatchFetch &&
+curBlocks.size >= maxBlocksInFlightPerAddress
+ if (curRequestSize >= targetRemoteRequestSize ||
readyForCollectingBlocks) {
// Add this FetchRequest
val mergedBlocks = mergeContinuousShuffleBlockIdsIfNeeded(curBlocks)
-collectedRemoteRequests += new FetchRequest(address, mergedBlocks)
-logDebug(s"Creating fetch request of $curRequestSize at $address "
- + s"with ${mergedBlocks.size} blocks")
+mergedBlocks.grouped(maxBlocksInFlightPerAddress).foreach {
mergedBlock =>
Review comment:
Yea, it makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL] Set isFinalPlan to true before posting the final AdaptiveSparkPlan event
manuzhang commented on a change in pull request #26983: [SPARK-30331][SQL] Set
isFinalPlan to true before posting the final AdaptiveSparkPlan event
URL: https://github.com/apache/spark/pull/26983#discussion_r361081996
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -39,6 +40,20 @@ class AdaptiveQueryExecSuite
setupTestData()
private def runAdaptiveAndVerifyResult(query: String): (SparkPlan,
SparkPlan) = {
+var finalPlanExists = false
Review comment:
now we check the final plan only exists once
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
