[GitHub] [spark] AmplabJenkins removed a comment on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang
AmplabJenkins removed a comment on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang URL: https://github.com/apache/spark/pull/26673#issuecomment-558506232 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang
AmplabJenkins commented on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang URL: https://github.com/apache/spark/pull/26673#issuecomment-558506239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114442/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang
AmplabJenkins removed a comment on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang URL: https://github.com/apache/spark/pull/26673#issuecomment-558506239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114442/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang
AmplabJenkins commented on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang URL: https://github.com/apache/spark/pull/26673#issuecomment-558506232 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang
SparkQA removed a comment on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang URL: https://github.com/apache/spark/pull/26673#issuecomment-558466953 **[Test build #114442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114442/testReport)** for PR 26673 at commit [`6c9bb01`](https://github.com/apache/spark/commit/6c9bb0132e2c30e5eb14c68b65bad82aab84abe7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang
SparkQA commented on issue #26673: [SPARK-30030][BUILD][FOLLOWUP] Remove unused org.apache.commons.lang URL: https://github.com/apache/spark/pull/26673#issuecomment-558505727 **[Test build #114442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114442/testReport)** for PR 26673 at commit [`6c9bb01`](https://github.com/apache/spark/commit/6c9bb0132e2c30e5eb14c68b65bad82aab84abe7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoanFM commented on issue #26608: [SPARK-29877][GRAPHX] static PageRank allow checkPoint from previous computations
JoanFM commented on issue #26608: [SPARK-29877][GRAPHX] static PageRank allow checkPoint from previous computations URL: https://github.com/apache/spark/pull/26608#issuecomment-558505434 There is no expected change in behavior This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangshuo128 edited a comment on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch
wangshuo128 edited a comment on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-558501869 I applied patch #21356 in my cluster. Found that the `AsyncEventQueue` thread was set interrupted when queue.take() sometimes. I guess it's interrupted by some other thread asynchronously. Unfortunately, I didn't find which thread (in Spark or HDFS) did this. Here is the log: ``` java.net.SocketTimeoutException: 7 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.132.165.35:46887 remote=/10.132.78.10:50010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2319) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1087) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1056) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1197) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:942) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:453) 19/11/24 03:58:01 ERROR spark-listener-group-eventLog Utils: uncaught error in thread spark-listener-group-eventLog, stopping SparkContext java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:439) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1303) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) 19/11/24 03:58:01 ERROR spark-listener-group-eventLog Utils: throw uncaught fatal error in thread spark-listener-group-eventLog java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:439) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1303) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) ``` Stopping the entire queue when interrupted in dispatch maybe not the best choice. If it's an important queue (e.g. dynamic resource allocation), I think it's better to
[GitHub] [spark] cloud-fan commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
cloud-fan commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558503966 yea a migration guide is needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangshuo128 commented on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch
wangshuo128 commented on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-558501869 I applied patch #21356 in my cluster. Found that the `AsyncEventQueue` thread was set interrupted when queue.take() sometimes. I guess it's interrupted by some other thread asynchronously. Unfortunately, I didn't find which thread (in Spark or HDFS) did this. Here is the log: ``` java.net.SocketTimeoutException: 7 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.132.165.35:46887 remote=/10.132.78.10:50010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2319) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1087) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1056) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1197) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:942) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:453) 19/11/24 03:58:01 ERROR spark-listener-group-eventLog Utils: uncaught error in thread spark-listener-group-eventLog, stopping SparkContext java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:439) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1303) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) 19/11/24 03:58:01 ERROR spark-listener-group-eventLog Utils: throw uncaught fatal error in thread spark-listener-group-eventLog java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:439) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1303) at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) ``` Stopping the entire queue when interrupted in dispatch maybe not the best choice. If it's an important queue (e.g. dynamic resource allocation), I think it's better to stop the
[GitHub] [spark] maropu commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
maropu commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558501425 > Actually, literals like 1E2 are parsed as Double before Spark2.1, but changed to Decimal after #14828 due to "The difference between the two confuses most users" as it said. But I also see support(DB2 test) of original behavior at #14828 (comment). Ah, I see. Thanks for that info. Either way, I personally think its worth adding a note about this behaivour change in the migration guide. Can you update? > hmm... as for this, I'd say yes. But I agree that things here may be a little awkward when we switch to PostgreSQLas we need to back to legacy Decimal mode again. NVM, if necessary, we can update that behaivour in follow-up. Thanks, anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch
AmplabJenkins removed a comment on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-558497285 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch
AmplabJenkins commented on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-558499389 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch
AmplabJenkins commented on issue #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-558497285 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangshuo128 opened a new pull request #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch
wangshuo128 opened a new pull request #26674: [SPARK-24309][CORE][FOLLOWUP]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674 ### What changes were proposed in this pull request? PR #21356 stop `AsyncEventQueue` when interrupted in `postToAll`. However, if it's interrupted in `AsyncEventQueue#dispatch`, SparkContext would be stopped. This PR proposes to stop `AsyncEventQueue` when interrupted in dispatch, rather than stop the SparkContext. ### Why are the changes needed? Avoid stopping the SparkContext when interrupted in `AsyncEventQueue#dispatch`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? New UT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
Ngone51 commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558497134 Actually, literals like `1E2` are parsed as `Double` before Spark2.1, but changed to `Decimal` after #14828 due to "The difference between the two confuses most users" as it said. But I also see support(DB2 test) of original behavior at https://github.com/apache/spark/pull/14828#issuecomment-251805403. > btw, when the PgSQL dialect enabled, should we follow the current Spark behaviour (that is, in both cases, we regard them as decimal)? hmm... as for this, I'd say yes. But I agree that things here may be a little awkward when we switch to PostgreSQLas we need to back to legacy Decimal mode again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues URL: https://github.com/apache/spark/pull/25307#issuecomment-558495569 cc @jiangxb1987 @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE]
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r350573157
##
File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
##
@@ -46,8 +46,18 @@ private class AsyncEventQueue(
// Cap the capacity of the queue so we get an explicit error (rather than an
OOM exception) if
// it's perpetually being added to more quickly than it's being drained.
- private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](
-conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+ // The capacity can be configured by
spark.scheduler.listenerbus.eventqueue.${name}.capacity,
+ // if no such conf is specified, use the value specified in
+ // LISTENER_BUS_EVENT_QUEUE_CAPACITY
Review comment:
We need to update the conf description of LISTENER_BUS_EVENT_QUEUE_CAPACITY.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE]
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r350573375
##
File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
##
@@ -46,8 +46,18 @@ private class AsyncEventQueue(
// Cap the capacity of the queue so we get an explicit error (rather than an
OOM exception) if
// it's perpetually being added to more quickly than it's being drained.
- private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](
-conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+ // The capacity can be configured by
spark.scheduler.listenerbus.eventqueue.${name}.capacity,
+ // if no such conf is specified, use the value specified in
+ // LISTENER_BUS_EVENT_QUEUE_CAPACITY
+ private[scheduler] def capacity: Int = {
+val queuesize =
conf.getInt(s"spark.scheduler.listenerbus.eventqueue.${name}.capacity",
Review comment:
Instead of hard-coded here, can we define it in
core/src/main/scala/org/apache/spark/internal/config/package.scala ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues
gatorsmile commented on a change in pull request #25307: [SPARK-28574][CORE]
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r350572738
##
File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
##
@@ -46,8 +46,18 @@ private class AsyncEventQueue(
// Cap the capacity of the queue so we get an explicit error (rather than an
OOM exception) if
// it's perpetually being added to more quickly than it's being drained.
- private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](
-conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+ // The capacity can be configured by
spark.scheduler.listenerbus.eventqueue.${name}.capacity,
+ // if no such conf is specified, use the value specified in
+ // LISTENER_BUS_EVENT_QUEUE_CAPACITY
+ private[scheduler] def capacity: Int = {
+val queuesize =
conf.getInt(s"spark.scheduler.listenerbus.eventqueue.${name}.capacity",
+conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
Review comment:
Nit: indent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file
gatorsmile commented on issue #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file URL: https://github.com/apache/spark/pull/25134#issuecomment-558493116 cc @Ngone51 @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file
gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE]
Logging in driver when loading single large unsplittable file
URL: https://github.com/apache/spark/pull/25134#discussion_r350570804
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1180,6 +1180,14 @@ package object config {
.intConf
.createWithDefault(1)
+ private[spark] val IO_WARNING_LARGEFILETHRESHOLD =
+ConfigBuilder("spark.io.warning.largeFileThreshold")
+ .internal()
+ .doc("When spark loading one single large file, if file size exceed this
" +
Review comment:
Please update the description to
> If the size in bytes of a file loaded by Spark exceeds this threshold, a
warning is logged with the possible reasons.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file
gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE]
Logging in driver when loading single large unsplittable file
URL: https://github.com/apache/spark/pull/25134#discussion_r350570917
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1180,6 +1180,14 @@ package object config {
.intConf
.createWithDefault(1)
+ private[spark] val IO_WARNING_LARGEFILETHRESHOLD =
+ConfigBuilder("spark.io.warning.largeFileThreshold")
+ .internal()
+ .doc("When spark loading one single large file, if file size exceed this
" +
+"threshold, then log warning with possible reasons.")
+ .longConf
Review comment:
updated it to `.bytesConf(ByteUnit.BYTE)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file
gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE]
Logging in driver when loading single large unsplittable file
URL: https://github.com/apache/spark/pull/25134#discussion_r350570917
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1180,6 +1180,14 @@ package object config {
.intConf
.createWithDefault(1)
+ private[spark] val IO_WARNING_LARGEFILETHRESHOLD =
+ConfigBuilder("spark.io.warning.largeFileThreshold")
+ .internal()
+ .doc("When spark loading one single large file, if file size exceed this
" +
+"threshold, then log warning with possible reasons.")
+ .longConf
Review comment:
Please update it to `.bytesConf(ByteUnit.BYTE)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
yaooqinn commented on a change in pull request #26662: [SPARK-30026][SQL]
Whitespaces can be identified as delimiters in interval string
URL: https://github.com/apache/spark/pull/26662#discussion_r350571074
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -464,14 +462,17 @@ object IntervalUtils {
if (input == null) {
throwIAE("interval string cannot be null")
}
-// scalastyle:off caselocale .toLowerCase
-val s = input.trimAll().toLowerCase
-// scalastyle:on
-val bytes = s.getBytes
-if (bytes.isEmpty) {
+val strs =
input.trimAll().toLowerCase().split(UTF8String.fromString("interval\\s+"), -1)
Review comment:
I'll think about that
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE] Logging in driver when loading single large unsplittable file
gatorsmile commented on a change in pull request #25134: [SPARK-28366][CORE]
Logging in driver when loading single large unsplittable file
URL: https://github.com/apache/spark/pull/25134#discussion_r350570804
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -1180,6 +1180,14 @@ package object config {
.intConf
.createWithDefault(1)
+ private[spark] val IO_WARNING_LARGEFILETHRESHOLD =
+ConfigBuilder("spark.io.warning.largeFileThreshold")
+ .internal()
+ .doc("When spark loading one single large file, if file size exceed this
" +
Review comment:
If the size in bytes of a file loaded by Spark exceeds this threshold, a
warning is logged with the possible reasons.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
AmplabJenkins removed a comment on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558491744 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
SparkQA commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558492182 **[Test build #114447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114447/testReport)** for PR 26629 at commit [`038b764`](https://github.com/apache/spark/commit/038b764ad7aa955b8c3e71a580b3a7801c3f92e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
AmplabJenkins removed a comment on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558491754 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114438/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9
AmplabJenkins removed a comment on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9 URL: https://github.com/apache/spark/pull/26672#issuecomment-558491441 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9
AmplabJenkins removed a comment on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9 URL: https://github.com/apache/spark/pull/26672#issuecomment-558491446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114440/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
AmplabJenkins commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558491754 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114438/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
MaxGekk commented on a change in pull request #26662: [SPARK-30026][SQL]
Whitespaces can be identified as delimiters in interval string
URL: https://github.com/apache/spark/pull/26662#discussion_r350570074
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -464,14 +462,17 @@ object IntervalUtils {
if (input == null) {
throwIAE("interval string cannot be null")
}
-// scalastyle:off caselocale .toLowerCase
-val s = input.trimAll().toLowerCase
-// scalastyle:on
-val bytes = s.getBytes
-if (bytes.isEmpty) {
+val strs =
input.trimAll().toLowerCase().split(UTF8String.fromString("interval\\s+"), -1)
Review comment:
For 9 units from 1726 -> 2618. This is slowing down by 50% which is
significant. Is it possible to implement the same without regexps and split?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
AmplabJenkins commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558491744 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9
AmplabJenkins commented on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9 URL: https://github.com/apache/spark/pull/26672#issuecomment-558491446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114440/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9
AmplabJenkins commented on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9 URL: https://github.com/apache/spark/pull/26672#issuecomment-558491441 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
SparkQA removed a comment on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558436654 **[Test build #114438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114438/testReport)** for PR 26437 at commit [`a6b8485`](https://github.com/apache/spark/commit/a6b8485eb52162faa35f5bba3ea7c85f7cdf9b5c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
SparkQA commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558491209 **[Test build #114438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114438/testReport)** for PR 26437 at commit [`a6b8485`](https://github.com/apache/spark/commit/a6b8485eb52162faa35f5bba3ea7c85f7cdf9b5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9
SparkQA removed a comment on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9 URL: https://github.com/apache/spark/pull/26672#issuecomment-558456896 **[Test build #114440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114440/testReport)** for PR 26672 at commit [`2ca2921`](https://github.com/apache/spark/commit/2ca29215b09f7fd3c080a08f9d46b0860c263cbf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9
SparkQA commented on issue #26672: [WIP][SPARK-30035][BUILD] Upgrade to Apache Commons Lang 3.9 URL: https://github.com/apache/spark/pull/26672#issuecomment-558490768 **[Test build #114440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114440/testReport)** for PR 26672 at commit [`2ca2921`](https://github.com/apache/spark/commit/2ca29215b09f7fd3c080a08f9d46b0860c263cbf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
HeartSaVioR commented on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558490228 @gengliangwang Thanks for understanding and bearing with me. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558490002 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19282/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558490002 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19282/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558489994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558489994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
maropu edited a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558489184 Actually, it seems this behaviour depends on implementations (the standard says so though). In PgSQL; ``` postgres=# select pg_typeof(1E2); pg_typeof --- numeric (1 row) postgres=# select pg_typeof(1.2); pg_typeof --- numeric (1 row) ``` Just in case, can you check the other system behaviour other than PgSQL/Presto? btw, when the PgSQL dialect enabled, should we follow the current Spark behaviour (that is, in both cases, we regard them as `decimal`)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
maropu edited a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558489184 Actually, it seems this behaviour depends on implementations (the standard says so though). In PgSQL; ``` postgres=# select pg_typeof(1E2); pg_typeof --- numeric (1 row) postgres=# select pg_typeof(1.2); pg_typeof --- numeric (1 row) ``` Just in case, can you check the other system behaviour other than PgSQL/Presto? btw, when the PgSQL dialect enabled, should we follow the current Spark behaviour (that is, in both cases, we should regard them as `decimal`)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
maropu commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558489184 Actually, it seems this behaviour depends on implementations (the standard says so though). In PgSQL; ``` postgres=# select pg_typeof(1E2); pg_typeof --- numeric (1 row) postgres=# select pg_typeof(1.2); pg_typeof --- numeric (1 row) ``` Just in case, can you check the other system behaviour other than PgSQL/Presto? btw, when the PgSQL dialect enabled, should we follow the current Spark behaviour (that is, in both cases, we regard them as `decimal`)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
AngersZh commented on a change in pull request #26594: [SPARK-29957][TEST]
Bump MiniKdc to 3.2.0
URL: https://github.com/apache/spark/pull/26594#discussion_r350567964
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
##
@@ -136,6 +137,20 @@ class KafkaTestUtils(
kdcConf.setProperty(MiniKdc.DEBUG, "true")
kdc = new MiniKdc(kdcConf, kdcDir)
kdc.start()
+val krb5Conf = Source.fromFile(kdc.getKrb5conf, "UTF-8").getLines()
+val rewriteKrb5Conf = krb5Conf.map(s => if (s.contains("libdefaults")) {
+ s + "\n" +
+"default_tkt_enctypes=aes128-cts-hmac-sha1-96\n" +
+"default_tgs_enctypes=aes128-cts-hmac-sha1-96 "
Review comment:
Check Kafka's `MiniKdc`, it rewrite from hadoop-2.7's MiniKDC, and make
`krb5.conf` customize.
Add spark's own MiniKDC like Kafka is better. This is also beneficial to
future work about kerberos test case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
huaxingao commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558488833 Thanks! @viirya @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
cloud-fan commented on a change in pull request #26667: [SPARK-29922][SQL] SHOW
FUNCTIONS should do multi-catalog resolution
URL: https://github.com/apache/spark/pull/26667#discussion_r350567336
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##
@@ -455,6 +455,25 @@ class ResolveSessionCatalog(
ShowTablePropertiesCommand(
tableName.asTableIdentifier,
propertyKey)
+
+case ShowFunctionsStatement(scope, pattern, functionName) =>
+ import ShowFunctionsStatement._
+ val userScope = scope.map(s => s == ALL || s == USER).getOrElse(true)
+ val systemScope = scope.map(s => s == ALL || s == SYSTEM).getOrElse(true)
+ val (db, function) = functionName match {
Review comment:
the function name may contain catalog name, we should resolve it first. We
should match `ShowFunctionsStatement(..., SessionCatalog(_, functionName))`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
gengliangwang commented on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558488299 OK, I think the current PR is good :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
yaooqinn commented on a change in pull request #26662: [SPARK-30026][SQL]
Whitespaces can be identified as delimiters in interval string
URL: https://github.com/apache/spark/pull/26662#discussion_r350566942
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -464,14 +462,17 @@ object IntervalUtils {
if (input == null) {
throwIAE("interval string cannot be null")
}
-// scalastyle:off caselocale .toLowerCase
-val s = input.trimAll().toLowerCase
-// scalastyle:on
-val bytes = s.getBytes
-if (bytes.isEmpty) {
+val strs =
input.trimAll().toLowerCase().split(UTF8String.fromString("interval\\s+"), -1)
Review comment:
this vs master vs static UTF8String.fromString("interval\\s+")
```
[info] Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1
[info] Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
[info] cast strings to intervals:Best Time(ms) Avg
Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative
[info]
[info] prepare string w/ interval 342
383 36 2.9 342.2 1.0X
[info] prepare string w/o interval 312
325 16 3.2 311.6 1.1X
[info] 1 units w/ interval1002
1019 15 1.01002.3 0.3X
[info] 1 units w/o interval815
817 1 1.2 815.0 0.4X
[info] 2 units w/ interval1090
1092 2 0.91089.8 0.3X
[info] 2 units w/o interval918
926 8 1.1 917.9 0.4X
[info] 3 units w/ interval1618
1620 2 0.61618.0 0.2X
[info] 3 units w/o interval 1550
1733 241 0.61549.7 0.2X
[info] 4 units w/ interval1751
1755 7 0.61750.8 0.2X
[info] 4 units w/o interval 1580
1587 7 0.61579.5 0.2X
[info] 5 units w/ interval1874
1877 4 0.51873.6 0.2X
[info] 5 units w/o interval 1696
1705 8 0.61695.9 0.2X
[info] 6 units w/ interval2025
2054 35 0.52025.1 0.2X
[info] 6 units w/o interval 1836
1843 10 0.51835.9 0.2X
[info] 7 units w/ interval2280
2284 4 0.42280.4 0.2X
[info] 7 units w/o interval 2090
2096 8 0.52089.7 0.2X
[info] 8 units w/ interval2363
2368 7 0.42362.6 0.1X
[info] 8 units w/o interval 2170
2182 13 0.52170.4 0.2X
[info] 9 units w/ interval2618
2626 11 0.42618.1 0.1X
[info] 9 units w/o interval 2432
2439 9 0.42432.2 0.1X
[info] 10 units w/ interval 2622
2635 14 0.42621.7 0.1X
[info] 10 units w/o interval 2455
2462 11 0.42455.2 0.1X
[info] 11 units w/ interval 3080
3086 9 0.33079.9 0.1X
[info] 11 units w/o interval 2900
2907 12 0.32899.6 0.1X
[info]
[info] Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1
[info] Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
[info] cast strings to intervals:Best Time(ms) Avg
Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative
[info]
[info] prepare string w/ interval 343
384 35 2.9 343.4 1.0X
[info] prepare string w/o interval 319
328
[GitHub] [spark] cloud-fan commented on a change in pull request #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
cloud-fan commented on a change in pull request #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#discussion_r350566754 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala ## @@ -475,3 +475,17 @@ case class ShowCurrentNamespaceStatement() extends ParsedStatement case class ShowTablePropertiesStatement( tableName: Seq[String], propertyKey: Option[String]) extends ParsedStatement + +/** + * SHOW FUNCTIONS statement, as parsed from SQL + */ +case class ShowFunctionsStatement( +scope: Option[String], Review comment: shall we simply have 2 boolean flags `userScope` and `systemScope`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558485203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19281/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558485196 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558485203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19281/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558485196 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on issue #26569: [SPARK-29938] [SQL] Add batching support in Alter table add partition flow
prakharjain09 commented on issue #26569: [SPARK-29938] [SQL] Add batching support in Alter table add partition flow URL: https://github.com/apache/spark/pull/26569#issuecomment-558485057 @dongjoon-hyun @cloud-fan I have addressed all the review comments. Can you please recheck this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
SparkQA commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-558484801 **[Test build #114446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114446/testReport)** for PR 26656 at commit [`8bfff6f`](https://github.com/apache/spark/commit/8bfff6f60ca15c96c6d2d1cda2fb2efc495ee389). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
SparkQA commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558484795 **[Test build #114445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114445/testReport)** for PR 26667 at commit [`7007274`](https://github.com/apache/spark/commit/7007274756d57287d3c11202cf43d7aa9137582b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558310499 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
cloud-fan commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558484414 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-558482968 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-558482979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19280/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins commented on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-558482968 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression
AmplabJenkins removed a comment on issue #26656: [SPARK-27986][SQL] Support ANSI SQL filter clause for aggregate expression URL: https://github.com/apache/spark/pull/26656#issuecomment-558482979 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19280/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558482151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114437/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558482145 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558482151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114437/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
AmplabJenkins commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558482145 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
SparkQA removed a comment on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558433178 **[Test build #114437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114437/testReport)** for PR 26629 at commit [`6f6a19e`](https://github.com/apache/spark/commit/6f6a19eb0f84ada4a1f75dd93e25251a5f801b48). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions
SparkQA commented on issue #26629: [SPARK-29768][SQL] Column pruning through nondeterministic expressions URL: https://github.com/apache/spark/pull/26629#issuecomment-558481736 **[Test build #114437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114437/testReport)** for PR 26629 at commit [`6f6a19e`](https://github.com/apache/spark/commit/6f6a19eb0f84ada4a1f75dd93e25251a5f801b48). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on issue #26638: [SPARK-29681][WebUI] Support column sorting in Environment tab
prakharjain09 commented on issue #26638: [SPARK-29681][WebUI] Support column sorting in Environment tab URL: https://github.com/apache/spark/pull/26638#issuecomment-558481475 @dongjoon-hyun My Apache ID - https://issues.apache.org/jira/secure/ViewProfile.jspa?name=prakharjain09 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
wangyum commented on a change in pull request #26594: [SPARK-29957][TEST] Bump
MiniKdc to 3.2.0
URL: https://github.com/apache/spark/pull/26594#discussion_r350561381
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
##
@@ -136,6 +137,20 @@ class KafkaTestUtils(
kdcConf.setProperty(MiniKdc.DEBUG, "true")
kdc = new MiniKdc(kdcConf, kdcDir)
kdc.start()
+val krb5Conf = Source.fromFile(kdc.getKrb5conf, "UTF-8").getLines()
+val rewriteKrb5Conf = krb5Conf.map(s => if (s.contains("libdefaults")) {
+ s + "\n" +
+"default_tkt_enctypes=aes128-cts-hmac-sha1-96\n" +
+"default_tgs_enctypes=aes128-cts-hmac-sha1-96 "
Review comment:
Maybe we should first customize `krb5.conf` to test Kafka delegation token:
https://issues.apache.org/jira/browse/SPARK-30037 @gaborgsomogyi
Add then add these two configurations to `krb5.conf`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
MaxGekk commented on a change in pull request #26662: [SPARK-30026][SQL]
Whitespaces can be identified as delimiters in interval string
URL: https://github.com/apache/spark/pull/26662#discussion_r350561136
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -464,14 +462,17 @@ object IntervalUtils {
if (input == null) {
throwIAE("interval string cannot be null")
}
-// scalastyle:off caselocale .toLowerCase
-val s = input.trimAll().toLowerCase
-// scalastyle:on
-val bytes = s.getBytes
-if (bytes.isEmpty) {
+val strs =
input.trimAll().toLowerCase().split(UTF8String.fromString("interval\\s+"), -1)
Review comment:
oh, no. Compiling the regular expression per each call? Could you run the
benchmark. Just wondering how much are you going to slow it down.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
AmplabJenkins removed a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558478929 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19279/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
AmplabJenkins commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558478924 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
AmplabJenkins commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558478929 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19279/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
AmplabJenkins removed a comment on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558478924 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double
SparkQA commented on issue #26595: [SPARK-29956][SQL] A literal number with an exponent should be parsed to Double URL: https://github.com/apache/spark/pull/26595#issuecomment-558478623 **[Test build #11 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/11/testReport)** for PR 26595 at commit [`32f6b64`](https://github.com/apache/spark/commit/32f6b6470bbece1daedc332a3a80e4987909af58). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
HeartSaVioR edited a comment on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558478147 Ah yes you're right that it cannot be reverted cleanly - so there's physically no clean revert. Maybe I overthought here; I thought about how we deal with JIRA issue for SPARK-26081/SPARK-2. If we reopen them (at least SPARK-26081) and open a chance to try to do the right fix, it'd be ideal if we have a "minimized" commit to revert the SPARK-26081 - so we can track how SPARK-26081 was introduced and reverted later, and re-introduced. If we would want to abandon the original idea of SPARK-26081 and close the issue as won't fix, any approach would be OK for me. Btw, would you mind if I ask for elaboration on the new suggestion on the new UT? > We can also add a new test case to check the behavior that an empty Dataframe will output exactly one empty file. I'm not familiar enough to understand the expectations/requirements on file sink; I feel the UT in SPARK-2 can reside with reverting commit as the UT tests the regression what we've broken - we're reverting and adding the guard to prevent we don't break again. Is the new UT same case - did SPARK-26081 break the expectation? If not, that sounds to be on different purpose. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
HeartSaVioR commented on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558478147 Ah yes you're right that it cannot be reverted cleanly - so there's physically no clean revert. Maybe I overthought here; I thought about how we deal with JIRA issue for SPARK-26081/SPARK-2. If we reopen them (at least SPARK-26081) and open a chance to try to do the right fix, it'd be ideal if we have a "minimized" commit to revert the SPARK-26081 - so we can track how SPARK-26081 was introduced and reverted later, and re-introduced. If we would want to abandon the original idea of SPARK-26081 and close the issue as won't fix, any approach would be OK for me. Btw, would you mind if I ask for elaboration on the new suggestion on the new UT? > We can also add a new test case to check the behavior that an empty Dataframe will output exactly one empty file. I'm not familiar enough to understand the expectations/requirements on file sink; I feel the UT in SPARK-2 can reside with reverting commit as the UT tests the regression what we've broken - we're reverting and adding the guard to prevent we don't break again. Is the new UT same case - did SPARK-26081 break the expectation? If not, that sounds to be different purpose. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
cloud-fan commented on a change in pull request #26664:
[SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf
affected by spark.sql.adaptive.enabled
URL: https://github.com/apache/spark/pull/26664#discussion_r350558598
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -353,59 +353,67 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
- val SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE =
-buildConf("spark.sql.adaptive.shuffle.targetPostShuffleInputSize")
- .doc("The target post-shuffle input size in bytes of a task.")
- .bytesConf(ByteUnit.BYTE)
- .createWithDefault(64 * 1024 * 1024)
-
val FETCH_SHUFFLE_BLOCKS_IN_BATCH_ENABLED =
buildConf("spark.sql.adaptive.shuffle.fetchShuffleBlocksInBatch.enabled")
.doc("Whether to fetch the continuous shuffle blocks in batch. Instead
of fetching blocks " +
"one by one, fetching continuous shuffle blocks for the same map task
in batch can " +
-"reduce IO and improve performance. Note, this feature also depends on
a relocatable " +
-"serializer and the concatenation support codec in use.")
+"reduce IO and improve performance. Note, multiple continuous blocks
exist in single " +
+s"fetch request only happen when '${ADAPTIVE_EXECUTION_ENABLED.key}'
is enabled, " +
+"this feature also depends on a relocatable serializer and the
concatenation support " +
+"codec in use.")
.booleanConf
.createWithDefault(true)
val REDUCE_POST_SHUFFLE_PARTITIONS_ENABLED =
buildConf("spark.sql.adaptive.shuffle.reducePostShufflePartitions.enabled")
-.doc("When true and adaptive execution is enabled, this enables reducing
the number of " +
- "post-shuffle partitions based on map output statistics.")
+.doc(s"When true and '${ADAPTIVE_EXECUTION_ENABLED.key}' is enabled, this
enables reducing " +
+ "the number of post-shuffle partitions based on map output statistics.")
.booleanConf
.createWithDefault(true)
val SHUFFLE_MIN_NUM_POSTSHUFFLE_PARTITIONS =
buildConf("spark.sql.adaptive.shuffle.minNumPostShufflePartitions")
- .doc("The advisory minimum number of post-shuffle partitions used in
adaptive execution.")
+ .doc("The advisory minimum number of post-shuffle partitions used when "
+
+s"'${ADAPTIVE_EXECUTION_ENABLED.key}' and " +
+s"'${REDUCE_POST_SHUFFLE_PARTITIONS_ENABLED.key}' is enabled.")
.intConf
.checkValue(_ > 0, "The minimum shuffle partition number " +
"must be a positive integer.")
.createWithDefault(1)
+ val SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE =
+buildConf("spark.sql.adaptive.shuffle.targetPostShuffleInputSize")
+ .doc("The target post-shuffle input size in bytes of a task. This
configuration only has " +
+s"an effect when '${ADAPTIVE_EXECUTION_ENABLED.key}' and " +
+s"'${REDUCE_POST_SHUFFLE_PARTITIONS_ENABLED.key}' is enabled.")
+ .bytesConf(ByteUnit.BYTE)
+ .createWithDefault(64 * 1024 * 1024)
+
val SHUFFLE_MAX_NUM_POSTSHUFFLE_PARTITIONS =
buildConf("spark.sql.adaptive.shuffle.maxNumPostShufflePartitions")
.doc("The advisory maximum number of post-shuffle partitions used in
adaptive execution. " +
"This is used as the initial number of pre-shuffle partitions. By
default it equals to " +
-"spark.sql.shuffle.partitions")
+"spark.sql.shuffle.partitions. This configuration only has an effect
when " +
+s"'${ADAPTIVE_EXECUTION_ENABLED.key}' is enabled.")
Review comment:
and also `REDUCE_POST_SHUFFLE_PARTITIONS_ENABLED` is enabled
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
cloud-fan commented on a change in pull request #26664:
[SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf
affected by spark.sql.adaptive.enabled
URL: https://github.com/apache/spark/pull/26664#discussion_r350558463
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -353,59 +353,67 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
- val SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE =
-buildConf("spark.sql.adaptive.shuffle.targetPostShuffleInputSize")
- .doc("The target post-shuffle input size in bytes of a task.")
- .bytesConf(ByteUnit.BYTE)
- .createWithDefault(64 * 1024 * 1024)
-
val FETCH_SHUFFLE_BLOCKS_IN_BATCH_ENABLED =
buildConf("spark.sql.adaptive.shuffle.fetchShuffleBlocksInBatch.enabled")
.doc("Whether to fetch the continuous shuffle blocks in batch. Instead
of fetching blocks " +
"one by one, fetching continuous shuffle blocks for the same map task
in batch can " +
-"reduce IO and improve performance. Note, this feature also depends on
a relocatable " +
-"serializer and the concatenation support codec in use.")
+"reduce IO and improve performance. Note, multiple continuous blocks
exist in single " +
+s"fetch request only happen when '${ADAPTIVE_EXECUTION_ENABLED.key}'
is enabled, " +
Review comment:
and also `REDUCE_POST_SHUFFLE_PARTITIONS_ENABLED` is true? normal shuffle
only fetch one block from each mapper.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
cloud-fan closed pull request #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
cloud-fan commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558476797 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
AngersZh commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558476143 > For top-level limit, we use a special physical plan `CollectLimitExec`, which doesn't have a shuffle. The perf should be the same, but we can avoid adding a new physical node and planner change. Yea, I will try these way to see the change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
AmplabJenkins removed a comment on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 URL: https://github.com/apache/spark/pull/26594#issuecomment-558474576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114443/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
AmplabJenkins removed a comment on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 URL: https://github.com/apache/spark/pull/26594#issuecomment-558474567 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
AmplabJenkins commented on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 URL: https://github.com/apache/spark/pull/26594#issuecomment-558474567 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
AmplabJenkins commented on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 URL: https://github.com/apache/spark/pull/26594#issuecomment-558474576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114443/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
SparkQA removed a comment on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 URL: https://github.com/apache/spark/pull/26594#issuecomment-558466959 **[Test build #114443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114443/testReport)** for PR 26594 at commit [`dfdf589`](https://github.com/apache/spark/commit/dfdf5899fbc529281cb0eb3a463d6006976e585b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0
SparkQA commented on issue #26594: [SPARK-29957][TEST] Bump MiniKdc to 3.2.0 URL: https://github.com/apache/spark/pull/26594#issuecomment-558474467 **[Test build #114443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114443/testReport)** for PR 26594 at commit [`dfdf589`](https://github.com/apache/spark/commit/dfdf5899fbc529281cb0eb3a463d6006976e585b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on issue #26472: [SPARK-29838][SQL] PostgreSQL dialect: cast to timestamp
amanomer commented on issue #26472: [SPARK-29838][SQL] PostgreSQL dialect: cast to timestamp URL: https://github.com/apache/spark/pull/26472#issuecomment-558470761 @Ngone51 I have updated this PR as per your reviews. Kindly review. cc @maropu @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
AmplabJenkins removed a comment on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558470147 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114435/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
AmplabJenkins removed a comment on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558470139 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
AmplabJenkins commented on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558470147 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114435/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
AmplabJenkins commented on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558470139 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26671: Revert "[SPARK-26081][SPARK-29999]"
SparkQA commented on issue #26671: Revert "[SPARK-26081][SPARK-2]" URL: https://github.com/apache/spark/pull/26671#issuecomment-558469713 **[Test build #114435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114435/testReport)** for PR 26671 at commit [`5bddefe`](https://github.com/apache/spark/commit/5bddefe956729dbba176bf39b7ffa92f0ef2cb2f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
