[GitHub] [spark] MaxGekk commented on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting
MaxGekk commented on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting URL: https://github.com/apache/spark/pull/26256#issuecomment-549705876 @cloud-fan Can we continue with this optimization since #26283 has been closed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
cloud-fan closed pull request #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
cloud-fan commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549701170 This has bigger impacts than I expect, e.g. it breaks `functions.window` which is a streaming-related public API. I'm closing it to avoid breaking public APIs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549700536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113244/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549700536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113244/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549700532 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549700532 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
SparkQA removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549682257 **[Test build #113244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113244/testReport)** for PR 26393 at commit [`a2955d8`](https://github.com/apache/spark/commit/a2955d8f8c2623cca5aa48d9ee4415de9ec29006). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
SparkQA commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549700274 **[Test build #113244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113244/testReport)** for PR 26393 at commit [`a2955d8`](https://github.com/apache/spark/commit/a2955d8f8c2623cca5aa48d9ee4415de9ec29006). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549699347 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113240/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549699340 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549699187 **[Test build #113240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113240/testReport)** for PR 26338 at commit [`ac76822`](https://github.com/apache/spark/commit/ac768227739dcab2f16dfe942d7a4e12a2f54159). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AlterTableSerDePropertiesStatement(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549699340 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549699347 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113240/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549673269 **[Test build #113240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113240/testReport)** for PR 26338 at commit [`ac76822`](https://github.com/apache/spark/commit/ac768227739dcab2f16dfe942d7a4e12a2f54159). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng edited a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
zhengruifeng edited a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549694050 IIRC, when `Summarizer` was created, I had done similar tests, and at that time `Summarizer` was much slower than `MultivariateOnlineSummarizer`. I am surprised by the results and will look into whether it works in other places. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549697554 There is a transitive dependency from `joda-time` via hive-common for `hadoop-3.2`: ``` [info] joda-time:joda-time:2.8.1 [info] +-org.apache.hive:hive-common:2.3.6 [info] +-org.apache.spark:spark-hive_2.12:3.0.0-SNAPSHOT [S] ``` which still exists in hive's master: https://github.com/apache/hive/blob/master/common/pom.xml#L101-L105 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
AmplabJenkins commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#issuecomment-549696311 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
AmplabJenkins commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394#issuecomment-549696352 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
AmplabJenkins commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394#issuecomment-549696356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18158/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
AmplabJenkins removed a comment on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#issuecomment-549696311 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
AmplabJenkins commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#issuecomment-549696315 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18159/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
AmplabJenkins removed a comment on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394#issuecomment-549696352 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
AmplabJenkins removed a comment on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394#issuecomment-549696356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18158/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
AmplabJenkins removed a comment on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#issuecomment-549696315 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18159/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
SparkQA commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394#issuecomment-549695881 **[Test build #113247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113247/testReport)** for PR 26394 at commit [`a02d8d9`](https://github.com/apache/spark/commit/a02d8d9e1dc494ab02a3ffa9bfe487981e5b2a01). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
SparkQA commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#issuecomment-549695913 **[Test build #113248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113248/testReport)** for PR 26360 at commit [`7dda882`](https://github.com/apache/spark/commit/7dda88298b5f4a19ea78189b911a6734eb309d64). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
cloud-fan commented on a change in pull request #26394:
[SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
URL: https://github.com/apache/spark/pull/26394#discussion_r342409645
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -93,24 +99,7 @@ class AdaptiveQueryExecSuite
assert(smj.size == 1)
val bhj = findTopLevelBroadcastHashJoin(adaptivePlan)
assert(bhj.size == 1)
- checkNumLocalShuffleReaders(adaptivePlan, 2)
-}
- }
-
- test("Change merge join to broadcast join and reduce number of shuffle
partitions") {
Review comment:
This test case is duplicated with the above one now. We add local reader to
both sides of broadcast join and won't apply "reduce number of shuffle
partitions".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
cloud-fan commented on issue #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394#issuecomment-549695040 cc @maryannxue @JkSelf This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan opened a new pull request #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust
cloud-fan opened a new pull request #26394: [SPARK-29752][SQL][TEST] make AdaptiveQueryExecSuite more robust URL: https://github.com/apache/spark/pull/26394 ### What changes were proposed in this pull request? instead of checking the exact number of local shuffle readers, we should check whether the number of shuffles is equal to the number of local readers. ### Why are the changes needed? AQE is known to have randomness. We may pick different build side for broadcast join depending on which query stage finishes first. The decision to build side may add/remove shuffles downstream, so it's flaky to check the exact number of local shuffle readers. ### Does this PR introduce any user-facing change? no ### How was this patch tested? test only PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
zhengruifeng commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549694512 friendly ping @srowen @WeichenXu123 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable
AmplabJenkins removed a comment on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable URL: https://github.com/apache/spark/pull/26337#issuecomment-549694062 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18157/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable
AmplabJenkins removed a comment on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable URL: https://github.com/apache/spark/pull/26337#issuecomment-549694058 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable
AmplabJenkins commented on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable URL: https://github.com/apache/spark/pull/26337#issuecomment-549694058 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable
AmplabJenkins commented on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable URL: https://github.com/apache/spark/pull/26337#issuecomment-549694062 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18157/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
zhengruifeng commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549694050 IIRC, when `Summarizer` was created, I had done similar tests, and at that time `Summarizer` was much slower than `MultivariateOnlineSummarizer`. I am surprised by the results and will look for whether it works in other places. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xyxiaoyou commented on issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint propagation
xyxiaoyou commented on issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint propagation URL: https://github.com/apache/spark/pull/17186#issuecomment-549693775 > @xyxiaoyou SQLConf.get ? What I want to express is that I wonder if this issue has been resolved in spark2.3/2.4, Is there a more appropriate solution? Because I found that more complex SQL would cause the excutor stuck. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable
SparkQA commented on issue #26337: [SPARK-29679][SQL] Make interval type comparable and orderable URL: https://github.com/apache/spark/pull/26337#issuecomment-549693710 **[Test build #113246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113246/testReport)** for PR 26337 at commit [`d1d60a9`](https://github.com/apache/spark/commit/d1d60a9148ea893788bfa4ad27f6ba00d11d6d62). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
dongjoon-hyun commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549693133 I pinged you the on-going `Arrow upgrade` PR. Let's resume this PR after that PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549692184 > Is that the last one? Or, just one dependency instance? I bumped arrow version to 0.15.1 locally, so, there is no dependency from joda-time anymore. Not sure that is is safe to migrate to arrow vector 0.15.1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
maropu commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#issuecomment-549691716 > I'm not sure this insert.sql is worthy to add and keep this in Apache Spark repository. Yea, to be honest, I'm also not sure we need to merge this. I think its important to check a regression test file query-by-query though, a decision not to merge it looks reasonable to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26133: [WIP][SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.0
dongjoon-hyun commented on issue #26133: [WIP][SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.0 URL: https://github.com/apache/spark/pull/26133#issuecomment-549691365 HI, @BryanCutler and @HyukjinKwon . Can we remove `[WIP]`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26133: [WIP][SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.0
dongjoon-hyun commented on issue #26133: [WIP][SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.0 URL: https://github.com/apache/spark/pull/26133#issuecomment-549691047 cc @MaxGekk since he is trying to remove `joda-time` with this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
dongjoon-hyun commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549690514 Is that the last one? Or, just one dependency instance? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint propagation
gatorsmile commented on issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint propagation URL: https://github.com/apache/spark/pull/17186#issuecomment-549690510 @xyxiaoyou SQLConf.get ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549689138 arrow vector requires joda time in 12.x: ``` [info] joda-time:joda-time:2.9.9 [info] +-org.apache.arrow:arrow-vector:0.12.0 ``` https://github.com/apache/arrow/blob/apache-arrow-0.12.0/java/vector/pom.xml#L34-L38 but arrow 0.15.1 does not require it anymore: https://github.com/apache/arrow/blob/apache-arrow-0.15.1/java/vector/pom.xml This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #26223: [WIP][SPARK-29564][SQL] Cluster deploy mode should support Spark Thrift server
dongjoon-hyun closed pull request #26223: [WIP][SPARK-29564][SQL] Cluster deploy mode should support Spark Thrift server URL: https://github.com/apache/spark/pull/26223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26223: [WIP][SPARK-29564][SQL] Cluster deploy mode should support Spark Thrift server
dongjoon-hyun commented on issue #26223: [WIP][SPARK-29564][SQL] Cluster deploy mode should support Spark Thrift server URL: https://github.com/apache/spark/pull/26223#issuecomment-549688495 Hi, @LantaoJin . I also have the same concern like @jerryshao . I'm closing this PR for now. Please feel free to reopen this PR when you are ready. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
AmplabJenkins removed a comment on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549687007 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113239/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
AmplabJenkins removed a comment on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549686982 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
AmplabJenkins commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549686982 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
SparkQA removed a comment on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549664041 **[Test build #113239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113239/testReport)** for PR 26283 at commit [`a4ffd00`](https://github.com/apache/spark/commit/a4ffd005431eb8ff3cce02886885d84e3081664d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
AmplabJenkins commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549687007 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113239/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format
SparkQA commented on issue #26283: [SPARK-29622][SQL] do not allow leading 'interval' in the interval string format URL: https://github.com/apache/spark/pull/26283#issuecomment-549686809 **[Test build #113239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113239/testReport)** for PR 26283 at commit [`a4ffd00`](https://github.com/apache/spark/commit/a4ffd005431eb8ff3cce02886885d84e3081664d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command
dongjoon-hyun commented on issue #25573: [SPARK-28833][DOCS][SQL] Document ALTER VIEW command URL: https://github.com/apache/spark/pull/25573#issuecomment-549686884 Gentle ping. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xyxiaoyou commented on issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint propagation
xyxiaoyou commented on issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint propagation URL: https://github.com/apache/spark/pull/17186#issuecomment-549686880 May I ask how to solve this issue in spark2.3? I see that this flag has been removed in PruneFilters and EliminateOuterJoin. @viirya @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #21027: [SPARK-23943][MESOS][DEPLOY] Improve observability of MesosRestServer/MesosClusterDi…
dongjoon-hyun commented on issue #21027: [SPARK-23943][MESOS][DEPLOY] Improve observability of MesosRestServer/MesosClusterDi… URL: https://github.com/apache/spark/pull/21027#issuecomment-549686589 Hi, All. I'm closing this PR due to author's inactivity. Please feel free to reopen this when the review comments are addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
MaxGekk commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549686541 > Sorry, but I'm -1 to downgrade like this. @dongjoon-hyun You are so quick. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #21027: [SPARK-23943][MESOS][DEPLOY] Improve observability of MesosRestServer/MesosClusterDi…
dongjoon-hyun closed pull request #21027: [SPARK-23943][MESOS][DEPLOY] Improve observability of MesosRestServer/MesosClusterDi… URL: https://github.com/apache/spark/pull/21027 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25898: [SPARK-29209][BUILD] Print build environment variables to Github
dongjoon-hyun commented on issue #25898: [SPARK-29209][BUILD] Print build environment variables to Github URL: https://github.com/apache/spark/pull/25898#issuecomment-549685906 Gentle ping, @wangyum . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549684968 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18156/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549684961 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549684968 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18156/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549684961 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549684586 **[Test build #113245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113245/testReport)** for PR 26338 at commit [`64a68fc`](https://github.com/apache/spark/commit/64a68fcadb7e27e05196b4e9da7c903b71c9921c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
dongjoon-hyun commented on issue #26360: [SPARK-29714][SQL][TESTS] Port
insert.sql
URL: https://github.com/apache/spark/pull/26360#issuecomment-549684325
Actually, only 3 original queries works. I'm not sure this `insert.sql` is
worthy to add and keep this in Apache Spark repository.
```
--
-- TOASTed value test
--
insert into inserttest values(30, 50, repeat('x', 1));
select col1, col2, char_length(col3) from inserttest;
drop table inserttest;
```
However, If we need to have this for completeness, I'm not against this.
BTW, cc @gatorsmile .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
jiangxb1987 commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-549683186 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549682706 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins removed a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549682712 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18155/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng edited a comment on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
zhengruifeng edited a comment on issue #26393: [SPARK-29751][ML] Scalers use
Summarizer instead of MultivariateOnlineSummarizer
URL: https://github.com/apache/spark/pull/26393#issuecomment-549682239
test code
```scala
import org.apache.spark.ml.feature._
scala> var df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
19/11/05 13:47:02 WARN LibSVMFileFormat: 'numFeatures' option not specified,
determining the number of features by going though the input. If you know the
number in advance, please specify it via 'numFeatures' option to avoid the
extra scan.
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]
scala> df.persist()
res0: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [label:
double, features: vector]
scala> df.count
res1: Long = 32561
scala> (0 until 8).foreach(_ => df = df.union(df))
scala> df.count
res3: Long = 8335616
val durations1 = (0 until 50).map{i => val tic = System.currentTimeMillis;
val scaler = new MaxAbsScaler().setInputCol("features"); val model =
scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}
durations1.takeRight(30).sum.toDouble / 30
val durations2 = (0 until 50).map{i => val tic = System.currentTimeMillis;
val scaler = new MinMaxScaler().setInputCol("features"); val model =
scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}
durations2.takeRight(30).sum.toDouble / 30
val durations3 = (0 until 50).map{i => val tic = System.currentTimeMillis;
val scaler = new StandardScaler().setInputCol("features"); val model =
scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}
durations3.takeRight(30).sum.toDouble / 30
```
Results: （the last 30 fitting are taken into account）
|MaxAbsScaler(Old)| MinMaxScaler(Old) | StandardScaler(Old) |
MaxAbsScaler(New) | MinMaxScaler(New) | StandardScaler(New) |
|--|--||--||--|
|6768.1|6875.2|6899.9|5862.1|5880.3|5889.7|
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549682706 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
AmplabJenkins commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549682712 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18155/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on a change in pull request #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
jiangxb1987 commented on a change in pull request #26078: [SPARK-29151][CORE]
Support fractional resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#discussion_r342398775
##
File path:
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala
##
@@ -86,8 +103,8 @@ trait ResourceAllocator {
s"address $address doesn't exist.")
}
val isAvailable = addressAvailabilityMap(address)
- if (!isAvailable) {
-addressAvailabilityMap(address) = true
+ if (isAvailable < slotsPerAddress) {
+addressAvailabilityMap(address) = addressAvailabilityMap(address) + 1
Review comment:
sure, it worth a separate jira.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
SparkQA commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393#issuecomment-549682257 **[Test build #113244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113244/testReport)** for PR 26393 at commit [`a2955d8`](https://github.com/apache/spark/commit/a2955d8f8c2623cca5aa48d9ee4415de9ec29006). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
zhengruifeng commented on issue #26393: [SPARK-29751][ML] Scalers use
Summarizer instead of MultivariateOnlineSummarizer
URL: https://github.com/apache/spark/pull/26393#issuecomment-549682239
test code
```scala
import org.apache.spark.ml.feature._
scala> var df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
19/11/05 13:47:02 WARN LibSVMFileFormat: 'numFeatures' option not specified,
determining the number of features by going though the input. If you know the
number in advance, please specify it via 'numFeatures' option to avoid the
extra scan.
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]
scala> df.persist()
res0: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [label:
double, features: vector]
scala> df.count
res1: Long = 32561
scala> (0 until 8).foreach(_ => df = df.union(df))
scala> df.count
res3: Long = 8335616
val durations1 = (0 until 50).map{i => val tic = System.currentTimeMillis;
val scaler = new MaxAbsScaler().setInputCol("features"); val model =
scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}
durations1.takeRight(30).sum.toDouble / 30
val durations2 = (0 until 50).map{i => val tic = System.currentTimeMillis;
val scaler = new MinMaxScaler().setInputCol("features"); val model =
scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}
durations2.takeRight(30).sum.toDouble / 30
val durations3 = (0 until 50).map{i => val tic = System.currentTimeMillis;
val scaler = new StandardScaler().setInputCol("features"); val model =
scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}
durations3.takeRight(30).sum.toDouble / 30
```
Results:
|MaxAbsScaler(Old)| MinMaxScaler(Old) | StandardScaler(Old) |
MaxAbsScaler(New) | MinMaxScaler(New) | StandardScaler(New) |
|--|--||--||--|
|6768.1|6875.2|6899.9|5862.1|5880.3|5889.7|
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
dongjoon-hyun commented on a change in pull request #26360:
[SPARK-29714][SQL][TESTS] Port insert.sql
URL: https://github.com/apache/spark/pull/26360#discussion_r342397723
##
File path: sql/core/src/test/resources/sql-tests/inputs/postgreSQL/insert.sql
##
@@ -0,0 +1,652 @@
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+-- INSERT
+--
https://github.com/postgres/postgres/blob/REL_12_STABLE/src/test/regress/sql/insert.sql
+
+--
+-- insert with DEFAULT in the target_list
+--
+-- [SPARK-19842] Informational Referential Integrity Constraints Support in
Spark
+-- [SPARK-29119] DEFAULT option is not supported in Spark
+create table inserttest (col1 int, col2 int /* NOT NULL */, col3 string /*
default 'testing' */) using parquet;
+-- [SPARK-29119] DEFAULT option is not supported in Spark
+-- [SPARK-20845] Support specification of column names in INSERT INTO
+-- Skip a test below because the PK constraint is violated and the query fails
in PostgreSQL
+-- insert into inserttest (col1, col2, col3) values (DEFAULT, DEFAULT,
DEFAULT);
+-- insert into inserttest (col2, col3) values (3, DEFAULT);
+insert into inserttest values (NULL, 3, 'testing');
+-- insert into inserttest (col1, col2, col3) values (DEFAULT, 5, DEFAULT);
+insert into inserttest values (NULL, 5, 'testing');
+-- insert into inserttest values (DEFAULT, 5, 'test');
+insert into inserttest values (NULL, 5, 'test');
+-- insert into inserttest values (DEFAULT, 7);
+insert into inserttest values (NULL, 7, 'testing');
+
+select * from inserttest;
+
+--
+-- insert with similar expression / target_list values (all fail)
+--
+-- [SPARK-20845] Support specification of column names in INSERT INTO
+-- [SPARK-29119] DEFAULT option is not supported in Spark
+-- insert into inserttest (col1, col2, col3) values (DEFAULT, DEFAULT);
+-- insert into inserttest (col1, col2, col3) values (1, 2);
+-- insert into inserttest (col1) values (1, 2);
+-- insert into inserttest (col1) values (DEFAULT, DEFAULT);
+
+-- select * from inserttest;
+
+--
+-- VALUES test
+--
+-- [SPARK-29119] DEFAULT option is not supported in Spark
+-- [SPARK-29715] Support SELECT statements in VALUES of INSERT INTO
+-- insert into inserttest values(10, 20, '40'), (-1, 2, 'testing' /* DEFAULT
*/),
+-- ((select 2), (select i from (values(3)) as foo (i)), 'values are fun!');
+
+-- select * from inserttest;
+
+--
+-- TOASTed value test
+--
+insert into inserttest values(30, 50, repeat('x', 1));
+
+select col1, col2, char_length(col3) from inserttest;
+
+drop table inserttest;
+
+--
+-- check indirection (field/array assignment), cf bug #14265
+--
+-- these tests are aware that transformInsertStmt has 3 separate code paths
+--
+
+-- [SPARK-29716] Support [CREATE|DROP] TYPE
+-- create type insert_test_type as (if1 int, if2 array);
+
+-- create table inserttest (f1 int, f2 int[],
+-- f3 insert_test_type, f4 insert_test_type[]);
+--
+-- insert into inserttest (f2[1], f2[2]) values (1,2);
+-- insert into inserttest (f2[1], f2[2]) values (3,4), (5,6);
+-- insert into inserttest (f2[1], f2[2]) select 7,8;
+-- insert into inserttest (f2[1], f2[2]) values (1,default); -- not supported
+--
+-- insert into inserttest (f3.if1, f3.if2) values (1,array['foo']);
+-- insert into inserttest (f3.if1, f3.if2) values (1,'{foo}'), (2,'{bar}');
+-- insert into inserttest (f3.if1, f3.if2) select 3, '{baz,quux}';
+-- insert into inserttest (f3.if1, f3.if2) values (1,default); -- not
supported
+--
+-- insert into inserttest (f3.if2[1], f3.if2[2]) values ('foo', 'bar');
+-- insert into inserttest (f3.if2[1], f3.if2[2]) values ('foo', 'bar'),
('baz', 'quux');
+-- insert into inserttest (f3.if2[1], f3.if2[2]) select 'bear', 'beer';
+--
+-- insert into inserttest (f4[1].if2[1], f4[1].if2[2]) values ('foo', 'bar');
+-- insert into inserttest (f4[1].if2[1], f4[1].if2[2]) values ('foo', 'bar'),
('baz', 'quux');
+-- insert into inserttest (f4[1].if2[1], f4[1].if2[2]) select 'bear', 'beer';
+--
+-- select * from inserttest;
+
+-- also check reverse-listing
+-- create table inserttest2 (f1 bigint, f2 string);
+-- [SPARK-29717] Support [CREATE|DROP] RULE - define a new plan rewrite rule
+-- create rule irule1 as on insert to inserttest2 do also
+-- insert into inserttest (f3.if2[1], f3.if2[2])
+-- values (new.f1,new.f2);
+-- create rule irule2 as on insert to inserttest2 do also
+-- insert into inserttest (f4[1].if1, f4[1].if2[2])
+-- values (1,'fool'),(new.f1,new.f2);
+-- create rule irule3 as on insert to inserttest2 do also
+-- insert into inserttest (f4[1].if1, f4[1].if2[2])
+-- select new.f1, new.f2;
+-- \d+ inserttest2
+
+-- drop table inserttest2;
+-- drop table inserttest;
+-- [SPARK-29716] Support [CREATE|DROP] TYPE
+-- drop type insert_test_type;
+
+-- direct partition inserts should check partition bound constraint
+-- [SPARK-29718] Support PARTITION BY [RANGE|LIST|HASH] and PARTITION OF in
CREATE TABLE
+-- create table
[GitHub] [spark] AmplabJenkins removed a comment on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
AmplabJenkins removed a comment on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549680788 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
AmplabJenkins removed a comment on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549680792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18154/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
AmplabJenkins commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549680788 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
AmplabJenkins commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549680792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18154/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng opened a new pull request #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer
zhengruifeng opened a new pull request #26393: [SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer URL: https://github.com/apache/spark/pull/26393 ### What changes were proposed in this pull request? use `ml.Summarizer` instead of `mllib.MultivariateOnlineSummarizer` ### Why are the changes needed? 1, I found that using `ml.Summarizer` is faster than current impl; 2, `mllib.MultivariateOnlineSummarizer` maintain all arrays, while `ml.Summarizer` only maintain necessary arrays 3, using `ml.Summarizer` will avoid vector conversions to `mlllib.Vector` ### Does this PR introduce any user-facing change? No ### How was this patch tested? existing testsuites This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
SparkQA commented on issue #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#issuecomment-549680472 **[Test build #113243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113243/testReport)** for PR 26392 at commit [`02e9bee`](https://github.com/apache/spark/commit/02e9bee249ec7c7627224633d24fddbb7b977e68). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26360: [SPARK-29714][SQL][TESTS] Port insert.sql
dongjoon-hyun commented on a change in pull request #26360: [SPARK-29714][SQL][TESTS] Port insert.sql URL: https://github.com/apache/spark/pull/26360#discussion_r342397024 ## File path: sql/core/src/test/resources/sql-tests/inputs/postgreSQL/insert.sql ## @@ -0,0 +1,652 @@ +-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group +-- +-- INSERT +-- https://github.com/postgres/postgres/blob/REL_12_STABLE/src/test/regress/sql/insert.sql + +-- +-- insert with DEFAULT in the target_list +-- +-- [SPARK-19842] Informational Referential Integrity Constraints Support in Spark +-- [SPARK-29119] DEFAULT option is not supported in Spark +create table inserttest (col1 int, col2 int /* NOT NULL */, col3 string /* default 'testing' */) using parquet; +-- [SPARK-29119] DEFAULT option is not supported in Spark +-- [SPARK-20845] Support specification of column names in INSERT INTO +-- Skip a test below because the PK constraint is violated and the query fails in PostgreSQL +-- insert into inserttest (col1, col2, col3) values (DEFAULT, DEFAULT, DEFAULT); +-- insert into inserttest (col2, col3) values (3, DEFAULT); +insert into inserttest values (NULL, 3, 'testing'); +-- insert into inserttest (col1, col2, col3) values (DEFAULT, 5, DEFAULT); +insert into inserttest values (NULL, 5, 'testing'); +-- insert into inserttest values (DEFAULT, 5, 'test'); +insert into inserttest values (NULL, 5, 'test'); +-- insert into inserttest values (DEFAULT, 7); +insert into inserttest values (NULL, 7, 'testing'); + +select * from inserttest; + +-- +-- insert with similar expression / target_list values (all fail) +-- +-- [SPARK-20845] Support specification of column names in INSERT INTO +-- [SPARK-29119] DEFAULT option is not supported in Spark +-- insert into inserttest (col1, col2, col3) values (DEFAULT, DEFAULT); +-- insert into inserttest (col1, col2, col3) values (1, 2); +-- insert into inserttest (col1) values (1, 2); +-- insert into inserttest (col1) values (DEFAULT, DEFAULT); + +-- select * from inserttest; + +-- +-- VALUES test +-- +-- [SPARK-29119] DEFAULT option is not supported in Spark +-- [SPARK-29715] Support SELECT statements in VALUES of INSERT INTO +-- insert into inserttest values(10, 20, '40'), (-1, 2, 'testing' /* DEFAULT */), Review comment: Since we are not running this, shall we keep the original? ``` insert into inserttest values(10, 20, '40'), (-1, 2, DEFAULT), ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679718 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113237/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679693 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113238/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679688 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679714 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549659899 **[Test build #113238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113238/testReport)** for PR 26338 at commit [`9e7a678`](https://github.com/apache/spark/commit/9e7a678f8e1351082b16fa14a009f47d615e969a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549679803 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113233/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549679803 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113233/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA removed a comment on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549658568 **[Test build #113237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113237/testReport)** for PR 26338 at commit [`fb7564a`](https://github.com/apache/spark/commit/fb7564ad1f6036dafae40e4d8ec12581079cb449). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time
dongjoon-hyun commented on a change in pull request #26392: [WIP][SPARK-29750][BUILD] Avoid dependency from joda-time URL: https://github.com/apache/spark/pull/26392#discussion_r342396376 ## File path: dev/deps/spark-deps-hadoop-3.2 ## @@ -139,6 +139,7 @@ jersey-server-2.29.jar jetty-webapp-9.4.18.v20190429.jar jetty-xml-9.4.18.v20190429.jar jline-2.14.6.jar +joda-time-2.8.1.jar Review comment: Ur, are you suggesting to have two difference (lower) joda-time version in this commit??? ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549679802 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549679802 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679592 **[Test build #113237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113237/testReport)** for PR 26338 at commit [`fb7564a`](https://github.com/apache/spark/commit/fb7564ad1f6036dafae40e4d8ec12581079cb449). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679693 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113238/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679714 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679688 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
SparkQA commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679570 **[Test build #113238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113238/testReport)** for PR 26338 at commit [`9e7a678`](https://github.com/apache/spark/commit/9e7a678f8e1351082b16fa14a009f47d615e969a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
AmplabJenkins commented on issue #26338: [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax URL: https://github.com/apache/spark/pull/26338#issuecomment-549679718 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113237/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
