[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19760 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83977/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19760 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19760 **[Test build #83977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83977/testReport)** for PR 19760 at commit [`8afc56a`](https://github.com/apache/spark/commit/8afc56a825fdb4bce8aac683348bbd4223a29acc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19760 **[Test build #83977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83977/testReport)** for PR 19760 at commit [`8afc56a`](https://github.com/apache/spark/commit/8afc56a825fdb4bce8aac683348bbd4223a29acc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 > my future plan is to move config related stuff to a new maven module Sounds good. I'll try to update this before leaving on vacation, but may not be able to... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 I'm ok to move the deprecated config keys of `MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM` and `LISTENER_BUS_EVENT_QUEUE_CAPACITY` to `SparkConf` if the deprecation message really matters. But I'd like to keep `withAlternatives`. Generally it's a better interface and my future plan is to move config related stuff to a new maven module, so it can be used in modules that don't depend on the core module(e.g. the network module). It will be annoying if everytime we wanna deprecate a conf we need to change the config module. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 (And even `SparkSubmit` now creates `SparkConf`.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 There's quite a lot of applications that use just `SparkContext`. And `SparkConf` is generally created before anything else (e.g. in yarn-cluster mode, the Spark code will instantiate `SparkConf` before even calling any user code). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 Since now `SparkSession` is the main entry, we just need `spark.conf.get(xxx)` to print deprecation message(still keep the old deprecated configs in `SparkConf`). I think then it's possible to deprecate SQL configs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 It would be possible for configs declared in core; because then we can force SparkConf to initialize those classes. (e.g., referencing any config constant causes the `o.a.s.internal.config` package object to be loaded and all config constants to be initialized). The only way to do that for other modules would be to move their configs to core, so that SparkConf can initialize them the same way. (Well, you could do it with reflection I guess, but ugh.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 Generally having the deprecation message is good, but I hope that can be done within the config entry, instead of a central place in `SparkConf`. Is this possible? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 > the name withAlternatives says it's an alternative, That's the other thing. Having alternative names is just confusing; there should be a single name for a config, with others deprecated. The two configs that had alternatives were basically deprecating the old names and replacing them with the new ones. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19760 I don't exactly follow the purpose here, the name `withAlternatives` says it's an alternative, not deprecated, so it should not print deprecation message. Besides, I think `withAlternatives` is easier to use and maintain, as it's defined with the config entry. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19760 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83916/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19760 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19760 **[Test build #83916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83916/testReport)** for PR 19760 at commit [`fcf3fbf`](https://github.com/apache/spark/commit/fcf3fbfc47fbf4ea7290f74e65f324eee68f7aba). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19760 **[Test build #83916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83916/testReport)** for PR 19760 at commit [`fcf3fbf`](https://github.com/apache/spark/commit/fcf3fbfc47fbf4ea7290f74e65f324eee68f7aba). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19760: [SPARK-22533][core] Handle deprecated names in ConfigEnt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19760 @cloud-fan since you made the change I'm (mostly) undoing. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
