[GitHub] spark issue #12456: [Spark-14686] Allow setting local properties that are no...
Github user marcintustin commented on the issue: https://github.com/apache/spark/pull/12456 Feel free to close -- Marcin Tustin Tel: +1 917 553 3974 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12456: [Spark-14686] Allow setting local properties that are no...
Github user marcintustin commented on the issue: https://github.com/apache/spark/pull/12456 I am not actively working on this. -- Marcin Tustin Tel: +1 917 553 3974 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14685] [CORE] Document heritability of ...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-216422062 @rxin Tests have passed, can we merge? This still seems low risk for Spark-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14685] [CORE] Document heritability of ...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-212673073 @rxin Are you cool to go ahead with this before we resolve the deal on #12456? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14686] Allow setting local properties t...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12456#issuecomment-212672809 @jerryshao > there's no way for user to set it in the job thread specifically, Can you explain what you mean by that? Or link us to some code? It's totally possible that it is impossible, or that it's possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14685] [CORE] Document heritability of ...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-211164186 The test failures are probably bogus, as per http://mail-archives.apache.org/mod_mbox/spark-dev/201604.mbox/%3CCAMFhwAYRbN0yJGwzvrY8atzS9CCudzioF%3DbcGogCwPq3gPC6Uw%40mail.gmail.com%3E --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14686][CORE,SQL,STREAMING]
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12456#issuecomment-211080503 @rxin Derp on my part. Of course this needs a better title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14686][CORE,SQL,STREAMING]
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12456#issuecomment-211078297 I should add that this will also need new tests. I haven't added any, again pending overall agreement on design. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14686][CORE,SQL,STREAMING]
GitHub user marcintustin opened a pull request: https://github.com/apache/spark/pull/12456 [Spark-14686][CORE,SQL,STREAMING] ## What changes were proposed in this pull request? This PR adds a uninheritableLocalPropertyFacility and ports sql.execution.id to be set with that facility. If this is to go forward, the changes should probably be folded into a Properties type which accommodates hierarchical access rather than a tuple. ## How was this patch tested? Running tests. @rxin @JoshRosen PR opened for comments. As noted above, this should probably have a little more engineering done, but I'd like to (a) get feedback on the overall approach; and (b) see which tests fail in jenkins, as I have some tests failing locally which may or may not be bogus. You can merge this pull request into a Git repository by running: $ git pull https://github.com/marcintustin/spark SPARK-14686 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12456.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12456 commit aecb305577a7d16065738afcf4bbeee6397b4f53 Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-16T23:41:22Z [SPARK-14685] Document inheritability of localProperties commit 9964e2e937718194521c6705ba345deba11f1f3d Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-17T14:49:51Z Add test for heritability of local properties commit b96cde118c1265bf37ac7036581b8bb1bef80ee0 Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-17T15:09:38Z SPARK-14685 add test to ensure no crosstalk between threads on localProperties Work with me in NYC: https://www.handy.com/careers/73115?gh_jid=73115_src=o5qcxn commit bc302e014f280cca9d9b7f104c14cdf7de6d5df4 Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-17T18:35:36Z [SPARK-14686] First cut of non-inheritable localProperties --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14685] [CORE] Document heritability of ...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-211060379 @JoshRosen This may well be of interest. As mentioned above, my enhancement PR is pending (running tests right now). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14685] Document heritability of localPr...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/12455#issuecomment-211042961 @rxin Documentation and tests PR. Enhancement PR to follow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14685] Document heritability of localPr...
GitHub user marcintustin opened a pull request: https://github.com/apache/spark/pull/12455 [Spark-14685] Document heritability of localProperties ## What changes were proposed in this pull request? This updates the java-/scala- doc for setLocalProperty to document heritability of localProperties. This also adds tests for that behaviour. ## How was this patch tested? Tests pass. New tests were added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/marcintustin/spark SPARK-14685 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12455.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12455 commit aecb305577a7d16065738afcf4bbeee6397b4f53 Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-16T23:41:22Z [SPARK-14685] Document inheritability of localProperties commit 9964e2e937718194521c6705ba345deba11f1f3d Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-17T14:49:51Z Add test for heritability of local properties commit b96cde118c1265bf37ac7036581b8bb1bef80ee0 Author: Marcin Tustin <marcin.tus...@gmail.com> Date: 2016-04-17T15:09:38Z SPARK-14685 add test to ensure no crosstalk between threads on localProperties Work with me in NYC: https://www.handy.com/careers/73115?gh_jid=73115_src=o5qcxn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14163][CORE] SumEvaluator and countAppr...
Github user marcintustin commented on the pull request: https://github.com/apache/spark/pull/11981#issuecomment-201951574 FYI I have a more parsimonious change here: #11982 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14163][CORE] SumEvaluator and countAppr...
Github user marcintustin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11981#discussion_r57520937
--- Diff: core/src/main/scala/org/apache/spark/partial/SumEvaluator.scala
---
@@ -42,6 +42,14 @@ private[spark] class SumEvaluator(totalOutputs: Int,
confidence: Double)
new BoundedDouble(counter.sum, 1.0, counter.sum, counter.sum)
} else if (outputsMerged == 0) {
new BoundedDouble(0, 0.0, Double.NegativeInfinity,
Double.PositiveInfinity)
+} else if (counter.count == 0) {
+ new BoundedDouble(0, 0.0, Double.NegativeInfinity,
Double.PositiveInfinity)
--- End diff --
Why not just use an || to avoid repeating code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
