[GitHub] spark pull request: SPARK-5841 [CORE] [HOTFIX 2] Memory leak in Di...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4690#issuecomment-75016363 No, this was already fixed in https://github.com/apache/spark/commit/49c19fdbad57f0609bbcc9278f9eaa8115a73604 I don't think you have that commit yet here. The problem is not do with it being `null`; I don't think it can be and we shouldn't just let the exception occur. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4689#issuecomment-75015004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27710/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4689#issuecomment-75014993 [Test build #27710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27710/consoleFull) for PR 4689 at commit [`d41d688`](https://github.com/apache/spark/commit/d41d68868fc0cc2057e0686344e02f92cb3e054e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Checking data types when resolving types
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-75020636 [Test build #27718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27718/consoleFull) for PR 4685 at commit [`72e88db`](https://github.com/apache/spark/commit/72e88dbb9ac906862c7dc235a562e085c5c90064). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5841 [CORE] [HOTFIX 2] Memory leak in Di...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4690#issuecomment-75019602 Ah right. I was glancing at the number of commits and thought it hadn't been rebased. The problem isn't this shutdown hook but another one in `yarn.ApplicationMaster`. Dang. Well I don't think the problem is a `null` shutdown hook, and I don't think it is an error per se. Maybe it's not even worth logging because it's OK? that is, if we find `close()` is called during shutdown and so we can't remove a hook, just keep going? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5841 [CORE] [HOTFIX 2] Memory leak in Di...
Github user nishkamravi2 commented on the pull request: https://github.com/apache/spark/pull/4690#issuecomment-75020545 Agreed. That's why a try-catch block seems enough for this one. Ok with removing logging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3172 and SPARK-3577
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2504#issuecomment-75015873 [Test build #27713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27713/consoleFull) for PR 2504 at commit [`b38fe51`](https://github.com/apache/spark/commit/b38fe51c6eccc4c9dd7886153ab9acef23263272). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark-5708: Add Slf4jSink to Spark Metrics
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4644#issuecomment-75017213 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75017727 [Test build #27716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27716/consoleFull) for PR 4677 at commit [`3e74372`](https://github.com/apache/spark/commit/3e743723f128f5d87f66dc05a92d3f68e6fd01cb). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75017703 [Test build #27712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27712/consoleFull) for PR 4677 at commit [`77549a9`](https://github.com/apache/spark/commit/77549a9b74c510f480df31fad314d3395313812d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75017712 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27712/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark-5708: Add Slf4jSink to Spark Metrics
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4644#issuecomment-75017728 [Test build #27717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27717/consoleFull) for PR 4644 at commit [`ef837c0`](https://github.com/apache/spark/commit/ef837c0b7c79a21982624cb2954376abf8e6e75b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5904][SQL] DataFrame API fixes.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4686#issuecomment-75013638 [Test build #27714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27714/consoleFull) for PR 4686 at commit [`fd9b199`](https://github.com/apache/spark/commit/fd9b199e6ce4db18449bf25ad8b9f5d051354dd4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5741][SQL] Support the path contains co...
Github user watermen commented on the pull request: https://github.com/apache/spark/pull/4532#issuecomment-75016913 @yhuai Can you review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75016836 [Test build #27711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27711/consoleFull) for PR 4677 at commit [`e008936`](https://github.com/apache/spark/commit/e0089363220f11b53a83bb1afb29af939cbb711f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75016842 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27711/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3172 and SPARK-3577
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2504#issuecomment-75015886 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27713/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Checking data types when resolving types
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-75018745 [Test build #27715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27715/consoleFull) for PR 4685 at commit [`4933dac`](https://github.com/apache/spark/commit/4933dac3c552e4e937acfc4fdd05ead15919b883). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Checking data types when resolving types
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-75018748 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27715/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5841 [CORE] [HOTFIX 2] Memory leak in Di...
Github user nishkamravi2 commented on the pull request: https://github.com/apache/spark/pull/4690#issuecomment-75018795 I have that commit. The problem continues to manifest itself. This is a workaround, not a fix at the source. I don't think the fix is trivial. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5559] [Streaming] [Test] Remove oppotun...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4337#issuecomment-75127417 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5775] BugFix: GenericRow cannot be cast...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4697#issuecomment-75128095 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5559] [Streaming] [Test] Remove oppotun...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4337#issuecomment-75127627 [Test build #27733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27733/consoleFull) for PR 4337 at commit [`33357e3`](https://github.com/apache/spark/commit/33357e3cb25241727a42a6ed55346cca5e765361). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5775] BugFix: GenericRow cannot be cast...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4697#issuecomment-75128074 /cc @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5775] BugFix: GenericRow cannot be cast...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4697#issuecomment-75128452 [Test build #27734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27734/consoleFull) for PR 4697 at commit [`22cec52`](https://github.com/apache/spark/commit/22cec5206091580e9922f997ef8052ded393d225). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5825] [Spark Submit] Remove the double ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4611#issuecomment-75128516 @andrewor14 Ah! right of course. I looked right past that. Yes that's a good change then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1537][WIP]: ats support for spark-on-ya...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4683#issuecomment-75128247 @zhzhan the problem is not creating 'alpha' features, but consuming them from elsewhere. These APIs don't exist at all until recent versions of YARN, so Spark needs another build profile and module to even compile this code without breaking existing users. It won't work for people using Spark with less than the latest YARN. That much is OK; it's only usable to people who want to build or package their own, although it won't benefit most Spark users yet. But this happens any time you want to add features that can only work with a newer version of, say, Hadoop. You just have to do this, if you really need to take advantage of some advanced functionality. The build complexity is a bit painful. Only recently was YARN alpha support dropped and maintaining the two was a headache. The overhead is smaller for this kind of feature. But it's a modest nice-to-have, and, may end up requiring yet a second implementation if any API changes between, say, 2.6 and 2.7 again. If it were vital, it might be something that just has to be done, but IMHO I sympathize with waiting for stable APIs for a nice-to-have feature. Hence, anything that can be done to bless the APIs that are required in YARN for 2.7 seems like the best use of time. Last question: I know the build already dodges around some tiny differences in API across YARN versions with reflection. Is that feasible here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4588 [MLLIB] [WIP] Add API for feature a...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4460#issuecomment-75129379 I'm OK with a type hierarchy as long as it stays simple (and doesn't turn into a type system parallel to the DataFrame system). To support any type of DataFrame (with Structs and Arrays), we'll need to support nesting for sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5904][SQL] DataFrame API fixes.
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4686#issuecomment-75129950 Btw, this fixed my issue with Java examples using DataFrame --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5494][SQL] SparkSqlSerializer Ignores K...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4693#issuecomment-75130162 There is no reason to allow custom kryo registration. The SQL serializer is only ever used to serialize SQL types. Have you seen some bug with us missing a type? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2555] Support configuration spark.sched...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1462#discussion_r25022847 --- Diff: docs/configuration.md --- @@ -831,7 +831,7 @@ Apart from these, the following properties are also available, and may be useful td0/td td The minimum ratio of registered resources (registered resources / total expected resources) -(resources are executors in yarn mode, CPU cores in standalone mode) +(resources are executors in yarn mode, CPU cores in standalone mode and coarse mesos mode) --- End diff -- coarse-grained mesos mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2555] Support configuration spark.sched...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1462#issuecomment-75130879 @pwendell @kayousterhout what is the verdict of this? Should we just remove the ratio altogether? What about backward compatibility? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5902] [ml] Made PipelineStage.transform...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4682#issuecomment-75122689 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27729/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4695#issuecomment-75123777
If we return a `JavaPairRDD`, the user code looks like the following:
~~~
for (Tuple2Long, Int assignment: assignments.collect()) {
... assignment._1() ...
... assignment._2() ...
}
~~~
With the current setting, this is
~~~
for (Assignment assignment: assignments.toJavaRDD().collect()) {
... assignment.id() ...
... assignment.cluster() ...
}
~~~
The latter is more readable to me. There is a cost on the user side if we
force using a special class in the input, for example, `Rating` for `ALS` and
`Document` for `LDA`. But for return types, the cost is not that high. Well,
this is not a strong argument.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5904][SQL] DataFrame API fixes.
Github user davies commented on the pull request: https://github.com/apache/spark/pull/4686#issuecomment-75124929 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5825] [Spark Submit] Remove the double ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4611#issuecomment-75125804 @srowen it actually does a `=~` which should be equivalent to a grep: ``` if [[ abc =~ abc ]]; then echo woohoo; fi # woohoo if [[ ffabc =~ abc ]]; then echo woohoo; fi # woohoo if [[ ffabcff =~ abc ]]; then echo woohoo; fi # woohoo ``` This seems like what we want, so I'm going to merge this into master and 1.3 thanks @chenghao-intel --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5873][SQL] Allow viewing of partially a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4684#issuecomment-75126780 [Test build #27732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27732/consoleFull) for PR 4684 at commit [`aee1e6a`](https://github.com/apache/spark/commit/aee1e6a027a17ce7cf2f522744a94dfb6217a047). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4423] Improve foreach() documentation t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4696#issuecomment-75119774 [Test build #27728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27728/consoleFull) for PR 4696 at commit [`d374d3a`](https://github.com/apache/spark/commit/d374d3a8e1086ae315dd8c1ca3fcc0ff3c105fcc). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4423] Improve foreach() documentation t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4696#issuecomment-75119787 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27728/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4695#issuecomment-75121366 For PIC, the names Assignment / id / cluster sound good to me. Those would be applicable to other clustering methods if ever needed. Assignment is a little generic, but ClusteringAssignment seems too verbose to me. * Just wondering, why go for a new type rather than returning a JavaPairRDD via javaAssignments()? (This seems analogous to the choice in LDA of whether to provide a Document type or take a JavaPairRDD.) The FPGrowth names and setup sound good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1537][WIP]: ats support for spark-on-ya...
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/4683#issuecomment-75124964 @srowen Yarn timeline is in alpha phase, but hadoop is embracing it as it provides a lot of extra functionalities. I will upload the patch and design doc to the JIRA. Before that, can you provide some clarification on alpha features in spark? In hadoop, AFAIK such features will be disabled by default and user can enable it if they want to try it. In spark, how to handle those alpha features that people may be interested in and want to play with it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-75124862 Jenkins, retest this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5825] [Spark Submit] Remove the double ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4611 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4695#issuecomment-75125808 LGTM except for that one comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5904][SQL] DataFrame API fixes.
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4686 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5867] [SPARK-5892] [doc] [ml] [mllib] D...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4675#issuecomment-75122512 @mbofb Thanks for noticing the issues with rows and cols documentation! I'll follow that JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5902] [ml] Made PipelineStage.transform...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4682#issuecomment-75122676 [Test build #27729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27729/consoleFull) for PR 4682 at commit [`6f02357`](https://github.com/apache/spark/commit/6f023576aed738124dba432211ad1b24060c679d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-75123029 [Test build #27730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27730/consoleFull) for PR 4688 at commit [`f8fe694`](https://github.com/apache/spark/commit/f8fe694efd117d707313748c02cef42240a3aec7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class UpdateCredentials(newCredentials: SerializableBuffer)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-75123043 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27730/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4695#discussion_r25020128
--- Diff: docs/mllib-frequent-pattern-mining.md ---
@@ -74,11 +74,12 @@ Calling `FPGrowth.run` with transactions returns an
that stores the frequent itemsets with their frequencies.
{% highlight java %}
-import java.util.Arrays;
import java.util.List;
import scala.Tuple2;
--- End diff --
no longer needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4695#issuecomment-75125659 Yeah, after reading the code, I like the special classes since the field names make the code more legible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4695#issuecomment-75126769 [Test build #27731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27731/consoleFull) for PR 4695 at commit [`9c0e590`](https://github.com/apache/spark/commit/9c0e59029eb61690974dbf8c5eecf80270bb6c6d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5873][SQL] Allow viewing of partially a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4684#issuecomment-75130185 [Test build #27735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27735/consoleFull) for PR 4684 at commit [`25753d2`](https://github.com/apache/spark/commit/25753d2abeedd0017b8b16b6b216297527c82929). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5347][CORE] Change FileSplit to InputSp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4150#issuecomment-75131653 Hi @shenh062326 since this is a duplicate would you mind closing this PR? The associated JIRA is already closed. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2555] Support configuration spark.sched...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1462#discussion_r25022830
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -62,6 +62,11 @@ private[spark] class MesosSchedulerBackend(
var classLoader: ClassLoader = null
+ if
(!sc.getConf.getOption(spark.scheduler.minRegisteredResourcesRatio).isEmpty) {
--- End diff --
`sc.conf.contains(...)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5095] Support capping cores and launch ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-75131737 Hey @tnachen can you add `[Mesos]` to the title? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5873][SQL] Allow viewing of partially a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4684#issuecomment-75131793 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27735/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5570: No docs stating that `new SparkCon...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4665#issuecomment-75132649 [Test build #27736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27736/consoleFull) for PR 4665 at commit [`f149ba1`](https://github.com/apache/spark/commit/f149ba1de60ffdab69b3b223461deccc582f0a7f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5873][SQL] Allow viewing of partially a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4684#issuecomment-75131788 [Test build #27735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27735/consoleFull) for PR 4684 at commit [`25753d2`](https://github.com/apache/spark/commit/25753d2abeedd0017b8b16b6b216297527c82929). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5902] [ml] Made PipelineStage.transform...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/4682#issuecomment-75132587 Merged into master and branch-1.3. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4808] Remove Spillable minimum threshol...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3656#issuecomment-75132672 Hey @lawlerd this seems to be superseded by #4420. Would you mind closing this PR? Please follow the latest discussion there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5902] [ml] Made PipelineStage.transform...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5570: No docs stating that `new SparkCon...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4665#issuecomment-75132183 Hi @andrewor14 - Can you please review the language for the other driver options and let me know if you agree with it? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4695#issuecomment-75134609 [Test build #27731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27731/consoleFull) for PR 4695 at commit [`9c0e590`](https://github.com/apache/spark/commit/9c0e59029eb61690974dbf8c5eecf80270bb6c6d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` class Assignment(val id: Long, val cluster: Int)` * `class FPGrowthModel[Item: ClassTag](val freqItemsets: RDD[FreqItemset[Item]]) extends Serializable` * ` class FreqItemset[Item](val items: Array[Item], val freq: Long) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4682 [CORE] Consolidate various 'Clock' ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4514#discussion_r25024948
--- Diff: core/src/main/scala/org/apache/spark/util/Clock.scala ---
@@ -21,9 +21,36 @@ package org.apache.spark.util
* An interface to represent clocks, so that they can be mocked out in
unit tests.
*/
private[spark] trait Clock {
- def getTime(): Long
+ def getTimeMillis(): Long
+ def waitTillTime(targetTime: Long): Long
}
-private[spark] object SystemClock extends Clock {
- def getTime(): Long = System.currentTimeMillis()
+private[spark] class SystemClock extends Clock {
+
+ val minPollTime = 25L
+
+ def getTimeMillis(): Long = System.currentTimeMillis()
+
+ def waitTillTime(targetTime: Long): Long = {
--- End diff --
I find this is used in `org.apache.spark.streaming.util.RecurringTimer`
which is still needed in `JobGenerator` and `BlockGenerator`. I can make the
other change though, one sec...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5900][MLLIB] make PIC and FPGrowth Java...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4695#issuecomment-75134621 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27731/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5873][SQL] Allow viewing of partially a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4684#issuecomment-75135398 [Test build #27732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27732/consoleFull) for PR 4684 at commit [`aee1e6a`](https://github.com/apache/spark/commit/aee1e6a027a17ce7cf2f522744a94dfb6217a047). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `class CheckAnalysis ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5873][SQL] Allow viewing of partially a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4684#issuecomment-75135416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27732/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5846] Correctly set job description and...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4631#issuecomment-75136332 Mind closing this now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5846] Correctly set job description and...
Github user kayousterhout closed the pull request at: https://github.com/apache/spark/pull/4631 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4689#discussion_r24975506
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/CacheManager.scala
---
@@ -39,7 +39,7 @@ private case class CachedData(plan: LogicalPlan,
cachedRepresentation: InMemoryR
private[sql] class CacheManager(sqlContext: SQLContext) extends Logging {
@transient
- private val cachedData = new
scala.collection.mutable.ArrayBuffer[CachedData]
+ val cachedData = new scala.collection.mutable.ArrayBuffer[CachedData]
--- End diff --
Seems no need to make this public? Didn't find this referenced anywhere
other than `CacheManager`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4689#issuecomment-75024215 This PR also added a `CLEAR CACHE` statement, would be good to add this in the PR description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4689#discussion_r24977332
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala ---
@@ -280,4 +280,43 @@ class CachedTableSuite extends QueryTest {
assert(intercept[RuntimeException](table(t1)).getMessage.startsWith(Table
Not Found))
assert(!isCached(t2))
}
+
+ test(Drop cached temporary table when the table gets overwritten) {
+val query1 = SELECT key FROM testData LIMIT 10
+val df1 = sql(query1)
+df1.registerTempTable(t1)
+sql(sCACHE TABLE t2 AS $query1)
+assert(isCached(t2))
+// t1 will be cached too because it has the same plan as t2.
+assert(isCached(t1))
+assert(cacheManager.lookupCachedData(df1).isDefined)
+
+val query2 = SELECT key FROM testData LIMIT 5
+val df2 = sql(query2)
+sql(sCACHE TABLE t2 AS $query2)
+// t1 will not be cached because it has not been explicitly cached.
+assert(!isCached(t1))
+assert(isCached(t2))
+assert(cacheManager.lookupCachedData(df2).isDefined)
+dropTempTable(t2)
+
+assert(cacheManager.lookupCachedData(df1).isEmpty)
+assert(cacheManager.lookupCachedData(df2).isEmpty)
+ }
--- End diff --
How about this case:
```scala
val df1 = sql(SELECT * FROM testData LIMIT 10)
df1.registerTempTable(t1)
// Cache t1 explicitly
sql(CACHE TABLE t1)
// t1 and t2 share the same query plan
sql(CACHE TABLE t2 AS SELECT * FROM testData LIMIT 10)
// Replace t2 with a different query plan
sql(CACHE TABLE t2 AS SELECT * FROM testData LIMIT 5)
// Should t1 remain cached here?
```
To my understanding, with this PR, `t1` is implicitly uncached, which may
not be the behavior we want. I think we also need a map from query plans to
table names to prevent unexpected uncache operations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark-5708: Add Slf4jSink to Spark Metrics
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4644#issuecomment-75029758 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27717/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark-5708: Add Slf4jSink to Spark Metrics
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4644#issuecomment-75029743 [Test build #27717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27717/consoleFull) for PR 4644 at commit [`ef837c0`](https://github.com/apache/spark/commit/ef837c0b7c79a21982624cb2954376abf8e6e75b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Checking data types when resolving types
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-75031044 [Test build #27718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27718/consoleFull) for PR 4685 at commit [`72e88db`](https://github.com/apache/spark/commit/72e88dbb9ac906862c7dc235a562e085c5c90064). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Checking data types when resolving types
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-75031054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27718/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-75034020 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27719/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-75034010 [Test build #27719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27719/consoleFull) for PR 4674 at commit [`cc5ac0b`](https://github.com/apache/spark/commit/cc5ac0b08ca39c3c339fdca905779bb3b037f8fa). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GraphLoaderSuite extends FunSuite with LocalSparkContext ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4689#discussion_r24976568
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSQLParser.scala
---
@@ -74,9 +75,15 @@ private[sql] class SparkSQLParser(fallback: String =
LogicalPlan) extends Abstr
}
private lazy val uncache: Parser[LogicalPlan] =
-UNCACHE ~ TABLE ~ ident ^^ {
- case tableName = UncacheTableCommand(tableName)
-}
+(
+ UNCACHE ~ TABLE ~ ident ^^ {
+case tableName = UncacheTableCommand(tableName)
+ }
+|
+ CLEAR ~ CACHE ^^ {
+case _ ~ _ = ClearCacheCommand
+ }
+)
--- End diff --
Nit: This can be simplified to
```scala
( UNCACHE ~ TABLE ~ ident ^^ {
case tableName = UncacheTableCommand(tableName)
}
| CLEAR ~ CACHE ^^^ ClearCacheCommand
)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5881][SQL] RDD remains cached after the...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4689#discussion_r24976046
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -888,6 +893,13 @@ class SQLContext(@transient val sparkContext:
SparkContext)
* only during the lifetime of this instance of SQLContext.
*/
private[sql] def registerDataFrameAsTable(df: DataFrame, tableName:
String): Unit = {
+if (catalog.tableExists(Seq(tableName))
cacheManager.lookupCachedData(df).isEmpty) {
+ // If the table already exists and the data of df has not already
been cached
+ // (we are trying to overwrite an existing temporary table),
+ // we will try to uncache the InMemoryRelation associated with the
existing table.
+ cacheManager.tryUncacheQuery(table(tableName))
--- End diff --
Should we add `blocking = false` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5904][SQL] DataFrame API fixes.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4686#issuecomment-75026734 [Test build #27714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27714/consoleFull) for PR 4686 at commit [`fd9b199`](https://github.com/apache/spark/commit/fd9b199e6ce4db18449bf25ad8b9f5d051354dd4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5904][SQL] DataFrame API fixes.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4686#issuecomment-75026746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27714/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user ankurdave commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-75022368 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user MechCoder commented on the pull request:
https://github.com/apache/spark/pull/4677#issuecomment-75026265
@jkbradley I have fixed up your comments.
Btw, why are there are both a train and a run, which seems to me do the
same thing. Is it not better to have one way of doing things.
Also a doubt in the case of the Classification problem. It seems to me for
each iteration, the problem is changed explicitly to a Regression problem with
labels mapped to {-1, 1}. Is it okay to break when this regression error no
longer reduces on the validation data for a classification problem (which seems
slightly awkward to me)?
Note that I had to explicitly set it to Regression to make sure that this
test passes,
(https://github.com/apache/spark/pull/4677/files#diff-d3159b88ae0ed6ff096ff8850ecac26eR207)
. Otherwise, the classification error seems to be the same for both with and
without validation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-75022575 [Test build #27719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27719/consoleFull) for PR 4674 at commit [`cc5ac0b`](https://github.com/apache/spark/commit/cc5ac0b08ca39c3c339fdca905779bb3b037f8fa). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5841 [CORE] [HOTFIX 2] Memory leak in Di...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4690#discussion_r24979807
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -148,7 +148,13 @@ private[spark] class DiskBlockManager(blockManager:
BlockManager, conf: SparkCon
/** Cleanup local dirs and stop shuffle sender. */
private[spark] def stop() {
// Remove the shutdown hook. It causes memory leaks if we leave it
around.
-Runtime.getRuntime.removeShutdownHook(shutdownHook)
+if(shutdownHook != null) {
+ try {
+Runtime.getRuntime.removeShutdownHook(shutdownHook)
+ } catch {
+case e: Exception = None
--- End diff --
Narrow this to `IllegalStateException`, to only squash this particular
case, hopefully? I think leaving the `null` check in is OK even if I'm not sure
it can happen since the variable is assigned to non-`null` in its constructor.
Nit: put a space after `if`. I'll pause a beat for others to weigh in today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75029645 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27716/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75029633 [Test build #27716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27716/consoleFull) for PR 4677 at commit [`3e74372`](https://github.com/apache/spark/commit/3e743723f128f5d87f66dc05a92d3f68e6fd01cb). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Partitioner(object):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5889] Remove pid file after stopping se...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4676#discussion_r24978988 --- Diff: sbin/spark-daemon.sh --- @@ -169,6 +169,7 @@ case $option in if [[ $(ps -p $TARGET_ID -o args=) =~ $command ]]; then echo stopping $command kill $TARGET_ID --- End diff -- Concretely, can there be an `if` statement here to check whether `kill` succeeded, and only `rm` if so? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75042207 [Test build #27720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27720/consoleFull) for PR 4677 at commit [`55e5c3b`](https://github.com/apache/spark/commit/55e5c3b22c39ef1f71e35eca0fda06a9080d22d7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75052502 [Test build #27720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27720/consoleFull) for PR 4677 at commit [`55e5c3b`](https://github.com/apache/spark/commit/55e5c3b22c39ef1f71e35eca0fda06a9080d22d7). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5436] [MLlib] Validate GradientBoostedT...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4677#issuecomment-75052509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27720/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Checking data types when resolving types
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-75015080 [Test build #27715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27715/consoleFull) for PR 4685 at commit [`4933dac`](https://github.com/apache/spark/commit/4933dac3c552e4e937acfc4fdd05ead15919b883). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5651][SQL] Add input64 in blacklist and...
GitHub user watermen reopened a pull request: https://github.com/apache/spark/pull/4427 [SPARK-5651][SQL] Add input64 in blacklist and add test suit for create table within backticks Now spark version is only support ```create table table_in_database_creation.test1 as select * from src limit 1;``` in HiveContext. This patch is used to support ```create table `table_in_database_creation.test2` as select * from src limit 1;``` in HiveContext. You can merge this pull request into a Git repository by running: $ git pull https://github.com/watermen/spark SPARK-5651 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4427.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4427 commit 1f0e42e6465ce89cea95ff67b3e4c50671fce74a Author: q00251598 qiyad...@huawei.com Date: 2015-02-10T03:55:33Z add input64 in blacklist and add test suit commit c5c8ed18abc27e9b2b48ed74a5771071746a901f Author: watermen qiyadong2...@gmail.com Date: 2015-02-10T04:19:48Z add the generated golden files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5651][SQL] Add input64 in blacklist and...
Github user watermen commented on the pull request: https://github.com/apache/spark/pull/4427#issuecomment-75016616 @marmbrus can you --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-75016589 **[Test build #27709 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27709/consoleFull)** for PR 4688 at commit [`2b0d745`](https://github.com/apache/spark/commit/2b0d745ec7b76c3dd992660c24ddac556ba1de6a) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-75016591 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27709/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5651][SQL] Add input64 in blacklist and...
Github user watermen closed the pull request at: https://github.com/apache/spark/pull/4427 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5494][SQL] SparkSqlSerializer Ignores K...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4693#issuecomment-75075748 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
