[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58315661 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21449/consoleFull) for PR 2529 at commit [`7b7e6e3`](https://github.com/apache/spark/commit/7b7e6e327aaa2942b30f2de1ee64417ca091269e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3710] Fix Yarn integration tests on Had...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2682#issuecomment-58315769 Great, let's take it for a spin. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58315840 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21448/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3710] Fix Yarn integration tests on Had...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-58315865 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21447/consoleFull) for PR 1616 at commit [`935fed6`](https://github.com/apache/spark/commit/935fed670bd951ca90072d2ebe24665485d164f3). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-58315866 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21447/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1685#issuecomment-58316202 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412] [PySpark] Replace Epydoc with Sph...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2689#issuecomment-58316342 This PR should point to SPARK-3420, I guess. We should leave SPARK-3412 open and close SPARK-3420 instead. @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58316539 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1685#issuecomment-58316638 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21450/consoleFull) for PR 1685 at commit [`57a2315`](https://github.com/apache/spark/commit/57a23158973872f17d577730b85cab808fc99142). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58316998 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21451/consoleFull) for PR 2529 at commit [`7b7e6e3`](https://github.com/apache/spark/commit/7b7e6e327aaa2942b30f2de1ee64417ca091269e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][Doc] Keep Spark SQL README.md up to date
GitHub user Ishiihara opened a pull request: https://github.com/apache/spark/pull/2706 [SQL][Doc] Keep Spark SQL README.md up to date @marmbrus Update README.md to be consistent with Spark 1.1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Ishiihara/spark SparkSQL-readme Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2706.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2706 commit 33b9d4b83fbc514d9895a1590155f276c26fb46f Author: Liquan Pei liquan...@gmail.com Date: 2014-10-08T06:57:13Z keep README.md up to date --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58318815 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21449/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58318810 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21449/consoleFull) for PR 2529 at commit [`7b7e6e3`](https://github.com/apache/spark/commit/7b7e6e327aaa2942b30f2de1ee64417ca091269e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58319419 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21451/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][Doc] Keep Spark SQL README.md up to date
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2706#issuecomment-58319420 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21452/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58319412 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21451/consoleFull) for PR 2529 at commit [`7b7e6e3`](https://github.com/apache/spark/commit/7b7e6e327aaa2942b30f2de1ee64417ca091269e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-58319574 @vanzin yeah I agree - I think it was just sort of a red herring based on the example in @ScrapCodes description. AFAIK this is totally unrelated to Hadoop. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58320520 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58320856 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21453/consoleFull) for PR 2529 at commit [`7b7e6e3`](https://github.com/apache/spark/commit/7b7e6e327aaa2942b30f2de1ee64417ca091269e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2667#issuecomment-58320921 @srowen Ranking metrics are different from multiclass metrics. In general, multiclass metrics do not consider the ordering of the predictions, but just hits and misses, and they don't truncate the result. Ranking metrics consider the ordering of predictions, applying discounts and truncating the result. I think they are two sets of evaluation metrics. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1685#issuecomment-58322010 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21450/consoleFull) for PR 1685 at commit [`57a2315`](https://github.com/apache/spark/commit/57a23158973872f17d577730b85cab808fc99142). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1685#issuecomment-58322014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21450/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
GitHub user mengxr opened a pull request: https://github.com/apache/spark/pull/2707 [SPARK-3844][UI] Truncate appName in WebUI if it is too long Truncate appName in WebUI if it is too long. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mengxr/spark truncate-app-name Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2707.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2707 commit c7111dcba885cb261bcc2a037052d4a6af0bc14e Author: Xiangrui Meng m...@databricks.com Date: 2014-10-08T07:57:51Z truncate appName in WebUI if it is too long --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3841] [mllib] Pretty-print params for M...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2700#issuecomment-58323685 That doesn't look good. I think it is still nice to put the full set of parameters in the WebUI, which is easier to check than the logs. https://github.com/apache/spark/pull/2707 fixes the UI issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2707#issuecomment-58324074 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21454/consoleFull) for PR 2707 at commit [`c7111dc`](https://github.com/apache/spark/commit/c7111dcba885cb261bcc2a037052d4a6af0bc14e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58324547 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21455/consoleFull) for PR 2388 at commit [`ca8e6f2`](https://github.com/apache/spark/commit/ca8e6f296a2f7ed674dd3a5cde49d4301d3d6d14). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
GitHub user chouqin opened a pull request: https://github.com/apache/spark/pull/2708 [SPARK-3158][MLLIB]Avoid 1 extra aggregation for DecisionTree training Currently, the implementation does one unnecessary aggregation step. The aggregation step for level L (to choose splits) gives enough information to set the predictions of any leaf nodes at level L+1. We can use that info and skip the aggregation step for the last level of the tree (which only has leaf nodes). ### Implementation Details Each node now has a `impurity` field and the `predict` is changed from type `Double` to type `Predict`(this can be used to compute predict probability in the future) When compute best splits for each node, we also compute impurity and predict for the child nodes, which is used to constructed newly allocated child nodes. So at level L, we have set impurity and predict for nodes at level L +1. If level L+1 is the last level, then we can avoid aggregation. What's more, calculation of parent impurity in Top nodes for each tree needs to be treated differently because we have to compute impurity and predict for them first. In `binsToBestSplit`, if current node is top node(level == 0), we calculate impurity and predict first. after finding best split, top node's predict and impurity is set to the calculated value. Non-top nodes's impurity and predict are already calculated and don't need to be recalculated again. I have considered to add a initialization step to set top nodes' impurity and predict and then we can treat all nodes in the same way, but this will need a lot of duplication of code(all the code to do seq operation(BinSeqOp) needs to be duplicated), so I choose the current way. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chouqin/spark avoid-agg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2708.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2708 commit 6cc0333bc02332bcf94d75c00b6850ea4d4e79f6 Author: Qiping Li liqiping1...@gmail.com Date: 2014-10-08T04:03:35Z SPARK-3158: Avoid 1 extra aggregation for DecisionTree training commit e41d715bf35bc1dd948fdb2c60317fd66f86fdec Author: Qiping Li liqiping1...@gmail.com Date: 2014-10-08T04:16:01Z fix bug in test suite commit 822c91274526e77528ef0a1c4a0e92a14f5696a5 Author: Qiping Li liqiping1...@gmail.com Date: 2014-10-08T07:32:19Z add comments and unit test commit 7ad7a71a0022ff808fb0066b68fe07a8c1a830b4 Author: Qiping Li liqiping1...@gmail.com Date: 2014-10-08T07:47:57Z fix unit test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58325016 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21456/consoleFull) for PR 2708 at commit [`7ad7a71`](https://github.com/apache/spark/commit/7ad7a71a0022ff808fb0066b68fe07a8c1a830b4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58325117 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21453/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3412][SQL]add missing row api
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2529#issuecomment-58325112 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21453/consoleFull) for PR 2529 at commit [`7b7e6e3`](https://github.com/apache/spark/commit/7b7e6e327aaa2942b30f2de1ee64417ca091269e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2520#discussion_r18570229
--- Diff: project/SparkBuild.scala ---
@@ -170,6 +178,24 @@ object SparkBuild extends PomBuild {
}
+object YARNCommon {
+ lazy val settings = Seq(
+scalaSource in Compile := baseDirectory.value / common/src/main/scala
+ )
+}
+
+object YARNAlpha {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
alpha/src/main/scala
+ )
+}
+
+object YARNStable {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
stable/src/main/scala
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2520#discussion_r18570223
--- Diff: project/SparkBuild.scala ---
@@ -170,6 +178,24 @@ object SparkBuild extends PomBuild {
}
+object YARNCommon {
+ lazy val settings = Seq(
+scalaSource in Compile := baseDirectory.value / common/src/main/scala
+ )
+}
+
+object YARNAlpha {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
alpha/src/main/scala
--- End diff --
This is an unnecessary change, I just verified. The reason is this step is
done by sbt pom reader plugin.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Minor change in the comment of spark-defaults....
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/2709 Minor change in the comment of spark-defaults.conf.template spark-defaults.conf is used in spark-shell as well, and this PR added this into the comment. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dbtsai/spark docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2709.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2709 commit b3e1ff1b808380707d04277c2379bf5b03556662 Author: DB Tsai dbt...@alpinenow.com Date: 2014-10-08T08:53:25Z add spark-shell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-58328853 I did not check up on maven part, it looks okay. Apart from two comments above, LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3343] [SQL] Add serde support for CTAS
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2570#issuecomment-58329046 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21458/consoleFull) for PR 2570 at commit [`d49596b`](https://github.com/apache/spark/commit/d49596b868094570d4238720ff59e49aab263020). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Minor change in the comment of spark-defaults....
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2709#issuecomment-58329045 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21457/consoleFull) for PR 2709 at commit [`b3e1ff1`](https://github.com/apache/spark/commit/b3e1ff1b808380707d04277c2379bf5b03556662). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2707#issuecomment-58330380 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21454/consoleFull) for PR 2707 at commit [`c7111dc`](https://github.com/apache/spark/commit/c7111dcba885cb261bcc2a037052d4a6af0bc14e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2707#issuecomment-58330386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21454/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58330730 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21456/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58330725 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21456/consoleFull) for PR 2708 at commit [`7ad7a71`](https://github.com/apache/spark/commit/7ad7a71a0022ff808fb0066b68fe07a8c1a830b4). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58330999 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21455/consoleFull) for PR 2388 at commit [`ca8e6f2`](https://github.com/apache/spark/commit/ca8e6f296a2f7ed674dd3a5cde49d4301d3d6d14). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class Params(inputFile: String = null, threshold: Double = 0.1)` * `class TopicModelingKryoRegistrator extends KryoRegistrator ` * `class Word2VecModel(object):` * `class Word2Vec(object):` * ` class SparkIMain(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1405][MLLIB] topic modeling on Graphx
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58331009 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21455/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58333659 @scwf Jenkins still fails... Actually I just realized that `HiveThriftServer2Suite` calls `start-thriftserver.sh` to start a separate server process, and the log level of the new process can't be controlled by the test code. On the other hand, I can reproduce this issue by putting a `log4j.properties` under `conf/` and adding this line to surpress INFO logs: ``` log4j.rootCategory=WARN, console ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3343] [SQL] Add serde support for CTAS
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2570#issuecomment-58334350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21458/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3343] [SQL] Add serde support for CTAS
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2570#issuecomment-58334343 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21458/consoleFull) for PR 2570 at commit [`d49596b`](https://github.com/apache/spark/commit/d49596b868094570d4238720ff59e49aab263020). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` logDebug(Found class for $serdeName)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3568 [mllib] add ranking metrics
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2667#issuecomment-58334664 @mengxr Yes I understand these metrics. Precision / recall are binary classifier metrics at heart (but not nDCG for example). Precision@k needs ranking. That's why precision/recall turn up in `MulticlassMetrics` as a special case -- you can compute precision of class-_i_ vs not-class-_i_. I understand it is two views on the same metric, and these are collections of different metrics. Hm, maybe this makes more sense if the metrics here are clearly precision@k, recall@k, with an argument _k_. The other class has F-measure, true positive rate, etc. I suppose those could be implemented here too for consistency, but don't know how useful they are. Meh, anything to rationalize the purposeful difference between these two would help. Maybe it's just me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] [Windows] Wrong EOL character in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58335548 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21459/consoleFull) for PR 2612 at commit [`44469ab`](https://github.com/apache/spark/commit/44469abbb1e9dc693be771234885a405776134be). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2520#discussion_r18572940
--- Diff: project/SparkBuild.scala ---
@@ -170,6 +178,24 @@ object SparkBuild extends PomBuild {
}
+object YARNCommon {
+ lazy val settings = Seq(
+scalaSource in Compile := baseDirectory.value / common/src/main/scala
+ )
+}
+
+object YARNAlpha {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
alpha/src/main/scala
--- End diff --
Really?
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
has 100+ length line but sbt/sbt -Pyarn -Phadoop-2.2 scalastyle couldn't
verified.
I think, all projects except for yarn has source directory line project
name/src but yarn project has source directory yarn/common/src instead of
yarn/src so I thought scalastyle couldn't verify.
How do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Minor change in the comment of spark-defaults....
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2709#issuecomment-58336192 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21457/consoleFull) for PR 2709 at commit [`b3e1ff1`](https://github.com/apache/spark/commit/b3e1ff1b808380707d04277c2379bf5b03556662). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Minor change in the comment of spark-defaults....
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2709#issuecomment-58336202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21457/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/2542#issuecomment-58338302 I tried this on hive: CREATE TABLE t1(x INT); CREATE TABLE t2(a STRUCTx: INT, k INT); SELECT a.x FROM t1 a JOIN t2 b; And hive can resolve this. However it is reasonable to report ambiguous references here, should we follow hive for this case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Github user tianyi commented on the pull request: https://github.com/apache/spark/pull/2542#issuecomment-58338634 @cloud-fan I had create a test case named test ambiguousReferences resolved as hive , the case is almost the same as yours. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Github user tianyi commented on the pull request: https://github.com/apache/spark/pull/2542#issuecomment-58339330 I think we should follow hive syntax as much as possible, which could save lots of work when people move their application form hive to spark, @marmbrus what's your opinion? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58339521 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21460/consoleFull) for PR 2708 at commit [`c41b1b6`](https://github.com/apache/spark/commit/c41b1b6c7cbf11c6298e25a5eb28f65138d91f3d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2520#discussion_r18574479
--- Diff: project/SparkBuild.scala ---
@@ -170,6 +178,24 @@ object SparkBuild extends PomBuild {
}
+object YARNCommon {
+ lazy val settings = Seq(
+scalaSource in Compile := baseDirectory.value / common/src/main/scala
+ )
+}
+
+object YARNAlpha {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
alpha/src/main/scala
--- End diff --
This setting line is about yarn-alpha and not common ?
So changes related to common is acceptable. But you can safely delete this
setting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2520#discussion_r18575095
--- Diff: project/SparkBuild.scala ---
@@ -170,6 +178,24 @@ object SparkBuild extends PomBuild {
}
+object YARNCommon {
+ lazy val settings = Seq(
+scalaSource in Compile := baseDirectory.value / common/src/main/scala
+ )
+}
+
+object YARNAlpha {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
alpha/src/main/scala
--- End diff --
Only for scalastyle, settings for alpha and stable are not needed but if we
don't set those settings, we cannot build because the sources in common cannot
resolve reference to the sources in alpha and stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] [Windows] Wrong EOL character in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58341737 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21459/consoleFull) for PR 2612 at commit [`44469ab`](https://github.com/apache/spark/commit/44469abbb1e9dc693be771234885a405776134be). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] [Windows] Wrong EOL character in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58341742 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21459/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2520#discussion_r18575772
--- Diff: project/SparkBuild.scala ---
@@ -170,6 +178,24 @@ object SparkBuild extends PomBuild {
}
+object YARNCommon {
+ lazy val settings = Seq(
+scalaSource in Compile := baseDirectory.value / common/src/main/scala
+ )
+}
+
+object YARNAlpha {
+ lazy val settings = Seq(
+unmanagedSourceDirectories in Compile += baseDirectory.value /
alpha/src/main/scala
--- End diff --
@ScrapCodes If we remove settings for alpha and stable from
SparkBuild.scala, we fail to build with 'sbt/sbt -Pyarn -Phadoop-2.2 assembly'
even though it can pass scalastyle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58345233 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21460/consoleFull) for PR 2708 at commit [`c41b1b6`](https://github.com/apache/spark/commit/c41b1b6c7cbf11c6298e25a5eb28f65138d91f3d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58345239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21460/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise does not work in H...
GitHub user ravipesala opened a pull request: https://github.com/apache/spark/pull/2710 [SPARK-3814][SQL] Bitwise does not work in Hive Currently there is no support of Bitwise in Spark HiveQl and Spark SQL as well. So this PR support the same. Author : ravipesala ravindra.pes...@huawei.com You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/spark SPARK-3814 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2710.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2710 commit 41e840b0ce6fe8321b131a80c8444ad70ff24c0b Author: ravipesala ravindra.pes...@huawei.com Date: 2014-10-08T11:58:46Z Supporting Bitwise in Spark HiveQl and SQL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise does not work in H...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2710#issuecomment-58348032 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][Doc] Keep Spark SQL README.md up to date
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2706#issuecomment-58350942 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/286/consoleFull) for PR 2706 at commit [`33b9d4b`](https://github.com/apache/spark/commit/33b9d4b83fbc514d9895a1590155f276c26fb46f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3654][SQL][WIP] Unifies SQL and HiveQL ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58353197 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21461/consoleFull) for PR 2698 at commit [`a4b0fc4`](https://github.com/apache/spark/commit/a4b0fc4fc4e3c0ffe7d251dc3e22e96ad68c44dc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2675#issuecomment-58353194
I also realized that. But i think there is some unkown reason here, i moved
```
// HACK: Hive is too noisy by default.
org.apache.log4j.LogManager.getCurrentLoggers.foreach { log =
log.asInstanceOf[org.apache.log4j.Logger].setLevel(org.apache.log4j.Level.WARN)
}
``` into method ```startThriftServerWithin```, then
sbt/sbt -Phive,hadoop-2.3 test-only
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2Suite failed.
So can you remove the hack in TestHive.scala and then retest this? I think
the problem is still there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3247][SQL] An API for adding forei...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/2475#discussion_r18580229
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -254,6 +254,8 @@ private[sql] abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
def numPartitions = self.numPartitions
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
+ case l @ foreign.LogicalRelation(t: foreign.TableScan) =
+ExistingRdd(l.output, t.buildScan()) :: Nil
--- End diff --
Will we add rules for `PrunedScan` and `FilteredScan` in this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL]fix HiveThriftServer2Suite to...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/2671#issuecomment-58354129 we will fix it in #2675 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL]fix HiveThriftServer2Suite to...
Github user scwf closed the pull request at: https://github.com/apache/spark/pull/2671 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] [Windows] Wrong EOL character in ...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58354515 I added lint-scripts for checking style of scripts. The script checks following things. * EOL * extra spaces before EOL * tabs * multibyte characters --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2675#issuecomment-58356235
Wow, very very strange!!! If i add a print in the hack, then
HiveThriftServer2Suite success.
```
// HACK: Hive is too noisy by default.
org.apache.log4j.LogManager.getCurrentLoggers.foreach { log =
println(s## ${log})
log.asInstanceOf[org.apache.log4j.Logger].setLevel(org.apache.log4j.Level.WARN)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][Doc] Keep Spark SQL README.md up to date
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2706#issuecomment-58356613 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/286/consoleFull) for PR 2706 at commit [`33b9d4b`](https://github.com/apache/spark/commit/33b9d4b83fbc514d9895a1590155f276c26fb46f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2711 [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_PATH instead of -Djava.library.path This is another implementation about #1031 You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-1719 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2711 commit 033201d6a7f9c2d48f67613715013386313783b6 Author: GuoQiang Li wi...@qq.com Date: 2014-10-08T13:32:23Z use LD_LIBRARY_PATH instead of -Djava.library.path --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58358351 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21462/consoleFull) for PR 2711 at commit [`033201d`](https://github.com/apache/spark/commit/033201d6a7f9c2d48f67613715013386313783b6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3654][SQL][WIP] Unifies SQL and HiveQL ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58359162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21461/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3654][SQL][WIP] Unifies SQL and HiveQL ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58359149 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21461/consoleFull) for PR 2698 at commit [`a4b0fc4`](https://github.com/apache/spark/commit/a4b0fc4fc4e3c0ffe7d251dc3e22e96ad68c44dc). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` protected case class Keyword(str: String)` * `class SqlLexical(val keywords: Seq[String]) extends StdLexical ` * ` case class FloatLit(chars: String) extends Token ` * `class SqlParser extends AbstractSparkSQLParser ` * `case class SetCommand(kv: Option[(String, Option[String])]) extends Command ` * `case class ShellCommand(cmd: String) extends Command` * `case class SourceCommand(filePath: String) extends Command` * `case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58359171 OK, I removed the log level hack from `TestHiveContext`, let's see what happens. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3788] [yarn] Fix compareFs to do the ri...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2649#issuecomment-58359605 test failure is in streaming-flume, totally unrelated to this. changes look good, thanks @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58359794 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21463/consoleFull) for PR 2675 at commit [`a4219c4`](https://github.com/apache/spark/commit/a4219c43fbc0a4343b91108f1fe03f47ecc2ca67). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3788] [yarn] Fix compareFs to do the ri...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2650#issuecomment-58359865 changes look good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3788] [yarn] Fix compareFs to do the ri...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2649 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3339][SQL] Support for skipping json li...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2680#issuecomment-58360514 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21464/consoleFull) for PR 2680 at commit [`b4a3632`](https://github.com/apache/spark/commit/b4a36329375a629cfe7570938558c2114eb0fcf1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
GitHub user james64 opened a pull request: https://github.com/apache/spark/pull/2712 [SPARK-3121] Wrong implementation of implicit bytesWritableConverter val path = ... //path to seq file with BytesWritable as type of both key and value val file = sc.sequenceFile[Array[Byte],Array[Byte]](path) file.take(1)(0)._1 This prints incorrect content of byte array. Actual content starts with correct one and some random bytes and zeros are appended. BytesWritable has two methods: getBytes() - return content of all internal array which is often longer then actual value stored. It usually contains the rest of previous longer values copyBytes() - return just begining of internal array determined by internal length property It looks like in implicit conversion between BytesWritable and Array[byte] getBytes is used instead of correct copyBytes. @dbtsai You can merge this pull request into a Git repository by running: $ git pull https://github.com/james64/spark 3121-bugfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2712.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2712 commit 480f9cdaf69254dd429b949d9ccc6d0b2c617ad0 Author: Dubovsky Jakub dubov...@avast.com Date: 2014-10-08T13:49:41Z Bug 3121 fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58361701 Jenkins, please start the test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58361679 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3559][SQL] Remove unnecessary columns f...
GitHub user gvramana opened a pull request: https://github.com/apache/spark/pull/2713 [SPARK-3559][SQL] Remove unnecessary columns from List of needed Column Ids in Hive Conf Author: Venkata Ramana G ramana.gollam...@huawei.com You can merge this pull request into a Git repository by running: $ git pull https://github.com/gvramana/spark remove_unnecessary_columns Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2713.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2713 commit 6a934593d1126f44ddc4576d04cc84412ea7cb9b Author: Venkata Ramana Gollamudi ramana.gollam...@huawei.com Date: 2014-10-08T13:38:32Z cloned hiveconf for each TableScanOperators so that only required columns are added commit b7ba7683434c602dd649d4468e5a7054f5efc408 Author: Venkata Ramana Gollamudi ramana.gollam...@huawei.com Date: 2014-10-08T14:14:34Z Added comment and checkstyle fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3559][SQL] Remove unnecessary columns f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2713#issuecomment-58364730 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] [Windows] Wrong EOL character in ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58364698 Hm, can't git just manage this for us? Looks like you could tell git that .cmd files are always text with Windows line endings with `.gitattributes`? https://help.github.com/articles/dealing-with-line-endings/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58368014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21462/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3339][SQL] Support for skipping json li...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2680#issuecomment-58368109 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21464/consoleFull) for PR 2680 at commit [`b4a3632`](https://github.com/apache/spark/commit/b4a36329375a629cfe7570938558c2114eb0fcf1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3339][SQL] Support for skipping json li...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2680#issuecomment-58368116 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21464/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2704 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2714 [SPARK-3781] code Style format code Style format, modify the third time :( You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2714.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2714 commit e00f13a554fb4a95996250e6cd65403fcc20b6e2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 271e2a4a60ec2412f32966b65448c8ca25dc377f Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 033af6a1bd2011b847633934ba719716d459aca8 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T15:03:24Z code style format commit 5732fae2d1ad94c820cc0a2fbc246aeee3142c00 Author: shijinkui shijinkui...@163.com Date: 2014-10-08T01:30:07Z resolve conflic commit 1f96a878815fd65035e11f5755426e6c9fda3ece Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 4a685cc2ea323f0fc54d6593e0ee568d829ebb02 Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format commit 5be64c970c2f5d7fb895910863eb6aa5d1e923bf Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:41:58Z Merge branch 'master' of git://git.apache.org/spark commit 42a9d81773d198c9fab7f77e0f2673d16c71 Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:42:29Z resolve test fail --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2704#issuecomment-58369606 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21465/consoleFull) for PR 2704 at commit [`42a9d81`](https://github.com/apache/spark/commit/42a9d81773d198c9fab7f77e0f2673d16c71). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2704#issuecomment-58369787 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21465/consoleFull) for PR 2704 at commit [`42a9d81`](https://github.com/apache/spark/commit/42a9d81773d198c9fab7f77e0f2673d16c71). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class FieldAccessFinder(output: mutable.Map[Class[_], mutable.Set[String]]) extends ClassVisitor(ASM4) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2704#issuecomment-58369791 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21465/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2704#issuecomment-58369964 @AmplabJenkins @andrewor14 review and run test again please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2714#issuecomment-58370041 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/2714#issuecomment-58370144 @AmplabJenkins @andrewor14 review and run test again please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
