[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #79181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79181/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). * This patch **fails Python style tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79181/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #79181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79181/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/14963 Hi, are you still working on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76550/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #76550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76550/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #76550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76550/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 So under the current jenkins set up this isn't running the Python style checks in Jenkins so it would be good to make sure these work in jenkins (can you update a python file)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73451/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #73451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73451/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #73451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73451/testReport)** for PR 14963 at commit [`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 That's a reasonable question, I'll try and look at this some next week but if you've got a chance to look at/address the current comments on it that would be great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Any hope this patch might be integrated ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70518/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #70518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70518/testReport)** for PR 14963 at commit [`4e27af2`](https://github.com/apache/spark/commit/4e27af2a87f5bb5c60425a98336667bc81cd9cbd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #70518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70518/testReport)** for PR 14963 at commit [`4e27af2`](https://github.com/apache/spark/commit/4e27af2a87f5bb5c60425a98336667bc81cd9cbd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 I agree, just email me :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 Also if it would be useful to have a quick chat off-line and then circle back to the PR with the result of our chat let me know - I'd really like to see us get something like this in so we can have better/more consistent linting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 @Stibbons however the problem is the virtualenv that gets created when a user is in a conda enviroment is partially broken. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66846/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66846 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66846/consoleFull)** for PR 14963 at commit [`cd98471`](https://github.com/apache/spark/commit/cd9847143869df11d5b4fb17d190817d5df023db). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66848/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66848/consoleFull)** for PR 14963 at commit [`132fec8`](https://github.com/apache/spark/commit/132fec81232c08a6adca910cc7b8333af722aa29). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66837/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66837 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66837/consoleFull)** for PR 14963 at commit [`278ff07`](https://github.com/apache/spark/commit/278ff07680a0ddf3f4a78266d0fa152aa3d2c93e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66836/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66836/consoleFull)** for PR 14963 at commit [`8a6d659`](https://github.com/apache/spark/commit/8a6d6599f96b6bc4b20f7101313701c9e051b158). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66848 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66848/consoleFull)** for PR 14963 at commit [`132fec8`](https://github.com/apache/spark/commit/132fec81232c08a6adca910cc7b8333af722aa29). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66846/consoleFull)** for PR 14963 at commit [`cd98471`](https://github.com/apache/spark/commit/cd9847143869df11d5b4fb17d190817d5df023db). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Ho, I have these pylint errors on my ubuntu! Probably I did not rebased correctly. Fixed with new check ignore: - deprecated-method - unsubscriptable-object, - used-before-assignment, of course, once merged. i can work on fixing this pylint errors ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66838/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66838 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66838/consoleFull)** for PR 14963 at commit [`93395fd`](https://github.com/apache/spark/commit/93395fdef27fbe2bc269d4df010640d85b7dcf82). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66835/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66835/consoleFull)** for PR 14963 at commit [`087d1a0`](https://github.com/apache/spark/commit/087d1a04a5c323754de1f2ea295260a11740f478). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 I really think using Spark with Anaconda is a **must have**. Deploying jobs that runs inside a Conda environment is so fast and efficient. I really want to push for this pull request #14180 that also add anaconda support for executors (thanks to jeff's work). Actually before this PR I never tried Anaconda, and now I use it everyday on jupyter. Anaconda on the driver and executor make totally sense! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 You and I agree, actually: - PySpark can run inside Anaconda, and indeed this is greatly valuable. This will make available to the "driver" all the package provided by Anaconda (in client mode) - Making the same environment available for executor is a bit messy for the moment. Here we have an NFS share that every executor use, but having it automatically deployed would be so cool. that's why the other PR :) For development, PySpark will still use the anaconda environment if the user wishes so. Just for the `lint-python` execution, we revert back to a virtualenv, but only for the duration of the script --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 @Stibbons the jenkins environment for Spark is not OS X, but since a lot of the developers work in OS X I figured it would be good to test there too. I think we should probably figure out why @sethah is getting a different result than you and I are. As for Conda I'm not sure I 100% agree, looking at https://www.continuum.io/blog/developer-blog/using-anaconda-pyspark-distributed-language-processing-hadoop-cluster and http://blog.cloudera.com/blog/2016/02/making-python-on-apache-hadoop-easier-with-anaconda-and-cdh/ it seems like many people are looking at ways to use conda together with PySpark so I think we should consider the possibility of running on a machine with Conda installed (but we can investigate other work arounds besides the one I proposed in the PR to your branch) :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66838/consoleFull)** for PR 14963 at commit [`93395fd`](https://github.com/apache/spark/commit/93395fdef27fbe2bc269d4df010640d85b7dcf82). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Here is my proposal: I leave the environment and let the script create the virtual env, plus minor improvement on the linter script. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66837/consoleFull)** for PR 14963 at commit [`278ff07`](https://github.com/apache/spark/commit/278ff07680a0ddf3f4a78266d0fa152aa3d2c93e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66836/consoleFull)** for PR 14963 at commit [`8a6d659`](https://github.com/apache/spark/commit/8a6d6599f96b6bc4b20f7101313701c9e051b158). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66835/consoleFull)** for PR 14963 at commit [`087d1a0`](https://github.com/apache/spark/commit/087d1a04a5c323754de1f2ea295260a11740f478). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 I have just tested this pull request on MacOS X with latest version of Spark, no error: ``` ... Checking Pep8... PEP8 checks passed. Checking Pylint... Pylint checks passed. rm -rf _build/* pydoc checks passed. ``` Is spark compiled with MacOS on Jenkins ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Indeed, I have not taken into account Anaconda environment, first because this tool provide quick and efficient way of having an almost good working environment to run jobs with numpy, pandas, and so on. There are many conda packages available to download, it could almost make pip irrelevant. But actually not. Indeed, Anaconda and pip do not really play in the same garden. Anaconda is great for one shot job or numpy script that needs to run is a quite well controlled environment, but this level of control is not that great. You can say you want numpy v1.1.0 but the user could have chosen another "channel" that the default one because someelse has compiled for the architecture. For example you can use `conda install -c intel` to download intel optimized libraries. This is great but there is too much latitude errors to occur. Reproducibility is not as much guaranteed than with pip At least on pypi you have the default libraries and you are guaranteed to have the original package, unpatched, and if the developer has provided "wheels", you might even download the precompiled binaries (see [numpy](https://pypi.python.org/pypi/numpy) packages). And you want to have a great control on the deployment of your master environment. Services such as heroku or similar does not provide a conda environment by default for the deployment of the application, if user wants it is can download it of course. **TL;DR**, executing the Spark master inside a conda environment is a BAD idea. Executing Driver and Executor inside a Conda environment makes totally sense on the other hand, and this is the proposal for this PR we are working on with Jeff: #14180. So, for this PR, for this PR we have several choices: - trust the user: he knows what he does. If we are inside a conda environment, we can install packages with `conda install`. I must admit this is not my favorite choice, since altering the conda environment might have unwanted behavior (if you dont use the right channel, you might replace some dependencies without the user acknowledge). Using pip from within a conda environment is not recommend at all - detect if we are inside a conda environment, leave it unconditionally inside at the script. Of course this only impact the `lint-python` script. I prefer this solution since we fallback into a nice controlled virtualenv for lint script I use quite a lot conda nowadays, playing with sckitlearn and alike, I must admit I am well impressed by the quality of the features and precompiled packages it provides, and I find it really useful for computation jobs, even for Spark job. For the output, I'll make better use of the `--debug` argument --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user sethah commented on the issue: https://github.com/apache/spark/pull/14963 I tested this on macosx, and got some pylint errors: + echo 'Checking Pylint...' Checking Pylint... + for to_be_checked in '"$PATHS_TO_CHECK"' + '[' false == true ']' + pylint --rcfile=/Users/sethhendrickson/Development/spark/python/pylintrc ./python/pyspark/ ./examples/src/main/python/ ./dev/sparktestsupport ./dev/run-tests.py ./python/run-tests.py ./dev/run-tests-jenkins.py + '[' 6 -ne 0 ']' + lint_status=1 + echo 'Pylint checks failed.' Pylint checks failed. + cat /Users/sethhendrickson/Development/spark/dev/pylint-report.txt * Module pyspark.daemon E:108,19: Value 'ex' is unsubscriptable (unsubscriptable-object) * Module pyspark.sql.streaming E: 23,20: Using variable 'long' before assignment (used-before-assignment) * Module pyspark.sql.tests W:1432, 8: Using deprecated method assertRaisesRegexp() (deprecated-method) W:1435, 8: Using deprecated method assertRaisesRegexp() (deprecated-method) ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 Another minor question is running the lint script now produces a lot of output about the package versions - it might make sense to silence this output except on failures (as we do with the actually linting its self). Although I can see the argument go the other way too (if the user is on a slow network connection the first time the run lint-python we need to be sure they understand its going to take awhile to download stuff). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 So looking at https://github.com/ContinuumIO/anaconda-issues/issues/305 it seems like we might cause problems for Conda users (and I've setup Conda locally to verify that this is indeed the case with the current version of Conda). This is sort of a Conda bug, but it might make sense to include the workaround for Conda users in the PR (I'll make a PR against your branch with a workaround for Conda users you can take a look at @Stibbons :)). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66819/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66819/consoleFull)** for PR 14963 at commit [`9e5d8d4`](https://github.com/apache/spark/commit/9e5d8d476118fc9145f19de88afe4b9f1b7f9575). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66819/consoleFull)** for PR 14963 at commit [`9e5d8d4`](https://github.com/apache/spark/commit/9e5d8d476118fc9145f19de88afe4b9f1b7f9575). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Thanks both of you for your reviews. It also fix the version of sphinx for documentation build during the `lint-python` My point of view is that for **application**, all versions of tools used to build should be frozen (in `requirements.txt`), to ensure full reproductibility. Upgrading any tool from version A to version B should be as much controlled as any other change in the source code, since it may have unwanted behavior. Pylint adds and remove checks from time to time and you don't want you code to stop building when pylint release a new version on Pypi, without a easy way to go back to a known state. I am ready to submit new PR with currently-ignored pylint checks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 This PR does somehting useful by its self namely it uses virtual env / `requirements.txt` to fix the versions of `pylint` and `pep8` and reenables `pylint` checking of the code. There is a setup cost associated with installing the virtual env the first time we run this, but only for developers who run `./dev/pylint`. Since this changes effectively adds back some linter rules its possible that mainline has changed since the last jenkins run and a merge to master performs poorly. Thinking about it some more, since we aren't under a rush for this, I'd like to double check this on Tuesday/Thursday on some of my coworkers machines that run OSX just to be safe because I know we've had some differences in the past with linter behaviour their (although this PR should remove those differences going forward its probably not a bad idea to double check). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14963 Just so I'm doubly clear, this PR is ready to go, does something useful by itself? that is it's not just like a change that only makes sense with others? I'm OK to merge it if the Python folks think it's all a thumbs up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 Also @Stibbons I'm sorry the process has been so challenging and thanks for hanging in with it :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 Right I think the virtualenv support for PySpark in general is interesting - but its a large enough change we would need to get some substantail reviewer time dedicated to it (and I don't see that happening with the current somewhat bare bones PySpark committer team) but this is a bit more of a clear win and since it only has Spark developer tool implications - not end user implications, it's a bit easier to do new things. How about we see if @davies has some time to look at this until Tuesday but if not - and if @srowen or one of the committers who has worked more on build related things feels comfortable - since its only a build change it might make sense to not be blocked too strongly on a Python specific reviewer. I'll take another look through the change tomorrow myself as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 sorry I wrote on the wrong pull request, I was talking about the virtualenv support for executor (#14180) :( This one is indeed only to reenable pylint. Merging this would allow me to submit new patch with minor code style corrections, and get rid of this weird manual pep8 package download. virtualenv and pip have been made to make life of python developers easy, make deployment reproductible and cheap to maintain, I just would like to instill this to the Spark project. There are many reason why playing with the PYTHONPATH is wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14963 Python really isn't my area, so I don't feel qualified to evaluate this. It sounds reasonable and has only build-time consequences, so I imagine it's low-risk. Improve style checking is good. I can merge this if there aren't objections. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 I would love to have a bit more feedback on this matter but it does not seem to interest core developers, sadly :( It's a bit disappointing, seeing how Python support on Spark is great, being able to deploy job as easy as with java (i.e., developers prepares the job + describes all the dependencies independently) would be so useful for Spark. For the moment, we have to ask our IT guys to install a given python module on each Executor when one needs a new one. We found a way using a nfs share but this is not convient. Automatic virtualenv creation with pip installing all dependencies described in a requirements.txt + support for wheels and python distribution package (sdist or bdist) is so useful and scalable. Each job can use different libraries and even the same library with different version, just like what can occur for java jars with the --packages arguments. I think of starting maintaining a fork of Spark with latest version of the source code, I would call it "Python Friendly Spark". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 I think this could be a good change for allowing more developers to onboard with PySpark - is there any interest in the current PySpark/Build focused committers [ @davies @srowen @rxin ] in seeing this change (or something similar)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66387/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66387/consoleFull)** for PR 14963 at commit [`0ab6ef5`](https://github.com/apache/spark/commit/0ab6ef580f41af25df36a6a2470d3abde21e78c9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66387/consoleFull)** for PR 14963 at commit [`0ab6ef5`](https://github.com/apache/spark/commit/0ab6ef580f41af25df36a6a2470d3abde21e78c9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66044/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66044/consoleFull)** for PR 14963 at commit [`3f74d2d`](https://github.com/apache/spark/commit/3f74d2d80e2ed221510d1c81397ab14c60d3b46c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 Maybe @davies, @srowen, or @yhuai will have some thoughts as people who have worked on python linting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66044/consoleFull)** for PR 14963 at commit [`3f74d2d`](https://github.com/apache/spark/commit/3f74d2d80e2ed221510d1c81397ab14c60d3b46c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66041/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66041/consoleFull)** for PR 14963 at commit [`3f74d2d`](https://github.com/apache/spark/commit/3f74d2d80e2ed221510d1c81397ab14c60d3b46c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #66041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66041/consoleFull)** for PR 14963 at commit [`3f74d2d`](https://github.com/apache/spark/commit/3f74d2d80e2ed221510d1c81397ab14c60d3b46c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Hello. Does anyone had time to review this patch? Is there anything I should do to help it being merged? This reenables Pylint checks using `lint-python` script and there are some errors that would need to be fixed thanks pylint. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65760/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #65760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65760/consoleFull)** for PR 14963 at commit [`ca2dcb2`](https://github.com/apache/spark/commit/ca2dcb2dc327f3f79570cc16dbf759df63eaae38). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #65760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65760/consoleFull)** for PR 14963 at commit [`ca2dcb2`](https://github.com/apache/spark/commit/ca2dcb2dc327f3f79570cc16dbf759df63eaae38). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Pull Request rebased --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 oh actually it seems like the PR dashboard updated its self (it lives at https://spark-prs.appspot.com/ and is sometimes used by reviewers to decide which PRs to look at). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 What is the PR dashboard ? I usually rebase this patch one or twice a week, I'll do it tomorrow --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 So even though this passes tests - its showing up in the PR dashboard as not for me. It might be easier to get reviewers to take a look if it was listed correctly, if you merge in the latest master the tests will re-run or we can explicitly ask jenkins to retest this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65481/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #65481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65481/consoleFull)** for PR 14963 at commit [`dff7b23`](https://github.com/apache/spark/commit/dff7b23eac8b7dd062bf09eb88cc963b1aedface). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14963 **[Test build #65481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65481/consoleFull)** for PR 14963 at commit [`dff7b23`](https://github.com/apache/spark/commit/dff7b23eac8b7dd062bf09eb88cc963b1aedface). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user Stibbons commented on the issue: https://github.com/apache/spark/pull/14963 Hello, sorry to bother you, but if this patch gets merged, I can work on the pylint errors and submit new PR I had to add in the ignore list of pylint. If I reenable most of them, here is an extract of errors I get: ``` * Module pyspark.shell W: 78, 8: Use of exec (exec-used) * Module pyspark.cloudpickle R:767,16: Redefinition of v type from property to staticmethod (redefined-variable-type) * Module pyspark.tests R:782,12: Redefinition of a type from pyspark.rdd.RDD to pyspark.rdd.PipelinedRDD (redefined-variable-type) R:783,12: Redefinition of b type from pyspark.rdd.RDD to pyspark.rdd.PipelinedRDD (redefined-variable-type) * Module pyspark.rdd R:1467,12: Redefinition of ser type from pyspark.serializers.AutoBatchedSerializer to pyspark.serializers.BatchedSerializer (redefined-variable-type) * Module pyspark.context R:130,12: Redefinition of self.serializer type from pyspark.serializers.AutoBatchedSerializer to pyspark.serializers.BatchedSerializer (redefined-variable-type) * Module pyspark.worker W:104,17: Use of eval (eval-used) * Module pyspark.shuffle C:732,32: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary) * Module pyspark.ml.tests R:1459, 8: Redefinition of a type from pyspark.ml.linalg.DenseVector to pyspark.ml.linalg.SparseVector (redefined-variable-type) * Module pyspark.ml.linalg E:683, 4: __len__ does not return non-negative integer (invalid-length-returned) R:1079, 8: Redefinition of colPtrs type from list to str (redefined-variable-type) R:1078, 8: Redefinition of rowIndices type from list to str (redefined-variable-type) * Module pyspark.streaming.kafka R:132,12: Redefinition of ser type from pyspark.serializers.PairDeserializer to pyspark.serializers.AutoBatchedSerializer (redefined-variable-type) * Module pyspark.streaming.tests R:419, 8: Redefinition of initial type from list to pyspark.rdd.RDD (redefined-variable-type) E:506,12: Bad first argument 'StreamingListener' given to super() (bad-super-call) * Module pyspark.streaming.context C:298,12: Consider using enumerate instead of iterating with range and len (consider-using-enumerate) ``` These are not big errors, but it would be quite easy to fix them and have pylint runing at each pullrequest. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14963 cc @davies ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14963 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
