[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1632 I think it is mostly the test output that pollutes the CI logs... So, if I read TravisCI logs right - with 2 new profiles `Elapsed time 31 min`, and on `master` i.e [yesterday](https://travis-ci.org/apache/zeppelin/builds/179087724) `Elapsed time 31 min 14 sec` Looks great to me, merging to master if there is no further discussion! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 @felixcheung @bzz The final remaining travis error is unrelated, it seems to be occurring with other recent PRs (https://travis-ci.org/apache/zeppelin/builds/179341529). Perhaps we should focus on making the logs less verbose, since this profile already produces a log file that's bigger 3 MB on master. In any case, is this good to finally merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 @felixcheung so I looked at the travis log more carefully, and I am pretty sure the main thing I was missing was an additional "-Pscala-2.11" to the maven build profile. Hopefully it will finally be green this time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1632 looks like transient network error but python 3 test failed. could you rebase and kick this off again? https://api.travis-ci.org/jobs/178966601/log.txt?deansi=true --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 Oh for the love of... looks like I will need to update the cache again. Let me run this one more time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1632 I see, I think generally if we could isolate test sets so that they don't overlap (ie. python tests not running docker tests), we would likely have more stable CI runs. I think that's really our main concerns - CI stability is kinda high priority since it affects everyone. But if CI passes for these last 2 runs then I don't mind either way :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 Correct, the main reason being that caching it actually increases the build time. I still have one slight nitpick myself, but I'll leave that up to others if that is worth pursuing right now (the newly added docker and conda tests have increased the build time for the python profiles by a good bit, so perhaps they should be excluded somehow since testing them again here has no benefit) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1632 great! a few minor comments (like like you addressed one already!) CI passed the last time. I think this is looking good. So to recap, we are able to cache R but not miniconda, correct? Any other comment - merging when CI complete. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1632 Thanks for spending more time investigating this. I'd agree with your assessment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 To be honest, I don't see it being that big of a deal. Both profiles take about 5 minutes each which I think is fairly short, since up until recently we were spending 4 minutes just installing the R dependencies for all of the other profiles. Additionally, this would make it much easier to test everything for both python 2 and 3, which can be quite a pain to do manually with conda / virtualenv (eg, fiddling with the PATH before running tests for each case). Also, matplotlib and pandas are most likely used for a large percentage of python / pyspark use cases in Zeppelin, so I think having that integration tested would be really useful. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1632 Thank you for CI failure assessment @agoodm ! How do you think, how crucial are those tests? May be just documenting on how to run them manually could be an option? Generally, we tried to avoid exploding number of CI profiles to keep the CI build time minimum, but if more profiles is not going cost substantial build time that defility could be an option. In that case any of the suggested solutions sounds good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 @felixcheung @bzz So the pyspark cluster test has two failures. One is a result of matplotlib always printing a warning message when it is first loaded, and the other is how python 3 handles unicode. For these reason I think we should either move the python 3 tests to another profile which doesn't perform this test, or have it in a separate profile as I had already done once before. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 I think the CI failures were due to me somehow inadvertently removing the -Ppyspark from the build profile. Let's see how it goes now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1632 @agoodm as CI is failing - could you please rebase on latest master, it has some CI improvements and then open\close the PR to trigger a new CI here again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 @bzz done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1632 @agoodm thank you, looks great! There has been some changes in `.travis.yaml`, could you please rebase? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1632 That's great. Thanks for looking into it. I guess we will let Travis run this through a few times at different time of the day (where load varies) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 @felixcheung so I have tested caching the miniconda directory and found that it actually slows down the build by 1-2 minutes on average. From what I can tell the other big python projects are not fully caching their conda installations either, at the very least not the dependencies themselves. I think the only compelling reason to cache the miniconda installation at this point is to prevent the build from failing due to download errors. In my own experience this has been very rare until recently, and the errors are always happening after conda gets updated. This is making me think that the problems are most likely some bug in the latest version of conda, so I think I will test this some more with an older version of conda and see how stable the build is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 Thank you for the review @felixcheung , see my responses. Although this is beyond the scope of the original intent of the PR, I have added the R dependencies to the cache, which could potentially save several minutes of time for most of the build profile. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1632: [ZEPPELIN-1639] Add tests with external python depende...
Github user agoodm commented on the issue: https://github.com/apache/zeppelin/pull/1632 ping @bzz @felixcheung @Leemoonsoo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
