[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Only the last CI profile fails on very strange curcumstances: ``` 16/12/19 04:54:08 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1482123134765_0003 16/12/19 04:54:08 ERROR ApplicationMaster: Failed to cleanup staging dir .sparkStaging/application_1482123134765_0003 java.net.ConnectException: Call From testing-docker-7a57c244-e7fb-44fb-9664-5856f0534506/172.17.0.5 to localhost:39393 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 Hi all, This PR looks like unpopular from the point of view design. I will close this issue. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz Thanks for the quick reply. 1. I will post CI logs :) 2. I'v just created an issue for the livy intergration test failure: [ZEPPELIN-1806](https://issues.apache.org/jira/browse/ZEPPELIN-1806) ([tried to search](https://issues.apache.org/jira/issues/?jql=project%20%3D%20%22zeppelin%22%20AND%20text%20~%20%22testSparkSQLInterpreter%22), but couldn't find similar issues) 3. We can reduce build time later i think ð --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Thanks for explanation, makes sense to me On CI failures - from my experience, if the reason of failure is posted here - there speeds up the merge dramatically. I have update [ZEPPELIN-1797](https://issues.apache.org/jira/browse/ZEPPELIN-1797) \w Scala 2.10 profile failure on DependencyInterpreter, that is a first failng profile here. And the last one loooks like something new, possibly [related to Livy](https://api.travis-ci.org/jobs/183791719/log.txt?deansi=true), do we have a JIRA issue for that? Other that that - current master build time is `Elapsed time 32 min 6 sec`, this branch `Elapsed time 34 min 4 sec`... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz Sorry for confusing. Let me explain more details. - without `-Ploose-checkstyle`, maven will print all WARNINGs. So people will see these warnings while developing (**default**) - with `-Ploose-checkstyle`, maven will **not** print warnings at all. This prevent 4MB log size limitation in travis CI - Initially, `CHECKSTYLE=1` profile was introduced to get people awared about warnings but it causes build failure since we have verbose zeppelin log (too many info log. it's not good). So I removed that profile. Additionally, `CHECKSTYLE=1` profile is meaningless because other profiles do checkstyle also currently. (FYI, splitting checkstyle doesn't reduce total build time much. approx 5 secs as i know) So I would like to add `CHECKSTYLE=1` profile after regulate some logging levels, fixing checkstyle rules (WARN -> ERROR) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Iâm a bit confused though - this PR seems not to have separate checkstyle CI profile, right? Didnât we want to have `CHECKSTLYLE=1 â¦` profile for all projects, and then skip checkstyle (I assume that is what `loose checkstyle` option is for) on all other profiles? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 I'm sorry for a confusion - as soon as this issue is not about CI, but about `Update checkstyle rule file, plugin`. Let's move the CI discussion to the dev@ mailing list so all people in the community will be included, I have created [ZEPPELIN-1797](https://issues.apache.org/jira/browse/ZEPPELIN-1797) Merging this to master as CI failure is not relevant to the changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 \cc @zjffdu regarding to the `spark-csv` dependency issue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue:
https://github.com/apache/zeppelin/pull/1689
Interesting, that is the first time I see it! AFAIK `spark-csv` has been
included in Spark since 2.0
The failing profile is Spark 1.6 with **Scala 2.11**, but it fails on
fetching `spark-csv_2.10` which seems to me as almost the right thing to happen
(to avoid runtime errors)...
This artefact seems to be fine, published on central and sonatype
https://mvnrepository.com/artifact/com.databricks/spark-csv_2.10/1.3.0
May be we need to update `./dev/change_scala_version.sh` to take care of
the version literal here as well and see if that helps?
Another way is to let maven `resource-plugin` replace it with actual
version on build time, and use smth like `${scala.version}` in the Java code.
Default property value can be set to 2.10 and override it though SCALA_VERSION
env var, but this is not very IDE-friendly as they tend to do custom builds for
you, skipping lifecycle definition from time to time, but the default value in
the code like `${scala.version}` would not be very useful. So resource-plugin
way actually seems to me to complicated and we usually try to avoid it.
What do you think, @1ambda ?
Or another, simplest, way could be - create a separate JIRA issue for Scala
version mismatch in tests(if that is the case here), move the
feedback\discussion there and handle it in subsequent PRs. But just re-trigger
CI here, to see if it is reproducible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz A test failed but it is irrelevant with this PR ``` Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.156 sec <<< FAILURE! - in org.apache.zeppelin.dep.DependencyResolverTest testLoad(org.apache.zeppelin.dep.DependencyResolverTest) Time elapsed: 1.142 sec <<< ERROR! org.sonatype.aether.RepositoryException: Cannot fetch dependencies for com.databricks:spark-csv_2.10:1.3.0 at org.apache.zeppelin.dep.DependencyResolver.getArtifactsWithDep(DependencyResolver.java:181) at org.apache.zeppelin.dep.DependencyResolver.loadFromMvn(DependencyResolver.java:131) at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:79) at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:96) at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:88) at org.apache.zeppelin.dep.DependencyResolverTest.testLoad(DependencyResolverTest.java:77) Results : Tests in error: DependencyResolverTest.testLoad:77 » Repository Cannot fetch dependencies for ... ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz Let me know if there is anything to do before merging this :) thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz CI failed but it's due to Flaky pyspark test. I think this can be merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 I see what you mean, for profile not included in build roster by default we would need to update the CI profile. Well, it's 12s * 10 (number of profile we have no), which looks like 2min time, but we can always optimize that later. Could you please elaborate on CI failures here? In my experience, that helps to merge things faster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz Recently, i found that checkstyle doesn't need too much time. It takes apprx 12 seconds in my labtop. So i believe it is such a big problem of executing checkstyle plugin in all profiles. ``` $ # if you fail due to dependency, try before `mvn install -DskipTest` $ mvn validate -Denforcer.skip=true ``` ``` [INFO] Building Zeppelin: Packaging distribution 0.7.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle-fail-build) @ zeppelin-distribution --- [INFO] [INFO] --- maven-resources-plugin:2.7:copy-resources (copy-resources) @ zeppelin-distribution --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 17 resources [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce) @ zeppelin-distribution --- [INFO] Skipping Rule Enforcement. [INFO] [INFO] Reactor Summary: [INFO] [INFO] Zeppelin ... SUCCESS [ 1.392 s] [INFO] Zeppelin: Interpreter .. SUCCESS [ 1.347 s] [INFO] Zeppelin: Zengine .. SUCCESS [ 0.859 s] [INFO] Zeppelin: Display system apis .. SUCCESS [ 0.200 s] [INFO] Zeppelin: Spark dependencies ... SUCCESS [ 0.648 s] [INFO] Zeppelin: Spark SUCCESS [ 0.415 s] [INFO] Zeppelin: Markdown interpreter . SUCCESS [ 0.213 s] [INFO] Zeppelin: Angular interpreter .. SUCCESS [ 0.182 s] [INFO] Zeppelin: Shell interpreter SUCCESS [ 0.183 s] [INFO] Zeppelin: Livy interpreter . SUCCESS [ 0.574 s] [INFO] Zeppelin: HBase interpreter SUCCESS [ 0.308 s] [INFO] Zeppelin: Apache Pig Interpreter ... SUCCESS [ 0.231 s] [INFO] Zeppelin: PostgreSQL interpreter ... SUCCESS [ 0.337 s] [INFO] Zeppelin: JDBC interpreter . SUCCESS [ 0.259 s] [INFO] Zeppelin: File System Interpreters . SUCCESS [ 0.232 s] [INFO] Zeppelin: Flink SUCCESS [ 0.317 s] [INFO] Zeppelin: Apache Ignite interpreter SUCCESS [ 0.394 s] [INFO] Zeppelin: Kylin interpreter SUCCESS [ 0.206 s] [INFO] Zeppelin: Python interpreter ... SUCCESS [ 0.289 s] [INFO] Zeppelin: Lens interpreter . SUCCESS [ 0.359 s] [INFO] Zeppelin: Apache Cassandra interpreter . SUCCESS [ 0.300 s] [INFO] Zeppelin: Elasticsearch interpreter SUCCESS [ 0.273 s] [INFO] Zeppelin: BigQuery interpreter . SUCCESS [ 0.273 s] [INFO] Zeppelin: Alluxio interpreter .. SUCCESS [ 0.271 s] [INFO] Zeppelin: Scio . SUCCESS [ 0.379 s] [INFO] Zeppelin: web Application .. SUCCESS [ 0.176 s] [INFO] Zeppelin: Server ... SUCCESS [ 0.695 s] [INFO] Zeppelin: Packaging distribution ... SUCCESS [ 0.204 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 11.878 s [INFO] Finished at: 2016-12-02T00:28:12+09:00 [INFO] Final Memory: 69M/859M [INFO] ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Sounds great! What is rationale for having checkstyle running everywhere + a separate profile? Would it make sense to measure the CI build time with\without separate profile and see if there is a net gain in time, having it separately (and skipping it everywhere else)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz https://github.com/apache/zeppelin/pull/1689/commits/56dfa5232ef3cc58c147f8512cd37d4d3d6e8c4f This commit added the `CHECKSTYLE=1` profile which printing all WARN. And rest of profiles remain same as before (DO NOT SKIP checkstyle) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 With number of PRs we have I'm a bit lost, which one will introduce a separate checkstyle CI profile and skip it in all the rest? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz I'v just rebased I will add some commits enumerated above and then mention you when ready. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Go it, that sounds great. Then shall we finish and merge this guy first, and then get back to #1668 merge it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 @bzz I will work on here and add some small details from #1696 later (e.g adding profile for checkstyle not to skip checkstyle in all other profiles) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 @1ambda I have created a HOTFIX for CI Spark download in #1709 so after merging it, let's rebase and see if that fixed the issue. I'm a bit confused, do you want to continue working here or in #1696 though? Or they can be rebased one after another --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Here are more detailed logs ``` [WARNING] Retrying (4 more) Downloading: http://archive.apache.org/dist/spark/spark-2.0.1/spark-2.0.1.tgz org.apache.maven.wagon.TransferFailedException: Failed to transfer file: http://archive.apache.org/dist/spark/spark-2.0.1/spark-2.0.1.tgz. Return code is: 503 , ReasonPhrase:Service Unavailable. at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:1039) at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:977) at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116) at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88) at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61) at com.googlecode.WGet.doGet(WGet.java:293) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Prev plugin version seems to have same behavior ``` [ERROR] Failed to execute goal com.googlecode.maven-download-plugin:download-maven-plugin:1.2.1:wget (download-pyspark-files) on project zeppelin-spark-dependencies_2.10: IO Error: Could not get content -> [Help 1] ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Looks great to me, thank you @1ambda ! May be, as soon as CI is green let's try to squash some commits, before merging it in - 37 may be a bit too much to be useful in merge commit message summary. Going to merge to master, if there is no further discussion, right after that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1689 Thank you @1ambda ! I think it's a great idea to have checkstyle off the most of profiles, but add a new one with just checkstyle. After CI finishes, let's post the CI time statistics in PR description, in order to verify the hypothesis that this change actually does speed up the CI build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 Failed due to log size limitation of travis CI ``` The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over). ``` We might disable `true` only in CI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1689: [ZEPPELIN-1709] Update checkstyle rule file, plugin
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1689 \cc @bzz for review thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
