[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 First and second failing CI profile hit [ZEPPELIN-1797](https://issues.apache.org/jira/browse/ZEPPELIN-1797) Spark 1.5 had another troubles \w `DepInterpreterTest` ``` 16/12/13 10:47:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. Results : Tests in error: DepInterpreterTest.testDefault:96 » NullPointer Tests run: 24, Failures: 0, Errors: 1, Skipped: 0 ``` And Selenium profiles also fails on test, related to `DepInterpreter` ``` ailed tests: SparkParagraphIT.testDep:234 First paragraph from SparkParagraphIT of testDep status: Expected: "FINISHED" but: was "ERROR" Tests in error: ZeppelinIT.testSparkInterpreterDependencyLoading:234->AbstractZeppelinIT.waitForParagraph:70->AbstractZeppelinIT.pollingWait:96 » Timeout Tests run: 18, Failures: 1, Errors: 1, Skipped: 0 ``` Though I belive none of these have to do anything with the changed introduced in this PR, so merging it to master if there is no further discussion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 Looks great to me, thank you @1ambda @mfelgamal Let's wait for CI results and merge to master, if nothing comes up and there is no further discussion! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @1ambda that sounds awesome job, thank you very much. Now the PR is merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal I agree with you opinion. We can directly use `usr/local/zeppelin` instead of `start-zeppelin.sh`. I'v just created PR for removing `start-zeppelin.sh` related files, codes, docs. https://github.com/mfelgamal/zeppelin/pull/3/files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @bzz @1ambda are there further discussion? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @bzz @1ambda the script `start-zeppelin.sh` isn't important, it just enforce using port `8080` and zeppelin home dir `/usr/local/zeppelin`, we can ignore it and use `zeppelin.sh`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal could you please explain, why do you think one more script is needed - `start-zeppelin.sh` and why `CMD["/bin/zeppelin.sh"]` can not be used as entrypoint for the image? @1ambda has the point, and we should try to reduce the number of shell scripts that need to be supported later on, as well as possible issues with setting up classpath, etc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal The reason, we can't run zeppelin binary on this docker image is because of **dumb-init**. It prevent from running zeppelin well by just installing **dumb-init**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal that sounds awesome job, thank you very much. Please let me test it tomorrow and get back to you here, but otherwise I think it's ready to be merged! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @bzz I removed `curl build-base make gcc g++` for decreasing the image size, now it's `301 MB`. And `R` is run well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1538 Sort of. There's Conda for R: https://www.continuum.io/content/preliminary-support-r-conda But generally some R package are compiled on installation; knitr is a relatively bigger one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal yes, exactly. Do you think this is possible? I wonder if the image size would go down, if we remove those guys after getting `knitr`. The goal would be to have a _minimal_ image that includes all necessary dependencies only to run Zeppelin. I'm not very familiar with R ecosystem, but isn't there some way of installing packages that comes with everything compiled, like [.whl](https://pypi.python.org/pypi/wheel) in [Python](http://pythonwheels.com/)? Then we could skip building layers of image \w gcc all together... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @bzz I think that we need to install `build-base make gcc g++` to build `knitr` package in R. And I wonder if we could remove them after building `knitr`. Is this right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 Now this issue is like more about building base image :) I will start working on a new JIRA issue as @bzz mentioned about creating runnable zeppelin images per version based on this. That would be the next step. https://issues.apache.org/jira/browse/ZEPPELIN-1711 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 to recap - theare are two final things, that would be very nice to have in this image: - find a way to reduce base image size from ~500mb i.e by installing R without `build-base make gcc g++` - re-use `./testing/install_external_dependencies.sh` as soon as #1632 is merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 > if we want to make dockerfile to the latest version from zeppelin which haven't a binary version This better be handled under a separate JIRA issue as this one is about `[ZEPPELIN-1386] Docker images for running Apache Zeppelin releases `. It can be done later i.e by setting up CI automation that runs nightly build \w something similar to `create_release.sh`. Important thing there would be to make sure that's clearly marked as non-release, but just as a developer's artefact. So all that sounds great, and image looks good (except for fat R dependencies, but I'm not sure if we can do something about it).. How do you guys think, is there anything that's left here? Or shall we merge this, run `docker build` for old binary releases using this image? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal @bzz I think we should dockerize binary zeppelin images first because more users use binary versions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 Hi @bzz So far, we have binary versions from 0.5.0 to 0.6.2, which help us building docker image to each version instead of building the source code, but I mean that if you want to make dockerfile to the latest version from zeppelin which haven't a binary version and is on master branch, so we may need to build the zeppelin in the docker. if the latest version isn't necessary now, we can ignore this, what do you think? * Yes, only `R` package requires `gcc` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user bzz commented on the issue: https://github.com/apache/zeppelin/pull/1538 Great job @mfelgamal ! The idea was to try to avoid building separate artefacts for Docker and use official convenience binaries from Apache release. Am I mimssing something, or what is the reason such images can not be published under https://hub.docker.com/r/apache/zeppelin automatically as a part of release? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @1ambda It's good if we did that in apache zeppelin repo. but in this case instead of using the created binary versions, we should build the source code inside the docker. Like that: ``` RUN git clone https://git-wip-us.apache.org/repos/asf/zeppelin.git "/usr/local/zeppelin" && \ cd /usr/local/zeppelin/ && \ git checkout v0.6.1 && mvn clean package -DskipTests ``` I think that alpine doesn't play well with node, so we could use ubuntu, what do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal Hi :) ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal First of all, I really appreciate this PR. This definitely helps users to install and start zeppelin in one-command. As you replied, you have automatically built docker images in your repository https://github.com/mfelgamal/zeppelin-dockers **But not this PR** So I am asking, what about adding `Dockerfile`s which can be automatically built into the https://github.com/apache/zeppelin repository like you did in your own. After that, we can create PR in https://github.com/docker-library/official-images for requesting automated build. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @astroshim @bzz @1ambda * Now I added a start-zeppelin.script to zeppelin-base image. It allow user to modify some configs by passing env variables. Also the user can share the logs and notebook folder in the container to the host machine by passing `-v $PWD/logs:path-in-docker/logs`. So you can share the spark folder to the container and assign `SPARK_HOME` in the container; according to what @khalidhuseynov have suggested. * And I think the previous issue needs more investigation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @1ambda I already use automated build of dockerhub, when push any new commit to my repo on github, the dockerhub builds the images recently. For point 2, It's good point and I will take a look in it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @1ambda Thank you for your review, The issue is solved, you can check again. And Thank you for the suggested enhancement for zeppelin.sh script, I think it will make the image better --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/1538 Additionally, It would be better to - run zeppelin immediately (using `zeppelin.sh` because we can run docker as daemon mode) - allow user to modify config important config files by passing env variables. You can refer this docker init script https://github.com/wurstmeister/kafka-docker/blob/master/start-kafka.sh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @astroshim I think it is just a message and the shell run normally. you can run with this command `docker run --rm -it -p 8080:8080 -p 8081:8081 mahmoudelgamal/zeppelin-release:0.5.0 -c bash`. Also there're an issue in current zeppelin binary versions form 0.6.0v to 0.6.2v with embedded spark. What you do think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @astroshim R and Python are installed and I think that the tutorials should be run without errors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user astroshim commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal Ping. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user AhyoungRyu commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal Awesome! There are some places to inform the page's existence to the users. Please check [documentation guide#adding-new-pages](http://zeppelin.apache.org/contribution/documentation.html#adding-new-pages) :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal first of all thanks for initiative on this one, and few points here: * as i can see here Zeppelin images will be with hardcoded `$ZEPPELIN_HOME/conf` folder and in order to configure it, you'll need to rebuild your image. As a possible solution here would be read `conf/` from certain place inside container and mount `conf` folder from your local drive to there. that may need little change in docker file and run command. * perhaps user can run with interactive (-it) as well as daemon (-d) modes, depending on preference. * also regarding running docker with, say, Spark - the easiest should be running it in local mode. if you want to point to your cluster, then there will be problems since at least port `7077` needs to be available. Note that these points may not need to be addressed right in this PR, it's more like things to consider and maybe future improvement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user astroshim commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal I checked point 1 and 3. Thank you fix properly. Could you also fix to run the `R Tutorial` and `Python Tutorial` too? I think all tutorial should be ran without error. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @astroshim Thank you for your reviews, Can you check the docker image again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user astroshim commented on the issue: https://github.com/apache/zeppelin/pull/1538 @mfelgamal Thank you for contributing this. I have several questions. 1. I think it needs `-it` option for running docker. so The command to run zeppelin docker should be `docker run --rm -it -p 8080:8080 -p 8081:8081 mahmoudelgamal/zeppelin-release:0.5.0 bash` 2. What do you think of adding documents for this like [this](http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/spark_cluster_mode.html#spark-standalone-mode) ? 3. I got an following error when I run the `Zeppelin Tutorial`, Could you take care a look this?  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1538: [ZEPPELIN-1386] Docker images for running Apache Zeppe...
Github user mfelgamal commented on the issue: https://github.com/apache/zeppelin/pull/1538 @bzz Thank you for your feedback. A MD documentation page is added to the install section of Zeppelin documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
