[GitHub] flink issue #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3494 Great, thanks @uce , I will take rebase an push the fix for FLINK-6003 (#3500) now. I will put you as a reviewer for this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3500: [FLINK-6003] Add docker image based on the openjdk...
Github user iemejia closed the pull request at: https://github.com/apache/flink/pull/3500 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3500: [FLINK-6003] Add docker image based on the openjdk one (d...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3500 Hey I probably should be more patient, I will wait until #3494 is merged and rebase from it. So closing for the moment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3500: [FLINK-6003] Add docker image based on the openjdk one (d...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3500 Sorry @greghogan to put you here as a reviewer but you are the only committer that I know who can take a look, if you think somebody else can do it please refer him. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/3494#discussion_r105171748
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory or a tarball on the local system
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
- mkdir -p $FLINK_INSTALL_PATH && \
- apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s
https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
--- End diff --
@patricklucas I'd prefer to prioritize the other JIRA since this is more
important (having the official image available), these cleanups we can do them
afterwards. And maybe try to go ahead with #3500.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/3494#discussion_r105171729
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory or a tarball on the local system
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
- mkdir -p $FLINK_INSTALL_PATH && \
- apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s
https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
--- End diff --
Hi, yes @jgrier the good thing is that we are all aligned on this, in
wanting flink developers and users to have the best experience possible. And
this is a discussion to follow in the other JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3500: [FLINK-6003] Add docker image based on the openjdk one (d...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3500 R: @greghogan cc: @jgrier @patricklucas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3500: [FLINK-6003] Add docker image based on the openjdk...
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/3500
[FLINK-6003] Add docker image based on the openjdk one (debian+alpine)
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink FLINK-6003-openjdk-docker-image
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3500.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3500
commit df8cb6723c53daed32b5d76d9790d5573e714d5c
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-08T10:47:24Z
[FLINK-6003] Change base docker image from java to openjdk
The java image is deprecated from the official docker images since
2016/12.
commit 84e22af005c343ec74e833e2c1abdc0eb1bb2b19
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-08T10:47:24Z
[FLINK-6003] Trim unneeded files from the official docker flink image
commit 50975f60e51e8a244ccdc86c076ca0fe4c9874d0
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-08T21:09:06Z
[FLINK-6003] Move Dockerfile into a specific alpine directory
commit 1457571fc8d30c1c0194798dfb5503803a47b10f
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-08T21:13:59Z
[FLINK-6003] Upgrade default version of the flink image to 1.2.0
commit f48882d2e3d0cac072e5cda2f2fec358a28540c5
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-08T22:39:39Z
[FLINK-6003] Refactor alpine Dockerfile
commit 32727cdef71e62d562486b546e9955b849ff8d25
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-08T22:39:59Z
[FLINK-6003] Clean docker-entrypoint after analysis with shellcheck
commit 7e40f28ea0937bbad9f10e84074931b36197537d
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-09T13:50:48Z
[FLINK-6003] Add dockerfile for pure openjdk (debian-based)
commit f4386a277529f23751d05cce2f99eb89a2319355
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-09T13:51:55Z
[FLINK-6003] Update docker README
commit 08d5ce4664a20d227368573a5a688bfef4de466e
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2017-03-09T13:55:48Z
[FLINK-6003] Remove quotes that broke host resolution on docker-compose
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3493: [FLINK-3026] Publish the flink docker container to...
Github user iemejia closed the pull request at: https://github.com/apache/flink/pull/3493 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3493: [FLINK-3026] Publish the flink docker container to the do...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3493 I am closing this to create a new one based on the new JIRA --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/3494#discussion_r104984654
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory or a tarball on the local system
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
- mkdir -p $FLINK_INSTALL_PATH && \
- apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s
https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
--- End diff --
I am not talking about a published image on docker hub, but the image that
exists in the flink repo, the whole reason I jumped in and worked in this image
was because I was aware that the community was around the image, and it will be
improved/updated, people can't be aware of the image existance/maintenance if
we don't show it somewhere.
Ok for the image on dataArtisans it makes sense but I have to be honest I
would prefer it to be still at Apache, at least I hope it preserves the license
(I hadn't seen you work there, nice).
And yes, when you propose an image, you have to show that you plan to
maintain it, or that you are part of the community.
https://github.com/docker-library/official-images#maintainership
This is important because they (docker) want images to get security updates
/ improvements.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3493: [FLINK-3026] Publish the flink docker container to the do...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3493 I oups not the ubuntu, the non-alpine one :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3493: [FLINK-3026] Publish the flink docker container to the do...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3493 I will, and add the commits for the ubuntu image, probably I will create a JIRA for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/3494#discussion_r104980420
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory or a tarball on the local system
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
- mkdir -p $FLINK_INSTALL_PATH && \
- apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s
https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
--- End diff --
Pefect, makes sense, but we should probably keep updating the existing
image for some time, because this is the way most people are aware that it
exists, or add a README that it was moved and maintained in the new repo.
Are you going to be the official maintainer (it is ok with me if you want
to take the lead there, I just want to learn the whole process of creating an
official image), but I can also help if you need with maintenance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/3494#discussion_r104919360
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory or a tarball on the local system
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
- mkdir -p $FLINK_INSTALL_PATH && \
- apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s
https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
--- End diff --
The easiest way to do this is to create a docker hub account for this and
try to build the image from your own fork of flink, that's what I did before to
test it:
https://hub.docker.com/r/iemejia/flink/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3494: [FLINK-5635] [docker] Improve Docker tooling
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/3494#discussion_r104907373
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory or a tarball on the local system
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
- mkdir -p $FLINK_INSTALL_PATH && \
- apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s
https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
--- End diff --
We will break compatibility with docker hub if we move this out. Docker hub
does not invoke a shell script but an specific Dockerfile hosted in a github
repo, so this should not be done, or at least we should validate that we can
build the image from docker hub.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3493: [FLINK-3026] Publish the flink docker container to the do...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3493 Yes, maybe I was not clear enough, my goal is to support the two variants of openjdk that you mention. I was just mentioning ubuntu/debian because this is the base of the non-alpine base container for openjdk. https://github.com/docker-library/openjdk/blob/445f8b8d18d7c61e2ae7fda76d8883b5d51ae0a5/8-jre So we will need some small tweaks and an extra Dockerfile to support both. For other images that support both openjdk base and the alpine version. https://hub.docker.com/_/solr/ https://hub.docker.com/_/jenkins/ https://hub.docker.com/_/tomcat/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3493: [FLINK-3026] Publish the flink docker container to the do...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3493 Cool ! Just ping me when you are done and I will rebase to finish the clean ups, I am going to start contacting the docker people to see what else do we need and continue to work in the ubuntu version meanwhile (I think having both is worth even if it was me who slimmed the thing into alpine). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3493: [FLINK-3026] Publish the flink docker container to the do...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3493 R: @greghogan cc: @jgrier @patricklucas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3204: [FLINK-5634] Flink should not always redirect stdout to a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3204 @greghogan Definitely a misunderstanding, I had the impression you just didn't like my solution, so I didn't continue pushing for it (even if we finished with similar ideas). For ref, Greg's PR, so someone (a committer) reviews it https://github.com/apache/flink/pull/3351 I will move the docker scripts to use start-foreground once it is merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 @jagatfx Yes, this is still in my TODO list, I have not had a lot of time to follow the process to make an image official, but I hope I will in the next days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 And remember that my ultimate goal is not to host that image but that we can convert it into an official one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 Of course, who do you think I am :P You can check a ready to run build here in case you don't want to test that one. But it is true that we need some kind of automatized tests for the image. https://hub.docker.com/r/iemejia/flink/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 @mxm this should be ready now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2340#discussion_r74261518
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
RUN apk add --no-cache bash snappy
# Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
ARG HADOOP_VERSION=27
ARG SCALA_VERSION=2.11
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
# Install build dependencies and flink
RUN set -x && \
+ mkdir -p /opt && \
apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
- awk '/preferred/ {gsub(/"/,""); print
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
- tar xvz -C /usr/local/ && \
- ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
- sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g"
/usr/local/flink/bin/flink-daemon.sh && \
+ curl -s
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
| \
--- End diff --
Notice that I have another argument against the mirrors and it is the
security issue, imagine that a malign mirror puts a wrong version of flink, I
wonder what is the recomended approach to validate this case (of course notice
that I asume that hacking apache would be harder).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 We should maybe wait for 1.1.1 to be up before doing this merge, I will add the commit for this and probably rebase everything once it is ready. Is it ok ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2340#discussion_r74259291
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
RUN apk add --no-cache bash snappy
# Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
ARG HADOOP_VERSION=27
ARG SCALA_VERSION=2.11
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
# Install build dependencies and flink
RUN set -x && \
+ mkdir -p /opt && \
apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
- awk '/preferred/ {gsub(/"/,""); print
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
- tar xvz -C /usr/local/ && \
- ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
- sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g"
/usr/local/flink/bin/flink-daemon.sh && \
+ curl -s
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
| \
+ tar xvz -C /opt/ && \
--- End diff --
Ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2340#discussion_r74243197
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -22,25 +22,30 @@ FROM java:8-jre-alpine
RUN apk add --no-cache bash snappy
# Configure Flink version
-ARG FLINK_VERSION=1.0.3
+ARG FLINK_VERSION=1.1.0
ARG HADOOP_VERSION=27
ARG SCALA_VERSION=2.11
+# Flink environment variables
+ENV FLINK_HOME /opt/flink
+ENV PATH $PATH:$FLINK_HOME/bin
+
# Install build dependencies and flink
RUN set -x && \
+ mkdir -p /opt && \
apk --update add --virtual build-dependencies curl && \
- curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?as_json\=1 | \
- awk '/preferred/ {gsub(/"/,""); print
$2}')flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz
| \
- tar xvz -C /usr/local/ && \
- ln -s /usr/local/flink-$FLINK_VERSION /usr/local/flink && \
- sed -i -e "s/echo \$mypid >> \$pid/echo \$mypid >> \$pid \&\& wait/g"
/usr/local/flink/bin/flink-daemon.sh && \
+ curl -s
https://archive.apache.org/dist/flink/flink-$FLINK_VERSION/flink-$FLINK_VERSION-bin-hadoop$HADOOP_VERSION-scala_$SCALA_VERSION.tgz
| \
--- End diff --
I agree, I just changed it because I find the older code so ugly (and
fragile) for what it does, also my other argument to do this was the fact that
people will probably won't build the image as much as they will pull it from
the official hub.docker image (once we publish this image as the official one).
I can let it like it was if this is better for you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 I added two additional changes, an extra security fix (default user is not root now) and I changed the path to the more appropriate /opt. Now this should be ready for review/merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 (notice than when I said need to improve I refer at the missing automation to change the default release version on the Dockerfile). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2340: [FLINK-3155] Update docker flink container to the latest ...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2340 I referred to the older FLINK issue since this is something we need to still improve. I can try to fix it in a subsequent PR, do you have any hints of how can we achieve this ? R: @aljoscha or @mxm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2340: [FLINK-3155] Update docker flink container to the ...
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/2340
[FLINK-3155] Update docker flink container to the latest release
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink new
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2340.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2340
commit 299a7536db6cb51dd9920989caaaf527a4cb4feb
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-08-07T10:57:48Z
[FLINK-3155] Update docker flink container to the latest release
I also simplified some stuff from the basic script
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 Ok, I will check how is the procedure and I will tell you if we need something else. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 About the flink image version, we can support multiple versions via tags, those are pulled with the: separator, e.g. docker pull flink:1.0.3-hadoop26-scala_2.10 And we can agree on a default (e.g. most recent versions which would be -> :latest). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 Great, closing now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2240: [FLINK-4209] Fix issue on docker with multiple NIC...
Github user iemejia closed the pull request at: https://github.com/apache/flink/pull/2240 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 One last thing, are you interested on getting the flink image into becoming an official docker image ? I took a look around it and it seems we can ask for this. The benefit is that users can just make 'docker pull flink' and get the flink image from the hub.docker.com repository ready to use (e.g. I am publishing for the moment my own 'unofficial' builds at https://hub.docker.com/r/iemejia/flink/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 Yes, it is done, I am really happy of getting rid of python/supervisord in the image now. Please merge if you consider it ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2239: [FLINK-4208] Support Running Flink processes in fo...
Github user iemejia closed the pull request at: https://github.com/apache/flink/pull/2239 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Fix issue on docker with multiple NICs and r...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 I added this last commit to remove the dependency on supervisord, thanks to @greghogan for the 'wait' idea. Now flink has the thinnest docker image possible :). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 Thanks Greg, I was probably too tired last night because I put the wait in a weird place, I just tried now and everything is working, it is still not 'real' foreground, since Ctrl-C gets captured by the wait, but it fixes the issue for the docker image so I am going to close this pull request and the issue. If you or anybody else is still interested I will keep it, but I am going to close it if you don't mind. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2239
I tried not to change the current daemon behavior, that's the reason why I
took
the decision to add an additional option. I am not sure if using wait may
work for what I want but if it does, perfect, can you give me hints of how to
test this ? I naively did this and it does not seem to work.
```
if [[ ${mypid} =~ ${IS_NUMBER} ]] && kill -0 $mypid > /dev/null
2>&1 ; then
echo $mypid >> $pid
+ wait $mypid # I also tried with $pid and it does not work either
else
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2239: [FLINK-4208] Support Running Flink processes in fo...
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2239#discussion_r70873347
--- Diff: flink-dist/src/main/flink-bin/bin/flink-daemon.sh ---
@@ -77,31 +77,36 @@ if [[ ${JAVA_VERSION} =~ ${IS_NUMBER} ]]; then
fi
case $STARTSTOP in
+(start|start-foreground)
+ # Rotate log files
+ rotateLogFile $log
+ rotateLogFile $out
+
+ # Print a warning if daemons are already running on host
+ if [ -f $pid ]; then
+active=()
+while IFS='' read -r p || [[ -n "$p" ]]; do
+ kill -0 $p >/dev/null 2>&1
+ if [ $? -eq 0 ]; then
+active+=($p)
+ fi
+done < "${pid}"
-(start)
-# Rotate log files
-rotateLogFile $log
-rotateLogFile $out
-
-# Print a warning if daemons are already running on host
-if [ -f $pid ]; then
- active=()
- while IFS='' read -r p || [[ -n "$p" ]]; do
-kill -0 $p >/dev/null 2>&1
-if [ $? -eq 0 ]; then
- active+=($p)
-fi
- done < "${pid}"
-
- count="${#active[@]}"
+count="${#active[@]}"
- if [ ${count} -gt 0 ]; then
-echo "[INFO] $count instance(s) of $DAEMON are already running
on $HOSTNAME."
- fi
+if [ ${count} -gt 0 ]; then
+ echo "[INFO] $count instance(s) of $DAEMON are already running
on $HOSTNAME."
fi
+ fi
+
+ if [[ $STARTSTOP == "start-foreground" ]]; then
+echo "Starting $DAEMON as a foreground process on host $HOSTNAME."
+$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}"
-classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null
+ fi
+ if [[ $STARTSTOP == "start" ]]; then
echo "Starting $DAEMON daemon on host $HOSTNAME."
-$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}"
-classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
+nohup $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS}
"${log_setting[@]}" -classpath "`manglePathList
"$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN}
"${ARGS[@]}" > "$out" 2>&1 < /dev/null &
--- End diff --
I added this because I was trying to grasp what makes a daemon a daemon and
I found a reference that convinced me that nohup was missing:
https://stackoverflow.com/questions/3430330/best-way-to-make-a-shell-script-daemon
Additionally when I looked for inspiration for my changes (the
start-foreground name), I look
at how they started the daemon in zookeeper and I noticed they use nohup
too.
https://github.com/apache/zookeeper/blob/trunk/bin/zkServer.sh#L219
This is an extra thing and not the core of the Pull Request, if you don't
agree
I can rebase and remove that commit, but I think it is worth the addition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 Hi, Yes, this is exactly the situation, in a previous pull request I was optimizing the flink docker image, however I found that the image used supervisord to catch and keep alive those daemons, so I wanted to remove this dependency (because it adds around 40MB to the image + python and some extra stuff). Can you give me some hints on the best way to address this ? Or how can I improve my current approach (notice that I took the start-foreground idea from zookeeper). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 I hope this gets into 1.1.0 so I can push the changes to the docker image once this is available in the official binary distribution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2239 @aljoscha As discussed in FLINK-4118 I am doing this to support this use case and eventually remove the dependency on python and supervisord from the docker image. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2240: [FLINK-4209] Docker image breaks with multiple NICs
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2240 I added some additional fixes for example to exclude build dependencies from the final image, and to make the docker-compose script simpler. @aljoscha This is a set of small fixes that I missed at FLINK-4118. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2240: [FLINK-4209] Docker image breaks with multiple NIC...
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/2240
[FLINK-4209] Docker image breaks with multiple NICs
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink FLINK-4209
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2240.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2240
commit 1af7c871a1aa9d446436df96f4b1e2d9115aee34
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-07-05T02:16:07Z
[FLINK-4209] Change hostname resolution from IP to name
This solves issues when a host has multiple IPs
commit 385cb7f2ec2d629912b2a7af03f1ba1c76f49483
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-07-05T02:17:09Z
[FLINK-4209] Separate build dependencies in the docker image and remove
them once it is ready
commit 88102e54fde4c967831ae1cc3e4025543e5fed57
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-07-05T07:23:46Z
[FLINK-4209] Add debug information of the build steps
commit b365fbd47db025dcf9d8d6d797ed7bdd85093b3f
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-07-11T15:23:17Z
[FLINK-4209] Simplify docker-compose script (volumes are now local)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2239: [FLINK-4208] Support Running Flink processes in fo...
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/2239
[FLINK-4208] Support Running Flink processes in foreground mode
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink FLINK-4208
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2239.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2239
commit af9ac6cdb3f6601d6248abe82df4fd44de4453e5
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-07-13T13:55:13Z
[FLINK-4208] Add support to run flink as a foreground process
commit 5a27c3ce0bfaf7cf862aba3929f7712a3897bc19
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-07-13T13:56:13Z
[FLINK-4208] Make the flink daemon a real daemon
nohup ignores the HUP terminal signals, so the process is still alive
even after the terminal ends
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Great, thanks for your review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 This should be ok now. In further commits I expect to fix the daemon thing + maybe add a HA version using zookeeper of the docker-compose file. One more question aljoscha, I intend to add the Beam Flink runner and contribute a similar version into Beam, however I don't know what is the best approach for this, I just tried naively to put the jars in $FLINK_HOME/lib but it didn't work, any ideas ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69306327
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your
version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L
https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname
-s`-`uname -m` > /usr/local/bin/docker-compose
+sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive
when running containers.
+docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You
can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of
flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+docker build --build-arg FLINK_VERSION=1.0.3 --build-arg
HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t
"flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen
session)
-`docker-compose up`
+docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+docker-compose scale taskmanager=
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+docker exec -it $(docker ps --filter name=flink_jobmanager
--format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 root@localhost:/`
+for i in $(docker ps --filter name=flink --format={{.ID}}); do
+docker cp $i:/
+done
- Run a topology
-`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c
`
+docker run -it --rm flink:latest flink run -m -c
--- End diff --
Yes, you are right, I hesitated about this, if you noticed, I added a third
option apart of jobmanager/taskmanager to let the image open ended, my goal was
in part that people could start an container with flink as a client, or for any
other use. However this can be confusing (because people mostly expect to send
ther local jar, but I will change it as you suggest in the README, and advanced
users can do as they wish.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69306386
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your
version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L
https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname
-s`-`uname -m` > /usr/local/bin/docker-compose
+sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive
when running containers.
+docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You
can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of
flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+docker build --build-arg FLINK_VERSION=1.0.3 --build-arg
HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t
"flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen
session)
-`docker-compose up`
+docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+docker-compose scale taskmanager=
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+docker exec -it $(docker ps --filter name=flink_jobmanager
--format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 root@localhost:/`
+for i in $(docker ps --filter name=flink --format={{.ID}}); do
--- End diff --
Nice I didn't know that flink took care of this, fix in mins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Hi, I tested it with the basic word count and with the beam pipeline
example that @ecesena put for his flink/beam demo.
I don't know if your case is maybe because you are running on docker for
mac, but check two things.
1. docker assigns an internal IP to each running container (the one
corresponding to the jobmanager is the one to put in the -m argument).
2. The path you pass in the examples must be accesible from the container,
e.g. you must copy the file in all the nodes (as I mention in the README), or
mount them with a volume (-v on docker).
e.g.
```
$FLINK_HOME/bin/flink run -m 172.18.0.2:6123 examples/batch/WordCount.jar
--input file:///tmp/kinglear.txt
```
The kinglear file must be in all the nodes. I copy those like this:
```
for i in $(docker ps --filter name=flink --format={{.ID}}); do
docker cp ~/Desktop/flink-beam/kinglear.txt $i:/tmp/
done
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Awasome, thanks @aljoscha, let's merge ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/2176#discussion_r69150030 --- Diff: flink-contrib/docker-flink/README.md --- @@ -1,80 +1,75 @@ -#Apache Flink cluster deployment on Docker using Docker-Compose +Apache Flink cluster deployment on docker using docker-compose -##Installation -###Install Docker +# Installation +Install the most recent stable version of docker https://docs.docker.com/installation/ -if you have issues with Docker-Compose versions incompatible with your version of Docker try +Install the most recent stable version of docker-compose +https://docs.docker.com/compose/install/ -`curl -sSL https://get.docker.com/ubuntu/ | sudo sh` +# Build -###Install Docker-Compose +Images are based on the official Java Alpine (OpenJDK 8) image and run +supervisord to stay alive when running containers. If you want to build the +flink image run: -``` -curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose +sh build.sh -chmod +x /usr/local/bin/docker-compose -``` - -###Get the repo - -###Build the images +or -Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers. +docker build -t "flink" flink --- End diff -- Nice catch, just fixed :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Nice, I just fixed as you suggested. I have three questions: 1. This container is based on the Java JRE (to keep it small), Does Flink in any part do some magic that requires a full JDK (like live recompiles) ? If no, I think this is almost perfect now. 2. Are you aware of any other flink dependency that uses any native OS library (I ask this since this because if it is the case it must be added to the container, I did this for snappy because I found the issue while testing a Beam pipeline, but I don't know if there are others). 3. In the docker image I left supervisor because I didn't find an easy way to start flink in normal mode, the scripts to start both taskmanager and jobmanager go into daemon mode immediately, is there something that can be done to change this (this will reduce the image in 40 more MB), but well I can work on that for a different PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Sorry I had to rebase my previous PR but this is the definitive one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: Flink 4118
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 The docker images script was simplified and the image size was reduced. Previous image: flink latest 6475add651c7 24 minutes ago 711.6 MB Image after FLINK-4118 flink latest 555e60f24c10 20 seconds ago 252.5 MB --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2176: Flink 4118
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/2176
Flink 4118
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink FLINK-4118
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2176
commit 7f7f24a1c31aa3e9b011fb492280389ac500fff4
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-23T16:39:50Z
[FLINK-4118] Update docker image to 1.0.3 and remove unneeded deps
Some of the changes include:
- Remove unneeded dependencies (nano, wget)
- Remove apt lists to reduce image size
- Reduce number of layers on the docker image (best docker practice)
- Remove useless variables and base the code in generic ones e.g.
FLINK_HOME
- Change the default JDK from oracle to openjdk-8-jre-headless, based on
two reasons:
1. You cannot legally repackage the oracle jdk in docker images
2. The open-jdk headless is more appropriate for a server image (no GUI
stuff)
- Return port assignation to the standard FLINK one:
Variable: docker-flink -> flink
taskmanager.rpc.port: 6121 -> 6122
taskmanager.data.port: 6122 -> 6121
jobmanager.web.port: 8080 -> 8081
commit 04c18edbbb6b109c1b23c28c17f82bde080b8686
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-24T14:52:22Z
[FLINK-4118] Base the image on the official java alpine and remove ssh
commit eff306e80ffe605f6f94d2b5764520fe6462b1f7
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-27T11:23:10Z
[FLINK-4118] Remove unused configuration files and fix README
commit e386b5e99c086e3279454a3e574edc7a02c54cdb
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-27T15:01:13Z
[FLINK-4118] Update compose to v2 + add new entrypoint for direct execution
commit 34f18849cd610199805bc396b2ac850242f5ab6d
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-27T16:52:48Z
[FLINK-4118] Change default entrypoint so users can run flink as a client
too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2164: [FLINK-4118] The docker-flink image is outdated (1...
Github user iemejia closed the pull request at: https://github.com/apache/flink/pull/2164 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2164: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2164 The docker images script was simplified and the image size was reduced. Previous image: flink latest 6475add651c724 minutes ago 711.6 MB Image after FLINK-4118 flink latest 555e60f24c1020 seconds ago 252.5 MB --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2164: [FLINK-4118] The docker-flink image is outdated (1...
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/2164
[FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed
down
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink FLINK-4118
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2164.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2164
commit 15bf574bfec2b12ee50f4a21f1b1a43421f632e2
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-23T16:39:50Z
[FLINK-4118] Update docker image to 1.0.3 and remove unneeded deps
Some of the changes include:
- Remove unneeded dependencies (nano, wget)
- Remove apt lists to reduce image size
- Reduce number of layers on the docker image (best docker practice)
- Remove useless variables and base the code in generic ones e.g.
FLINK_HOME
- Change the default JDK from oracle to openjdk-8-jre-headless, based on
two reasons:
1. You cannot legally repackage the oracle jdk in docker images
2. The open-jdk headless is more appropriate for a server image (no GUI
stuff)
- Return port assignation to the standard FLINK one:
Variable: docker-flink -> flink
taskmanager.rpc.port: 6121 -> 6122
taskmanager.data.port: 6122 -> 6121
jobmanager.web.port: 8080 -> 8081
commit 523e8b0ece28537635792cc9514d4e9bbaa2c53c
Author: Ismaël MejÃa <ieme...@gmail.com>
Date: 2016-06-24T14:52:22Z
[FLINK-4118] Base the image on the official java alpine and remove ssh
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
