[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Great, thanks for your review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2176 That's great to hear! I'll write something on the Beam ML thread. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 This should be ok now. In further commits I expect to fix the daemon thing + maybe add a HA version using zookeeper of the docker-compose file. One more question aljoscha, I intend to add the Beam Flink runner and contribute a similar version into Beam, however I don't know what is the best approach for this, I just tried naively to put the jars in $FLINK_HOME/lib but it didn't work, any ideas ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2176 I had two more comments about the README but after that it should be good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2176 You were right, I did exactly the same thing I did on OS X on a new Ubuntu 16.04 installation and it worked. ð --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2176 Yes, this is exactly what I was trying on OS X. I'm quickly setting up a ubuntu VM to see if it works there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Hi, I tested it with the basic word count and with the beam pipeline
example that @ecesena put for his flink/beam demo.
I don't know if your case is maybe because you are running on docker for
mac, but check two things.
1. docker assigns an internal IP to each running container (the one
corresponding to the jobmanager is the one to put in the -m argument).
2. The path you pass in the examples must be accesible from the container,
e.g. you must copy the file in all the nodes (as I mention in the README), or
mount them with a volume (-v on docker).
e.g.
```
$FLINK_HOME/bin/flink run -m 172.18.0.2:6123 examples/batch/WordCount.jar
--input file:///tmp/kinglear.txt
```
The kinglear file must be in all the nodes. I copy those like this:
```
for i in $(docker ps --filter name=flink --format={{.ID}}); do
docker cp ~/Desktop/flink-beam/kinglear.txt $i:/tmp/
done
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2176 One last thing I would like to try is running a job from an existing Flink installation using `$FLINK_HOME/bin/flink run -m ` as described in the README. I suspect it has something to do with setting the right IP or network setting because Akka is very particular about the IP to which it is bound. Did you get this to work? I'm only managing to access the Web Dashboard. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Awasome, thanks @aljoscha, let's merge ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
Re 1. I think this should be fine. There is some dynamic code generation
but this uses Janino as a library so that shouldn't be a problem.
Re 2. I'm not aware of any but it should be easy to add them if we find any
in the future.
Re 3. This is not easily possible since the `&` is hardcoded in
`flink-daemon.sh`:
```
$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath
"`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
```
I'd be very happy if you'd like to work on adding a flag or setting to make
the daemons start non-daemonized. It should not be too hard to add that, IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Nice, I just fixed as you suggested. I have three questions: 1. This container is based on the Java JRE (to keep it small), Does Flink in any part do some magic that requires a full JDK (like live recompiles) ? If no, I think this is almost perfect now. 2. Are you aware of any other flink dependency that uses any native OS library (I ask this since this because if it is the case it must be added to the container, I did this for snappy because I found the issue while testing a Beam pipeline, but I don't know if there are others). 3. In the docker image I left supervisor because I didn't find an easy way to start flink in normal mode, the scripts to start both taskmanager and jobmanager go into daemon mode immediately, is there something that can be done to change this (this will reduce the image in 40 more MB), but well I can work on that for a different PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2176 Nice work! I didn't know how to use docker but I managed to set it up and use the new version on OS X without a problem. So it seems to work well, and the code is a lot simpler and the image is smaller. LGTM minus the one comment I had about the config file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 Sorry I had to rebase my previous PR but this is the definitive one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2176: Flink 4118
Github user iemejia commented on the issue: https://github.com/apache/flink/pull/2176 The docker images script was simplified and the image size was reduced. Previous image: flink latest 6475add651c7 24 minutes ago 711.6 MB Image after FLINK-4118 flink latest 555e60f24c10 20 seconds ago 252.5 MB --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
