Re: [DISCUSS] FLIP-111: Docker image unification

I am coming extremely late to this discussion since the vote already started but it is great we are finally getting into unification Enthusiast +1. Kudos to Andrey and the rest of the community for bringing all the useful and different perspectives. I just want to bring information on two tickets

Re: [DISCUSS] FLIP-111: Docker image unification

Hi, all, Thanks for the reply, Andrey! I have filed two new tickets tracking the problems: 1. FLINK-17033 for upgrading base Java Docker image, I pointed out some other problems the openjdk:8-jre-alpine could have in the ticket‘s

Re: [DISCUSS] FLIP-111: Docker image unification

Hi all, Thanks for the further feedback Niels and Canbin. @Niels I agree with Till, the comments about docker tags are valid concerns and we can discuss them in dedicated ML threads in parallel or after the general unification of Dockerfiles suggested by this FLIP. One thing to add about point

Re: [DISCUSS] FLIP-111: Docker image unification

Hi, all Thanks a lot for this FLIP and all the fruitable discussion. I am not sure whether the following questions are in the scope of this FLIP, but I still expect your reply: 1. Which docker base image do we plan to use for Java? As far as I see, openjdk:8-jre-alpine[1] is not officially

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for the feedback Niels. This is very helpful. 1. I agree `flink:latest` is nice to get started but in the long run people will want to pin their dependencies to a specific Flink version. I think the fix will happen as part of FLINK-15794. 2. SNAPSHOT docker images will be really helpful

Re: [DISCUSS] FLIP-111: Docker image unification

Hi all, Sorry for jumping in at this late point of the discussion. I see a lot of things I really like and I would like to put my "needs" and observations here too so you take them into account (where possible). I suspect that there will be overlap with things you already have taken into account.

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for the summary, Andrey. Good idea to link Patrick's document from the FLIP as a future direction so it doesn't get lost. Could you make sure to revive that discussion when FLIP-111 nears an end? This is good to go on my part. +1 to start the VOTE. @Till, @Yang: Thanks for the

Re: [DISCUSS] FLIP-111: Docker image unification

Hi everyone, Patrick and Ufuk, thanks a lot for more ideas and suggestions! I have updated the FLIP according to the current state of discussion. Now it also contains the implementation steps and future follow-ups. Please, review if there are any concerns. The order of the steps aims for keeping

Re: [DISCUSS] FLIP-111: Docker image unification

Hi everyone, just a small inline comment. On Fri, Apr 3, 2020 at 11:42 AM Ufuk Celebi wrote: > Hey Yang, > > thanks! See inline answers. > > On Fri, Apr 3, 2020 at 5:11 AM Yang Wang wrote: > > > Hi Ufuk, > > > > Thanks for make the conclusion and directly point out what need to be > done > >

Re: [DISCUSS] FLIP-111: Docker image unification

Hey Yang, thanks! See inline answers. On Fri, Apr 3, 2020 at 5:11 AM Yang Wang wrote: > Hi Ufuk, > > Thanks for make the conclusion and directly point out what need to be done > in > FLIP-111. I agree with you that we should narrow down the scope and focus > the > most important and basic part

Re: [DISCUSS] FLIP-111: Docker image unification

Hi Ufuk, Thanks for make the conclusion and directly point out what need to be done in FLIP-111. I agree with you that we should narrow down the scope and focus the most important and basic part about docker image unification. (1) Extend the entrypoint script in apache/flink-docker to start the

Re: [DISCUSS] FLIP-111: Docker image unification

Hey all, thanks for the proposal and the detailed discussion. In particular, thanks to Andrey for starting this thread and to Patrick for the additional ideas in the linked Google doc. I find many of the improvements proposed during the discussion (such as the unified entrypoint in Flink, proper

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks Andrey for working on this, and everyone else for your feedback. This FLIP inspired me to discuss and write down some ideas I've had for a while about configuring and running Flink (especially in Docker) that go beyond the scope of this FLIP, but don't contradict what it sets out to do.

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for summarising the discussion points, Till. # Configuration ## Env variables Agree, this looks like an independent effort. ## dynamic program arguments Indeed, jobmanager.sh needs small extension. It can be addressed independently but I think it has chance to be addressed in this

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for creating this FLIP Andrey. I like the general idea pretty much. I tried to group some of the above-mentioned points in order to give my 50 cent. # Configuration How to configure the Flink process seems to be the biggest question. Due to historical reasons we have a plethora of

Re: [DISCUSS] FLIP-111: Docker image unification

I second Thomas that we can support both Java 8 and 11. Best, Yangze Guo On Wed, Mar 18, 2020 at 12:12 PM Thomas Weise wrote: > > --> > > On Mon, Mar 16, 2020 at 1:58 AM Andrey Zagrebin wrote: >> >> Thanks for the further feedback Thomas and Yangze. >> >> > A generic, dynamic configuration

Re: [DISCUSS] FLIP-111: Docker image unification

--> On Mon, Mar 16, 2020 at 1:58 AM Andrey Zagrebin wrote: > Thanks for the further feedback Thomas and Yangze. > > > A generic, dynamic configuration mechanism based on environment variables > is essential and it is already supported via envsubst and an environment > variable that can supply a

Re: [DISCUSS] FLIP-111: Docker image unification

Hi Andrey, Thanks for your explanation. > About the logging What i mean is we could not forward the stdout/stderr to local files and docker stdout at the same time by using log4j. For the jobmanager.log/taskmanager.log, it works quite well since we only need to add a console appender in the

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for the further feedback Thomas and Yangze. > A generic, dynamic configuration mechanism based on environment variables is essential and it is already supported via envsubst and an environment variable that can supply a configuration fragment True, we already have this. As I understand

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for the reply, Andrey. Regarding building from local dist: - Yes, I bring this up mostly for development purpose. Since k8s is popular, I believe more and more developers would like to test their work on k8s cluster. I'm not sure should all developers write a custom docker file themselves

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for working on improvements to the Flink Docker container images. This will be important as more and more users are looking to adopt Kubernetes and other deployment tooling that relies on Docker images. A generic, dynamic configuration mechanism based on environment variables is essential

Re: [DISCUSS] FLIP-111: Docker image unification

Hi All, Thanks a lot for the feedback! *@Yangze Guo* - Regarding the flink_docker_utils#install_flink function, I think it > should also support build from local dist and build from a > user-defined archive. I suppose you bring this up mostly for development purpose or powerful users. Most of

Re: [DISCUSS] FLIP-111: Docker image unification

Hi Andrey, thanks a lot for this proposal. The variety of Docker files in the project has been causing quite some confusion. For the entrypoint, have you considered to also allow setting configuration via environment variables as in "docker run -e FLINK_REST_BIN_PORT=8081 ..."? This is quite

Re: [DISCUSS] FLIP-111: Docker image unification

Hi Andrey, Thanks for driving this significant FLIP. From the user ML, we could also know there are many users running Flink in container environment. Then the docker image will be the very basic requirement. Just as you say, we should provide a unified place for all various usage(e.g. session,

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for open this FLIP and summarize the current state of Dockerfiles, Andrey! +1 for this idea. I have some minor comments / questions: - Regarding the flink_docker_utils#install_flink function, I think it should also support build from local dist and build from a user-defined archive. - It

[DISCUSS] FLIP-111: Docker image unification

Hi All, If you have ever touched the docker topic in Flink, you probably noticed that we have multiple places in docs and repos which address its various concerns. We have prepared a FLIP [1] to simplify the perception of docker topic in Flink by users. It mostly advocates for an approach of