Hi Ismael, Thanks for the correction.
We will be following the same EOL policy as Apache Kafka, hence we have removed it from the KIP. Thanks and regards, Vedarth On Mon, Oct 23, 2023 at 12:17 PM Ismael Juma <m...@ismaeljuma.com> wrote: > Sorry, I noticed a typo in my message. I meant "Additionally, we should not > specify the EOL policy in this KIP" since it doesn't propose changing it. > > Ismael > > On Sun, Oct 22, 2023 at 10:56 PM Vedarth Sharma <vedarth.sha...@gmail.com> > wrote: > > > Hi Ismael, > > Thanks for the valuable feedback. > > > > 1. No docker image specific release process: This was one of our > > considered approaches, but we thought that docker image shouldn't > block > > AK > > release. Though I agree, treating docker image as another artifact for > > every AK release makes much more sense. Hence, releasing a new version > > of > > Kafka for the affected branch in such a scenario is a much cleaner > > approach. Added this as the accepted approach in the KIP. > > 2. EOL policy: Updated in the KIP. > > > > Thanks and regards, > > Vedarth > > > > On Sun, Oct 22, 2023 at 11:20 PM Ismael Juma <m...@ismaeljuma.com> wrote: > > > > > Hi Vedarth, > > > > > > I think we shouldn't introduce any new release process that is docker > > > specific. We should consider the software in the docker image in the > same > > > way as consider third party dependencies today - if there is a high > > > severity CVE affecting any of them, we aim to release a new version of > > > Kafka for the affected branch. It would include the latest Kafka code > > from > > > the branch. > > > > > > Additionally, we should specify the EOL policy in this KIP - we are not > > > changing it as part of it. One interesting detail is that the release > > > document claims we support the last 3 releases, but the reality has > been > > a > > > bit different - we tend to support the 2 most recent releases unless > > it's a > > > high severity CVE in Kafka itself (these tend to be much rarer, > > > thankfully). > > > > > > Ismael > > > > > > On Sun, Oct 22, 2023, 10:19 AM Vedarth Sharma < > vedarth.sha...@gmail.com> > > > wrote: > > > > > > > Hi Mickael, > > > > Thanks for going through the KIP and providing valuable feedback. > > > > > > > > 1. We will support the latest LTS version of Java supported by > > Apache > > > > Kafka. > > > > 2. We will provide support for the last three releases. We've > added > > a > > > > detailed example of this in the KIP under our EOL policy. > > > > 3. We can establish a nightly cron job using GitHub Actions and > > > leverage > > > > an open-source vulnerability scanning tool like trivy ( > > > > https://github.com/aquasecurity/trivy), to get vulnerability > > reports > > > on > > > > all supported images. This tool offers a straightforward way to > > > > integrate > > > > vulnerability checks directly into our GitHub Actions workflow. > > > > 4. That's a good suggestion to have a GitHub Actions workflow. We > > will > > > > implement a GitHub Actions workflow to automate the build and > > testing > > > > process. > > > > 5. Regarding the release process, we observed that there isn't an > > > > existing CI/CD pipeline. We can consider the addition of a GitHub > > > > workflow > > > > to facilitate the release process. > > > > > > > > Please let us know your thoughts on the above. > > > > > > > > Thanks and regards, > > > > Vedarth > > > > > > > > On Fri, Oct 20, 2023 at 7:34 PM Mickael Maison < > > mickael.mai...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi Krishna, > > > > > > > > > > Overall I'm supportive of having an official docker image. > > > > > I have a few questions: > > > > > - Can you clarify the process of selecting the Java version? Is the > > > > > proposal to only pick LTS versions? or to pick the highest version > > > > > supported by Kafka? > > > > > - Once a new Kafka version is released, what happens to the image > > > > > containing the previous release? Do we expect to still update it in > > > > > case of CVEs? If so for how long? > > > > > - How will we get notified that the base image has a CVE? > > > > > - Rather than having scripts PMC members have to run from their > > > > > machines, would it e possible to have a Jenkins job or GitHub > action? > > > > > > > > > > Thanks, > > > > > Mickael > > > > > > > > > > > > > > > > > > > > On Fri, Oct 20, 2023 at 12:51 PM Vedarth Sharma > > > > > <vedarth.sha...@gmail.com> wrote: > > > > > > > > > > > > Hi Manikumar, > > > > > > > > > > > > Thanks for the feedback! > > > > > > > > > > > > 1. We propose the addition of a new directory named "docker" at > the > > > > root > > > > > of > > > > > > the repository, where all Docker-related code will be stored. A > > > > detailed > > > > > > directory structure has been added in the KIP. > > > > > > 2. We request the creation of an Apache Kafka repository > > > (apache/kafka) > > > > > on > > > > > > DockerHub, to be administered under the The Apache Software > > > Foundation > > > > > > <https://hub.docker.com/u/apache>. The PMC members should have > the > > > > > > necessary permissions for pushing updates to the docker repo. > > > > > > > > > > > > Thanks and regards, > > > > > > Vedarth > > > > > > > > > > > > > > > > > > On Fri, Oct 20, 2023 at 2:44 PM Manikumar < > > manikumar.re...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Krishna, Vedarth, > > > > > > > > > > > > > > Thanks for the KIP. > > > > > > > > > > > > > > 1. Can we add directory structure of Docker Image related files > > in > > > > > Kafka > > > > > > > repo. > > > > > > > > > > > > > > 2. > Steps for the Docker image release will be included in the > > > > Release > > > > > > > Process doc of Apache Kafka > > > > > > > > > > > > > > Can we list down the requirements (repos, accounts) for > releasing > > > > > images to > > > > > > > docker hub. I am mainly asking because PMC needs to request > > docker > > > > hub > > > > > > > access/repos. > > > > > > > I can help in getting required repos/accounts. > > > > > > > https://infra.apache.org/docker-hub-policy.html > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Manikumar > > > > > > > > > > > > > > On Thu, Oct 19, 2023 at 8:22 PM Krishna Agarwal < > > > > > > > krishna0608agar...@gmail.com> wrote: > > > > > > > > > > > > > > > Hi Viktor, > > > > > > > > > > > > > > > > I've noticed there are two types of custom jar > configurations: > > > > > > > > > > > > > > > > 1. *Type 1*: In this case, only the class name is > > required(e.g > > > > > > > > *authorizer.class.name > > > > > > > > <http://authorizer.class.name>**)* This can be configured > > by > > > > the > > > > > > > > following steps: > > > > > > > > - Mount the jar in the container. > > > > > > > > - Configure the *CLASSPATH* environment variable (used > by > > > > > > > > *kafka-run-class.sh*) by providing the mounted path to > > it. > > > > > This can > > > > > > > > be passed as an environment variable to the docker > > > container. > > > > > > > > 2. *Type 2*: Here, in addition to the class name, > classpath > > > can > > > > > also > > > > > > > be > > > > > > > > configured (eg *remote.log.metadata.manager.class.name > > > > > > > > <http://remote.log.metadata.manager.class.name> *and > > > > > > > > *remote.log.metadata.manager.class.path*). This can be > > > > configured > > > > > by > > > > > > > the > > > > > > > > following steps: > > > > > > > > - Mount the jar in the container. > > > > > > > > - Configure the respective *class.path* property. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Krishna > > > > > > > > > > > > > > > > On Mon, Sep 25, 2023 at 11:41 PM Krishna Agarwal < > > > > > > > > krishna0608agar...@gmail.com> wrote: > > > > > > > > > > > > > > > > > Hi Viktor, > > > > > > > > > Thanks for the questions. > > > > > > > > > > > > > > > > > > 1. While the docker image outlined in KIP-975 is > designed > > > for > > > > > > > > > production environments, it is equally suitable for > > > > development > > > > > and > > > > > > > > testing > > > > > > > > > purposes. We will furnish the docker image, allowing > users > > > the > > > > > > > > flexibility > > > > > > > > > to employ it according to their specific needs. > > > > > > > > > 2. The configs will be injected into the docker > container > > > > > through > > > > > > > > > environment variables. These environment variables will > > > have a > > > > > > > prefix > > > > > > > > > allowing for efficient parsing to extract the relevant > > > > > > > > properties.(Will add > > > > > > > > > this implementation in the KIP as well once we converge > on > > > > > this.) > > > > > > > > > 3. Regarding this question, I'll conduct a test on my > end > > > > after > > > > > > > > > gaining a better understanding, and then provide you > with > > a > > > > > > > response. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Krishna > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass > > > > > > > > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > > > > > > > > > > > >> Hi Ismael, > > > > > > > > >> > > > > > > > > >> I'm not trying to advocate against the docker image, I > just > > > > > pointed > > > > > > > out > > > > > > > > >> that the current scoping of the KIP may be a bit too > generic > > > and > > > > > > > thought > > > > > > > > >> that KIP-974 and KIP-975 were aiming for mostly the same > > thing > > > > > and can > > > > > > > > be > > > > > > > > >> discussed under one umbrella. Apologies if this was rooted > > in > > > a > > > > > > > > >> misunderstanding. > > > > > > > > >> > > > > > > > > >> Kirshna, > > > > > > > > >> > > > > > > > > >> I think we need to refine the KIP a bit more. I think > there > > > are > > > > > some > > > > > > > > >> interfaces that we need to include in the KIP as Kafka has > > > > > plugins in > > > > > > > > >> certain cases where users are expected to provide > > > implementation > > > > > and I > > > > > > > > >> think it's worth discussing this in the KIP as they're > kind > > of > > > > > > > > interfaces > > > > > > > > >> for users. Here are my questions in order: > > > > > > > > >> 1. In what environments do you want the image to be used? > > As I > > > > > > > > understand > > > > > > > > >> it would replace the current testing image and serve as a > > > basis > > > > > for > > > > > > > > >> development, but would it aim at production use cases too > > > > > > > > (docker-compose, > > > > > > > > >> Kubernetes, etc.)? > > > > > > > > >> 2. How do you plan to forward configs to the broker? Do we > > > > expect > > > > > a > > > > > > > > >> populated server.properties file placed in a certain > > location > > > or > > > > > > > should > > > > > > > > >> the > > > > > > > > >> docker image create this file based on some input (like > env > > > > vars)? > > > > > > > > >> 3. Certain parts can be pluggable, like metric reporters > or > > > > > remote log > > > > > > > > >> implementations that were just introduced by KIP-405. > These > > > > > manifest > > > > > > > in > > > > > > > > >> jar > > > > > > > > >> files that must be put on the classpath of Kafka while > > certain > > > > > > > > classnames > > > > > > > > >> have to be configured. How do you plan to implement this, > > how > > > do > > > > > we > > > > > > > > >> allow users to configure such things? > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Viktor > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole > > > > > > > > >> <kevers...@cloudflare.com.invalid> wrote: > > > > > > > > >> > > > > > > > > >> > Hello, > > > > > > > > >> > > > > > > > > > >> > I think this would be a wonderful improvement to the > > > > ecosystem. > > > > > > > While > > > > > > > > >> > Viktor is correct that most Docker pipelines eventually > > lead > > > > to > > > > > a > > > > > > > > >> > kubernetes deployment, that should not stop us from > > creating > > > > an > > > > > > > > >> > Official Docker Image. Creating a Docker image would > allow > > > us > > > > to > > > > > > > > ensure > > > > > > > > >> a > > > > > > > > >> > level of quality and support for people who want to > deploy > > > > > Kafka as > > > > > > > a > > > > > > > > >> > container on baremetal machines, it could allow us to > > create > > > > > > > > >> > a sandbox/developer environment for new contributors and > > > > > developers > > > > > > > to > > > > > > > > >> test > > > > > > > > >> > and have a single agreed upon environment that kafka > works > > > in > > > > > for > > > > > > > > future > > > > > > > > >> > KIPs and would most likely spawn more contributions from > > > > people > > > > > > > > wanting > > > > > > > > >> to > > > > > > > > >> > optimize kafka for k8s. > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > I am 100% for this and will gladly help if approved. > > > > > > > > >> > > > > > > > > > >> > Kenneth > > > > > > > > >> > > > > > > > > > >> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma < > > > > m...@ismaeljuma.com> > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > >> > > Hi Viktor, > > > > > > > > >> > > > > > > > > > > >> > > I disagree. Docker is a very popular deployment tool > and > > > > it's > > > > > not > > > > > > > > only > > > > > > > > >> > used > > > > > > > > >> > > with Kubernetes. > > > > > > > > >> > > > > > > > > > > >> > > Ismael > > > > > > > > >> > > > > > > > > > > >> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass > > > > > > > > >> > > <viktor.somo...@cloudera.com.invalid> wrote: > > > > > > > > >> > > > > > > > > > > >> > > > Hi Krishna, > > > > > > > > >> > > > > > > > > > > > >> > > > I think you should merge this KIP and KIP-974 > > > > > > > > >> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974> > > > > as > > > > > > > there > > > > > > > > >> are > > > > > > > > >> > overlaps as > > > > > > > > >> > > > Federico pointed out on KIP-974 > > > > > > > > >> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974 > > > >. > > > > I > > > > > > > think > > > > > > > > >> you > > > > > > > > >> > should keep that one as it > > > > > > > > >> > > > has well defined goals (improve tests) while I feel > > this > > > > > one is > > > > > > > > too > > > > > > > > >> > > > generic. Docker is usually just a tool for either > > > testing > > > > or > > > > > > > > >> > Kubernetes, > > > > > > > > >> > > so > > > > > > > > >> > > > they have very well defined use-cases. In the case > of > > > > Flink > > > > > for > > > > > > > > >> > instance > > > > > > > > >> > > > the image is used for its kubernetes operator. The > use > > > > case > > > > > > > would > > > > > > > > >> > > determine > > > > > > > > >> > > > a lot of things and I think a generic image would > > likely > > > > > not fit > > > > > > > > the > > > > > > > > >> > > needs > > > > > > > > >> > > > of all use-cases. > > > > > > > > >> > > > > > > > > > > > >> > > > Best, > > > > > > > > >> > > > Viktor > > > > > > > > >> > > > > > > > > > > > >> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal < > > > > > > > > >> > > > krishna0608agar...@gmail.com> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > > > > > > > > > >> > > > > Hi, > > > > > > > > >> > > > > Apache Kafka does not have an official docker > image > > > > > currently. > > > > > > > > >> > > > > I want to submit a KIP to publish a docker image > for > > > > > Apache > > > > > > > > Kafka. > > > > > > > > >> > > > > > > > > > > > > >> > > > > KIP-975 < > > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975 > >: > > > > > > > > >> > Docker Image for Apache Kafka > > > > > > > > >> > > > > < > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Regards, > > > > > > > > >> > > > > Krishna > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
