[jira] [Commented] (KAFKA-9774) Create official Docker image for Kafka Connect

2020-05-18 Thread Jordan Moore (Jira) 


[ 
https://issues.apache.org/jira/browse/KAFKA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109910#comment-17109910
 ] 

Jordan Moore commented on KAFKA-9774:
-

Done here 
https://github.com/OneCricketeer/apache-kafka-connect-docker#containerized-apache-kafka-connect

> Create official Docker image for Kafka Connect
> --
>
> Key: KAFKA-9774
> URL: https://issues.apache.org/jira/browse/KAFKA-9774
> Project: Kafka
>  Issue Type: Task
>  Components: build, KafkaConnect, packaging
>Affects Versions: 2.4.1
>Reporter: Jordan Moore
>Priority: Major
>  Labels: build, features, needs-kip
> Attachments: image-2020-03-27-05-04-46-792.png, 
> image-2020-03-27-05-05-59-024.png
>
>
> This is a ticket for creating an *official* apache/kafka-connect Docker 
> image. 
> Does this need a KIP?  -  I don't think so. This would be a new feature, not 
> any API change. 
> Why is this needed?
>  # Kafka Connect is stateless. I believe this is why a Kafka image is not 
> created?
>  # It scales much more easily with Docker and orchestrators. It operates much 
> like any other serverless / "microservice" web application 
>  # People struggle with deploying it because it is packaged _with Kafka_ , 
> which leads some to believe it needs to _*run* with Kafka_ on the same 
> machine. 
> I think there is separate ticket for creating an official Docker image for 
> Kafka but clearly none exist. I reached out to Confluent about this, but 
> heard nothing yet.
> !image-2020-03-27-05-05-59-024.png|width=740,height=196!
>  
> Zookeeper already has one , btw  
> !image-2020-03-27-05-04-46-792.png|width=739,height=288!
> *References*: 
> [Docs for Official 
> Images|[https://docs.docker.com/docker-hub/official_images/]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-9774) Create official Docker image for Kafka Connect

2020-03-29 Thread Jordan Moore (Jira) 


[ 
https://issues.apache.org/jira/browse/KAFKA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070641#comment-17070641
 ] 

Jordan Moore commented on KAFKA-9774:
-

Hi [~ewencp]. Thanks for the response. 

{quote}
it's a significant public support, compatibility, and maintenance commitment 
for the project
{quote}

1. Public support and maintenance can be tracked in JIRA, mailing list, and 
other mediums as currently done, no? 
2. Compatibility is locked to the versions of dependencies locked in Gradle. 
The existing Kafka Compatibility Matrix page is good enough for all clients, 
including Connect. If you mean compatibility of various Java base images or 
OS's, then there should be a vote on that, perhaps?

{quote}
what's the commitment to publication during releases
{quote}

Well, the jib-gradle-plugin can be added to release as part of a standard AK 
release, so sure, the release process will have one new build artifact that 
would push into DockerHub. The RM would maintain a DockerHub credential, maybe 
initialized by a PMC, I assume. 

{quote}
periodic updates to get base image CVE updates
{quote}

I'd say this would be done per AK release, as needed. Base images I can think 
of would be openjdk , adoptopenjdk, or the new RedHat UBI images. 

{quote}
verification, testing
{quote}

What kinds of testing would be needed outside of the existing Connect test 
framework? A Docker image is just a build artifact like a JAR, no? After JARs 
are built, what extra testing is being done? 



> Create official Docker image for Kafka Connect
> --
>
> Key: KAFKA-9774
> URL: https://issues.apache.org/jira/browse/KAFKA-9774
> Project: Kafka
>  Issue Type: Task
>  Components: build, KafkaConnect, packaging
>Affects Versions: 2.4.1
>Reporter: Jordan Moore
>Priority: Major
>  Labels: build, features
> Attachments: image-2020-03-27-05-04-46-792.png, 
> image-2020-03-27-05-05-59-024.png
>
>
> This is a ticket for creating an *official* apache/kafka-connect Docker 
> image. 
> Does this need a KIP?  -  I don't think so. This would be a new feature, not 
> any API change. 
> Why is this needed?
>  # Kafka Connect is stateless. I believe this is why a Kafka image is not 
> created?
>  # It scales much more easily with Docker and orchestrators. It operates much 
> like any other serverless / "microservice" web application 
>  # People struggle with deploying it because it is packaged _with Kafka_ , 
> which leads some to believe it needs to _*run* with Kafka_ on the same 
> machine. 
> I think there is separate ticket for creating an official Docker image for 
> Kafka but clearly none exist. I reached out to Confluent about this, but 
> heard nothing yet.
> !image-2020-03-27-05-05-59-024.png|width=740,height=196!
>  
> Zookeeper already has one , btw  
> !image-2020-03-27-05-04-46-792.png|width=739,height=288!
> *References*: 
> [Docs for Official 
> Images|[https://docs.docker.com/docker-hub/official_images/]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-9774) Create official Docker image for Kafka Connect

2020-03-29 Thread Ewen Cheslack-Postava (Jira) 


[ 
https://issues.apache.org/jira/browse/KAFKA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070633#comment-17070633
 ] 

Ewen Cheslack-Postava commented on KAFKA-9774:
--

My first reaction to this was that is should have a KIP because it's a 
significant public support, compatibility, and maintenance commitment for the 
project... Maybe if it's very clearly highlighted as unofficial, no 
compatibility guarantees, etc, though I'm not sure how many people want that 
type of image being published from the project.

As simple off-the-top-of-my-head examples, what's the commitment to publication 
during releases (is this being added to 
[https://cwiki.apache.org/confluence/display/KAFKA/Release+Process]?), periodic 
updates to get base image CVE updates, verification, testing, etc?

> Create official Docker image for Kafka Connect
> --
>
> Key: KAFKA-9774
> URL: https://issues.apache.org/jira/browse/KAFKA-9774
> Project: Kafka
>  Issue Type: Task
>  Components: build, KafkaConnect, packaging
>Affects Versions: 2.4.1
>Reporter: Jordan Moore
>Priority: Major
>  Labels: build, features
> Attachments: image-2020-03-27-05-04-46-792.png, 
> image-2020-03-27-05-05-59-024.png
>
>
> This is a ticket for creating an *official* apache/kafka-connect Docker 
> image. 
> Does this need a KIP?  -  I don't think so. This would be a new feature, not 
> any API change. 
> Why is this needed?
>  # Kafka Connect is stateless. I believe this is why a Kafka image is not 
> created?
>  # It scales much more easily with Docker and orchestrators. It operates much 
> like any other serverless / "microservice" web application 
>  # People struggle with deploying it because it is packaged _with Kafka_ , 
> which leads some to believe it needs to _*run* with Kafka_ on the same 
> machine. 
> I think there is separate ticket for creating an official Docker image for 
> Kafka but clearly none exist. I reached out to Confluent about this, but 
> heard nothing yet.
> !image-2020-03-27-05-05-59-024.png|width=740,height=196!
>  
> Zookeeper already has one , btw  
> !image-2020-03-27-05-04-46-792.png|width=739,height=288!
> *References*: 
> [Docs for Official 
> Images|[https://docs.docker.com/docker-hub/official_images/]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-9774) Create official Docker image for Kafka Connect

2020-03-27 Thread Jordan Moore (Jira) 


[ 
https://issues.apache.org/jira/browse/KAFKA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069144#comment-17069144
 ] 

Jordan Moore commented on KAFKA-9774:
-

[~rhauch] & [~kkonstantine] , lemme know your thoughts here. 

Here's the current design
 # Use Jib to build the image 
 # Follow similar practices as the confluentinc/cp-docker image in that the 
CONNECT_ variables are templated into the worker property file. I'd planned on 
using FreeMarker to do this, but later realized that's probably overkill.
 # Wrap ConnectDistributed with a small class that just initializes that file 
from the environment variables.
 # Maybe add some additional verification around expected variables.
 # Write up some documentation that would be hosted in DockerHub
 # Ping someone in Docker official-images repo to get this put up there.

Yes, right now the above repo is built in Maven, and just a branch of some 
Kafka Stream stuff I did before. That is all fixable of course, no problem.

> Create official Docker image for Kafka Connect
> --
>
> Key: KAFKA-9774
> URL: https://issues.apache.org/jira/browse/KAFKA-9774
> Project: Kafka
>  Issue Type: Task
>  Components: build, KafkaConnect, packaging
>Affects Versions: 2.4.1
>Reporter: Jordan Moore
>Priority: Major
>  Labels: build, features
> Attachments: image-2020-03-27-05-04-46-792.png, 
> image-2020-03-27-05-05-59-024.png
>
>
> This is a ticket for creating an *official* apache/kafka-connect Docker 
> image. 
> Does this need a KIP?  -  I don't think so. This would be a new feature, not 
> any API change. 
> Why is this needed?
>  # Kafka Connect is stateless. I believe this is why a Kafka image is not 
> created?
>  # It scales much more easily with Docker and orchestrators. It operates much 
> like any other serverless / "microservice" web application 
>  # People struggle with deploying it because it is packaged _with Kafka_ , 
> which leads some to believe it needs to _*run* with Kafka_ on the same 
> machine. 
> I think there is separate ticket for creating an official Docker image for 
> Kafka but clearly none exist. I reached out to Confluent about this, but 
> heard nothing yet.
> !image-2020-03-27-05-05-59-024.png|width=740,height=196!
>  
> Zookeeper already has one , btw  
> !image-2020-03-27-05-04-46-792.png|width=739,height=288!
> *References*: 
> [Docs for Official 
> Images|[https://docs.docker.com/docker-hub/official_images/]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-9774) Create official Docker image for Kafka Connect

2020-03-27 Thread Jordan Moore (Jira) 


[ 
https://issues.apache.org/jira/browse/KAFKA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068519#comment-17068519
 ] 

Jordan Moore commented on KAFKA-9774:
-

Happy to take this on. And already working on a POC here (ignore the README and 
the project name, I started working on only the pom and the main class in this 
branch) - 
[https://github.com/cricket007/kafka-streams-jib-example/tree/feature/connect-distributed]

> Create official Docker image for Kafka Connect
> --
>
> Key: KAFKA-9774
> URL: https://issues.apache.org/jira/browse/KAFKA-9774
> Project: Kafka
>  Issue Type: Task
>  Components: build, KafkaConnect, packaging
>Affects Versions: 2.4.1
>Reporter: Jordan Moore
>Priority: Major
>  Labels: build, features
> Attachments: image-2020-03-27-05-04-46-792.png, 
> image-2020-03-27-05-05-59-024.png
>
>
> This is a ticket for creating an *official* apache/kafka-connect Docker 
> image. 
> Does this need a KIP?  -  I don't think so. This would be a new feature, not 
> any API change. 
> Why is this needed?
>  # Kafka Connect is stateless. I believe this is why a Kafka image is not 
> created?
>  # It scales much more easily with Docker and orchestrators. It operates much 
> like any other serverless / "microservice" web application 
>  # People struggle with deploying it because it is packaged _with Kafka_ , 
> which leads some to believe it needs to _*run* with Kafka_ on the same 
> machine. 
> I think there is separate ticket for creating an official Docker image for 
> Kafka but clearly none exist. I reached out to Confluent about this, but 
> heard nothing yet.
> !image-2020-03-27-05-05-59-024.png|width=740,height=196!
>  
> Zookeeper already has one , btw  
> !image-2020-03-27-05-04-46-792.png|width=739,height=288!
> *References*: 
> [Docs for Official 
> Images|[https://docs.docker.com/docker-hub/official_images/]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)