[jira] [Created] (KAFKA-9258) Connect ConnectorStatusMetricsGroup sometimes NPE
Cyrus Vafadari created KAFKA-9258: - Summary: Connect ConnectorStatusMetricsGroup sometimes NPE Key: KAFKA-9258 URL: https://issues.apache.org/jira/browse/KAFKA-9258 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Cyrus Vafadari java.lang.NullPointerException at org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:901) at org.apache.kafka.connect.runtime.Worker.awaitStopTask(Worker.java:720) at org.apache.kafka.connect.runtime.Worker.awaitStopTasks(Worker.java:740) at org.apache.kafka.connect.runtime.Worker.stopAndAwaitTasks(Worker.java:758) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.processTaskConfigUpdatesWithIncrementalCooperative(DistributedHerder.java:575) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:396) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:282) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8605) Warn users when they have same connector in their plugin-path more than once
[ https://issues.apache.org/jira/browse/KAFKA-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8605: -- Labels: (was: needs-kip) > Warn users when they have same connector in their plugin-path more than once > > > Key: KAFKA-8605 > URL: https://issues.apache.org/jira/browse/KAFKA-8605 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > Right now it is very easy to have multiple copies of the same connector in > the plugin-path and not realize it. > This can be problematic if a user is adding dependencies into the plugin, or > accidentally using the wrong version of the connector. > An unintrusive improvement would be to log a warning if the same connector > appears in the plugin-path more than once -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KAFKA-8749) Connect SinkTask.put(...) to specify ArrayList in Signature
[ https://issues.apache.org/jira/browse/KAFKA-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8749: -- Labels: needs-kip (was: ) > Connect SinkTask.put(...) to specify ArrayList in Signature > --- > > Key: KAFKA-8749 > URL: https://issues.apache.org/jira/browse/KAFKA-8749 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > Labels: needs-kip > > Detailed in > [KIP-502|https://cwiki.apache.org/confluence/display/KAFKA/KIP-502%3A+Connect+SinkTask.put%28...%29+to+specify+ArrayList%3CSinkRecord%3E+in+Signature] > > The key class in a Sink Connector is the `SinkTask` implementation. The > `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the > Connector-specific code through this entrypoint, and its current abstract > class specifies: > public abstract void put(Collection records); > Usually, the implementer will use a for-each loop to iterate through the > records one by one and act on them. There is only one caller, WorkerSinkTask > which always passes an ArrayList. > There are two shortcomings with this signature: > # The Connector implementation is not provide with any *ordering guarantees* > of Records. > # The decision values giving flexibility to the caller of the method over > the implementer of the method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KAFKA-8605) Warn users when they have same connector in their plugin-path more than once
[ https://issues.apache.org/jira/browse/KAFKA-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8605: -- Labels: needs-kip (was: ) > Warn users when they have same connector in their plugin-path more than once > > > Key: KAFKA-8605 > URL: https://issues.apache.org/jira/browse/KAFKA-8605 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > Labels: needs-kip > > Right now it is very easy to have multiple copies of the same connector in > the plugin-path and not realize it. > This can be problematic if a user is adding dependencies into the plugin, or > accidentally using the wrong version of the connector. > An unintrusive improvement would be to log a warning if the same connector > appears in the plugin-path more than once -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KAFKA-8749) Connect SinkTask.put(...) to specify ArrayList in Signature
[ https://issues.apache.org/jira/browse/KAFKA-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8749: -- Description: [ Detailed in KIP-502 |https://cwiki.apache.org/confluence/display/KAFKA/KIP-502%3A+Connect+SinkTask.put%28...%29+to+specify+ArrayList%3CSinkRecord%3E+in+Signature] The key class in a Sink Connector is the `SinkTask` implementation. The `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the Connector-specific code through this entrypoint, and its current abstract class specifies: public abstract void put(Collection records); Usually, the implementer will use a for-each loop to iterate through the records one by one and act on them. There is only one caller, WorkerSinkTask which always passes an ArrayList. There are two shortcomings with this signature: # The Connector implementation is not provide with any *ordering guarantees* of Records. # The decision values giving flexibility to the caller of the method over the implementer of the method. was: Detailed in KIP-502 The key class in a Sink Connector is the `SinkTask` implementation. The `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the Connector-specific code through this entrypoint, and its current abstract class specifies: public abstract void put(Collection records); Usually, the implementer will use a for-each loop to iterate through the records one by one and act on them. There is only one caller, WorkerSinkTask which always passes an ArrayList. There are two shortcomings with this signature: # The Connector implementation is not provide with any *ordering guarantees* of Records. # The decision values giving flexibility to the caller of the method over the implementer of the method. > Connect SinkTask.put(...) to specify ArrayList in Signature > --- > > Key: KAFKA-8749 > URL: https://issues.apache.org/jira/browse/KAFKA-8749 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > > [ > Detailed in KIP-502 > > > |https://cwiki.apache.org/confluence/display/KAFKA/KIP-502%3A+Connect+SinkTask.put%28...%29+to+specify+ArrayList%3CSinkRecord%3E+in+Signature] > The key class in a Sink Connector is the `SinkTask` implementation. The > `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the > Connector-specific code through this entrypoint, and its current abstract > class specifies: > public abstract void put(Collection records); > Usually, the implementer will use a for-each loop to iterate through the > records one by one and act on them. There is only one caller, WorkerSinkTask > which always passes an ArrayList. > There are two shortcomings with this signature: > # The Connector implementation is not provide with any *ordering guarantees* > of Records. > # The decision values giving flexibility to the caller of the method over > the implementer of the method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KAFKA-8749) Connect SinkTask.put(...) to specify ArrayList in Signature
[ https://issues.apache.org/jira/browse/KAFKA-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8749: -- Description: Detailed in [KIP-502|https://cwiki.apache.org/confluence/display/KAFKA/KIP-502%3A+Connect+SinkTask.put%28...%29+to+specify+ArrayList%3CSinkRecord%3E+in+Signature] The key class in a Sink Connector is the `SinkTask` implementation. The `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the Connector-specific code through this entrypoint, and its current abstract class specifies: public abstract void put(Collection records); Usually, the implementer will use a for-each loop to iterate through the records one by one and act on them. There is only one caller, WorkerSinkTask which always passes an ArrayList. There are two shortcomings with this signature: # The Connector implementation is not provide with any *ordering guarantees* of Records. # The decision values giving flexibility to the caller of the method over the implementer of the method. was: [ Detailed in KIP-502 |https://cwiki.apache.org/confluence/display/KAFKA/KIP-502%3A+Connect+SinkTask.put%28...%29+to+specify+ArrayList%3CSinkRecord%3E+in+Signature] The key class in a Sink Connector is the `SinkTask` implementation. The `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the Connector-specific code through this entrypoint, and its current abstract class specifies: public abstract void put(Collection records); Usually, the implementer will use a for-each loop to iterate through the records one by one and act on them. There is only one caller, WorkerSinkTask which always passes an ArrayList. There are two shortcomings with this signature: # The Connector implementation is not provide with any *ordering guarantees* of Records. # The decision values giving flexibility to the caller of the method over the implementer of the method. > Connect SinkTask.put(...) to specify ArrayList in Signature > --- > > Key: KAFKA-8749 > URL: https://issues.apache.org/jira/browse/KAFKA-8749 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > > Detailed in > [KIP-502|https://cwiki.apache.org/confluence/display/KAFKA/KIP-502%3A+Connect+SinkTask.put%28...%29+to+specify+ArrayList%3CSinkRecord%3E+in+Signature] > > The key class in a Sink Connector is the `SinkTask` implementation. The > `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the > Connector-specific code through this entrypoint, and its current abstract > class specifies: > public abstract void put(Collection records); > Usually, the implementer will use a for-each loop to iterate through the > records one by one and act on them. There is only one caller, WorkerSinkTask > which always passes an ArrayList. > There are two shortcomings with this signature: > # The Connector implementation is not provide with any *ordering guarantees* > of Records. > # The decision values giving flexibility to the caller of the method over > the implementer of the method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8749) Connect SinkTask.put(...) to specify ArrayList in Signature
Cyrus Vafadari created KAFKA-8749: - Summary: Connect SinkTask.put(...) to specify ArrayList in Signature Key: KAFKA-8749 URL: https://issues.apache.org/jira/browse/KAFKA-8749 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Cyrus Vafadari Detailed in KIP-502 The key class in a Sink Connector is the `SinkTask` implementation. The `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the Connector-specific code through this entrypoint, and its current abstract class specifies: public abstract void put(Collection records); Usually, the implementer will use a for-each loop to iterate through the records one by one and act on them. There is only one caller, WorkerSinkTask which always passes an ArrayList. There are two shortcomings with this signature: # The Connector implementation is not provide with any *ordering guarantees* of Records. # The decision values giving flexibility to the caller of the method over the implementer of the method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KAFKA-8694) Connect REST Endpoint for Transformations (SMTs) and other Plugins
[ https://issues.apache.org/jira/browse/KAFKA-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8694: -- Description: [KIP-494|https://cwiki.apache.org/confluence/display/KAFKA/KIP-494%3A+Connect+REST+Endpoint+for+Transformations+%28SMTs%29+and+other+Plugins] Proposes to add REST endpoints to Connect workers to enable them to return plugins of all types, not just Connector Plugins, as implemented currently by ConnectorPluginsResource. This will be an update to the REST API of the Connect Worker. was:KIP-494 Proposes to add REST endpoints to Connect workers to enable them to return plugins of all types, not just Connector Plugins, as implemented currently by ConnectorPluginsResource > Connect REST Endpoint for Transformations (SMTs) and other Plugins > -- > > Key: KAFKA-8694 > URL: https://issues.apache.org/jira/browse/KAFKA-8694 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > Labels: needs-kip > > [KIP-494|https://cwiki.apache.org/confluence/display/KAFKA/KIP-494%3A+Connect+REST+Endpoint+for+Transformations+%28SMTs%29+and+other+Plugins] > Proposes to add REST endpoints to Connect workers to enable them to return > plugins of all types, not just Connector Plugins, as implemented currently by > ConnectorPluginsResource. > This will be an update to the REST API of the Connect Worker. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (KAFKA-8694) Connect REST Endpoint for Transformations (SMTs) and other Plugins
[ https://issues.apache.org/jira/browse/KAFKA-8694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891421#comment-16891421 ] Cyrus Vafadari commented on KAFKA-8694: --- [~kkonstantine], these aren't the same improvement: KAFKA-5012 is about indexing plugins within the connect worker, and this ticket KAFKA-8694 is about REST APIs and updating REST API to support more types of plugin than just Connector type. I will update the description in this ticket to have more information and a link to the KIP However, I do now see that KAFKA-5012 does trivialize KAFKA-8605, which is a very simple corollary. > Connect REST Endpoint for Transformations (SMTs) and other Plugins > -- > > Key: KAFKA-8694 > URL: https://issues.apache.org/jira/browse/KAFKA-8694 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Cyrus Vafadari >Priority: Major > Labels: needs-kip > > KIP-494 Proposes to add REST endpoints to Connect workers to enable them to > return plugins of all types, not just Connector Plugins, as implemented > currently by ConnectorPluginsResource -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8694) Connect REST Endpoint for Transformations (SMTs) and other Plugins
Cyrus Vafadari created KAFKA-8694: - Summary: Connect REST Endpoint for Transformations (SMTs) and other Plugins Key: KAFKA-8694 URL: https://issues.apache.org/jira/browse/KAFKA-8694 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Cyrus Vafadari KIP-494 Proposes to add REST endpoints to Connect workers to enable them to return plugins of all types, not just Connector Plugins, as implemented currently by ConnectorPluginsResource -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8605) Warn users when they have same connector in their plugin-path more than once
Cyrus Vafadari created KAFKA-8605: - Summary: Warn users when they have same connector in their plugin-path more than once Key: KAFKA-8605 URL: https://issues.apache.org/jira/browse/KAFKA-8605 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Cyrus Vafadari Right now it is very easy to have multiple copies of the same connector in the plugin-path and not realize it. This can be problematic if a user is adding dependencies into the plugin, or accidentally using the wrong version of the connector. An unintrusive improvement would be to log a warning if the same connector appears in the plugin-path more than once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8447) New Metric to Measure Number of Tasks on a Connector
[ https://issues.apache.org/jira/browse/KAFKA-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyrus Vafadari updated KAFKA-8447: -- Summary: New Metric to Measure Number of Tasks on a Connector (was: Add Connector-level metric to count tasks) > New Metric to Measure Number of Tasks on a Connector > > > Key: KAFKA-8447 > URL: https://issues.apache.org/jira/browse/KAFKA-8447 > Project: Kafka > Issue Type: New Feature >Reporter: Cyrus Vafadari >Priority: Major > > KIP-475 > Worker-level metrics expose number of tasks on a worker, but for many > applications it is useful to have metrics on how many tasks each connector > has. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8447) Add Connector-level metric to count tasks
Cyrus Vafadari created KAFKA-8447: - Summary: Add Connector-level metric to count tasks Key: KAFKA-8447 URL: https://issues.apache.org/jira/browse/KAFKA-8447 Project: Kafka Issue Type: New Feature Reporter: Cyrus Vafadari KIP-475 Worker-level metrics expose number of tasks on a worker, but for many applications it is useful to have metrics on how many tasks each connector has. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6788) Grouping consumer requests per consumer coordinator in admin client
[ https://issues.apache.org/jira/browse/KAFKA-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530064#comment-16530064 ] Cyrus Vafadari commented on KAFKA-6788: --- [~guozhang] I abandoned it a while ago when it didn't get attention, so haven't actively worked on it in a while. I think if the code is moving towards using futures/promise it might be better to start from scratch. If you like this strategy though I'll fix merge conflicts and reopen the PR > Grouping consumer requests per consumer coordinator in admin client > --- > > Key: KAFKA-6788 > URL: https://issues.apache.org/jira/browse/KAFKA-6788 > Project: Kafka > Issue Type: Improvement > Components: admin >Reporter: Guozhang Wang >Assignee: Yishun Guan >Priority: Major > Labels: newbie++ > > In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we > will first try to get the coordinator for each requested group id, and then > send the corresponding request for that group id. However, different group > ids could be hosted on the same coordinator, and these requests do support > multi group ids be sent within the same request. So we can consider optimize > it by grouping the requests per coordinator destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
