[jira] [Created] (KAFKA-12259) Consolidated Status endpoint return 500 whe config provider can't find a config
Magesh kumar Nandakumar created KAFKA-12259: --- Summary: Consolidated Status endpoint return 500 whe config provider can't find a config Key: KAFKA-12259 URL: https://issues.apache.org/jira/browse/KAFKA-12259 Project: Kafka Issue Type: Bug Reporter: Magesh kumar Nandakumar The consolidated connectors endpoint connectors?expand=status return a `500` error when any of the connector's has an exception from the config provider. [https://github.com/apache/kafka/blob/e9edf104866822d9e6c3b637ffbf338767b5bf27/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L287] The status endpoint doesn't need the complete list of configs and just requires the Connector Class to infer if it's a `Source` or a `Sink`. The failed connector status should be returned as `Failed` instead of the endpoint returning a 500. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8974) Sink Connectors can't handle topic list with whitespaces
Magesh kumar Nandakumar created KAFKA-8974: -- Summary: Sink Connectors can't handle topic list with whitespaces Key: KAFKA-8974 URL: https://issues.apache.org/jira/browse/KAFKA-8974 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Sink connector allows the user to configure`topics` config which is a comma-separated list of topics. When the user inadvertently includes white spaces in the coman separated list like `topic1, topic2` the connector doesn't process `topic2` correctly. The fix is to trim the individual items before creating the consumer subscribption -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8819) Plugin path for converters not working as intended
Magesh kumar Nandakumar created KAFKA-8819: -- Summary: Plugin path for converters not working as intended Key: KAFKA-8819 URL: https://issues.apache.org/jira/browse/KAFKA-8819 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:- [https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader . This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8352) Connect System Tests are failing with 404
Magesh kumar Nandakumar created KAFKA-8352: -- Summary: Connect System Tests are failing with 404 Key: KAFKA-8352 URL: https://issues.apache.org/jira/browse/KAFKA-8352 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.3.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Fix For: 2.3.0 [https://github.com/apache/kafka/pull/6651] modified the way how connect Rest Server was started. As a result, the connect system tests are failing with a 404 error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8320) Connect Error handling is using the RetriableException from common package
Magesh kumar Nandakumar created KAFKA-8320: -- Summary: Connect Error handling is using the RetriableException from common package Key: KAFKA-8320 URL: https://issues.apache.org/jira/browse/KAFKA-8320 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar When a SourceConnector throws org.apache.kafka.connect.errors.RetriableException during the poll, connect runtime is supposed to ignore the error and retry per [https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect] . When the conenctors throw the execption its not handled gracefully. WorkerSourceTask is catching the execption from wrog package `org.apache.kafka.common.errors`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8312) Rest advertised name is required while setting the listener to 0.0.0.0
Magesh kumar Nandakumar created KAFKA-8312: -- Summary: Rest advertised name is required while setting the listener to 0.0.0.0 Key: KAFKA-8312 URL: https://issues.apache.org/jira/browse/KAFKA-8312 Project: Kafka Issue Type: Improvement Components: KafkaConnect Affects Versions: 1.1.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar [https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface] added the support for SSL and as part of it introduced a listener config. Prior to the change, one could specify not specify a `host.name` config and they would bind to default interface and this worked well for not requiring to configure the `rest.advertised.host.name` explicitly. If one has to use https they could still bind to default interface by not specifying the hostname but when you use 0.0.0.0 as the host in listeners, you are required to set the advertised.host.name explicitly. Connect framework can either use InetAddress.getLocalHost().getCanonicalHostName() as the default value or use the above only when the listeners host is 0.0.0.0. This might be an useful improvement while configuring connectors to listen on all network interfaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8265) Connect Client Config Override policy
Magesh kumar Nandakumar created KAFKA-8265: -- Summary: Connect Client Config Override policy Key: KAFKA-8265 URL: https://issues.apache.org/jira/browse/KAFKA-8265 Project: Kafka Issue Type: New Feature Components: KafkaConnect Reporter: Magesh kumar Nandakumar Right now, each source connector and sink connector inherit their client configurations from the worker properties. Within the worker properties, all configurations that have a prefix of "producer." or "consumer." are applied to all source connectors and sink connectors respectively. We should also provide connector-level overrides whereby connector properties that are prefixed with "producer." and "consumer." are used to feed into the producer and consumer clients embedded within source and sink connectors respectively. The prefixes will be removed via a String#substring() call, and the remainder of the connector property key will be used as the client configuration key. The value is fed directly to the client as the configuration value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7826) Connect Standalone System tests are failing
Magesh kumar Nandakumar created KAFKA-7826: -- Summary: Connect Standalone System tests are failing Key: KAFKA-7826 URL: https://issues.apache.org/jira/browse/KAFKA-7826 Project: Kafka Issue Type: Bug Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Connect standalone tests are failing since the rest servers are not started. The refactor done to add integration tests removed the rest start effectively affecting the standalone mode -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7686) Remove PowerMock from Connect Tests
Magesh kumar Nandakumar created KAFKA-7686: -- Summary: Remove PowerMock from Connect Tests Key: KAFKA-7686 URL: https://issues.apache.org/jira/browse/KAFKA-7686 Project: Kafka Issue Type: Task Components: KafkaConnect Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Remove PowerMock from Connect Tests -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7594) Make the time out for connect service to be dynamic
Magesh kumar Nandakumar created KAFKA-7594: -- Summary: Make the time out for connect service to be dynamic Key: KAFKA-7594 URL: https://issues.apache.org/jira/browse/KAFKA-7594 Project: Kafka Issue Type: Test Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Currently, the timeout for Connect service to be started in the ducktape tests are hardcoded at 60 sec. The proposal is to make it configurable via the service init. The default would still be 60 seconds. This will allow downstream connectors to pass their own timeout values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7551) Refactor to create both producer & consumer in Worker
Magesh kumar Nandakumar created KAFKA-7551: -- Summary: Refactor to create both producer & consumer in Worker Key: KAFKA-7551 URL: https://issues.apache.org/jira/browse/KAFKA-7551 Project: Kafka Issue Type: Task Components: KafkaConnect Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Fix For: 2.2.0 In distributed mode, the producer is created in the Worker and the consumer is created in the WorkerSinkTask. The proposal is to refactor it so that both of them are created in Worker. This will not affect any functionality and is just a refactoring to make the code consistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7068) ConfigTransformer doesn't handle null values
Magesh kumar Nandakumar created KAFKA-7068: -- Summary: ConfigTransformer doesn't handle null values Key: KAFKA-7068 URL: https://issues.apache.org/jira/browse/KAFKA-7068 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Magesh kumar Nandakumar Fix For: 2.0.0 ConfigTransformer fails with NPE when the input configs have keys with null values. This is a blocker for 2.0.0 since connectors configs can have null values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7067) ConnectRestApiTest fails assertion
Magesh kumar Nandakumar created KAFKA-7067: -- Summary: ConnectRestApiTest fails assertion Key: KAFKA-7067 URL: https://issues.apache.org/jira/browse/KAFKA-7067 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Fix For: 2.0.0 ConnectRestApiTest fails assertion for the test_rest_api. The test needs to be updated to include the new configs added in 2.0 in the expected result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7039) DelegatingClassLoader creates plugin instance even if its not Versioned
Magesh kumar Nandakumar created KAFKA-7039: -- Summary: DelegatingClassLoader creates plugin instance even if its not Versioned Key: KAFKA-7039 URL: https://issues.apache.org/jira/browse/KAFKA-7039 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Fix For: 2.0.0 The versioned interface was introduced as part of [KIP-285|https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin]. DelegatingClassLoader is now attempting to create an instance of all the plugins, even if it's not required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6997) Kafka run class doesn't exclude test-sources jar
Magesh kumar Nandakumar created KAFKA-6997: -- Summary: Kafka run class doesn't exclude test-sources jar Key: KAFKA-6997 URL: https://issues.apache.org/jira/browse/KAFKA-6997 Project: Kafka Issue Type: Improvement Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Fix For: 2.0.0 kafka-run-class.sh has a flag INCLUDE_TEST_JAR. This doesn't exclude test-sources jar files when the flag is set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6991) Connect Rest Extension Plugin issue with Class Loader
Magesh kumar Nandakumar created KAFKA-6991: -- Summary: Connect Rest Extension Plugin issue with Class Loader Key: KAFKA-6991 URL: https://issues.apache.org/jira/browse/KAFKA-6991 Project: Kafka Issue Type: Improvement Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar Fix For: 2.0.0 KAFKA-6776 implementation has 2 issues with the class loader. # If a RestExtension plugin is available in the classpath, then the plugin gets associated with all the plugin class loaders. This is happening because the Manifest resource is searched in the parent class loader as well by default. # The class name associated with the plugin class loader is incorrect. It should be the implementation class instead of the interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6776) Connect Rest Extension Plugin
Magesh kumar Nandakumar created KAFKA-6776: -- Summary: Connect Rest Extension Plugin Key: KAFKA-6776 URL: https://issues.apache.org/jira/browse/KAFKA-6776 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar This covers the connect Rest extension plugin covered at KIP-285 https://cwiki.apache.org/confluence/display/KAFKA/KIP+285+-+Connect+Rest+Extension+Plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
