GitHub user shibd added a comment to the discussion: Should Kafka Source Connector itsself after unrecoverable error?
> end the process which would in turn result in another pod being spun up if > this is a scenario where the connector is deployed in kubernetes In the Kubernetes runtime, It should be this behavior. > However I feel like the connector just sitting there and not consuming > messages from Kafka anymore is the wrong behavior. Am I missing a config > value? In pulsar-io-framework, if a connector throws an exception, it will trigger this process exit. So, maybe the Kafka source connector does not throw an exception on your case. My guess is the logic handled here: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java#L62-L64 I found a way to compare hacks to get the connector to throw exceptions. Maybe you can give it a try: set your `processingGuarantees = EFFECTIVELY_ONCE` https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java#L153-L156 Can you share the logs of your Kafka source connector? Maybe we can optimize for this scenario(let it throw an exception to trigger connector process exit). GitHub link: https://github.com/apache/pulsar/discussions/19880#discussioncomment-5925126 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
