tonisojandu commented on issue #20635: URL: https://github.com/apache/pulsar/issues/20635#issuecomment-1709616321
I think we are facing the same problem. [This issue might also be related](https://github.com/tonisojandu/pulsar-dlq-schema-issue-example). We face this issue from time to time, when due to engineer, we encounter and schema incompatibility between the received messages and the one consumer expects. We have disabled schema validations on broker side by choice. However, rather than just sending those messages to DLQ, Pulsar client side fails validation and goes into a producer creating loop. [Here is the example code to replicate the issue](https://github.com/tonisojandu/pulsar-dlq-schema-issue-example). We had a look at the client code and the problem seems to be in `AutoProduceBytesSchema.encode(byte[] message)` method since the `requireSchemaValidation` is flipped to true in the constructor, even though broker side validation is disabled. We were able to "fix" this problem when in `ConsumerImpl.initDeadLetterProducerIfNeeded()` instead of : ```java ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema))) ``` we created the producer with as: ```java ((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.BYTES)) ``` and in `ConsumerImpl.processPossibleToDLQ(MessageIdAdv messageId)` instead of ``` producerDLQ.newMessage(Schema.AUTO_PRODUCE_BYTES(message.getReaderSchema().get())) ``` we sent the message with ``` producerDLQ.newMessage(Schema.AUTO_PRODUCE_BYTES(Schema.BYTES)) ``` I am mot sure if this is proper way to fix it, as it leaves the DLQ with binary schema in registry. This would be fine for us, but not sure if someone else is relying on it having a more descriptive schema. However, I would like to imagine DLQ as a dumping ground that should be able to accept all types of messages. If this is an OK fix, I can create a pull request for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
