SEPURI-SAI-KRISHNA opened a new issue, #12926: URL: https://github.com/apache/seatunnel/issues/12926
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened A Kafka tombstone (a record with a key and a `null` value, which compacted topics use to mark a key deleted) fails an Avro Kafka source with a raw `NullPointerException`. Setting `format_error_handle_way = skip` does not help, because the skip handler itself throws a second `NullPointerException` on the same null value. The offset is never advanced past the record, so the tombstone is replayed after every restart. ### Measured on `dev` Driving `KafkaRecordEmitter.emitRecord` with a real tombstone (`value = null`), split offset starting at 42: | format | `format_error_handle_way` | outcome | offset after | | --- | --- | --- | --- | | json | fail | returned normally, 0 rows | 43 | | json | skip | returned normally, 0 rows | 43 | | avro | fail | `NullPointerException: Cannot read the array length because "message" is null` | **42** | | avro | skip | `NullPointerException: ... because "bytes" is null` | **42** | The two exception messages name different variables, which separates two independent defects. ### Defect 1: the Avro format has no tombstone guard `AvroDeserializationSchema.java:72` ```java return deserializePayload(message, 0, message.length); ``` `message.length` throws when the value is null. Every other SeaTunnel format guards this, several with that exact comment: | format | guards a null value | | --- | --- | | json, text, csv, ogg, canal, debezium, maxwell | yes | | **avro** | **no** | For example `DebeziumJsonDeserializationSchema`: ```java if (message == null || message.length == 0) { // skip tombstone messages return; } ``` ### Defect 2: the skip handler cannot skip a record with no value `KafkaRecordEmitter.java:100` ```java if (this.messageFormatErrorHandleWay == MessageFormatErrorHandleWay.SKIP) { logger.warn( "Deserialize message failed, skip this message, message: {}", new String(consumerRecord.value())); } ``` `new String((byte[]) null)` throws `NullPointerException`. That escapes `emitRecord`, so `splitState.setCurrentOffset(consumerRecord.offset() + 1)` on line 107 is never reached. This is reachable from any format that raises on a null value, not only Avro. The documented contract for the option is: "When skip is selected, data format error will skip this line data." ### Why no test catches it `KafkaRecordEmitterTest` has no case with a null value, and the Avro format tests never deserialize one. The existing E2E `avro/kafka_avro_to_assert.conf` does run Avro with `format_error_handle_way = skip`, but it only ever produces well formed records. ### What you expected to happen A tombstone is a normal Kafka record, not a format error. It should produce no row, the offset should advance past it, and the job should continue, which is what the other seven formats already do. ### SeaTunnel Version `dev` (verified against the live files at `?ref=dev`; both lines are unchanged). ### Note Closed #6795 is the same option failing for a different reason: `skip` was not wired up at all for a `NumberFormatException` in the text format. #7838 fixed that by adding the try/catch, and introduced the `new String(consumerRecord.value())` line that this report is about. So the two are complementary rather than duplicates. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
