hutiefang76 opened a new pull request, #12900: URL: https://github.com/apache/seatunnel/pull/12900
### Purpose of this pull request Yes, contributing this fix to Apache SeaTunnel. Fixes #12879. The derived Avro schema currently represents a ROW as a bare record, so serializing a null ROW fails. This adds an opt-in `avro_row_nullable` setting and wires it through Kafka and Pulsar sources and sinks. It covers null ROW fields, nested ROW fields, ARRAY elements and MAP values. ### Does this PR introduce _any_ user-facing change? Yes. Setting `avro_row_nullable = true` derives nullable Avro unions for ROW values. The default remains `false`: existing constructors, derived schemas and legacy bytes retain their current behavior. Kafka's explicitly supplied `avro_schema` remains authoritative. The writer and reader must agree on the layout. This does not detect or migrate existing raw Avro messages. The EN/ZH format and connector documentation explains that boundary and provides separate writer and reader examples. Kafka per-table configurations must repeat the setting; Pulsar uses its existing global/per-table configuration merge. ### How was this patch tested? JDK 17: * `./mvnw spotless:apply` and root `./mvnw -q -DskipTests verify` passed. * All 18 Avro module tests passed, including null ROW/ARRAY/MAP values, deeper nesting, explicit writer-schema precedence and legacy/default byte equality. The original implementation failed the three null-ROW reproductions. * All 204 Kafka and Pulsar connector unit tests passed. No failures, errors or skips in these affected suites. Real broker tests used the official E2E suites with the Zeta engine and its JDK 11 runtime. Each suite ran the two existing default-layout Avro tests and the new nullable source-to-sink round trip: 3 Kafka tests and 3 Pulsar tests, all passed with no skips. The new round trips assert three messages containing a null outer ROW, a null inner ROW and a fully present nested ROW. ```sh ./mvnw -B -pl seatunnel-formats/seatunnel-format-avro -Dtest=Avro*Test test ./mvnw -B -pl seatunnel-connectors-v2/connector-kafka,seatunnel-connectors-v2/connector-pulsar test RUN_ENGINES=seatunnel ./mvnw -B -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e -am -DskipUT=true -DskipIT=false '-Dit.test=KafkaIT#testKafkaAvroNullableRowRoundTrip+testFakeSourceToKafkaAvroFormat+testKafkaAvroToAssert' -DfailIfNoTests=false -Dit.failIfNoSpecifiedTests=false verify RUN_ENGINES=seatunnel ./mvnw -B -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-pulsar-e2e -am -DskipUT=true -DskipIT=false '-Dit.test=PulsarFormatIT#testPulsarAvroNullableRowRoundTrip+testFakeSourceToPulsarAvroFormat+testPulsarAvroToAssert' -DfailIfNoTests=false -Dit.failIfNoSpecifiedTests=false verify ``` The connector unit command used previously installed reactor dependencies. Broker images were Confluent Kafka 7.0.9 and Pulsar 2.3.1. The Pulsar fixture uses the existing `tables_configs`/`table_path` mapping to route to its output topic; it does not change sink routing. ARRAY/MAP coverage is in the format unit tests. Flink and Spark E2E runs were not performed. A broader dependency-reactor unit run hit the unchanged Protobuf module's missing native protoc 3.11.4 binary on macOS ARM64, tracked separately in #12897. That run is not claimed as passing; the affected suites and broker tests above were executed separately. AI assistance was used for implementation and review. The validation results above are from executed builds and tests. ### Check list * [x] No new JAR dependency or binary package is added. * [x] EN/ZH documentation describes the setting and wire-layout boundary. * [x] Existing default behavior is preserved; no incompatible default change is introduced. * [x] Existing connector tests and official E2E suites are extended; no new connector registration is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
