PDGGK opened a new pull request, #11022:
URL: https://github.com/apache/paimon/pull/11022

   ### Purpose
   
   Fixes #5092.
   
   With Maxwell's `output_ddl` enabled, DDL events such as `table-alter` and 
`table-create` go to the same Kafka topic as the data changes, unless 
`ddl_kafka_topic` is set. These events carry `def`, `old` and `sql`, and have 
no `data` field. `MaxwellRecordParser.extractRecords` therefore throws `Invalid 
maxwell-json format: missing 'data' field.`. The Kafka sync job fails, and 
after a restart it fails again on the same offset. Schema discovery at job 
start (`buildSchema`) fails the same way when it reaches such an event.
   
   The canal-json, aliyun-json and aws-dms-json parsers already skip their DDL 
records through `isDDL()`. This PR does the same for Maxwell, using the six DDL 
event types Maxwell emits: `database-create`, `database-alter`, 
`database-drop`, `table-create`, `table-alter` and `table-drop`. `buildSchema` 
returns null for them, so schema discovery moves on to the next record, and 
`extractRecords` returns no records.
   
   The reporter's earlier PR #5099 caught every `RuntimeException` around the 
`data` lookup. It was closed by the stale bot without review. Checking the 
event type instead keeps malformed data events failing as before.
   
   ### Tests
   
   New `MaxwellRecordParserTest#testSkipDdlEvents`, parameterized over the six 
DDL types. For each one, `buildSchema` returns null and `flatMap` emits 
nothing. An insert event parsed afterwards is still emitted.
   
   - `mvn -pl paimon-flink/paimon-flink-cdc -am -Pflink1,fast-build 
-Dtest=MaxwellRecordParserTest test` on JDK 8: 6 tests pass.
   - With only the change to `MaxwellRecordParser` reverted, all 6 fail with 
`Invalid maxwell-json format: missing 'data' field.`, the error from the issue.
   - `spotless:check` and `checkstyle:check` pass for `paimon-flink-cdc`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to