deepakpanda93 commented on issue #16104: URL: https://github.com/apache/hudi/issues/16104#issuecomment-4854986222
This is expected behavior given the source configuration, and it's resolved via config rather than a code change (mirroring the resolution on the original [#9143]). Why it happens: with Postgres REPLICA IDENTITY DEFAULT, Debezium DELETE events include only the primary key; the other columns (including your partition field) arrive as null/0. The timestamp keygen then converts 0 → 1970-01-01, so Hudi tries to delete from the wrong partition and the delete misses. A local (partition-scoped) index can't help here because the delete event doesn't carry the real partition value. Recommended fix — use a global index: - Set hoodie.index.type=GLOBAL_BLOOM (or another global index). It maintains uniqueness across partitions, so the delete locates and removes the record regardless of the missing/dummy partition value. Alternative (source-side): - Set Postgres REPLICA IDENTITY FULL (or USING <unique index> on the partition column) so Debezium's before image contains all columns and Hudi derives the correct partition. Note FULL has documented performance downsides on the Postgres side. Since deletes with REPLICA IDENTITY DEFAULT on a partitioned table inherently lack the partition value, a global index is the right approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
