wombatu-kun commented on PR #19018: URL: https://github.com/apache/hudi/pull/19018#issuecomment-5112120041
Rebased onto current master. On the maintenance cost: `transactionParticipants` is private and has exactly four write sites, all in this one file (constructor, the per-partition open, the per-partition close, and the stop-time clear), and the secondary index mirrors each one directly alongside it. The file has 6 commits in its entire history and its last behavioral change was in 2021; everything since has been mechanical (slf4j unification, logging cleanup, Lombok), so the surface where the invariant could drift is both small and rarely touched. On the gain: `put()` is the only per-record path in this class, and it was allocating a `TopicPartition` per record, which is 24 bytes on a 64-bit JVM with compressed oops (header, cached hash, partition, topic reference). That drops to zero for partition indices below 128, where the `Integer` key hits the autobox cache, and to 16 bytes above it. At Kafka Connect ingest rates that removes a continuous stream of young-gen garbage from the hot loop, in exchange for a strictly bounded amount of extra state. If the drift concern is the blocker, I am happy to add a small unit test pinning the two maps in sync across open, close and stop. There is no test coverage on `HoodieSinkTask` at all today, so that would be a net improvement regardless of how this PR lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
