aryash45 opened a new pull request, #15639: URL: https://github.com/apache/iceberg/pull/15639
When iceberg.tables.route-field is not set, records from all topics were being written to all tables if they shared the same id-columns name and similar schemas. This was due to the routing logic broadcasting to all tables instead of matching topics to table names. This change implements topic-based routing by: - Extracting the last dot-delimited segment from both topic and table names - Matching them case-insensitively - Only writing to matching tables - Falling back to broadcast only if no topic-to-table match is found This preserves backward compatibility while fixing the multi-topic routing issue reported in #15584. Changes: - SinkWriter.java: Added topic-based routing in routeRecordStatically() - Added lastSegment() helper method - TestSinkWriter.java: Added testTopicRoute(), testTopicRouteSecondTable(), and testTopicRouteFallbackBroadcast() test cases Questions For Maintainers -Should unmatched topic fallback log a warning to alert users of potential misconfiguration? - Is topic-based routing the officially recommended approach for multi-topic setups? - Should this routing behavior be documented in kafka-connect.md Closes #15584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
