vbhanuchander-lang opened a new pull request, #7747: URL: https://github.com/apache/hop/pull/7747
Addresses #2679. The Kafka producer publishes every row to one configured topic, so routing rows to different topics today means adding one producer transform per topic. This adds a **Get topic from field** option. When enabled, the topic is read from a field on each incoming row, so a single transform can fan rows out across topics. When disabled — the default, and what every existing pipeline gets — the transform behaves exactly as before. ### Naming The option follows the convention already used for the same idea in Table Output, which pairs a `tableNameInField` flag with a `tableNameField` name. Here that is `topicInField` / `topicField`, so the metadata and injection keys should read as familiar. ### Why one producer still suffices The topic is only ever part of the `ProducerRecord`, never of the producer configuration — `KafkaFactory.producer()` sets bootstrap servers, client id and serializers, and never looks at the topic. So a per-row topic needs no extra producer instances and opens no extra connections. ### Validation Field resolution happens once, when the first row arrives: - option enabled but no field configured → transform fails with a clear message - configured field not present in the input stream → transform fails naming the field - a row whose topic field is empty → fails naming the field and the line number Failing at first-row rather than at send time means a misconfiguration surfaces immediately instead of after a partial publish. The dialog disables whichever of the two topic widgets does not apply, and validates the field against the incoming stream before accepting the change, matching how the existing key/message field checks behave. ### Backward compatibility Pipelines saved before this change carry no `topicInField` element, so it deserializes to `false` and the fixed topic name keeps being used. There is a test pinning that default. ### Tests `KafkaProducerOutputMetaTest` gains coverage for the serialization round trip of both new properties, the off-by-default guarantee for older pipelines, and the copy constructor. The full module suite passes: 17 tests, 0 failures. Verified with `mvn -pl plugins/transforms/kafka test`, plus `spotless:check` and `apache-rat:check` (Unapproved: 0). ### Documentation `docs/.../kafkaproducer.adoc` is updated with both new options and a note that the fixed Topic field is disabled when the topic comes from a field. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. *(ran the module build plus `spotless:check` and `apache-rat:check`; the full `clean install` was not run locally)* - [x] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`. - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) you have to acknowledge this by using the following check-box. - [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) - [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
