Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1082#discussion_r199290836
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
---
@@ -197,6 +209,15 @@ public void init(Map stormConf, TopologyContext
topologyContext, WriterConfigura
return producerConfig;
}
+ public Optional<String> getKafkaTopic(JSONObject message) {
--- End diff --
I am assuming this logic applies to any topology that uses a `KafkaWriter`.
Would it be easy enough for a user to run into an infinite loop scenario if
they have any two sequential topologies both using a `KafkaWriter`?
Parser -> Stage1 -> Stage2 -> ...
* In the Parser, it ingests a message where the "outputTopic" = "stage1".
* This sends the message to my Stage1 processing
* If the Stage1 logic does not change the value of that field for whatever
reason, then the message will go right back to Stage1 and be reprocessed.
* Wash, rinse, repeat and you've got a mess on your hands that is difficult
to debug.
Maybe I am thinking too hard about this. There may be nothing we can
really do about that. With power comes responsibility. If an advanced user
wants to customize routing, then they need to own this risk.
---
