Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1082#discussion_r199610213
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
---
@@ -197,6 +209,15 @@ public void init(Map stormConf, TopologyContext
topologyContext, WriterConfigura
return producerConfig;
}
+ public Optional<String> getKafkaTopic(JSONObject message) {
--- End diff --
It's unclear to me that we really want to do that. It could be that users
are going to have logic that depends on that field downstream. It seems wrong
to me to remove a field that they're adding or have in the message.
I mean, in order to create a loop, one of two things would have to happen:
* the user specifies the input topic as the output topic (aka a simple
loop), which removing the kafka topic field wouldn't help because it is likely
to be computed (e.g. in the manual test script)
* the user creates a non-simple loop where by sensor A -> B -> ... C -> A,
but it's exceedingly unlikely that they're all going to be parsers of the same
type, so messages from C will unlikely fail to parse in A. In the case that
they do, it's likely that the kafka topic will be computed in first parser, so
it'll be recomputed and removing the field after the first parse won't have
helped.
TL;DR
Ultimately, I think removing the field won't appreciably help the situation
and puts us into the state of removing data, which makes me uncomfortable.
---
