[
https://issues.apache.org/jira/browse/METRON-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078136#comment-16078136
]
ASF GitHub Bot commented on METRON-1001:
----------------------------------------
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/621#discussion_r126156447
--- Diff:
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
---
@@ -176,16 +182,48 @@ public SimpleStormKafkaBuilder( Map<String, Object>
kafkaProps
, List<String> fieldsConfiguration
)
{
+ this(kafkaProps, toSubscription(topic), zkQuorum, fieldsConfiguration);
+ }
+
+ /**
+ * Create an object with the specified properties and exposing the
specified fields.
+ * @param kafkaProps The special kafka properties
+ * @param subscription The subscription to the kafka topic(s)
+ * @param zkQuorum The zookeeper quorum. We will use this to pull the
brokers from this.
+ * @param fieldsConfiguration The fields to expose in the storm tuple
emitted.
+ */
+ public SimpleStormKafkaBuilder( Map<String, Object> kafkaProps
+ , Subscription subscription
+ , String zkQuorum
+ , List<String> fieldsConfiguration
+ )
+ {
super( getBootstrapServers(zkQuorum, kafkaProps)
,
createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)),
DEFAULT_DESERIALIZER)
,
createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)),
DEFAULT_DESERIALIZER)
- , topic
+ , subscription
);
setProp(kafkaProps);
setRecordTranslator(new
SpoutRecordTranslator<>(FieldsConfiguration.toList(fieldsConfiguration)));
- this.topic = topic;
}
+
+ private static Subscription toSubscription(String topicOrSubscription) {
+ if (StringUtils.isEmpty(topicOrSubscription)) {
+ throw new IllegalStateException("Topic name is invalid: empty or
null");
--- End diff --
Could you make this IllegalArgumentException? It also might be nice to
include the value itself, because empty vs null seem like they could come up in
different ways.
> Allow metron to ingest parser metadata along with data
> ------------------------------------------------------
>
> Key: METRON-1001
> URL: https://issues.apache.org/jira/browse/METRON-1001
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
> Assignee: Casey Stella
>
> Currently, we only ingest data in Metron. Often, there is valuable metadata
> constructed up-stream of Metron that is relevant to enrichment and cross-cuts
> many data formats. Take, for instance, a multi-tenancy case where multiple
> sources come in and you'd like to tag the data with the customer ID. In this
> case you're stuck finding ways to add the metadata to each data source's
> format. Rather than do that, we should allow metadata to be ingested along
> with the data associated with it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
