[
https://issues.apache.org/jira/browse/METRON-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939117#comment-15939117
]
ASF GitHub Bot commented on METRON-322:
---------------------------------------
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/481#discussion_r107766918
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/IndexingConfigurations.java
---
@@ -73,7 +74,30 @@ public boolean isDefault(String sensorName, String
writerName) {
}
public int getBatchSize(String sensorName, String writerName ) {
- return getBatchSize(getSensorIndexingConfig(sensorName, writerName));
+ return getBatchSize(getSensorIndexingConfig(sensorName, writerName));
+ }
+
+ public int getBatchTimeout(String sensorName, String writerName ) {
+ return getBatchTimeout(getSensorIndexingConfig(sensorName,
writerName));
+ }
+
+ public List<Integer> getAllConfiguredTimeouts(String writerName) {
+ //The configuration infrastructure was not designed to enumerate
sensors, so we synthesize.
+ //The use of SENSOR_MARKER_STRING assumes the config key for a sensor
name consists of a
+ //prefix string (dependent on the ConfigurationType) followed by the
sensor name.
+ //This then allows us to determine the keyPrefixString, and thence
which of all the
+ //configurations.keySet are actually sensor keys. Then we look under
those for timeouts.
+ String markerKeyString = getKey(SENSOR_MARKER_STRING);
--- End diff --
Could you perhaps go a bit into why you're using this
`SENSOR_MARKER_STRING` again? I read the comment and I am still not entirely
sure. I think, at a high level, what you're trying to do is, for a given
writer (e.g. `hdfs`) you want a list of all of the configured timeouts across
all sensors, right?
> Global Batching and Flushing
> ----------------------------
>
> Key: METRON-322
> URL: https://issues.apache.org/jira/browse/METRON-322
> Project: Metron
> Issue Type: Improvement
> Reporter: Ajay Yadav
> Assignee: Matt Foley
>
> All Writers and other bolts that maintain an internal "batch" queue, need to
> have a timeout flush, to prevent messages from low-volume telemetries from
> sitting in their queues indefinitely. Storm has a timeout value
> (topology.message.timeout.secs) that prevents it from waiting for too long.
> If the Writer does not process the queue before the timeout, then Storm
> recycles the tuples through the topology. This has multiple undesirable
> consequences, including data duplication and waste of compute resources. We
> would like to be able to specify an interval after which the queues would
> flush, even if the batch size is not met.
> We will utilize the Storm Tick Tuple to trigger timeout flushing, following
> the recommendations of the article at
> http://hortonworks.com/blog/apache-storm-design-pattern-micro-batching/#CONCLUSION
> Since every Writer processes its queue somewhat differently, every bolt that
> has a "batchSize" parameter will be given a "batchTimeout" parameter too. It
> will default to 1/2 the value of "topology.message.timeout.secs", as
> recommended, and will ignore settings larger than the default, which could
> cause failure to flush in time. In the Enrichment topology, where two
> Writers may be placed one after the other (enrichment and threat intel), the
> default timeout interval will be 1/4 the value of
> "topology.message.timeout.secs". The default value of
> "topology.message.timeout.secs" in Storm is 30 seconds.
> In addition, Storm provides a limit on the number of pending messages that
> have not been acked. If more than "topology.max.spout.pending" messages are
> waiting in a topology, then Storm will recycle them through the topology.
> However, the default value of "topology.max.spout.pending" is null, and if
> set to non-null value, the user can manage the consequences by setting
> batchSize limits appropriately. Having the timeout flush will also
> ameliorate this issue. So we do not need to address
> "topology.max.spout.pending" directly in this task.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
