[
https://issues.apache.org/jira/browse/METRON-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613919#comment-15613919
]
Matt Foley edited comment on METRON-516 at 10/28/16 1:41 AM:
-------------------------------------------------------------
There are 10 non-test java files in which "batchSize" or "batchsize" occurs:
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/SensorEnrichmentConfig.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/SensorEnrichmentUpdateConfig.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/EnrichmentWriterConfiguration.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/ParserWriterConfiguration.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/SingleBatchConfigurationFacade.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/WriterConfiguration.java
*
metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/bolt/BatchHelper.java
*
metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/bolt/HBaseBolt.java
*
metron-platform/metron-management/src/main/java/org/apache/metron/management/EnrichmentConfigFunctions.java
*
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
Of these, a Tick Tuple mechanism for batch queue timeout is already implemented
in
*
metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/bolt/BatchHelper.java
(for HBaseBolt)
Tick Tuples are also used in
*
metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java
to delimit the sampling windows of the Profiler. (Its test file is also
affected.)
Currently this code assumes the Tick Tuple receipt interval will be exactly
what this bolt requested. That is only reliable if the bolt is alone in its
topology, or none of the other bolts in the topo utilize Tick Tuples.
Also, the offered configuration for window period is "periodDurationMillis",
but the granularity of Tick Tuples is seconds. It seems that a real timer
thread needs to be introduced here.
was (Author: mattf):
There are 10 non-test java files in which "batchSize" or "batchsize" occurs:
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/SensorEnrichmentConfig.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/SensorEnrichmentUpdateConfig.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/EnrichmentWriterConfiguration.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/ParserWriterConfiguration.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/SingleBatchConfigurationFacade.java
*
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/writer/WriterConfiguration.java
*
metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/bolt/BatchHelper.java
*
metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/bolt/HBaseBolt.java
*
metron-platform/metron-management/src/main/java/org/apache/metron/management/EnrichmentConfigFunctions.java
*
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
Of these, a Tick Tuple mechanism for batch queue timeout is already implemented
in
*
metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/bolt/BatchHelper.java
(for HBaseBolt)
Tick Tuples are also used in
*
metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java
to delimit the sampling windows of the Profiler. Currently this code assumes
the Tick Tuple receipt interval will be exactly what this bolt requested. That
is only reliable if the bolt is alone in its topology, or none of the other
bolts in the topo utilize Tick Tuples.
Also, the offered configuration for window period is "periodDurationMillis",
but the granularity of Tick Tuples is seconds. It seems that a real timer
thread needs to be introduced here.
> Add batchTimeout parameters with every use of batchSize parameters, and set
> topology.tick.tuple.freq.secs appropriately
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: METRON-516
> URL: https://issues.apache.org/jira/browse/METRON-516
> Project: Metron
> Issue Type: Sub-task
> Reporter: Matt Foley
> Assignee: Matt Foley
>
> This is the control mechanism for METRON-322, timeout-based flushing of batch
> queues. The behavior of the batchTimeout parameter is as follows:
> * The default value will be 1/2 the value of "topology.message.timeout.secs",
> minus delta, except in the Enrichment topology where the default value will
> be 1/4 the value of "topology.message.timeout.secs" (due to the possibility
> of having an enricher bolt and a threat intel bolt, both with batch queues,
> daisy-chained one after the other, in the same Enrichment topology).
> ** The default value of "topology.message.timeout.secs" is 30 sec, so the
> general default batchTimeout will be 14 sec, or 7 sec for Enrichment
> topologies.
> * Because there is a good default setting, the user does not need to set
> batchTimeout unless they wish to force more frequent flushing.
> * If the user sets a batchTimeout interval larger than the default based on
> "topology.message.timeout.secs", it will be ignored. You can flush more
> frequently than the default, but not less frequently. If you want less
> frequent flushing, you must set a larger "topology.message.timeout.secs" at
> the Storm level.
> ** Question: should we provide a Metron-level ZK parameter to override the
> Storm-level "topology.message.timeout.secs"?
> * The value of batchTimeout is not a guaranteed flush interval:
> ** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the
> smallest interval used by any of that bolt's queues' batchTimeout.
> ** The actual interval of Tick Tuple receipt by all bolts within a topology
> will be the smallest "topology.tick.tuple.freq.secs" interval specified for
> any bolt in the topology. (This is a Storm behavior.)
> ** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s)
> need flushing, based on whether each queue is actually older then that
> queue's batchTimeout.
> ** Altogether, this means that each queue will flush at an indeterminate time
> when its age is between \(i) the batchTimeout interval for that queue and
> (ii) the sum of that batchTimeout interval plus the actual Tick Tuple receipt
> interval -- unless it flushes first due to batchSize.
> Bolts that use batch queues will be modified to capture the value of
> "topology.message.timeout.secs" and set the value of
> "topology.tick.tuple.freq.secs" in the call to getComponentConfiguration()
> method. Because this apparently only happens at topology startup:
> * If the Metron administrator wishes to change
> "topology.message.timeout.secs" from the default value of 30 sec, she must do
> it in storm.yaml or in the CLI, and *must not attempt to change it* via
> custom java code in topology submission. Attempted changes at topology
> submission time (except in the CLI) will not be seen in time to correctly set
> "topology.tick.tuple.freq.secs" in response.
> * Authors of custom bolts may change "topology.message.timeout.secs" in the
> getComponentConfiguration() method at the same time as setting
> "topology.tick.tuple.freq.secs".
> * "topology.tick.tuple.freq.secs" cannot be changed during runtime; it
> requires restarting the topology (unless there are Storm GUI actions to
> change it in runtime, which has not yet been investigated). As a result,
> attempts during runtime to change batchTimeout to a *smaller* value than set
> at topology startup, will have this result: The affected queue will flush at
> an indeterminate time when its age is between \(i) the new smaller
> batchTimeout interval, and (ii) the sum of the new smaller batchTimeout plus
> the unchanged Tick Tuple receipt interval -- noting that the unchanged Tick
> Tuple receipt interval may be much larger than the new smaller batchTimeout,
> if the revised batchTimeout was set very small.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
