[
https://issues.apache.org/jira/browse/METRON-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated METRON-516:
------------------------------
Description:
This is the control mechanism for METRON-322, timeout-based flushing of batch
queues. The behavior of the batchTimeout parameter is as follows:
* The default value will be 1/2 the value of "topology.message.timeout.secs",
minus delta, except in the Enrichment topology where the default value will be
1/4 the value of "topology.message.timeout.secs" (due to the possibility of
having an enricher bolt and a threat intel bolt, both with batch queues,
daisy-chained one after the other, in the same Enrichment topology).
** The default value of "topology.message.timeout.secs" is 30 sec, so the
general default batchTimeout will be 14 sec, or 7 sec for Enrichment topologies.
* Because there is a good default setting, the user does not need to set
batchTimeout unless they wish to force more frequent flushing.
* If the user sets a batchTimeout interval larger than the default based on
"topology.message.timeout.secs", it will be ignored. You can flush more
frequently than the default, but not less frequently. If you want less
frequent flushing, you must set a larger "topology.message.timeout.secs" at the
Storm level.
** Question: should we provide a Metron-level ZK parameter to override the
Storm-level "topology.message.timeout.secs"?
* The value of batchTimeout is not a guaranteed flush interval:
** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the smallest
interval used by any of that bolt's queues' batchTimeout.
** -The actual interval of Tick Tuple receipt by all bolts within a topology
will be the smallest "topology.tick.tuple.freq.secs" interval specified for any
bolt in the topology. (This is a Storm behavior.)- (Tentatively striking as
incorrect, but still seeking clarification from Taylor G.)
** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s)
need flushing, based on whether each queue is actually older then that queue's
batchTimeout.
** Altogether, this means that each queue will flush at an indeterminate time
when its age is between \(i) the batchTimeout interval for that queue and (ii)
the sum of that batchTimeout interval plus the actual Tick Tuple receipt
interval -- unless it flushes first due to batchSize.
Bolts that use batch queues will be modified to capture the value of
"topology.message.timeout.secs" and set the value of
"topology.tick.tuple.freq.secs" in the call to getComponentConfiguration()
method. Because this apparently only happens at topology startup:
* If the Metron administrator wishes to change "topology.message.timeout.secs"
from the default value of 30 sec, she must do it in storm.yaml or in the CLI,
and *must not attempt to change it* via custom java code in topology
submission. Attempted changes at topology submission time (except in the CLI)
will not be seen in time to correctly set "topology.tick.tuple.freq.secs" in
response.
* -Authors of custom bolts may change "topology.message.timeout.secs" in the
getComponentConfiguration() method at the same time as setting
"topology.tick.tuple.freq.secs".- (Need to confirm whether this will really
work at the topology level.)
* "topology.tick.tuple.freq.secs" cannot be changed during runtime; it requires
restarting the topology (unless there are Storm GUI actions to change it in
runtime, which has not yet been investigated). As a result, changing
batchTimeout during runtime will have this result: The affected queue will
flush at an indeterminate time when its age is between \(i) the new
batchTimeout interval, and (ii) the sum of the new batchTimeout plus the
unchanged Tick Tuple receipt interval -- noting that the unchanged Tick Tuple
receipt interval may be much larger than the new batchTimeout, if the revised
batchTimeout was made much smaller.
was:
This is the control mechanism for METRON-322, timeout-based flushing of batch
queues. The behavior of the batchTimeout parameter is as follows:
* The default value will be 1/2 the value of "topology.message.timeout.secs",
minus delta, except in the Enrichment topology where the default value will be
1/4 the value of "topology.message.timeout.secs" (due to the possibility of
having an enricher bolt and a threat intel bolt, both with batch queues,
daisy-chained one after the other, in the same Enrichment topology).
** The default value of "topology.message.timeout.secs" is 30 sec, so the
general default batchTimeout will be 14 sec, or 7 sec for Enrichment topologies.
* Because there is a good default setting, the user does not need to set
batchTimeout unless they wish to force more frequent flushing.
* If the user sets a batchTimeout interval larger than the default based on
"topology.message.timeout.secs", it will be ignored. You can flush more
frequently than the default, but not less frequently. If you want less
frequent flushing, you must set a larger "topology.message.timeout.secs" at the
Storm level.
** Question: should we provide a Metron-level ZK parameter to override the
Storm-level "topology.message.timeout.secs"?
* The value of batchTimeout is not a guaranteed flush interval:
** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the smallest
interval used by any of that bolt's queues' batchTimeout.
** The actual interval of Tick Tuple receipt by all bolts within a topology
will be the smallest "topology.tick.tuple.freq.secs" interval specified for any
bolt in the topology. (This is a Storm behavior.)
** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s)
need flushing, based on whether each queue is actually older then that queue's
batchTimeout.
** Altogether, this means that each queue will flush at an indeterminate time
when its age is between \(i) the batchTimeout interval for that queue and (ii)
the sum of that batchTimeout interval plus the actual Tick Tuple receipt
interval -- unless it flushes first due to batchSize.
Bolts that use batch queues will be modified to capture the value of
"topology.message.timeout.secs" and set the value of
"topology.tick.tuple.freq.secs" in the call to getComponentConfiguration()
method. Because this apparently only happens at topology startup:
* If the Metron administrator wishes to change "topology.message.timeout.secs"
from the default value of 30 sec, she must do it in storm.yaml or in the CLI,
and *must not attempt to change it* via custom java code in topology
submission. Attempted changes at topology submission time (except in the CLI)
will not be seen in time to correctly set "topology.tick.tuple.freq.secs" in
response.
* Authors of custom bolts may change "topology.message.timeout.secs" in the
getComponentConfiguration() method at the same time as setting
"topology.tick.tuple.freq.secs".
* "topology.tick.tuple.freq.secs" cannot be changed during runtime; it requires
restarting the topology (unless there are Storm GUI actions to change it in
runtime, which has not yet been investigated). As a result, changing
batchTimeout during runtime will have this result: The affected queue will
flush at an indeterminate time when its age is between \(i) the new
batchTimeout interval, and (ii) the sum of the new batchTimeout plus the
unchanged Tick Tuple receipt interval -- noting that the unchanged Tick Tuple
receipt interval may be much larger than the new batchTimeout, if the revised
batchTimeout was made much smaller.
> Add batchTimeout parameters with every use of batchSize parameters, and set
> topology.tick.tuple.freq.secs appropriately
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: METRON-516
> URL: https://issues.apache.org/jira/browse/METRON-516
> Project: Metron
> Issue Type: Sub-task
> Reporter: Matt Foley
> Assignee: Matt Foley
>
> This is the control mechanism for METRON-322, timeout-based flushing of batch
> queues. The behavior of the batchTimeout parameter is as follows:
> * The default value will be 1/2 the value of "topology.message.timeout.secs",
> minus delta, except in the Enrichment topology where the default value will
> be 1/4 the value of "topology.message.timeout.secs" (due to the possibility
> of having an enricher bolt and a threat intel bolt, both with batch queues,
> daisy-chained one after the other, in the same Enrichment topology).
> ** The default value of "topology.message.timeout.secs" is 30 sec, so the
> general default batchTimeout will be 14 sec, or 7 sec for Enrichment
> topologies.
> * Because there is a good default setting, the user does not need to set
> batchTimeout unless they wish to force more frequent flushing.
> * If the user sets a batchTimeout interval larger than the default based on
> "topology.message.timeout.secs", it will be ignored. You can flush more
> frequently than the default, but not less frequently. If you want less
> frequent flushing, you must set a larger "topology.message.timeout.secs" at
> the Storm level.
> ** Question: should we provide a Metron-level ZK parameter to override the
> Storm-level "topology.message.timeout.secs"?
> * The value of batchTimeout is not a guaranteed flush interval:
> ** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the
> smallest interval used by any of that bolt's queues' batchTimeout.
> ** -The actual interval of Tick Tuple receipt by all bolts within a topology
> will be the smallest "topology.tick.tuple.freq.secs" interval specified for
> any bolt in the topology. (This is a Storm behavior.)- (Tentatively striking
> as incorrect, but still seeking clarification from Taylor G.)
> ** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s)
> need flushing, based on whether each queue is actually older then that
> queue's batchTimeout.
> ** Altogether, this means that each queue will flush at an indeterminate time
> when its age is between \(i) the batchTimeout interval for that queue and
> (ii) the sum of that batchTimeout interval plus the actual Tick Tuple receipt
> interval -- unless it flushes first due to batchSize.
> Bolts that use batch queues will be modified to capture the value of
> "topology.message.timeout.secs" and set the value of
> "topology.tick.tuple.freq.secs" in the call to getComponentConfiguration()
> method. Because this apparently only happens at topology startup:
> * If the Metron administrator wishes to change
> "topology.message.timeout.secs" from the default value of 30 sec, she must do
> it in storm.yaml or in the CLI, and *must not attempt to change it* via
> custom java code in topology submission. Attempted changes at topology
> submission time (except in the CLI) will not be seen in time to correctly set
> "topology.tick.tuple.freq.secs" in response.
> * -Authors of custom bolts may change "topology.message.timeout.secs" in the
> getComponentConfiguration() method at the same time as setting
> "topology.tick.tuple.freq.secs".- (Need to confirm whether this will really
> work at the topology level.)
> * "topology.tick.tuple.freq.secs" cannot be changed during runtime; it
> requires restarting the topology (unless there are Storm GUI actions to
> change it in runtime, which has not yet been investigated). As a result,
> changing batchTimeout during runtime will have this result: The affected
> queue will flush at an indeterminate time when its age is between \(i) the
> new batchTimeout interval, and (ii) the sum of the new batchTimeout plus the
> unchanged Tick Tuple receipt interval -- noting that the unchanged Tick Tuple
> receipt interval may be much larger than the new batchTimeout, if the revised
> batchTimeout was made much smaller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
