[
https://issues.apache.org/jira/browse/METRON-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated METRON-329:
------------------------------
Description:
Add new functionality to BulkMessageWriter and its related classes for
batchTimeout flushing.
This primarily affects:
* BulkMessageWriterBolt
* BulkWriterComponent
* IndexingConfigurations
* and newly added utilities in BatchTimeoutHelper
All of the other changes (from METRON-516) are basically boilerplate, just
laying "getBatchTimeout()" facade methods alongside existing "getBatchSize()"
facade methods. (There's a lot of layers!)
The flush-on-timeout logic is fairly straightforward. It was implemented by a
refactoring of BulkWriterComponent and basic Tick Tuple reception in
BulkMessageWriterBolt.
The tricky part was figuring out the appropriate setting for
'topology.tick.tuple.freq.secs' if the administrator configures non-default
batchTimeouts. It is necessary to enumerate the batchTimeout settings for all
configured sensorNames, which is implemented in
IndexingConfigurations$getAllConfiguredTimeouts(). Then multiple other factors
must be taken into account to determine the allowed and recommended settings,
which is implemented in BatchTimeoutHelper. If there are better ways to
accomplish these things, please share your ideas.
was:
Add new functionality to BulkMessageWriter and its related classes for
batchTimeout flushing.
This primarily affects:
* BulkMessageWriterBolt
* BulkWriterComponent
* IndexingConfigurations
* and newly added utilities in BatchTimeoutHelper
> Add new functionality to BulkMessageWriter and its related classes for
> batchTimeout flushing
> --------------------------------------------------------------------------------------------
>
> Key: METRON-329
> URL: https://issues.apache.org/jira/browse/METRON-329
> Project: Metron
> Issue Type: Sub-task
> Reporter: Ajay Yadav
> Assignee: Matt Foley
>
> Add new functionality to BulkMessageWriter and its related classes for
> batchTimeout flushing.
> This primarily affects:
> * BulkMessageWriterBolt
> * BulkWriterComponent
> * IndexingConfigurations
> * and newly added utilities in BatchTimeoutHelper
> All of the other changes (from METRON-516) are basically boilerplate, just
> laying "getBatchTimeout()" facade methods alongside existing "getBatchSize()"
> facade methods. (There's a lot of layers!)
> The flush-on-timeout logic is fairly straightforward. It was implemented by a
> refactoring of BulkWriterComponent and basic Tick Tuple reception in
> BulkMessageWriterBolt.
> The tricky part was figuring out the appropriate setting for
> 'topology.tick.tuple.freq.secs' if the administrator configures non-default
> batchTimeouts. It is necessary to enumerate the batchTimeout settings for all
> configured sensorNames, which is implemented in
> IndexingConfigurations$getAllConfiguredTimeouts(). Then multiple other
> factors must be taken into account to determine the allowed and recommended
> settings, which is implemented in BatchTimeoutHelper. If there are better
> ways to accomplish these things, please share your ideas.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
