Matt Foley created METRON-750:
---------------------------------
Summary: Add new functionality to ParserWriter related classes for
batchTimeout flushing
Key: METRON-750
URL: https://issues.apache.org/jira/browse/METRON-750
Project: Metron
Issue Type: Sub-task
Reporter: Ajay Yadav
Assignee: Matt Foley
Add new functionality to BulkMessageWriter and its related classes for
batchTimeout flushing.
This primarily affects:
* BulkMessageWriterBolt
* BulkWriterComponent
* IndexingConfigurations
* and newly added utilities in BatchTimeoutHelper
All of the other changes (from METRON-516) are basically boilerplate, just
laying "getBatchTimeout()" facade methods alongside existing "getBatchSize()"
facade methods. (There's a lot of layers!)
The flush-on-timeout logic is fairly straightforward. It was implemented by a
refactoring of BulkWriterComponent and basic Tick Tuple reception in
BulkMessageWriterBolt.
The tricky part was figuring out the appropriate setting for
'topology.tick.tuple.freq.secs' if the administrator configures non-default
batchTimeouts. It is necessary to enumerate the batchTimeout settings for all
configured sensorNames, which is implemented in
IndexingConfigurations$getAllConfiguredTimeouts(). Then multiple other factors
must be taken into account to determine the allowed and recommended settings,
which is implemented in BatchTimeoutHelper. If there are better ways to
accomplish these things, please share your ideas.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
