Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/481
@cestella said:
```
Since this is a core change, it bears probably developing an acceptance
test plan.
Something like creating a dummy sensor, creating a batch size of 10 and
sending
in 5 messages and ensuring things get flushed.
Also, it might be worth adjusting the IndexingIntegrationTest to set the
hdfs writer
config in metron-integration-test/src/main/config/indexing/test.json to a
large batch
size (say 50) and a timeout of, say, 5 seconds to ensure that batches get
flushed as
part of the integration tests.
```
Unfortunately a lot of existing tests have fairly sensitive dependence on
hdfs parameters in indexing/test.json, so tweaking it arbitrarily is not
feasible. However, I was able to enlarge the elasticsearch values, and I have
mocked procedures in the unit test that test the following cases:
1. Test that flush on batchSize continues to work correctly, without
time-based triggers going off.
1. Test that flush occurs on less-than-full queue, when a message comes in
and the queue is older than batchTimeout. (Age of the queue is determined as
currentTime() - timeOfEnqueingFirstMsg().)
1. Test that flush occurs on less-than-full queue, when a TickTuple comes
in and the queue is older than batchTimeout.
1. Test that default batchTimeout is correctly calculated as
(TOPOLOGY_MESSAGE_TIMEOUT_SECS / 2 / batchTimeoutDivisor - 1).
1. Test that when default batchTimeout is smaller than configured
batchTimeout, that the default is used.
1. Test that when default batchTimeout is greater than configured
batchTimeout, that the configured batchTimeout is used.
All of the above test cases are tested, with mocked writers and a
FakeClock, in the BulkMessageWriterBoltTest unit test file, with the latest
commit above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
