Github user bbende commented on the pull request:
https://github.com/apache/incubator-metron/pull/67#issuecomment-204616131
@merrimanr I think using the auto commits would be the way to go and gives
you the flexibility to tune the values to your needs. You could still leave the
optional commit in the writer in case it is needed for testing or anything
else.
In terms of the appropriate commit intervals, a lot of it is trial and
error depending on how much latency the application can tolerate to see
updates, vs. speed & amount of incoming data. Soft commits are lightweight
operations (some overhead) that will make updates visible, but the updates
aren't flushed to disk until a hard commit.
This post has a good description of the tread-offs:
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Metron might be the "Index-Heavy, Query-light" scenario? but not sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
