GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/286
METRON-326 Error Handling in ElasticsearchWriter
To facilitate partial successes writing in Elasticsearch (i.e. subset of
Tuples succeeds and another subset fails), the BulkMessageWriter.write() now
returns a BulkWriterResponse object which contains both the successful and
failed Tuples (So that we know which ones to ack successfully and which to send
to the error stream).
ElasticsearchWriter has been modified to be able to take advantage of
partial writes. The other implementations of the BulkWriter have only been
affected to produce total success at the end of their methods (which is the
same functionality previously implied by not hitting an Exception.
Unit tests have been added for the ElasticsearchWriter writers
implementation of the partial successes and failures, along with modifying
existing tests elsewhere to account for the BulkWriterResponse.
One question I'd like resolved in review, is where to actually place the
BulkWriterResponse in the code hierarchy. It's obviously closely tied to the
BulkMessageWriter (really only existing to add functionality to it), but that
lives in an interface package and BWR is a concrete class. If there are
opinions on where that should live, I'm happy to move it and update the PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron es_error_handling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/286.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #286
----
commit 8d7475d0f8857f40ab2a025a0d20b429ef084053
Author: justinjleet <[email protected]>
Date: 2016-09-27T15:02:14Z
Adding functionality for partial write success in Elasticsearch
commit 83161afd1b1407c47a5565340bdfa96173955e7f
Author: justinjleet <[email protected]>
Date: 2016-09-30T11:40:21Z
fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
