[
https://issues.apache.org/jira/browse/METRON-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536485#comment-15536485
]
ASF GitHub Bot commented on METRON-326:
---------------------------------------
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/286#discussion_r81378509
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriterTest.java
---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.writer;
+
+import backtype.storm.tuple.Tuple;
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.writer.BulkWriterResponse;
+import org.elasticsearch.action.bulk.BulkItemResponse;
+import org.elasticsearch.action.bulk.BulkResponse;
+import org.junit.Test;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.*;
+
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class ElasticsearchWriterTest {
+ @Test
+ public void testSingleSuccesses() throws Exception {
+ Tuple tuple1 = mock(Tuple.class);
+
+ BulkResponse response = mock(BulkResponse.class);
+ when(response.hasFailures()).thenReturn(false);
+
+ BulkWriterResponse expected = new BulkWriterResponse();
+ expected.addSuccess(tuple1);
+
+ ElasticsearchWriter esWriter = new ElasticsearchWriter();
+ BulkWriterResponse actual =
esWriter.buildWriteReponse(ImmutableList.of(tuple1), response);
+
+ assertEquals("Response should have no errors and single success",
expected, actual);
+ }
+
+ @Test
+ public void testMultipleSuccesses() throws Exception {
+ Tuple tuple1 = mock(Tuple.class);
+ Tuple tuple2 = mock(Tuple.class);
+
+ BulkResponse response = mock(BulkResponse.class);
+ when(response.hasFailures()).thenReturn(false);
+
+ BulkWriterResponse expected = new BulkWriterResponse();
+ expected.addSuccess(tuple1);
+ expected.addSuccess(tuple2);
+
+ ElasticsearchWriter esWriter = new ElasticsearchWriter();
+ BulkWriterResponse actual =
esWriter.buildWriteReponse(ImmutableList.of(tuple1, tuple2), response);
+
+ assertEquals("Response should have no errors and two successes",
expected, actual);
+ }
+
+ @Test
+ public void testSingleFailure() throws Exception {
+ Tuple tuple1 = mock(Tuple.class);
+
+ BulkResponse response = mock(BulkResponse.class);
+ when(response.hasFailures()).thenReturn(true);
+
+ Exception e = new IllegalStateException();
+ BulkItemResponse itemResponse = buildBulkItemFailure(e);
+
when(response.iterator()).thenReturn(ImmutableList.of(itemResponse).iterator());
+
+ BulkWriterResponse expected = new BulkWriterResponse();
+ expected.addError(e, tuple1);
+
+ ElasticsearchWriter esWriter = new ElasticsearchWriter();
+ BulkWriterResponse actual =
esWriter.buildWriteReponse(ImmutableList.of(tuple1), response);
+
+ assertEquals("Response should have no errors and two successes",
expected, actual);
--- End diff --
You are correct, will update
> Error Handling in ElasticsearchWriter
> -------------------------------------
>
> Key: METRON-326
> URL: https://issues.apache.org/jira/browse/METRON-326
> Project: Metron
> Issue Type: Bug
> Reporter: Ajay Yadav
> Assignee: Justin Leet
>
> In Elasticsearch writer we raise a exception if BulkResponse object has
> failures and that results in failing the whole batch even if some objects
> failed in it. This has spiral effect specially when there is continuous
> stream of bad messages and errorStream is tied to indexingBolt.
> If possible we should iterate through items in BulkResponse object and send
> only failed messages to errorStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
