[
https://issues.apache.org/jira/browse/METRON-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664181#comment-16664181
]
ASF GitHub Bot commented on METRON-1845:
----------------------------------------
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1247#discussion_r228299751
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
---
@@ -97,48 +118,81 @@ protected static InMemoryComponent startIndex() throws
Exception {
return es;
}
- protected static void loadTestData() throws ParseException, IOException {
+ protected static void loadTestData() throws Exception {
ElasticSearchComponent es = (ElasticSearchComponent) indexComponent;
+ // define the bro index template
+ String broIndex = "bro_index_2017.01.01.01";
JSONObject broTemplate = JSONUtils.INSTANCE.load(new
File(broTemplatePath), JSONObject.class);
addTestFieldMappings(broTemplate, "bro_doc");
-
es.getClient().admin().indices().prepareCreate("bro_index_2017.01.01.01")
- .addMapping("bro_doc",
JSONUtils.INSTANCE.toJSON(broTemplate.get("mappings"), false)).get();
+ es.getClient().admin().indices().prepareCreate(broIndex)
+ .addMapping("bro_doc",
JSONUtils.INSTANCE.toJSON(broTemplate.get("mappings"), false)).get();
+
+ // define the snort index template
+ String snortIndex = "snort_index_2017.01.01.02";
JSONObject snortTemplate = JSONUtils.INSTANCE.load(new
File(snortTemplatePath), JSONObject.class);
addTestFieldMappings(snortTemplate, "snort_doc");
-
es.getClient().admin().indices().prepareCreate("snort_index_2017.01.01.02")
- .addMapping("snort_doc",
JSONUtils.INSTANCE.toJSON(snortTemplate.get("mappings"), false)).get();
-
- BulkRequestBuilder bulkRequest = es.getClient().prepareBulk()
- .setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
- JSONArray broArray = (JSONArray) new JSONParser().parse(broData);
- for (Object o : broArray) {
- JSONObject jsonObject = (JSONObject) o;
- IndexRequestBuilder indexRequestBuilder = es.getClient()
- .prepareIndex("bro_index_2017.01.01.01", "bro_doc");
- indexRequestBuilder = indexRequestBuilder.setId((String)
jsonObject.get("guid"));
- indexRequestBuilder =
indexRequestBuilder.setSource(jsonObject.toJSONString());
- indexRequestBuilder = indexRequestBuilder
- .setTimestamp(jsonObject.get("timestamp").toString());
- bulkRequest.add(indexRequestBuilder);
+ es.getClient().admin().indices().prepareCreate(snortIndex)
+ .addMapping("snort_doc",
JSONUtils.INSTANCE.toJSON(snortTemplate.get("mappings"), false)).get();
+
+ // setup the classes required to write the test data
+ AccessConfig accessConfig = createAccessConfig();
+ ElasticsearchClient client =
ElasticsearchUtils.getClient(createGlobalConfig());
+ ElasticsearchRetrieveLatestDao retrieveLatestDao = new
ElasticsearchRetrieveLatestDao(client);
+ ElasticsearchColumnMetadataDao columnMetadataDao = new
ElasticsearchColumnMetadataDao(client);
+ ElasticsearchRequestSubmitter requestSubmitter = new
ElasticsearchRequestSubmitter(client);
+ ElasticsearchUpdateDao updateDao = new ElasticsearchUpdateDao(client,
accessConfig, retrieveLatestDao);
+ ElasticsearchSearchDao searchDao = new ElasticsearchSearchDao(client,
accessConfig, columnMetadataDao, requestSubmitter);
--- End diff --
All of this because I need an `UpdateDao` to write the tests data and a
`SearchDao` to make sure the test data is loaded. Is there a simpler way?
> Correct Test Data Load in Elasticsearch Integration Tests
> ---------------------------------------------------------
>
> Key: METRON-1845
> URL: https://issues.apache.org/jira/browse/METRON-1845
> Project: Metron
> Issue Type: Sub-task
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> The Elasticsearch integration tests use the legacy Transport client to load
> test data into the search indexes before running the tests. Loading the test
> data like this does not accurately reflect how the indices will appear in a
> production environment.
> This should be changed to use our existing ElasticsearchUpdateDao to write
> the test data. This ensures that any changes made to the 'write' portion of
> our Elasticsearch code will function correctly with the 'read' portion. This
> ensures that telemetry written into Elasticsearch by 'Indexing' can be read
> correctly by the Alerts UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
