[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/840 ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159866359
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
I played with this a bit and I'm not convinced this works in master
anymore, so if this was necessary to get this running, I'd rather push fixing
coverage off to a follow on task.
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159681034
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
It's the top level pom, so no code coverage at all, iirc. I haven't looked
at it in awhile. I'll play with it a bit and see what's going on
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159680669
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
The build was failing with this argLine. Though, I can't recall the exact
error now as it's been a while. What's the impact of leaving it off? No code
coverage for this module?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159680299
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -33,7 +42,217 @@ For instance, an `es.date.format` of `.MM.dd.HH`
would have the consequence
roll hourly, whereas an `es.date.format` of `.MM.dd` would have the
consequence that the indices would
roll daily.
-## Using Metron with Elasticsearch 2.x
+## Upgrading to 5.6.2
+
+Users should be prepared to re-index when migrating from Elasticsearch
2.3.3 to 5.6.2. There are a number of template changes, most notably around
+string type handling, that may cause issues when upgrading.
+
+[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html)
+
+Be aware that if you add a new string value and want to be able to filter
and search on this value from the Alerts UI, you **must** add a mapping for
that type to
+the appropriate Elasticsearch template. Below is more detail on how to
choose the appropriate mapping type for your string value.
+
+## Type Mappings
+
+Type mappings have changed quite a bit from ES 2.x -> 5.x. Here is a brief
rundown of the biggest changes. More detailed references from Elasticsearch
+are provided in the [Type Mapping References](#type-mapping-references)
section below.
+* string fields replaced by text/keyword type
+* strings have new default mappings as follows
+
+```
+{
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+```
+
+* There is no longer a `_timestamp` field that you can set "enabled" on.
This field now causes an exception on templates.
+Replace with an application-created timestamp of "date" type.
+
+The semantics for string types have changed. In 2.x, you have the concept
of index settings as either "analyzed" or "not_analyzed" which basically means
"full text" and "keyword", respectively.
+Analyzed text basically means the indexer will split the text using a text
analyzer thus allowing you to search on substrings within the original text.
"New York" is split and indexed as two buckets,
+ "New" and "York", so you can search or query for aggregate counts for
those terms independently and will match against the individual terms "New" or
"York." "Keyword" means that the original text
+ will not be split/analyzed during indexing and instead treated as a whole
unit, i.e. "New" or "York" will not match in searches against the document
containing "New York", but searching on "New York"
+ as the full city name will. In 5.x language instead of using the "index"
setting, you now set the "type" to either "text" for full text, or "keyword"
for keywords.
+
+Below is a table depicting the changes to how String types are now handled.
+
+
+
+ sort, aggregate, or access values
+ ES 2.x
+ ES 5.x
+ Example
+
+
+ no
+
+"my_property" : {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "text"
+}
+
+Additional defaults: "index": "true", "fielddata": "false"
+
+
+ "New York" handled via in-mem search as "New" and "York"
buckets. No aggregation or sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fielddata": "true"
+}
+
+
+
+ "New York" handled via in-mem search as "New" and "York" buckets.
Can aggregate and sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "not_analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "keyword"
+}
+
+
+
+ "New York" searchable as single value. Can aggregate
and sort. A search for "New" or "York" will not match against the whole value.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+
+
+
+ "New York" searchable as single value or as text document, can
aggregate and sort on the sub term "keyword."
+
+
+
+
+If you want to set default string behavior for all strings for a given
index and type, you can do so with a
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159680218 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- All set ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159674941 --- Diff: metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md --- @@ -441,7 +441,7 @@ Client - Kibana: * Set "kibana_es_url" to `http://:9200`. "replace_with_elasticsearch_master_hostname" is the IP of the node where you assigned ElasticSearch Master on the Assign Master tab. -* Change kibana_default_application to "dashboard/Metron-Dashboard" +* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9" --- End diff -- They changed the links to point to index keys. I was bummed about this as well. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159273549
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
This was in for code coverage via JaCoCo. It basically overrides the empty
argline from above that got deleted. Was it causing problems with running
things with it there?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159266301 --- Diff: metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md --- @@ -441,7 +441,7 @@ Client - Kibana: * Set "kibana_es_url" to `http://:9200`. "replace_with_elasticsearch_master_hostname" is the IP of the node where you assigned ElasticSearch Master on the Assign Master tab. -* Change kibana_default_application to "dashboard/Metron-Dashboard" +* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9" --- End diff -- Is it possible to provide an easily identifiable name, or are we stuck with the "AV-..." ugliness? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159265938 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- As a heads up, #883 is in now, so this will have to be taken care of when you merge master to deconflict. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159122591 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- Can you please add the license header to this? https://github.com/apache/metron/pull/884 is close to going in and enforcing this, so I'm hoping to avoid impact to master. ``` ``` ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r157540768
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -33,7 +42,217 @@ For instance, an `es.date.format` of `.MM.dd.HH`
would have the consequence
roll hourly, whereas an `es.date.format` of `.MM.dd` would have the
consequence that the indices would
roll daily.
-## Using Metron with Elasticsearch 2.x
+## Upgrading to 5.6.2
+
+Users should be prepared to re-index when migrating from Elasticsearch
2.3.3 to 5.6.2. There are a number of template changes, most notably around
+string type handling, that may cause issues when upgrading.
+
+[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html)
+
+Be aware that if you add a new string value and want to be able to filter
and search on this value from the Alerts UI, you **must** add a mapping for
that type to
+the appropriate Elasticsearch template. Below is more detail on how to
choose the appropriate mapping type for your string value.
+
+## Type Mappings
+
+Type mappings have changed quite a bit from ES 2.x -> 5.x. Here is a brief
rundown of the biggest changes. More detailed references from Elasticsearch
+are provided in the [Type Mapping References](#type-mapping-references)
section below.
+* string fields replaced by text/keyword type
+* strings have new default mappings as follows
+
+```
+{
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+```
+
+* There is no longer a `_timestamp` field that you can set "enabled" on.
This field now causes an exception on templates.
+Replace with an application-created timestamp of "date" type.
+
+The semantics for string types have changed. In 2.x, you have the concept
of index settings as either "analyzed" or "not_analyzed" which basically means
"full text" and "keyword", respectively.
+Analyzed text basically means the indexer will split the text using a text
analyzer thus allowing you to search on substrings within the original text.
"New York" is split and indexed as two buckets,
+ "New" and "York", so you can search or query for aggregate counts for
those terms independently and will match against the individual terms "New" or
"York." "Keyword" means that the original text
+ will not be split/analyzed during indexing and instead treated as a whole
unit, i.e. "New" or "York" will not match in searches against the document
containing "New York", but searching on "New York"
+ as the full city name will. In 5.x language instead of using the "index"
setting, you now set the "type" to either "text" for full text, or "keyword"
for keywords.
+
+Below is a table depicting the changes to how String types are now handled.
+
+
+
+ sort, aggregate, or access values
+ ES 2.x
+ ES 5.x
+ Example
+
+
+ no
+
+"my_property" : {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "text"
+}
+
+Additional defaults: "index": "true", "fielddata": "false"
+
+
+ "New York" handled via in-mem search as "New" and "York"
buckets. No aggregation or sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fielddata": "true"
+}
+
+
+
+ "New York" handled via in-mem search as "New" and "York" buckets.
Can aggregate and sort.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "not_analyzed"
+}
+
+
+
+"my_property" : {
+ "type": "keyword"
+}
+
+
+
+ "New York" searchable as single value. Can aggregate
and sort. A search for "New" or "York" will not match against the whole value.
+
+
+
+
+ yes
+
+
+"my_property": {
+ "type": "string",
+ "index": "analyzed"
+}
+
+
+
+"my_property": {
+ "type": "text",
+ "fields": {
+"keyword": {
+ "type": "keyword",
+ "ignore_above": 256
+}
+ }
+}
+
+
+
+ "New York" searchable as single value or as text document, can
aggregate and sort on the sub term "keyword."
+
+
+
+
+If you want to set default string behavior for all strings for a given
index and type, you can do so with a
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r157540388 --- Diff: metron-platform/metron-elasticsearch/README.md --- @@ -1,5 +1,14 @@ # Elasticsearch in Metron +## Table of Contents + +* [Introduction](#introduction) +* [Properties](#properties) +* [Upgrading to 5.6.2](#upgrading-to-562) +* [Type Mappings](#type-mappings) +* [Using Metron with Elasticsearch 5.x](#using-metron-with-elasticsearch-5x) --- End diff -- This link is broken, because the section name doesn't line up with the actual name ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r153289006
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
---
@@ -102,13 +94,25 @@
"match_mapping_type": "*"
}
},
- {
-"threat_triage_reason": {
- "mapping": {
-"type": "string"
- },
- "match": "threat:triage:rules:*:reason",
- "match_mapping_type": "*"
+{
+ "threat_triage_reason": {
+"mapping": {
+ "type": "text",
+ "fielddata": "true"
+},
+"match": "threat.triage.rules:*:reason",
+"match_mapping_type": "*"
+ }
+},
+{
+ "threat_triage_name": {
+"mapping": {
+ "type": "text",
+ "fielddata": "true"
+},
+"match": "threat.triage.rules:*:name",
+"match_mapping_type": "*"
+ }
}
},
--- End diff --
This brace is extraneous, I'm guessing a merge broke it. Drop it, but keep
the comma and we should be good. I'd just get the formatting lined back up
while you're in there.
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150282327 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Heh, yep. That extra import broke the Kibana install hard. I've made most of your requested changes and will have another commit coming shortly. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150281496
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --
Yeah, I think you're right. I really like having data inline so that you
can see the full context of your tests without having to follow multiple
branches to piece together what actually happens. I think it's ok for the
system under test, but the tests themselves should be much more intuitive and
easy to follow. So I think the right answer here is that I've included a lot of
extra noise that is unnecessary for the test. I'm going to prune the "_source"
object, which will make this very lean and much easier to read. Really good
catch @ottobackwards.
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150274541 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -99,7 +148,15 @@ --- End diff -- Yes, absolutely agreed. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150274108 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Hm, I'm not sure where that came from. I suspect IntelliJ added that in the middle of my edits. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150269845
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
---
@@ -48,19 +51,30 @@ def slave():
content=InlineTemplate(params.elastic_env_sh_template)
)
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --
Hm, adding to my list of cleanup tasks.
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150264674 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -99,7 +148,15 @@ --- End diff -- Maybe a comment as to why we are doing relocations will help future maintainers ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150262722
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
---
@@ -48,19 +51,30 @@ def slave():
content=InlineTemplate(params.elastic_env_sh_template)
)
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --
I don't think "Cre" is what you wanted here, typo?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150263736 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Is the ansible module something we are going to distribute? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150264904
--- Diff: metron-platform/metron-data-management/pom.xml ---
@@ -231,11 +231,11 @@
httpclient
${httpcore.version}
--- End diff --
Why not just remove these?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150259383 --- Diff: dependencies_with_url.csv --- @@ -310,6 +310,29 @@ org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RE org.springframework.kafka:spring-kafka:jar:1.1.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-kafka ch.hsr:geohash:jar:1.3.0:compile,ASLv2,https://github.com/kungfoo/geohash-java org.locationtech.spatial4j:spatial4j:jar:0.6:compile,ASLv2,https://github.com/locationtech/spatial4j --- End diff -- can we remove any of the older dependencies? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150266275
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --
Is there a tipping point where this should be a file vs. multiline?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150265577 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java --- @@ -139,13 +141,17 @@ protected SearchResponse search(SearchRequest searchRequest, QueryBuilder queryB searchRequest.getSort().forEach(sortField -> searchSourceBuilder.sort(sortField.getField(), getElasticsearchSortOrder(sortField.getSortOrder(; Optionalfields = searchRequest.getFields(); if (fields.isPresent()) { --- End diff -- why not just remove? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150265417 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExport.java --- @@ -0,0 +1,76 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.elasticsearch.bulk; + +import com.fasterxml.jackson.core.type.TypeReference; +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.FileReader; +import java.io.FileWriter; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import org.apache.metron.common.utils.JSONUtils; + --- End diff -- javadoc ---
