[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248335#comment-16248335 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 The only issue with it, is if we did that in the old version, then this would be a regression. Other than that, I'm just waiting to review the docs ;) > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248260#comment-16248260 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 Whew, that's great news! Hahahahaha, that's the default colors from the Kibana widgets... I usually refrain from bringing it up, but I'm partially colorblind (even though I used to do a fair amount of Photoshop graphic design work on the side, but it's a bit more effort for me). Anyhow, Kibana had no easy way to pick color names or supply RGB values, so I punted on it for now because I figured that's an easy change. If you would be willing to take a screenshot or the xy coordinate of colors in the color picker that you'd prefer, I'd be more than happy to change it. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248249#comment-16248249 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Ran up in full dev, everything running normally, kibana dashboard has data. Is the color scheme supposed to be so 'miami vice'? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248180#comment-16248180 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @ottobackwards You should be good to try it now. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created
[ https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248138#comment-16248138 ] ASF GitHub Bot commented on METRON-1289: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/824 The latest commits address some outstanding tasks and address a couple other issues: - disabling updates to meta alert objects means we can't update a meta alert name or comments (or other fields that may be added in the future). The patch implementation was added to the ElasticsearchMetaAlertDao to allow updates to fields other than `alert` and `status` which require special handling and have dedicated endpoints - a `getAllLatest` implementation was added to the HBaseDao along with an integration test - a bug was discovered where the number of search results returned defaults to 10 meaning changes to alerts are only propagated to the first 10 meta alerts and vice-versa. This was corrected in the ElasticsearchDao.searchByGuids by explicitly setting the size and fixed in the alert/meta alert lookup queries by paging through all the results if necessary. A test was added to the integration test to catch this case in the future. I feel like this PR is in pretty good shape from a functional perspective. Still working on javadocs/documentation but it's ready for functional testing/code review. > Alert fields are lost when a MetaAlert is created > - > > Key: METRON-1289 > URL: https://issues.apache.org/jira/browse/METRON-1289 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > When a MetaAlert is created, the included results are being updated > incorrectly with only the "metaalert" field. This causes subsequent findOne > operations to only return the "metaalert field for that alert. All fields > should continue to be present. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248026#comment-16248026
]
ASF GitHub Bot commented on METRON-1302:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150336506
--- Diff:
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter
$METRON_HOME/config/hdfs.properties
--- End diff --
Very true. I think you're right, @ottobackwards
> Split up Indexing Topology into batch and random access sections
>
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We
> should split these up and configure them separately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247946#comment-16247946
]
ASF GitHub Bot commented on METRON-1302:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150317133
--- Diff:
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter
$METRON_HOME/config/hdfs.properties
--- End diff --
I almost think that if it is going to be hdfs, we should call it hdfs. If
we have multiple topologies based on the batch indexer, each will end up with a
unique name anyways ( S3 indexer etc ).
> Split up Indexing Topology into batch and random access sections
>
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We
> should split these up and configure them separately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247944#comment-16247944
]
ASF GitHub Bot commented on METRON-1302:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150316808
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
---
@@ -162,34 +164,77 @@ def start_indexing_topology(self, env):
self.__params.metron_principal_name,
execute_user=self.__params.metron_user)
-start_cmd_template =
"""{0}/bin/start_elasticsearch_topology.sh \
--s {1} \
--z {2}"""
-start_cmd =
start_cmd_template.format(self.__params.metron_home,
- self.__indexing_topology,
-
self.__params.zookeeper_quorum)
+start_cmd_template = """{0}/bin/start_hdfs_topology.sh"""
+start_cmd =
start_cmd_template.format(self.__params.metron_home)
Execute(start_cmd, user=self.__params.metron_user, tries=3,
try_sleep=5, logoutput=True)
else:
-Logger.info('Indexing topology already running')
+Logger.info('Batch Indexing topology already running')
-Logger.info('Finished starting indexing topology')
+Logger.info('Finished starting batch indexing topology')
-def stop_indexing_topology(self, env):
-Logger.info('Stopping ' + self.__indexing_topology)
+def start_random_access_indexing_topology(self, env):
+Logger.info('Starting ' + self.__random_access_indexing_topology)
--- End diff --
I like this a lot. I would like to see the indexing be like the parses,
where you start with a name parameter. I think the flux files make it
difficult though. I would be +1 exploring this approach though, maybe a
feature branch?
> Split up Indexing Topology into batch and random access sections
>
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We
> should split these up and configure them separately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247940#comment-16247940 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 I got full_dev failure on connecting to metron web. Can you post when you think full dev is ready and I'll try again then? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[ https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247931#comment-16247931 ] ASF GitHub Bot commented on METRON-1302: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r150315275 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-indexing-env.xml --- @@ -73,18 +98,31 @@ Indexing Update Column Family -indexing_workers +ra_indexing_workers +Number of Indexing Topology Workers --- End diff -- Actually, I see the display name is different. Doh. But here we are using the terms "Elasticsearch" and "HDFS", rather than "Random Access" and "Batch". We should probably stick with one nomenclature. > Split up Indexing Topology into batch and random access sections > > > Key: METRON-1302 > URL: https://issues.apache.org/jira/browse/METRON-1302 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > Currently we have the indexing topology handle writing to both random access > indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We > should split these up and configure them separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247928#comment-16247928
]
ASF GitHub Bot commented on METRON-1302:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150314839
--- Diff:
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter
$METRON_HOME/config/hdfs.properties
--- End diff --
In the MPack we refer to this as the "batch" topology. But in this script
we refer to it as the "HDFS" topology. We should probably stick with one
nomenclature.
> Split up Indexing Topology into batch and random access sections
>
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We
> should split these up and configure them separately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[ https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247926#comment-16247926 ] ASF GitHub Bot commented on METRON-1302: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r150314308 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-indexing-env.xml --- @@ -73,18 +98,31 @@ Indexing Update Column Family -indexing_workers +ra_indexing_workers +Number of Indexing Topology Workers --- End diff -- With the name and description the same for the random access and batch topologies; can the user distinguish which is which? > Split up Indexing Topology into batch and random access sections > > > Key: METRON-1302 > URL: https://issues.apache.org/jira/browse/METRON-1302 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > Currently we have the indexing topology handle writing to both random access > indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We > should split these up and configure them separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247923#comment-16247923
]
ASF GitHub Bot commented on METRON-1302:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150314117
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
---
@@ -162,34 +164,77 @@ def start_indexing_topology(self, env):
self.__params.metron_principal_name,
execute_user=self.__params.metron_user)
-start_cmd_template =
"""{0}/bin/start_elasticsearch_topology.sh \
--s {1} \
--z {2}"""
-start_cmd =
start_cmd_template.format(self.__params.metron_home,
- self.__indexing_topology,
-
self.__params.zookeeper_quorum)
+start_cmd_template = """{0}/bin/start_hdfs_topology.sh"""
+start_cmd =
start_cmd_template.format(self.__params.metron_home)
Execute(start_cmd, user=self.__params.metron_user, tries=3,
try_sleep=5, logoutput=True)
else:
-Logger.info('Indexing topology already running')
+Logger.info('Batch Indexing topology already running')
-Logger.info('Finished starting indexing topology')
+Logger.info('Finished starting batch indexing topology')
-def stop_indexing_topology(self, env):
-Logger.info('Stopping ' + self.__indexing_topology)
+def start_random_access_indexing_topology(self, env):
+Logger.info('Starting ' + self.__random_access_indexing_topology)
--- End diff --
First off, I think we definitely need to make this happen. Each index
destination is going to have very different performance characteristics that
need to be tuned in isolation. I think this is a step in the right direction.
As I read this we have effectively hard-coded two indexing topologies;
random access and batch. This is definitely the most logical way to get to
separate topologies based on our existing code base. But I am wondering if we
might think about this in a slightly different way.
What I really like about indexing is that we have the idea of multiple,
independent destinations. For example, my indexing configuration could look
like this.
```
{
"elasticsearch": {
"index": "foo",
"enabled" : true
},
"hdfs": {
"index": "foo",
"batchSize": 1,
"enabled" : true
}
}
```
What if we introduced logic that consumes the indexing configuration,
determines that it needs to launch 2 topologies in this case, and then launches
those 2 separate topologies? If I had 3 destinations configured, then it would
launch 3 topologies; one for each destination?
I can definitely see the extra complexity in doing this. You have to make
sure the user can independently configure each of the topologies. You have to
respond to configuration changes made by the user. And probably a few other
complications.
But these are already complications that we need to deal with in Parsing.
A user can define 1 to N Parsing topologies. It seems like if we can solve
these challenges for Parsing, we can do the same for Indexing.
Anywho, I can totally see this PR as a near-term solution to the immediate
problem, which might lead towards a longer-term solution like I propose. I
just wanted to see if anyone had related thoughts.
> Split up Indexing Topology into batch and random access sections
>
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We
> should split these up and configure them separately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247911#comment-16247911 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 Ok, looks like 2 more minor issues. 1. params.py missing the kibana_server_host, so it's not making its way to kibana.yml 2. The default index mapping template that ES creates when I index the Kibana configuration is not compatible with Kibana. I'm done fixing 1 already and have the solution for 2. I'm going to blow away the .kibana index, let Kibana recreate it with the expected template, and then extract that into a template file that's used during the Kibana server install. The only hitch is that this is one more thing that potentially needs to be modified/updated on future upgrades of ES and Kibana. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1311) Service Check Should Check Elasticsearch Index Templates
[ https://issues.apache.org/jira/browse/METRON-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247795#comment-16247795 ] ASF GitHub Bot commented on METRON-1311: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/839 I ran this through another full test after merge with master. Everything looks good to me. > Service Check Should Check Elasticsearch Index Templates > > > Key: METRON-1311 > URL: https://issues.apache.org/jira/browse/METRON-1311 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > The Service Check in Ambari does not validate that the Elasticsearch index > templates have been installed. Without these index templates bad things can > happen. For example, the Alerts UI will not be able to display any alerts. > The Elasticsearch index templates that are installed by Ambari should also be > checked as part of the Metron Service Check. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247784#comment-16247784 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Thanks @mmiklavc, those changes look good. Running up in full dev > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1270) fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common and metron-profiler-client
[
https://issues.apache.org/jira/browse/METRON-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247774#comment-16247774
]
ASF GitHub Bot commented on METRON-1270:
Github user dbist closed the pull request at:
https://github.com/apache/metron/pull/810
> fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client
> ---
>
> Key: METRON-1270
> URL: https://issues.apache.org/jira/browse/METRON-1270
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
>
> address the following warnings
> {noformat}
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/ProfileBuilder.java:66:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/ColumnBuilder.java:43:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/DateSpecifierPredicate.java:79:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/DayPredicates.java:109:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/HolidaysPredicate.java:82:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:50:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:63:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:77:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:91:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:103:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:115:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:130:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java:61:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java:393:
> warning - @return tag has no arguments.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247750#comment-16247750 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150282327 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Heh, yep. That extra import broke the Kibana install hard. I've made most of your requested changes and will have another commit coming shortly. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247749#comment-16247749
]
ASF GitHub Bot commented on METRON-939:
---
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150281496
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --
Yeah, I think you're right. I really like having data inline so that you
can see the full context of your tests without having to follow multiple
branches to piece together what actually happens. I think it's ok for the
system under test, but the tests themselves should be much more intuitive and
easy to follow. So I think the right answer here is that I've included a lot of
extra noise that is unnecessary for the test. I'm going to prune the "_source"
object, which will make this very lean and much easier to read. Really good
catch @ottobackwards.
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1311) Service Check Should Check Elasticsearch Index Templates
[ https://issues.apache.org/jira/browse/METRON-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247734#comment-16247734 ] ASF GitHub Bot commented on METRON-1311: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/839 Thanks @anandsubbu . I resolved the merge conflicts from master. > Service Check Should Check Elasticsearch Index Templates > > > Key: METRON-1311 > URL: https://issues.apache.org/jira/browse/METRON-1311 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > The Service Check in Ambari does not validate that the Elasticsearch index > templates have been installed. Without these index templates bad things can > happen. For example, the Alerts UI will not be able to display any alerts. > The Elasticsearch index templates that are installed by Ambari should also be > checked as part of the Metron Service Check. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247730#comment-16247730
]
ASF GitHub Bot commented on METRON-939:
---
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150276834
--- Diff: metron-platform/metron-data-management/pom.xml ---
@@ -231,11 +231,11 @@
httpclient
${httpcore.version}
--- End diff --
Yep, absolutely. More cleanup.
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247719#comment-16247719 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150274541 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -99,7 +148,15 @@ --- End diff -- Yes, absolutely agreed. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247715#comment-16247715 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150274108 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Hm, I'm not sure where that came from. I suspect IntelliJ added that in the middle of my edits. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247707#comment-16247707 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @mraliagha @cestella agreed on timestamp. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247703#comment-16247703 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Can you call the curator stuff out in your checklist above? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247702#comment-16247702 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Ok, killing my vagrant up then, until your commit > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1311) Service Check Should Check Elasticsearch Index Templates
[
https://issues.apache.org/jira/browse/METRON-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247696#comment-16247696
]
ASF GitHub Bot commented on METRON-1311:
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/839
+1 (non-binding) works fine, and a much needed fix as well ! If for some
reason, we were not able to install the templates during the startup, it would
be caught by the service check.
Validated this on a full dev environment. For a successful service check, I
see this now:
```
2017-11-10 15:48:06,150 - Checking Elasticsearch templates for Indexing
2017-11-10 15:48:06,151 - Execute['curl -s -XGET
"http://node1:9200/_template/yaf_index; | grep -o yaf_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
yaf_index
yaf_index
2017-11-10 15:48:06,343 - Execute['curl -s -XGET
"http://node1:9200/_template/error_index; | grep -o error_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
error_index
error_index
2017-11-10 15:48:06,385 - Execute['curl -s -XGET
"http://node1:9200/_template/snort_index; | grep -o snort_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
snort_index
snort_index
2017-11-10 15:48:06,433 - Execute['curl -s -XGET
"http://node1:9200/_template/bro_index; | grep -o bro_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
bro_index
bro_index
2017-11-10 15:48:06,491 - Execute['curl -s -XGET
"http://node1:9200/_template/metaalert_index; | grep -o metaalert_index']
{'logoutput': True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'],
'tries': 3, 'user': 'metron', 'try_sleep': 5}
metaalert_index
```
And the service check fails when I delete the ES template file.
```
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/service_check.py",
line 78, in
ServiceCheck().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/service_check.py",
line 51, in service_check
indexing_cmds.service_check(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
line 264, in service_check
self.check_elasticsearch_templates()
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
line 179, in check_elasticsearch_templates
metron_service.execute(cmd, user=self.__params.metron_user,
err_msg=err_msg)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/metron_service.py",
line 257, in execute
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Missing Elasticsearch index
template: name=yaf_index
```
@nickwallen - I see two conflicting files for the commit. Guess this would
be need to be resolved first?
> Service Check Should Check Elasticsearch Index Templates
>
>
> Key: METRON-1311
> URL: https://issues.apache.org/jira/browse/METRON-1311
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> The Service Check in Ambari does not validate that the Elasticsearch index
> templates have been installed. Without these index templates bad things can
> happen. For example, the Alerts UI will not be able to display any alerts.
> The Elasticsearch index templates that are installed by Ambari should also be
> checked as part of the Metron Service Check.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247695#comment-16247695 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @ottobackwards Oh, afa the data pruner, I believe we can use Curator now as a built-in mechanism for pruning data. Steps will be added to the documentation. https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247689#comment-16247689
]
ASF GitHub Bot commented on METRON-939:
---
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150269845
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
---
@@ -48,19 +51,30 @@ def slave():
content=InlineTemplate(params.elastic_env_sh_template)
)
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --
Hm, adding to my list of cleanup tasks.
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1270) fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common and metron-profiler-client
[
https://issues.apache.org/jira/browse/METRON-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247680#comment-16247680
]
ASF GitHub Bot commented on METRON-1270:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/810
It's merged. The commit message looks acceptable too. Not sure why it
didn't automatically close. Please close @dbist
> fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client
> ---
>
> Key: METRON-1270
> URL: https://issues.apache.org/jira/browse/METRON-1270
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
>
> address the following warnings
> {noformat}
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/ProfileBuilder.java:66:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/ColumnBuilder.java:43:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/DateSpecifierPredicate.java:79:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/DayPredicates.java:109:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/HolidaysPredicate.java:82:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:50:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:63:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:77:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:91:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:103:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:115:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:130:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java:61:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java:393:
> warning - @return tag has no arguments.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1270) fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common and metron-profiler-client
[
https://issues.apache.org/jira/browse/METRON-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247676#comment-16247676
]
ASF GitHub Bot commented on METRON-1270:
Github user dbist commented on the issue:
https://github.com/apache/metron/pull/810
@justinleet thanks for looking into this, as soon as I get a confirmation,
I'll close it.
> fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client
> ---
>
> Key: METRON-1270
> URL: https://issues.apache.org/jira/browse/METRON-1270
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
>
> address the following warnings
> {noformat}
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/ProfileBuilder.java:66:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/ColumnBuilder.java:43:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/DateSpecifierPredicate.java:79:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/DayPredicates.java:109:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/predicates/HolidaysPredicate.java:82:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:50:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:63:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:77:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:91:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:103:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:115:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/Window.java:130:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java:61:
> warning - @return tag has no arguments.
> [WARNING]
> /Users/aervits/NetBeansProjects/METRON/metron/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java:393:
> warning - @return tag has no arguments.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247675#comment-16247675 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Are there any required changes to the ansible roles? I thought we injected the templates from there. Or has the indexing service injection landed? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247673#comment-16247673
]
ASF GitHub Bot commented on METRON-939:
---
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150266275
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --
Is there a tipping point where this should be a file vs. multiline?
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247668#comment-16247668 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150265417 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExport.java --- @@ -0,0 +1,76 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.elasticsearch.bulk; + +import com.fasterxml.jackson.core.type.TypeReference; +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.FileReader; +import java.io.FileWriter; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import org.apache.metron.common.utils.JSONUtils; + --- End diff -- javadoc > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247671#comment-16247671 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150265577 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java --- @@ -139,13 +141,17 @@ protected SearchResponse search(SearchRequest searchRequest, QueryBuilder queryB searchRequest.getSort().forEach(sortField -> searchSourceBuilder.sort(sortField.getField(), getElasticsearchSortOrder(sortField.getSortOrder(; Optionalfields = searchRequest.getFields(); if (fields.isPresent()) { --- End diff -- why not just remove? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247667#comment-16247667
]
ASF GitHub Bot commented on METRON-939:
---
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150264904
--- Diff: metron-platform/metron-data-management/pom.xml ---
@@ -231,11 +231,11 @@
httpclient
${httpcore.version}
--- End diff --
Why not just remove these?
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247670#comment-16247670 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150264674 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -99,7 +148,15 @@ --- End diff -- Maybe a comment as to why we are doing relocations will help future maintainers > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247669#comment-16247669 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150259383 --- Diff: dependencies_with_url.csv --- @@ -310,6 +310,29 @@ org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RE org.springframework.kafka:spring-kafka:jar:1.1.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-kafka ch.hsr:geohash:jar:1.3.0:compile,ASLv2,https://github.com/kungfoo/geohash-java org.locationtech.spatial4j:spatial4j:jar:0.6:compile,ASLv2,https://github.com/locationtech/spatial4j --- End diff -- can we remove any of the older dependencies? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247672#comment-16247672
]
ASF GitHub Bot commented on METRON-939:
---
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150262722
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
---
@@ -48,19 +51,30 @@ def slave():
content=InlineTemplate(params.elastic_env_sh_template)
)
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --
I don't think "Cre" is what you wanted here, typo?
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247666#comment-16247666 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150263736 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Is the ansible module something we are going to distribute? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247581#comment-16247581 ] ASF GitHub Bot commented on METRON-939: --- Github user cestella commented on the issue: https://github.com/apache/metron/pull/840 @mraliagha Yeah, absolutely. This task purposefully leaves out major changes that we could do to improve things (e.g. moving back to `.` separated fields instead of `:`). The attempt is to touch as few things as possible. Your ask makes a lot of sense and we should add it as a follow-on task. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created
[
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247514#comment-16247514
]
ASF GitHub Bot commented on METRON-1289:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150239607
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/MetaAlertController.java
---
@@ -60,5 +63,37 @@
) throws RestException {
return new ResponseEntity<>(metaAlertService.create(createRequest),
HttpStatus.OK);
}
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/add/alert", method = RequestMethod.POST)
+ ResponseEntity addAlertsToMetaAlert(
+ @ApiParam(name = "request", value = "Meta Alert Create Request",
required = true)
+ @RequestBody final MetaAlertAddRemoveRequest
metaAlertAddRemoveRequest
+ ) throws RestException {
+return new
ResponseEntity<>(metaAlertService.addAlertsToMetaAlert(metaAlertAddRemoveRequest),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/remove/alert", method = RequestMethod.POST)
+ ResponseEntity removeAlertsFromMetaAlert(
+ @ApiParam(name = "request", value = "Meta Alert Create Request",
required = true)
+ @RequestBody final MetaAlertAddRemoveRequest
metaAlertAddRemoveRequest
+ ) throws RestException {
+return new
ResponseEntity<>(metaAlertService.removeAlertsFromMetaAlert(metaAlertAddRemoveRequest),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/update/status/{guid}/{status}", method =
RequestMethod.POST)
+ ResponseEntity updateMetaAlertStatus(
+ final @ApiParam(name = "guid", value = "Kafka topic name", required
= true)
+ @PathVariable String guid,
+ final @ApiParam(name = "status", value = "Kafka topic name",
required = true)
+ @PathVariable String status) throws RestException {
+return new
ResponseEntity<>(metaAlertService.updateMetaAlertStatus(guid,
+MetaAlertStatus.valueOf(status.toUpperCase())), HttpStatus.OK);
+ }
--- End diff --
I think we'll also need to be able to add comments and a name to the
metaalert as a whole, to support work in
https://github.com/apache/metron/pull/803/
@iraghumitra Is there anything else needed in terms of updates on meta
alerts themselves (underlying alerts aren't affected by this stuff)?
> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated
> incorrectly with only the "metaalert" field. This causes subsequent findOne
> operations to only return the "metaalert field for that alert. All fields
> should continue to be present.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created
[
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247449#comment-16247449
]
ASF GitHub Bot commented on METRON-1289:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150230733
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
---
@@ -256,59 +256,91 @@ public Document getLatest(final String guid, final
String sensorType) throws IOE
return ret.orElse(null);
}
+ @Override
+ public Iterable getAllLatest(final Collection guids,
final Collection sensorTypes) throws IOException {
+List documents = searchByGuids(
+guids
+, sensorTypes
+, hit -> {
+ Long ts = 0L;
+ String doc = hit.getSourceAsString();
+ String sourceType =
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+ try {
+return Optional.of(new Document(doc, hit.getId(), sourceType,
ts));
+ } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve latest: "
+ e.getMessage(), e);
+ }
+}
+
+);
+return documents;
+ }
+
+ Optional searchByGuid(String guid, String sensorType,
+ Function callback) {
+Collection sensorTypes = sensorType != null ?
Collections.singleton(sensorType) : null;
+List results = searchByGuids(Collections.singleton(guid),
sensorTypes, callback);
+if (results.size() > 0) {
+ return Optional.of(results.get(0));
+} else {
+ return Optional.empty();
+}
+ }
+
/**
* Return the search hit based on the UUID and sensor type.
* A callback can be specified to transform the hit into a type T.
* If more than one hit happens, the first one will be returned.
*/
- Optional searchByGuid(String guid, String sensorType,
+ List searchByGuids(Collection guids, Collection
sensorTypes,
Function callback) {
QueryBuilder query;
-if (sensorType != null) {
- query = QueryBuilders.idsQuery(sensorType + "_doc").ids(guid);
+if (sensorTypes != null) {
+ String[] types = sensorTypes.stream().map(sensorType -> sensorType +
"_doc").toArray(String[]::new);
+ query = QueryBuilders.idsQuery(types).ids(guids);
} else {
- query = QueryBuilders.idsQuery().ids(guid);
+ query = QueryBuilders.idsQuery().ids(guids);
}
SearchRequestBuilder request = client.prepareSearch()
--- End diff --
This is only going to return 10 max results, from the default size on the
query. This was fine before (since we only returned one), but it's going to be
incomplete with the refactor.
> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated
> incorrectly with only the "metaalert" field. This causes subsequent findOne
> operations to only return the "metaalert field for that alert. All fields
> should continue to be present.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
