subject:"\[jira\] \[Commented\] \(METRON\-939\) Upgrade ElasticSearch and Kibana"

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-02-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359635#comment-16359635
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user wardbekker commented on the issue:

https://github.com/apache/metron/pull/619
  
agreed @nickwallen 


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>Priority: Major
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-02-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359636#comment-16359636
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user wardbekker closed the pull request at:

https://github.com/apache/metron/pull/619


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>Priority: Major
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-02-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359575#comment-16359575
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/619
  
This functionality was completed in #840.  As mentioned in #840 this 
inspired much of that work.  Is there anything else needed from this PR?  If 
not, can you close this PR @wardbekker ?

Thanks




> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>Priority: Major
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316806#comment-16316806
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/840


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316804#comment-16316804
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/840
  
I want to pile on and give this my (non-binding since I contributed PRs 
against this PR) +1.  LGTM!


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316772#comment-16316772
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/840
  
At this point, I'm +1 since @merrimanr ran up the e2e tests.  A couple 
people have put a fair amount of testing into this, and it seems like at this 
point we're at parity and not finding more issues.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316771#comment-16316771
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/840
  
I ran this up in full dev again and verified the e2e tests now work similar 
to how they do in master.  I also manually tested several other areas including 
the Alerts UI, Kibana and Swagger.  Everything works as expected.  Assuming 
others are still testing but +1 from me.  


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314074#comment-16314074
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Worked with @merrimanr to fix the e2e test issue and just submitted a fix. 
This does not fix the full e2e test runs as this is being handled by 857, 
however it brings this PR back to parity with current functionality in master.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313030#comment-16313030
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159866359
  
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
 ${global_surefire_version}
 
 
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --

I played with this a bit and I'm not convinced this works in master 
anymore, so if this was necessary to get this running, I'd rather push fixing 
coverage off to a follow on task.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312073#comment-16312073
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/840
  
I spun this up in full dev and spent all day testing it.  From a functional 
perspective, I can not find anything wrong with it.  I ran through the test 
plan in this PR and everything worked as expected.  I also tested this 
exhaustively with the Alerts UI and Swagger UI and everything works great.

The only issue I found are with the Alerts UI e2e tests.  These no longer 
run at all and I suspect it's because the templates have changed with ES 5.6.2 
so loading e2e test data and template no longer works.  I'm not sure that this 
should hold up this PR since e2e tests are actively being worked on in 
https://github.com/apache/metron/pull/857 but I wanted everyone to be aware.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311516#comment-16311516
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159681034
  
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
 ${global_surefire_version}
 
 
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --

It's the top level pom, so no code coverage at all, iirc. I haven't looked 
at it in awhile.  I'll play with it a bit and see what's going on


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311514#comment-16311514
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159680669
  
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
 ${global_surefire_version}
 
 
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --

The build was failing with this argLine. Though, I can't recall the exact 
error now as it's been a while. What's the impact of leaving it off? No code 
coverage for this module?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311512#comment-16311512
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159680218
  
--- Diff: metron-deployment/README.md ---
@@ -1,3 +1,16 @@
+# Metron Deployment
--- End diff --

All set


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311466#comment-16311466
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
This is going to take some time to resolve, but everything else in this 
branch is still unhindered. 
https://github.com/apache/metron/commit/01c26a77b1041204b0bbbc544cc0a5d02e9339a8#diff-55e8119c8b8ae56260305e01c354d04b


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311460#comment-16311460
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159674941
  
--- Diff: 
metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md ---
@@ -441,7 +441,7 @@ Client
 
 - Kibana:
 * Set "kibana_es_url" to 
`http://:9200`. 
"replace_with_elasticsearch_master_hostname" is the IP of the node where you 
assigned ElasticSearch Master on the Assign Master tab.
-* Change kibana_default_application to "dashboard/Metron-Dashboard"
+* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9"
--- End diff --

They changed the links to point to index keys. I was bummed about this as 
well.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311458#comment-16311458
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Looking at the additional deployment readme merge conflicts now


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-02 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308402#comment-16308402
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159266301
  
--- Diff: 
metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md ---
@@ -441,7 +441,7 @@ Client
 
 - Kibana:
 * Set "kibana_es_url" to 
`http://:9200`. 
"replace_with_elasticsearch_master_hostname" is the IP of the node where you 
assigned ElasticSearch Master on the Assign Master tab.
-* Change kibana_default_application to "dashboard/Metron-Dashboard"
+* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9"
--- End diff --

 Is it possible to provide an easily identifiable name, or are we stuck 
with the "AV-..." ugliness?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-02 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308401#comment-16308401
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159273549
  
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
 ${global_surefire_version}
 
 
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --

This was in for code coverage via JaCoCo. It basically overrides the empty 
argline from above that got deleted. Was it causing problems with running 
things with it there?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2018-01-02 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308354#comment-16308354
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159265938
  
--- Diff: metron-deployment/README.md ---
@@ -1,3 +1,16 @@
+# Metron Deployment
--- End diff --

As a heads up, #883 is in now, so this will have to be taken care of when 
you merge master to deconflict.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-30 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306760#comment-16306760
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r159122591
  
--- Diff: metron-deployment/README.md ---
@@ -1,3 +1,16 @@
+# Metron Deployment
--- End diff --

Can you please add the license header to this? 
https://github.com/apache/metron/pull/884 is close to going in and enforcing 
this, so I'm hoping to avoid impact to master.

```

```


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295255#comment-16295255
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r157540388
  
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -1,5 +1,14 @@
 # Elasticsearch in Metron
 
+## Table of Contents
+
+* [Introduction](#introduction)
+* [Properties](#properties)
+* [Upgrading to 5.6.2](#upgrading-to-562)
+* [Type Mappings](#type-mappings)
+* [Using Metron with Elasticsearch 
5.x](#using-metron-with-elasticsearch-5x)
--- End diff --

This link is broken, because the section name doesn't line up with the 
actual name


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295254#comment-16295254
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r157540768
  
--- Diff: metron-platform/metron-elasticsearch/README.md ---
@@ -33,7 +42,217 @@ For instance, an `es.date.format` of `.MM.dd.HH` 
would have the consequence
 roll hourly, whereas an `es.date.format` of `.MM.dd` would have the 
consequence that the indices would
 roll daily.
 
-## Using Metron with Elasticsearch 2.x
+## Upgrading to 5.6.2
+
+Users should be prepared to re-index when migrating from Elasticsearch 
2.3.3 to 5.6.2. There are a number of template changes, most notably around
+string type handling, that may cause issues when upgrading.
+

+[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html)
+
+Be aware that if you add a new string value and want to be able to filter 
and search on this value from the Alerts UI, you **must** add a mapping for 
that type to
+the appropriate Elasticsearch template. Below is more detail on how to 
choose the appropriate mapping type for your string value.
+
+## Type Mappings
+
+Type mappings have changed quite a bit from ES 2.x -> 5.x. Here is a brief 
rundown of the biggest changes. More detailed references from Elasticsearch
+are provided in the [Type Mapping References](#type-mapping-references) 
section below.
+* string fields replaced by text/keyword type
+* strings have new default mappings as follows
+
+```
+{
+  "type": "text",
+  "fields": {
+"keyword": {
+  "type": "keyword",
+  "ignore_above": 256
+}
+  }
+}
+```
+
+* There is no longer a `_timestamp` field that you can set "enabled" on. 
This field now causes an exception on templates.
+Replace with an application-created timestamp of "date" type.
+
+The semantics for string types have changed. In 2.x, you have the concept 
of index settings as either "analyzed" or "not_analyzed" which basically means 
"full text" and "keyword", respectively.
+Analyzed text basically means the indexer will split the text using a text 
analyzer thus allowing you to search on substrings within the original text. 
"New York" is split and indexed as two buckets,
+ "New" and "York", so you can search or query for aggregate counts for 
those terms independently and will match against the individual terms "New" or 
"York." "Keyword" means that the original text
+ will not be split/analyzed during indexing and instead treated as a whole 
unit, i.e. "New" or "York" will not match in searches against the document 
containing "New York", but searching on "New York"
+ as the full city name will. In 5.x language instead of using the "index" 
setting, you now set the "type" to either "text" for full text, or "keyword" 
for keywords.
+
+Below is a table depicting the changes to how String types are now handled.
+
+
+
+   sort, aggregate, or access values
+   ES 2.x
+   ES 5.x
+   Example
+
+
+   no
+   
+"my_property" : {
+  "type": "string",
+  "index": "analyzed"
+}
+
+   
+   
+"my_property" : {
+  "type": "text"
+}
+
+Additional defaults: "index": "true", "fielddata": "false"
+   
+   
+   "New York" handled via in-mem search as "New" and "York" 
buckets. No aggregation or sort.
+   
+
+
+   
+   yes
+   
+   
+"my_property": {
+  "type": "string",
+  "index": "analyzed"
+}
+
+   
+   
+"my_property": {
+  "type": "text",
+  "fielddata": "true"
+}
+
+   
+   
+   "New York" handled via in-mem search as "New" and "York" buckets. 
Can aggregate and sort.
+   
+
+
+   
+   yes
+   
+   
+"my_property": {
+  "type": "string",
+  "index": "not_analyzed"
+}
+
+   
+   
+"my_property" : {
+  "type": "keyword"
+}
+
+   
+   
+   "New York" searchable as single value. Can aggregate 
and sort. A search for "New" or "York" will not match against the whole value.
+   
+
+
+   
+   yes
+   
+   
+"my_property": {
+  "type": "string",
+  "index": "analyzed"
+}
+
+   
+   
+"my_property": {
+  "type": "text",
+  "fields": {
+"keyword": {
+  "type": "keyword",
+  "ignore_above": 256
+}
+  }
+}
+
+   
+   
+   "New York"

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292889#comment-16292889
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Just a status update on this. We're currently waiting for 0.4.2 to roll out 
before this gets committed. We definitely want more eyes and testing on this PR 
considering its breadth and size. We do not have any +1's yet, and I would 
prefer to have at least 2 for good measure, if at all possible.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>  Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277579#comment-16277579
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
Yeah, I'm trying to think of the right way to do it, without having to have 
it be "officially maintained".
more like an informal /testing_stuff , with some descriptions, when it was 
valid/written against etc.

Something like that.  Then people could edit them and adapt them etc.





> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277562#comment-16277562
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
I don't think that's a bad idea. I definitely like having test scripts 
associated with the PR's because we can explicitly see what was done and/or 
recommended at the time the PR was being ushered into master. But we could also 
start to consolidate these into manual testing scripts that we reference 
ongoing. Then you could say something like:

```
Testing Plan
- es basic search test
- meta alerts search test
- kibana dashboard smoke test
```

That could be links, or we could copy-paste the current manual test plan 
from the source tree. I think the main thing I would want is to have the PR's 
test plan (here in the comments) be maintained statically for posterity based 
on the code at that time. For example, since we're removing the data pruner, 
the new test plan would not have a data pruner test, but I'd want to make sure 
any earlier PRs that did have a test plan for it can still be viewed 
statically. Basically, you'd either include the latest commit as part of the 
link or copy-paste the test plan. 


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277495#comment-16277495
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
So, this has me thinking   It is a shame to have these test materials ( 
and others like @JonZeolla creates ) embedded in these pr's.

Maybe we should have someplace to put them in the code tree?




> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-12-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277487#comment-16277487
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Ok, here is the remaining bit of the test plan for verifying ES. Thanks 
@cestella  for pulling together some of the public test scripts around this.

# Test Script

Testing Kibana dashboards - see 
[here](https://github.com/apache/metron/pull/840#issuecomment-348085037)

Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).

# Preliminaries

Set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.4.2` 

# Deploy the dummy parser
* Edit `$METRON_HOME/config/zookeeper/parsers/dummy.json`:
```
{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"dummy"
}
```
* Create the dummy kafka topic:
  `/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 
--create --topic dummy --partitions 1 --replication-factor 1`
* Persist config changes: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i 
$METRON_HOME/config/zookeeper -z node1:2181`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z 
node1:2181 -s dummy`

# Send dummy data through
* Edit `~/msg.json` with the following content:
```
{ "guid" : "guid0", "sensor.type" : "dummy", "timestamp" : 100 }
```
* Send `msg.json` through to kafka via `cat ~/msg.json | 
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
node1:6667 --topic dummy`
* Validate data has been written to the index:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' 
```

## Test Case: Update via patch
* Patch the message in ES and create a new field 'project' by executing
  the following:
```
curl -u user:password -X PATCH --header 'Content-Type: application/json' 
--header 'Accept: */*' -d '{
  "guid" : "guid0",
"sensorType" : "dummy",
"patch" : [
{
  "op": "add"
, "path": "/project"
, "value": "metron"
}
  ]
}' 'http://node1:8082/api/v1/update/patch'
```
* Validate that the message has a field 'project':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
  "_source" : [ "project" ]
}
'
```

## Test Case: Update via replace 
* Replace the message in ES and create a couple of modifications:
  * new field `new_field` == "brand new"
  * modified `timestamp` == 7
Execute the following:
```
curl -u user:password -X POST --header 'Content-Type: application/json' 
--header 'Accept: */*' -d '{
 "guid" : "guid0",
 "sensorType" : "dummy",
 "replacement" : {
   "source:type": "dummy",
   "guid" : "guid0",
   "new_field" : "brand new",
   "timestamp" : 7
  }
   }' 'http://node1:8082/api/v1/update/replace'
```
* Validate that the message has a field 'new_field':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
  "_source" : [ "new_field", "timestamp" ]
}
'
```
## Meta Alerts Test

### Set Up Base Data
We're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data
```
curl 'node1:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted 
store.size pri.store.size
green  open   snort_index_2017.09.06.14   1   01300
180.9kb180.9kb
green  open   bro_index_2017.09.06.14 1   01600
564.3kb564.3kb
green  open   .kibana 1   0 520 
71.2kb 71.2kb
green  open   metaalert_index 1   0  60 
62.3kb 62.3kb
```
In this case, we care about `snort_index_2017.09.06.14` and 
`metaalert_index`.  To make our lives easier, we'll add a couple of stripped 
down messages to the our snort index (Make sure to sub in the correct index 
name):
```
curl -XPUT 
'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H 
'Content-Type: application/json' -d'
{
  "msg": "snort test alert",
  "ip_dst_port": "8080",
  "ethsrc": "0A:00:27:00:00:00",
  "protocol": "TCP",
  "source:type": "snort",
  "ip_dst_addr": "192.168.66.121",
  "ip_src_addr": "192.168.66.1",
  "threat:triage:rules:0:score": 10,
  "timestamp": 1504708744000,
  "threat:triage:rules:0:reason": null,
  "threat:triage:score": 10,

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272207#comment-16272207
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
For reference, I've added some PDF snapshots of the dashboards to the Jira

- 
https://issues.apache.org/jira/secure/attachment/12899952/Metron-Dashboard%20-%20Kibana.pdf
- 
https://issues.apache.org/jira/secure/attachment/12899951/Metron-Error-Dashboard%20-%20Kibana.pdf


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - 
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272176#comment-16272176
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Ok, the Metron error dashboard is in now. I'll add some additional testing 
instructions tomorrow, but this PR should be ready for some more vigorous 
testing. The most easily accessible e2e place to start is with the Kibana 
dashboards. Checking Kibana minimally proves that data is flowing through the 
system with the new versions of Elasticsearch and Kibana. The default main 
dashboard should come up immediately with Bro and Snort data only. 

I recommend first stopping some services:

```
service monit stop
storm kill profiler
```

Yaf is not enabled by default, but you can start the Yaf topology and turn 
on all sensor stubs by doing the following:

```
# start yaf parser topology
export METRON_HOST=node1
export ZOOKEEPER=${METRON_HOST}:2181
export BROKERLIST=${METRON_HOST}:6667
export METRON_VERSION=0.4.2
export METRON_HOME=/usr/metron/${METRON_VERSION}
$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s 
yaf
# start the yaf sensor stub
service sensor-stubs start yaf
```

Now publish some intentionally bad data to  each of the topics so we can 
check the error dashboard.

```
# publish error data on a cycle
cycleval=1; 
while true; 
do 
if [ $cycleval -ne 0 ]; 
then 
datestamp=$(date "+%s");
cycleval=0;
else
cycleval=1;
fi;
echo "bro-garbage-" $datestamp | 
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
$BROKERLIST --topic bro;
echo "snort-garbage-" $datestamp | 
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
$BROKERLIST --topic snort;
echo "yaf-garbage-" $datestamp | 
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
$BROKERLIST --topic yaf;
sleep 2;
done;
```


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267592#comment-16267592
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
Yes, I agree. It completely makes sense to minimize the scope and work on 
stabilizing this version at this moment. 


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267250#comment-16267250
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r153289006
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
 ---
@@ -102,13 +94,25 @@
   "match_mapping_type": "*"
 }
   },
-  {
-"threat_triage_reason": {
-  "mapping": {
-"type": "string"
-  },
-  "match": "threat:triage:rules:*:reason",
-  "match_mapping_type": "*"
+{
+  "threat_triage_reason": {
+"mapping": {
+  "type": "text",
+  "fielddata": "true"
+},
+"match": "threat.triage.rules:*:reason",
+"match_mapping_type": "*"
+  }
+},
+{
+  "threat_triage_name": {
+"mapping": {
+  "type": "text",
+  "fielddata": "true"
+},
+"match": "threat.triage.rules:*:name",
+"match_mapping_type": "*"
+  }
 }
   },
--- End diff --

This brace is extraneous, I'm guessing a merge broke it. Drop it, but keep 
the comma and we should be good.  I'd just get the formatting lined back up 
while you're in there.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267222#comment-16267222
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
For reference, here is a list of some of the follow-on work we should 
consider:

- Improvements to Kibana dashboard
- Add new timestamp field to parsers and index templates to take place of 
_timestamp
- Fix Log4j logging problem - classpath issues
- Upgrade to ES 6.x
- Migrate to new ES REST client
- Improved config management for ES 5.x+ in Ambari
- New field name conventions?



> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267214#comment-16267214
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
@mraliagha I do think we should consider revisiting the field name 
conventions, but I'd push for that as a follow-on task. As discussed in other 
points on this thread, e.g. going straight to ES 6.x, this PR is already 
massive in scope, and adding further non-critical improvements will increase 
the surface area of those changes. 

Now that we have the meta alerts fixes in master, I strongly encourage the 
community to stabilize around as minimal an ES upgrade as possible with 
incremental improvements and feature enhancements once we've been able to 
convince ourselves through the unit and integration test suites and functional 
testing that we're in a healthy and stable place with minimal risk of adding 
regressions.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266590#comment-16266590
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
One of the issues that we had with the previous mpack was the lack of 
config segregation for Elasticsearch Master Nodes and Data Nodes. Hence, we 
have ended up hardcoding specific configuration outside Ambari. With ES 5 it 
will be even more complex due to the existence of Ingestion Nodes and ML Nodes. 
Therefore, it would be really nice if we can have multiple tabs in Ambari ES 
service. For example, one tab for generic configurations and another tab per 
each node type to segregate specific configurations from generic ones.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266513#comment-16266513
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
Is this the best time to ask for changing field name convention to avoid 
dot or colon? We are externally using Hive external tables on HDFS data, due to 
Hive limitations we need to change the Metron field convention. I heard there 
is a long term plan in future to use ORC files instead of JSON and maybe Hive 
table can be supported directly. If this is right maybe this is the best time 
we can move towards changing field seperators accordingly.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263304#comment-16263304
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Modified the Kibana chart colors per @ottobackwards request. I don't know 
if Kibana ever offered fine-grained control over the bar chart colors, but they 
do not offer it now. This is as close as I was able to get.


![image](https://user-images.githubusercontent.com/658443/33148607-b5d7236c-cf89-11e7-8536-7aebd70b9a4b.png)



> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-14 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252820#comment-16252820
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840

I was looking at that today as well, and I think that should be a follow-on
considering how large the change is moving from 2.x to 5.x. The changes
I've implemented (and Casey and Nick) for 5.x should make moving to 6.x
much easier once we're ready for it, but we should probably stabilize
master on this before another upgrade. Also, we have a number of fixes to
meta alerts that I need to merge with this branch, which will also take
some time. There are also some enhancements to the dashboard that we're
tossing around as well.

On Nov 14, 2017 6:46 PM, "ottobackwards"  wrote:

> 6.0 was released, should we consider going to that while we are at it?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>

> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).

--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-14 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252814#comment-16252814
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
6.0 was released, should we consider going to that while we are at it?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248335#comment-16248335
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
The only issue with it, is if we did that in the old version, then this 
would be a regression.  Other than that, I'm just waiting to review the docs ;)


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248260#comment-16248260
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Whew, that's great news!

Hahahahaha, that's the default colors from the Kibana widgets... I usually 
refrain from bringing it up, but I'm partially colorblind (even though I used 
to do a fair amount of Photoshop graphic design work on the side, but it's a 
bit more effort for me). Anyhow, Kibana had no easy way to pick color names or 
supply RGB values, so I punted on it for now because I figured that's an easy 
change. If you would be willing to take a screenshot or the xy coordinate of 
colors in the color picker that you'd prefer, I'd be more than happy to change 
it.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248249#comment-16248249
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
Ran up in full dev, everything running normally, kibana dashboard has data. 
 Is the color scheme supposed to be so 'miami vice'?



> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248180#comment-16248180
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
@ottobackwards You should be good to try it now.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247940#comment-16247940
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
I got full_dev failure on connecting to metron web.  Can you post when you 
think full dev is ready and I'll try again then?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247911#comment-16247911
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
Ok, looks like 2 more minor issues.
1. params.py missing the kibana_server_host, so it's not making its way to 
kibana.yml
2. The default index mapping template that ES creates when I index the 
Kibana configuration is not compatible with Kibana.

I'm done fixing 1 already and have the solution for 2. I'm going to blow 
away the .kibana index, let Kibana recreate it with the expected template, and 
then extract that into a template file that's used during the Kibana server 
install. The only hitch is that this is one more thing that potentially needs 
to be modified/updated on future upgrades of ES and Kibana.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247784#comment-16247784
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
Thanks @mmiklavc, those changes look good.  Running up in full dev


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247750#comment-16247750
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150282327
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py
 ---
@@ -24,6 +24,7 @@
 
 from ambari_commons.os_check import OSCheck
 from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl
--- End diff --

Heh, yep. That extra import broke the Kibana install hard. I've made most 
of your requested changes and will have another commit coming shortly.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247749#comment-16247749
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150281496
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
 ---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --

Yeah, I think you're right. I really like having data inline so that you 
can see the full context of your tests without having to follow multiple 
branches to piece together what actually happens. I think it's ok for the 
system under test, but the tests themselves should be much more intuitive and 
easy to follow. So I think the right answer here is that I've included a lot of 
extra noise that is unnecessary for the test. I'm going to prune the "_source" 
object, which will make this very lean and much easier to read. Really good 
catch @ottobackwards.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247730#comment-16247730
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150276834
  
--- Diff: metron-platform/metron-data-management/pom.xml ---
@@ -231,11 +231,11 @@
 httpclient
 ${httpcore.version}
 
--- End diff --

Yep, absolutely. More cleanup.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247719#comment-16247719
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150274541
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -99,7 +148,15 @@
   
 
 
--- End diff --

Yes, absolutely agreed.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247715#comment-16247715
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150274108
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py
 ---
@@ -24,6 +24,7 @@
 
 from ambari_commons.os_check import OSCheck
 from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl
--- End diff --

Hm, I'm not sure where that came from. I suspect IntelliJ added that in the 
middle of my edits.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247707#comment-16247707
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
@mraliagha @cestella agreed on timestamp.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247703#comment-16247703
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
Can you call the curator stuff out in your checklist above?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247702#comment-16247702
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
Ok, killing my vagrant up then, until your commit


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247695#comment-16247695
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
@ottobackwards Oh, afa the data pruner, I believe we can use Curator now as 
a built-in mechanism for pruning data. Steps will be added to the documentation.


https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247689#comment-16247689
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150269845
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
 ---
@@ -48,19 +51,30 @@ def slave():
  content=InlineTemplate(params.elastic_env_sh_template)
  )
 
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --

Hm, adding to my list of cleanup tasks.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247675#comment-16247675
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/840
  
Are there any required changes to the ansible roles?  I thought we injected 
the templates from there.  Or has the indexing service injection landed?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247673#comment-16247673
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150266275
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
 ---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --

Is there a tipping point where this should be a file vs. multiline?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247668#comment-16247668
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150265417
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExport.java
 ---
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.FileReader;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.common.utils.JSONUtils;
+
--- End diff --

javadoc


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247671#comment-16247671
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150265577
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -139,13 +141,17 @@ protected SearchResponse search(SearchRequest 
searchRequest, QueryBuilder queryB
 searchRequest.getSort().forEach(sortField -> 
searchSourceBuilder.sort(sortField.getField(), 
getElasticsearchSortOrder(sortField.getSortOrder(;
 Optional fields = searchRequest.getFields();
 if (fields.isPresent()) {
--- End diff --

why not just remove?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247667#comment-16247667
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150264904
  
--- Diff: metron-platform/metron-data-management/pom.xml ---
@@ -231,11 +231,11 @@
 httpclient
 ${httpcore.version}
 
--- End diff --

Why not just remove these?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247670#comment-16247670
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150264674
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -99,7 +148,15 @@
   
 
 
--- End diff --

Maybe a comment as to why we are doing relocations will help future 
maintainers


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247669#comment-16247669
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150259383
  
--- Diff: dependencies_with_url.csv ---
@@ -310,6 +310,29 @@ 
org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RE
 
org.springframework.kafka:spring-kafka:jar:1.1.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-kafka
 
ch.hsr:geohash:jar:1.3.0:compile,ASLv2,https://github.com/kungfoo/geohash-java
 
org.locationtech.spatial4j:spatial4j:jar:0.6:compile,ASLv2,https://github.com/locationtech/spatial4j
--- End diff --

can we remove any of the older dependencies?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247672#comment-16247672
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150262722
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
 ---
@@ -48,19 +51,30 @@ def slave():
  content=InlineTemplate(params.elastic_env_sh_template)
  )
 
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --

I don't think "Cre" is what you wanted here, typo?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247666#comment-16247666
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r150263736
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py
 ---
@@ -24,6 +24,7 @@
 
 from ambari_commons.os_check import OSCheck
 from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl
--- End diff --

Is the ansible module something we are going to distribute?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247581#comment-16247581
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/840
  
@mraliagha Yeah, absolutely.   This task purposefully leaves out major 
changes that we could do to improve things (e.g. moving back to `.` separated 
fields instead of `:`).  The attempt is to touch as few things as possible.  
Your ask makes a lot of sense and we should add it as a follow-on task.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247096#comment-16247096
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
It might not be completely related to this PR, but since _timestamp is 
removed in ES 5.x, can we add a specific time of indexing at indexing bolt to 
capture time of indexing. It is useful for the purpose of benchmarking as well 
as evaluation of SLA.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-11-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247028#comment-16247028
 ] 

ASF GitHub Bot commented on METRON-939:
---

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/840

METRON-939: Upgrade ElasticSearch and Kibana

DO NOT MERGE

This PR builds on the great work in 
https://github.com/apache/metron/pull/619

## Contributor Comments
I wanted to get eyes on this despite some outstanding work left. It was no 
small undertaking moving 3 major versions, so there's quite a bit to review. In 
the future I would like to recommend we make feature branches for changes of 
this magnitude. 

This work upgrades Elasticsearch and Kibana both to 5.6. I'm running a 
local full dev build now, but assuming no failures this will spin up full dev 
with everything, including the Kibana dashboard (had to be completely 
rewritten).

### TODO
- [ ] Another merge with master
- [ ] Update docs
- [ ] Docs for managing Kibana dashboards (no more pickle file - straight 
JSON using ES's bulk load API. This was not as easy as you would think it would 
be.)
- [ ] Test with Kerberos


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron elasticsearch562

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/840.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #840


commit 0805d20b95e3cdcf55bb0dfde91a08d8b9f58395
Author: Michael Miklavcic 
Date:   2017-10-02T18:51:23Z

Version changes for MPack

commit 889c1bd5d24ee359d7d8113f500539ea10750a40
Author: Michael Miklavcic 
Date:   2017-10-04T14:55:08Z

save demo code

commit 42f82180208dd7235214810406596bfe896665ab
Author: Michael Miklavcic 
Date:   2017-10-17T12:12:43Z

Working through tests

commit 084db5ab4258a1dacb792ba504c1aeb18838327d
Author:

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171632#comment-16171632
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/619
  
As a note, this ticket is slightly impacted by the metaalerts backend 
ticket (https://github.com/apache/metron/pull/734).  The alerts field in the 
various templates should be removed and the search queries for meta alerts 
updated according to 
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields,
 in order to allow for searches against metaalerts without having to have an 
alert field in each template.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-07-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103126#comment-16103126
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/619
  
Seems like this is failing on some un-related temporary test failures. Can 
we get Travis kicked, and see what's left to do on this?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-07-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077509#comment-16077509
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/619
  
Do we not have to re-write the ambari service?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-07-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075074#comment-16075074
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user wardbekker commented on the issue:

https://github.com/apache/metron/pull/619
  
@JonZeolla done. 


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-07-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075029#comment-16075029
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/619
  
@wardbekker Can you please merge master and deconflict?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-06-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064875#comment-16064875
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user wardbekker commented on the issue:

https://github.com/apache/metron/pull/619
  
hey @cestella, @simonellistonball, see updated contributor notes. It's not 
ready for a official pull request, but this gives a good idea on the impact on 
the code for a working ES5.x implementation. 


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-06-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054446#comment-16054446
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user wardbekker commented on the issue:

https://github.com/apache/metron/pull/619
  
Thanks for the review @cestella I've pushed a commit fixing the reported 
issues.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-06-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054057#comment-16054057
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/619#discussion_r122715927
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -102,40 +103,57 @@ public void start() throws UnableToStartException {
 throw new UnableToStartException("Unable to clean log or data 
directories", e);
 }
 
-Settings.Builder settingsBuilder = Settings.settingsBuilder()
-.put("node.http.enabled", true)
-.put("http.port", httpPort)
+Settings.Builder settingsBuilder = Settings.builder()
+.put("cluster.name", "metron")
 .put("path.logs",logDir.getAbsolutePath())
 .put("path.data",dataDir.getAbsolutePath())
 .put("path.home", indexDir.getAbsoluteFile())
-.put("index.number_of_shards", 1)
-.put("node.mode", "network")
-.put("index.number_of_replicas", 1);
+.put("transport.type", "netty4");
+
 
 if(extraElasticSearchSettings != null) {
 
 settingsBuilder = 
settingsBuilder.put(extraElasticSearchSettings);
-
 }
 
-node = 
NodeBuilder.nodeBuilder().settings(settingsBuilder).clusterName("metron").node();
-node.start();
+Collection plugins = Collections.singletonList(Netty4Plugin.class);
 
+node = new PluginConfigurableNode(settingsBuilder.build(), 
plugins);
 client = node.client();
 
-waitForCluster(client, ClusterHealthStatus.YELLOW, new 
TimeValue(6));
+waitForCluster(node, ClusterHealthStatus.YELLOW, new 
TimeValue(6));
 
 }
 
-public static void waitForCluster(ElasticsearchClient client, 
ClusterHealthStatus status, TimeValue timeout) throws UnableToStartException {
+private static class PluginConfigurableNode extends Node {
+PluginConfigurableNode(Settings settings, Collection classpathPlugins) {
+super(InternalSettingsPreparer.prepareEnvironment(settings, 
null), classpathPlugins);
+}
+}
+
+public static void waitForCluster(Node node, ClusterHealthStatus 
status, TimeValue timeout) throws UnableToStartException {
 try {
+   node.start();
+
 ClusterHealthResponse healthResponse =
-
(ClusterHealthResponse)client.execute(ClusterHealthAction.INSTANCE, new 
ClusterHealthRequest().waitForStatus(status).timeout(timeout)).actionGet();
+(ClusterHealthResponse) 
node.client().execute(ClusterHealthAction.INSTANCE, new 
ClusterHealthRequest().waitForStatus(status).timeout(timeout)).actionGet();
 if (healthResponse != null && healthResponse.isTimedOut()) {
 throw new UnableToStartException("cluster state is " + 
healthResponse.getStatus().name()
 + " and not " + status.name()
 + ", from here on, everything will fail!");
 }
+
+byte[] indexTemplate = new byte[0];
+try {
+indexTemplate = 
Files.readAllBytes(Paths.get("/Users/wbekker/metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template"));
--- End diff --

This path should be relative.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

2017-06-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054058#comment-16054058
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/619#discussion_r122716256
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -178,8 +195,11 @@ public boolean hasIndex(String indexName) {
 
 @Override
 public void stop() {
-node.close();
+try {
+node.close();
+} catch (IOException e) {
+e.printStackTrace();
--- End diff --

We either want to throw up the exception or at least log it with a logger, 
rather than stdout.


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

76 matches

Mail list logo