[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
ok - so mvn archetype:generate uses the newer version of the plugin if run 
in an empty directory.
If run in a directory where there is a pom with plugin setting for 2.4 and 
it will honor that.

So the issue with the missing catalog file and it not working has to do 
with the version you are running with and the version you are installing with, 
and if you are under a pom structure that keeps them the same.

I don't know how that helps, but it solves that mystery.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
hey @mmiklavc .  Looking back at my logs above, no matter that the 
maven-archetype-plugin was 2.4 in the pom at the time, it shows 3.0.1 in the 
output.  Can you check what version it shows in your output?  



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I also may have a fix for the missing catalog file issue ( or a way to run 
the command differently )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
Ok, I tried with a new M2 directory and the can't find local archetype 
issue returned.  

 
  

  maven-archetype-plugin
  3.0.1

  


Which is up from 2.4  resolved that issue, and after that I still cannot 
reproduce.
Is going to 3.0.1 viable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #682: METRON-1081: Fix Alerts and Ops UI Notices file

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/682
  
Are the modules/versions pulled in dependent on the version of node or 
something?  My list is different.  I have more things, and different versions 
for some.

[licenses.txt](https://github.com/apache/metron/files/1199144/licenses.txt)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I'm going to try with a new .m2 dir


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I hate to ask but are you sure you have the latest code?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #666: METRON-1051: Enable the ability to update indexed message...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/666
  
Thanks for making the changes, I'm +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #666: METRON-1051: Enable the ability to update indexed message...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/666
  
Alright, I think all of the concerns thus far are addressed.  Let me know 
if I missed anything @justinleet , et al


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Hrm, I completely blew away both archetype catalogs and re-installed our 
archetype again. It shows up with the correct version. However, the parser 
folders are still not being created correctly. As one would expect, this also 
causes building the extension to fail bc the dir names don't match the expected 
module names.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
```bash
┌─[ottofowler@Winterfell] - [~/tmp/HelloParser] - [Thu Aug 03, 19:12]
└─[$]> cat ~/.m2/repository/archetype-catalog.xml

http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0
 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd;

xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
  

  org.apache.metron
  metron-maven-parser-extension-archetype
  0.4.1
  Apache Maven Parser Extension Archetype for 
Metron

  

```

```bash
┌─[ottofowler@Winterfell] - [~/tmp/HelloParser] - [Thu Aug 03, 19:13]
└─[$]> cat ~/.m2/archetype-catalog.xml

http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0
 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd;

xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
  

  org.apache.nifi
  nifi-service-bundle-archetype
  1.2.0-SNAPSHOT
  Apache NiFi is an easy to use, powerful, and reliable 
system to process and distribute data.


  org.apache.nifi
  nifi-processor-bundle-archetype
  1.2.0-SNAPSHOT
  Apache NiFi is an easy to use, powerful, and reliable 
system to process and distribute data.


  org.apache.metron
  metron-maven-parser-extension-archetype
  0.4.1
  Apache Maven Parser Extension Archetype for 
Metron

  

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I left the Parser off of the HelloParser input, sorry


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
```
cat ~/.m2/repository/archetype-catalog.xml

http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0
 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd;

xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
  

  org.apache.metron
  metron-maven-parser-extension-archetype
  0.4.0
  Apache Maven Parser Extension Archetype for 
Metron

  


cat ~/.m2/archetype-catalog.xml

http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0
 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd;

xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
  
...

  org.apache.metron
  metron-maven-parser-extension-archetype
  0.4.1
  Apache Maven Parser Extension Archetype for 
Metron

  

```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I DO think that the archetype's input variables could use refactoring.  For 
example, I force the artifact name, and maybe I should not.

I do not know why the bundle plugin is looking for the $version, when the 
pom says $metronVersion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
```bash
┌─[ottofowler@Winterfell] - 
[~/.m2/repository/org/apache/metron/metron-maven-parser-extension-archetype/0.4.1]
 - [Thu Aug 03, 18:42]
└─[$]> ll
total 80
-rw-r--r--  1 ottofowler  staff   237B Aug  3 18:24 _remote.repositories
-rw-r--r--  1 ottofowler  staff29K Aug  3 18:24 
metron-maven-parser-extension-archetype-0.4.1.jar
-rw-r--r--  1 ottofowler  staff   1.8K Aug  3 18:17 
metron-maven-parser-extension-archetype-0.4.1.pom
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
```bash

┌─[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:36]
└─[$]> mvn archetype:generate -DarchetypeCatalog=local
[INFO] Scanning for projects...
[INFO]
[INFO] 

[INFO] Building Maven Stub Project (No POM) 1
[INFO] 

[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > 
generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < 
generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ 
standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart 
(org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
1: local -> org.apache.metron:metron-maven-parser-extension-archetype 
(Apache Maven Parser Extension Archetype for Metron)
Choose a number or apply filter (format: [groupId:]artifactId, case 
sensitive contains): : 1
Define value for property 'groupId': com.michaelmiklavcic
Define value for property 'artifactId': helloparser
Define value for property 'version' 1.0-SNAPSHOT: : 0.4.1
Define value for property 'package' com.michaelmiklavcic: :
Define value for property 'metronVersion': 0.4.1
Define value for property 'parserClassName' (should match expression 
'^[A-Z].*$'): Hello
Define value for property 'parserName' (should match expression 
'^[a-z]+[A-Z,a-z]+$'): hello
Confirm properties configuration:
groupId: com.michaelmiklavcic
artifactId: helloparser
version: 0.4.1
package: com.michaelmiklavcic
metronVersion: 0.4.1
parserClassName: Hello
parserName: hello
 Y: : y
[INFO] 

[INFO] Using following parameters for creating project from Archetype: 
metron-maven-parser-extension-archetype:0.4.1
[INFO] 

[INFO] Parameter: groupId, Value: com.michaelmiklavcic
[INFO] Parameter: artifactId, Value: helloparser
[INFO] Parameter: version, Value: 0.4.1
[INFO] Parameter: package, Value: com.michaelmiklavcic
[INFO] Parameter: packageInPathFormat, Value: com/michaelmiklavcic
[INFO] Parameter: parserName, Value: hello
[INFO] Parameter: package, Value: com.michaelmiklavcic
[INFO] Parameter: version, Value: 0.4.1
[INFO] Parameter: metronVersion, Value: 0.4.1
[INFO] Parameter: groupId, Value: com.michaelmiklavcic
[INFO] Parameter: parserClassName, Value: Hello
[INFO] Parameter: artifactId, Value: helloparser
[INFO] Parent element not overwritten in 
/Users/ottofowler/tmp/helloparser/metron-parser-hello/pom.xml
[INFO] Parent element not overwritten in 
/Users/ottofowler/tmp/helloparser/metron-parser-hello-bundle/pom.xml
[INFO] Parent element not overwritten in 
/Users/ottofowler/tmp/helloparser/metron-parser-hello-assembly/pom.xml
[INFO] Project created from Archetype in dir: 
/Users/ottofowler/tmp/helloparser
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 48.709 s
[INFO] Finished at: 2017-08-03T18:37:13-04:00
[INFO] Final Memory: 15M/245M
[INFO] 

┌─[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:37]
└─[$]> tree helloparser
helloparser
├── metron-parser-hello
│   ├── README.md
│   ├── pom.xml
│   └── src
│   ├── main
│   │   ├── config
│   │   │   ├── elasticsearch
│   │   │   │   └── hello_index.template
│   │   │   └── zookeeper
│   │   │   ├── enrichments
│   │   │   │   └── hello.json
│   │   │   ├── indexing
│   │   │   │   └── hello.json
│   │   │   └── parsers
│   │   │   └── hello.json
│   │   ├── java
│   │   │   └── com
│   │   │   └── michaelmiklavcic
│   │   │   └── hello
│   │   │   └── HelloParser.java
│   │   └── resources
│   │   ├── META-INF
 

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I'm not done the build, so I'll try this again, but this is what I get:  ( 
Note, I don't know how you are getting the packageInPathFormat option ):

┌─[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:18]
└─[$]> mvn archetype:generate -DarchetypeCatalog=local
[INFO] Scanning for projects...
[INFO]
[INFO] 

[INFO] Building Maven Stub Project (No POM) 1
[INFO] 

[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > 
generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < 
generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ 
standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart 
(org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
1: local -> org.apache.metron:metron-maven-parser-extension-archetype 
(Apache Maven Parser Extension Archetype for Metron)
Choose a number or apply filter (format: [groupId:]artifactId, case 
sensitive contains): : 1
Define value for property 'groupId': com.michaelmiklavcic
Define value for property 'artifactId': HelloParser
Define value for property 'version' 1.0-SNAPSHOT: : 1.0-SNAPSHOT
Define value for property 'package' com.michaelmiklavcic: : 
com.michaelmiklavcic
Define value for property 'metronVersion': 0.4.1
Define value for property 'parserClassName' (should match expression 
'^[A-Z].*$'): HelloParser
Define value for property 'parserName' (should match expression 
'^[a-z]+[A-Z,a-z]+$'): hello
Confirm properties configuration:
groupId: com.michaelmiklavcic
artifactId: HelloParser
version: 1.0-SNAPSHOT
package: com.michaelmiklavcic
metronVersion: 0.4.1
parserClassName: HelloParser
parserName: hello
 Y: : y
[INFO] 

[INFO] Using following parameters for creating project from Archetype: 
metron-maven-parser-extension-archetype:0.4.1
[INFO] 

[INFO] Parameter: groupId, Value: com.michaelmiklavcic
[INFO] Parameter: artifactId, Value: HelloParser
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: com.michaelmiklavcic
[INFO] Parameter: packageInPathFormat, Value: com/michaelmiklavcic
[INFO] Parameter: parserName, Value: hello
[INFO] Parameter: package, Value: com.michaelmiklavcic
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: metronVersion, Value: 0.4.1
[INFO] Parameter: groupId, Value: com.michaelmiklavcic
[INFO] Parameter: parserClassName, Value: HelloParser
[INFO] Parameter: artifactId, Value: HelloParser
[INFO] Parent element not overwritten in 
/Users/ottofowler/tmp/HelloParser/metron-parser-hello/pom.xml
[INFO] Parent element not overwritten in 
/Users/ottofowler/tmp/HelloParser/metron-parser-hello-bundle/pom.xml
[INFO] Parent element not overwritten in 
/Users/ottofowler/tmp/HelloParser/metron-parser-hello-assembly/pom.xml
[INFO] Project created from Archetype in dir: 
/Users/ottofowler/tmp/HelloParser
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 03:54 min
[INFO] Finished at: 2017-08-03T18:22:44-04:00
[INFO] Final Memory: 16M/309M
[INFO] 

┌─[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:22]
└─[$]> ls
HelloParser antlr   dummy   dummy-one   dummy-two
┌─[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:22]
└─[$]> tree HelloParser
HelloParser
├── metron-parser-hello
│   ├── README.md
│   ├── pom.xml
│   └── src
│   ├── main
│   │   ├── config
│   │   │   ├── elasticsearch
│   │   │   │   └── hello_index.template
│   │   │   └── zookeeper
│   │   │   ├── enrichments
│   │   │   │   └── hello.json
│   │   │   ├── indexing
│   │   │   │   └── hello.json
│   │   │   └── parsers
│   │   │   └── hello.json
│  

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
WRT: the archetype directories:  I don't see that ( but I'm going to 
re-build and try again ).
I don't use the same options you are using though.

Importantly -> the parser version MUST match the METRON version right now. 
So the 1.0-SNAPSHOT won't build.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #682: METRON-1081: Fix Alerts and Ops UI Notices file

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/682
  
I am not sure how to verify or review this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Ok, I see what that's doing now re: metron-parsers. As I was glancing over 
the project dirs my next question was going to be about what is in 
metron-parsers vs the extensions, but I follow. I do believe we discussed that 
a while back (this thread is long heh).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Will review your other comments shortly. Some additional questions:
I ran the archetype to create a new system parser:
```
[INFO] Parameter: groupId, Value: com.michaelmiklavcic
[INFO] Parameter: artifactId, Value: HelloParser
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: com.michaelmiklavcic
[INFO] Parameter: packageInPathFormat, Value: com/michaelmiklavcic
[INFO] Parameter: parserName, Value: hello
[INFO] Parameter: package, Value: com.michaelmiklavcic
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: metronVersion, Value: 0.4.1
[INFO] Parameter: groupId, Value: com.michaelmiklavcic
[INFO] Parameter: parserClassName, Value: HelloParser
[INFO] Parameter: artifactId, Value: HelloParser
```

Which results in the following directories and files. It seems that 
parserName should have been replaced, no?
```
HelloParser/metron-parser-__parserName__
HelloParser/metron-parser-__parserName__/pom.xml
HelloParser/metron-parser-__parserName__/README.md
HelloParser/metron-parser-__parserName__/src
HelloParser/metron-parser-__parserName__/src/main
HelloParser/metron-parser-__parserName__/src/main/config
HelloParser/metron-parser-__parserName__/src/main/config/elasticsearch

HelloParser/metron-parser-__parserName__/src/main/config/elasticsearch/hello_index.template
HelloParser/metron-parser-__parserName__/src/main/config/zookeeper

HelloParser/metron-parser-__parserName__/src/main/config/zookeeper/enrichments

HelloParser/metron-parser-__parserName__/src/main/config/zookeeper/enrichments/hello.json
HelloParser/metron-parser-__parserName__/src/main/config/zookeeper/indexing

HelloParser/metron-parser-__parserName__/src/main/config/zookeeper/indexing/hello.json
HelloParser/metron-parser-__parserName__/src/main/config/zookeeper/parsers

HelloParser/metron-parser-__parserName__/src/main/config/zookeeper/parsers/hello.json
HelloParser/metron-parser-__parserName__/src/main/java
HelloParser/metron-parser-__parserName__/src/main/java/com
HelloParser/metron-parser-__parserName__/src/main/java/com/michaelmiklavcic

HelloParser/metron-parser-__parserName__/src/main/java/com/michaelmiklavcic/hello

HelloParser/metron-parser-__parserName__/src/main/java/com/michaelmiklavcic/hello/HelloParserParser.java
...
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #682: modified: NOTICE

[james-sirota]

GitHub user james-sirota opened a pull request:

https://github.com/apache/metron/pull/682

modified:   NOTICE

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/james-sirota/metron jsirota/METRON-1081

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #682


commit aa516a642497db0f815f59fecc9e2cfd87adad28
Author: James Sirota 
Date:   2017-08-03T21:51:35Z

modified:   NOTICE




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
Ok - I think you are running find from /usr/metron/$V/
So - to explain the other config/zookeeper

So let's pretend there are two things here:
1. metron-parsers
2. parser-extensions

metron-parsers ( as described way above in PR descriptions) is:

- The Uber Jar loaded by storm
- Has all the Parser base classes and Interfaces
- Has the 'Raw' Parsers -> CSV, JSONMap, GROK

Any configuration for those parsers are installed into the root 
/usr/metron/$V/config... path.  They always where actually.  I did not move 
them, because these are not extensions.

Now, if you want to say that there should not be a default configuration 
for JSONMap under parsers... OK, but we already HAD these before.
https://user-images.githubusercontent.com/551/28945840-292df89c-7876-11e7-8c4d-5bbbdaf51ea4.png;>










---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
thanks @mmiklavc.  I don't think I've stated this in the actual PR

As a 3rd party ( or whatever you want to call it ) developer, who only 
makes Parser or possibly other extensions for metron, and does not maintain a 
fork of metron or change any metron code, I want to manage my extension as a 
completely self contained package.  That means everything that is needed to 
make my extension work is packaged and installed/deployed as one unit.   I do 
not want my enrichment configurations in some other library, or my indexing 
configurations for that matter.  I may never even have the metron code, only 
the maven repo.

This is a different mindset from the metron was obviously conceived, as a 
single project.

I am not certain what your second config/zookeeper refers to, can you give 
more detail.






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
I see that for parser extensions, the enrichment and indexing 
configurations have also been moved into this tree. This structure conflates 
parsers, enrichments, and indexing. Beyond that, I also see that we still have 
a config directory with zookeeper elements. I'm not sure we want to merge these 
concepts together in this fashion. Can you shed some light on this?

```
find extension_etc
extension_etc
extension_etc/parsers
extension_etc/parsers/yaf
extension_etc/parsers/yaf/config
extension_etc/parsers/yaf/config/zookeeper
extension_etc/parsers/yaf/config/zookeeper/parsers
extension_etc/parsers/yaf/config/zookeeper/parsers/yaf.json
extension_etc/parsers/yaf/config/zookeeper/indexing
extension_etc/parsers/yaf/config/zookeeper/indexing/yaf.json
extension_etc/parsers/yaf/config/zookeeper/enrichments
extension_etc/parsers/yaf/config/zookeeper/enrichments/yaf.json
extension_etc/parsers/yaf/config/elasticsearch
extension_etc/parsers/yaf/config/elasticsearch/yaf_index.template
extension_etc/parsers/yaf/patterns
extension_etc/parsers/yaf/patterns/yaf
extension_etc/parsers/yaf/patterns/common
extension_etc/parsers/snort
extension_etc/parsers/snort/config
extension_etc/parsers/snort/config/zookeeper
extension_etc/parsers/snort/config/zookeeper/parsers
extension_etc/parsers/snort/config/zookeeper/parsers/snort.json
extension_etc/parsers/snort/config/zookeeper/indexing
extension_etc/parsers/snort/config/zookeeper/indexing/snort.json
extension_etc/parsers/snort/config/zookeeper/enrichments
extension_etc/parsers/snort/config/zookeeper/enrichments/snort.json
extension_etc/parsers/snort/config/elasticsearch
extension_etc/parsers/snort/config/elasticsearch/snort_index.template
```
and
```
find config/zookeeper/
config/zookeeper/
config/zookeeper/parsers
config/zookeeper/parsers/jsonMap.json
config/zookeeper/indexing
config/zookeeper/indexing/error.json
config/zookeeper/bundle.properties
config/zookeeper/global.json
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131249786
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -43,10 +43,7 @@
 
 import java.io.File;
 import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
--- End diff --

Right you are; corrected. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131249214
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -43,10 +43,7 @@
 
 import java.io.File;
 import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
--- End diff --

We should avoid this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131246467
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

Ok, I made the JIRA 
[METRON-1082](https://issues.apache.org/jira/browse/METRON-1082)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131217265
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

Yeah, probably.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #681: METRON-1079 Add NaN as a keyword in STELLAR langua...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/681

METRON-1079 Add NaN as a keyword in STELLAR language

Add "NaN" to STELLAR as a keyword.  

Add NaN, and tests to verify that it works in line with the java 
specification.

## Testing
Build and Tests should work.
Execute statement similar to the added tests in Stellar Shell.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar_nan

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/681.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #681


commit 827566937bfb9527abcd6979d1bb503c78db5f16
Author: Otto Fowler 
Date:   2017-08-03T16:06:00Z

Add NaN as a keyword in STELLAR language




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

[Casey Stella]

Ok, those talks are added.

On Thu, Aug 3, 2017 at 3:44 PM, Casey Stella  wrote:

> Absolutely!
>
> On Thu, Aug 3, 2017 at 3:41 PM, Justin Leet  wrote:
>
>> Could we put these up on the wiki page for tech talks in the community?
>> That page could probably use some love, although I know we've had
>> discussions about what we should do with wiki content.
>>
>> https://cwiki.apache.org/confluence/display/METRON/Tech+Talks
>>
>> On Thu, Aug 3, 2017 at 10:32 AM, Casey Stella  wrote:
>>
>>> The Videos of talks that Simon Ball and I gave at DataWorks Summit are
>>> now up and on youtube:
>>>
>>> * Solving Cyber at Scale (business-level track) -
>>> https://www.youtube.com/watch?v=zVdRhwfum4Q
>>> * Model as a Service (technical track) - https://www.youtube.com/watc
>>> h?v=LkrOKvyAc0s
>>> * Metron Architecture (with demo from LANL data) (technical track) -
>>> https://www.youtube.com/watch?v=0LrrAQXhqGY
>>>
>>> These talks are mostly current based on the existing architecture and
>>> the demos reflect the alerting UI that is not committed yet.  There are
>>> blogs coming out in support of this over the next week or so.
>>>
>>> If anyone has any questions about the talks or want any more
>>> information, feel free to ask. :)
>>>
>>> Best,
>>>
>>> Casey
>>>
>>
>>
>

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131177279
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

Thanks for creating that JIRA.  I think that's a good way to handle it.

Do we also need/want a corresponding doc for metron-solr?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131174351
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

I added [METRON-1080](https://issues.apache.org/jira/browse/METRON-1080).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131173765
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

Thinking about it for a minute, I think we need to create a `README.md` in 
`metron-elasticsearch` that details some of the assumptions for how messages 
are written in ES.  I'll create a JIRA around that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131172968
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -90,6 +95,38 @@ private void cleanDir(File dir) throws IOException {
 }
 dir.mkdirs();
 }
+
+public BulkResponse add(String indexName, String sensorType, String... 
docs) throws IOException {
+List d = new ArrayList<>();
+for(String doc : docs) {
--- End diff --

Yep


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131172518
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
 ---
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.integration;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.collect.Iterables;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import 
org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.hbase.mock.MockHTable;
+import org.apache.metron.hbase.mock.MockHBaseTableProvider;
+import org.apache.metron.indexing.dao.*;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.File;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+public class ElasticsearchUpdateIntegrationTest {
+  private static final int MAX_RETRIES = 10;
+  private static final int SLEEP_MS = 500;
+  private static final String SENSOR_NAME= "test";
+  private static final String TABLE_NAME = "modifications";
+  private static final String CF = "p";
+  private static String indexDir = "target/elasticsearch_mutation";
+  private static String dateFormat = ".MM.dd.HH";
+  private static String index = SENSOR_NAME + "_index_" + new 
SimpleDateFormat(dateFormat).format(new Date());
+  private static MockHTable table;
+  private static IndexDao esDao;
+  private static IndexDao hbaseDao;
+  private static MultiIndexDao dao;
+  private static ElasticSearchComponent es;
+
+  @BeforeClass
+  public static void setup() throws Exception {
+Configuration config = HBaseConfiguration.create();
+MockHBaseTableProvider tableProvider = new MockHBaseTableProvider();
+tableProvider.addToCache(TABLE_NAME, CF);
+table = (MockHTable)tableProvider.getTable(config, TABLE_NAME);
+// setup the client
+es = new ElasticSearchComponent.Builder()
+.withHttpPort(9211)
+.withIndexDir(new File(indexDir))
+.build();
+es.start();
+
+hbaseDao = new HBaseDao();
+AccessConfig accessConfig = new AccessConfig();
+accessConfig.setTableProvider(tableProvider);
+Map globalConfig = new HashMap() {{
+  put("es.clustername", "metron");
+  put("es.port", "9300");
+  put("es.ip", "localhost");
+  put("es.date.format", dateFormat);
+  put(HBaseDao.HBASE_TABLE, TABLE_NAME);
+  put(HBaseDao.HBASE_CF, CF);
+}};
+accessConfig.setGlobalConfigSupplier(() -> globalConfig);
+
+esDao = new ElasticsearchDao();
+
+dao = new MultiIndexDao(hbaseDao, esDao);
+dao.init(accessConfig);
+
+  }
+
+  @AfterClass
+  public static void teardown() {
+if(es != null) {
+  es.stop();
+}
+  }
+
+
+
+  @Test
+  public void test() throws Exception {
+List> inputData = new ArrayList<>();
+for(int i = 0; i < 10;++i) {
+  final String name = "message" + i;
+  inputData.add(
+  new HashMap() {{
+put("source:type", SENSOR_NAME);
+

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131172237
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
 ---
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.integration;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.collect.Iterables;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import 
org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.hbase.mock.MockHTable;
+import org.apache.metron.hbase.mock.MockHBaseTableProvider;
+import org.apache.metron.indexing.dao.*;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.File;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+public class ElasticsearchUpdateIntegrationTest {
+  private static final int MAX_RETRIES = 10;
+  private static final int SLEEP_MS = 500;
+  private static final String SENSOR_NAME= "test";
+  private static final String TABLE_NAME = "modifications";
+  private static final String CF = "p";
+  private static String indexDir = "target/elasticsearch_mutation";
+  private static String dateFormat = ".MM.dd.HH";
+  private static String index = SENSOR_NAME + "_index_" + new 
SimpleDateFormat(dateFormat).format(new Date());
+  private static MockHTable table;
+  private static IndexDao esDao;
+  private static IndexDao hbaseDao;
+  private static MultiIndexDao dao;
+  private static ElasticSearchComponent es;
+
+  @BeforeClass
+  public static void setup() throws Exception {
+Configuration config = HBaseConfiguration.create();
+MockHBaseTableProvider tableProvider = new MockHBaseTableProvider();
+tableProvider.addToCache(TABLE_NAME, CF);
+table = (MockHTable)tableProvider.getTable(config, TABLE_NAME);
+// setup the client
+es = new ElasticSearchComponent.Builder()
+.withHttpPort(9211)
+.withIndexDir(new File(indexDir))
+.build();
+es.start();
+
+hbaseDao = new HBaseDao();
+AccessConfig accessConfig = new AccessConfig();
+accessConfig.setTableProvider(tableProvider);
+Map globalConfig = new HashMap() {{
+  put("es.clustername", "metron");
+  put("es.port", "9300");
+  put("es.ip", "localhost");
+  put("es.date.format", dateFormat);
+  put(HBaseDao.HBASE_TABLE, TABLE_NAME);
+  put(HBaseDao.HBASE_CF, CF);
+}};
+accessConfig.setGlobalConfigSupplier(() -> globalConfig);
+
+esDao = new ElasticsearchDao();
+
+dao = new MultiIndexDao(hbaseDao, esDao);
+dao.init(accessConfig);
+
+  }
+
+  @AfterClass
+  public static void teardown() {
+if(es != null) {
+  es.stop();
+}
+  }
+
+
+
+  @Test
+  public void test() throws Exception {
+List> inputData = new ArrayList<>();
+for(int i = 0; i < 10;++i) {
+  final String name = "message" + i;
+  inputData.add(
+  new HashMap() {{
+put("source:type", SENSOR_NAME);
+

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131170639
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

I'm not sure, where would you expect that to be documented?  I'm relying on 
this component interacting with the writer components in a seamless manner, so 
there are some assumptions being made.  Perhaps I should make a comment here 
and denote that connection?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131162332
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
 ---
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.integration;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.collect.Iterables;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import 
org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.hbase.mock.MockHTable;
+import org.apache.metron.hbase.mock.MockHBaseTableProvider;
+import org.apache.metron.indexing.dao.*;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.File;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+public class ElasticsearchUpdateIntegrationTest {
+  private static final int MAX_RETRIES = 10;
+  private static final int SLEEP_MS = 500;
+  private static final String SENSOR_NAME= "test";
+  private static final String TABLE_NAME = "modifications";
+  private static final String CF = "p";
+  private static String indexDir = "target/elasticsearch_mutation";
+  private static String dateFormat = ".MM.dd.HH";
+  private static String index = SENSOR_NAME + "_index_" + new 
SimpleDateFormat(dateFormat).format(new Date());
+  private static MockHTable table;
+  private static IndexDao esDao;
+  private static IndexDao hbaseDao;
+  private static MultiIndexDao dao;
+  private static ElasticSearchComponent es;
+
+  @BeforeClass
+  public static void setup() throws Exception {
+Configuration config = HBaseConfiguration.create();
+MockHBaseTableProvider tableProvider = new MockHBaseTableProvider();
+tableProvider.addToCache(TABLE_NAME, CF);
+table = (MockHTable)tableProvider.getTable(config, TABLE_NAME);
+// setup the client
+es = new ElasticSearchComponent.Builder()
+.withHttpPort(9211)
+.withIndexDir(new File(indexDir))
+.build();
+es.start();
+
+hbaseDao = new HBaseDao();
+AccessConfig accessConfig = new AccessConfig();
+accessConfig.setTableProvider(tableProvider);
+Map globalConfig = new HashMap() {{
+  put("es.clustername", "metron");
+  put("es.port", "9300");
+  put("es.ip", "localhost");
+  put("es.date.format", dateFormat);
+  put(HBaseDao.HBASE_TABLE, TABLE_NAME);
+  put(HBaseDao.HBASE_CF, CF);
+}};
+accessConfig.setGlobalConfigSupplier(() -> globalConfig);
+
+esDao = new ElasticsearchDao();
+
+dao = new MultiIndexDao(hbaseDao, esDao);
+dao.init(accessConfig);
+
+  }
+
+  @AfterClass
+  public static void teardown() {
+if(es != null) {
+  es.stop();
+}
+  }
+
+
+
+  @Test
+  public void test() throws Exception {
+List> inputData = new ArrayList<>();
+for(int i = 0; i < 10;++i) {
+  final String name = "message" + i;
+  inputData.add(
+  new HashMap() {{
+put("source:type", SENSOR_NAME);
+

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131161781
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
 ---
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.integration;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.collect.Iterables;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import 
org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.hbase.mock.MockHTable;
+import org.apache.metron.hbase.mock.MockHBaseTableProvider;
+import org.apache.metron.indexing.dao.*;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.File;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+public class ElasticsearchUpdateIntegrationTest {
+  private static final int MAX_RETRIES = 10;
+  private static final int SLEEP_MS = 500;
+  private static final String SENSOR_NAME= "test";
+  private static final String TABLE_NAME = "modifications";
+  private static final String CF = "p";
+  private static String indexDir = "target/elasticsearch_mutation";
+  private static String dateFormat = ".MM.dd.HH";
+  private static String index = SENSOR_NAME + "_index_" + new 
SimpleDateFormat(dateFormat).format(new Date());
+  private static MockHTable table;
+  private static IndexDao esDao;
+  private static IndexDao hbaseDao;
+  private static MultiIndexDao dao;
+  private static ElasticSearchComponent es;
+
+  @BeforeClass
+  public static void setup() throws Exception {
+Configuration config = HBaseConfiguration.create();
+MockHBaseTableProvider tableProvider = new MockHBaseTableProvider();
+tableProvider.addToCache(TABLE_NAME, CF);
+table = (MockHTable)tableProvider.getTable(config, TABLE_NAME);
+// setup the client
+es = new ElasticSearchComponent.Builder()
+.withHttpPort(9211)
+.withIndexDir(new File(indexDir))
+.build();
+es.start();
+
+hbaseDao = new HBaseDao();
+AccessConfig accessConfig = new AccessConfig();
+accessConfig.setTableProvider(tableProvider);
+Map globalConfig = new HashMap() {{
+  put("es.clustername", "metron");
+  put("es.port", "9300");
+  put("es.ip", "localhost");
+  put("es.date.format", dateFormat);
+  put(HBaseDao.HBASE_TABLE, TABLE_NAME);
+  put(HBaseDao.HBASE_CF, CF);
+}};
+accessConfig.setGlobalConfigSupplier(() -> globalConfig);
+
+esDao = new ElasticsearchDao();
+
+dao = new MultiIndexDao(hbaseDao, esDao);
+dao.init(accessConfig);
+
+  }
+
+  @AfterClass
+  public static void teardown() {
+if(es != null) {
+  es.stop();
+}
+  }
+
+
+
+  @Test
+  public void test() throws Exception {
+List> inputData = new ArrayList<>();
+for(int i = 0; i < 10;++i) {
+  final String name = "message" + i;
+  inputData.add(
+  new HashMap() {{
+put("source:type", SENSOR_NAME);
+

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131168587
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
 ---
@@ -0,0 +1,161 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.indexing.dao;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.Iterables;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.io.IOException;
+import java.util.*;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class MultiIndexDao implements IndexDao {
+  private List indices;
+
+  public MultiIndexDao( IndexDao... composedDao) {
+indices = new ArrayList<>();
+for(IndexDao dao: composedDao) {
--- End diff --

`Collections.addAll` again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131157143
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,99 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
--- End diff --

Do we document a requirement that it be of the form `_doc` 
anywhere?  I'm not opposed to doing it, but it is might currently be an 
undocumented requirement on how ES templates are defined in our project.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131162722
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java
 ---
@@ -90,6 +95,38 @@ private void cleanDir(File dir) throws IOException {
 }
 dir.mkdirs();
 }
+
+public BulkResponse add(String indexName, String sensorType, String... 
docs) throws IOException {
+List d = new ArrayList<>();
+for(String doc : docs) {
--- End diff --

Can be `Collections.addAll(d, docs);`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #675: METRON-379 STELLAR can differentiate between a val...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/675


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

[Casey Stella]

Absolutely!

On Thu, Aug 3, 2017 at 3:41 PM, Justin Leet  wrote:

> Could we put these up on the wiki page for tech talks in the community?
> That page could probably use some love, although I know we've had
> discussions about what we should do with wiki content.
>
> https://cwiki.apache.org/confluence/display/METRON/Tech+Talks
>
> On Thu, Aug 3, 2017 at 10:32 AM, Casey Stella  wrote:
>
>> The Videos of talks that Simon Ball and I gave at DataWorks Summit are
>> now up and on youtube:
>>
>> * Solving Cyber at Scale (business-level track) -
>> https://www.youtube.com/watch?v=zVdRhwfum4Q
>> * Model as a Service (technical track) - https://www.youtube.com/watc
>> h?v=LkrOKvyAc0s
>> * Metron Architecture (with demo from LANL data) (technical track) -
>> https://www.youtube.com/watch?v=0LrrAQXhqGY
>>
>> These talks are mostly current based on the existing architecture and the
>> demos reflect the alerting UI that is not committed yet.  There are blogs
>> coming out in support of this over the next week or so.
>>
>> If anyone has any questions about the talks or want any more information,
>> feel free to ask. :)
>>
>> Best,
>>
>> Casey
>>
>
>

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

[Justin Leet]

Could we put these up on the wiki page for tech talks in the community?
That page could probably use some love, although I know we've had
discussions about what we should do with wiki content.

https://cwiki.apache.org/confluence/display/METRON/Tech+Talks

On Thu, Aug 3, 2017 at 10:32 AM, Casey Stella  wrote:

> The Videos of talks that Simon Ball and I gave at DataWorks Summit are now
> up and on youtube:
>
> * Solving Cyber at Scale (business-level track) - https://www.youtube.com/
> watch?v=zVdRhwfum4Q
> * Model as a Service (technical track) - https://www.youtube.com/
> watch?v=LkrOKvyAc0s
> * Metron Architecture (with demo from LANL data) (technical track) -
> https://www.youtube.com/watch?v=0LrrAQXhqGY
>
> These talks are mostly current based on the existing architecture and the
> demos reflect the alerting UI that is not committed yet.  There are blogs
> coming out in support of this over the next week or so.
>
> If anyone has any questions about the talks or want any more information,
> feel free to ask. :)
>
> Best,
>
> Casey
>

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/667#discussion_r131155065
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
 ---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+  @Stellar(name = "FUZZY_SCORE",
+  description =
+  "Returns the Fuzzy Score which indicates the similarity score 
between two Strings "
+  +
+  "One point is given for every matched character. Subsequent 
matches yield two bonus "
+  +
+  "points. A higher score indicates a higher similarity",
+  params = {
+  "string - The full term that should be matched against",
+  "string - The query that will be matched against a term",
+  "string - The IETF BCP 47 language code to use"
+  },
+  returns = "integer representing the score")
+  /**
+   * FuzzyScoreFunction exposes the Apache Commons Text Similarity 
FuzzyScore through
+   * Stellar.
+   */
+  public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args: 
[string, string, string]");
+  }
+  String term = (String) list.get(0);
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: SimpleEnrichmentFlatFileLoaderIntegrationTest …..

[Casey Stella]

That is puzzling for sure.  Another intermittent failing test mystery (dum
dum dum), brought to you by the letter 'o' and the number Math.rand().

On Thu, Aug 3, 2017 at 3:15 PM, Otto Fowler  wrote:

> Results :
>
> Failed tests:
>
>   SimpleEnrichmentFlatFileLoaderIntegrationTest.testLocalLineByLine_gz:362
> expected:<1000> but was:<991>
>
>
> Is there a reason why this seems to be happening more often now?
>

[GitHub] metron issue #675: METRON-379 STELLAR can differentiate between a value pass...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/675
  
+1 by inspection, this is good work, @ottobackwards 

Sorry for jumping to conclusions!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #675: METRON-379 STELLAR can differentiate between a value pass...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/675
  
hah no, a JIRA will be necessary ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #675: METRON-379 STELLAR can differentiate between a value pass...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/675
  
I'll create a Jira for the NaN thing, unless you have already coded it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #675: METRON-379 STELLAR can differentiate between a value pass...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/675
  
Yeah, I'm totally ok with this now.  Really sorry about jumping to the 
wrong conclusion; I saw the errors in the aftermath of the math PR that got in 
this morning and jumped to the wrong conclusion about default behavior.

To that end, that `NaN` change you have, shouldn't have been necessary.  We 
need `NaN` as a language keyword.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #679: METRON-1075: Add faceted search capabilities

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/679#discussion_r131151368
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -179,4 +206,43 @@ public void init(Map globalConfig, 
AccessConfig config) {
 return latestIndices.values().toArray(new 
String[latestIndices.size()]);
   }
 
+  public void addFacetFields(SearchSourceBuilder searchSourceBuilder, 
List fields) {
+for(String field: fields) {
+  searchSourceBuilder = searchSourceBuilder.aggregation(new 
TermsBuilder(getAggregationName(field)).field(field));
+}
+  }
+
+  public Map> getFacetCounts(List 
fields, Aggregations aggregations, Map commonColumnMetadata) 
{
+Map> fieldCounts = new HashMap<>();
+for (String field: fields) {
+  Map valueCounts = new HashMap<>();
+  Aggregation aggregation = 
aggregations.get(getAggregationName(field));
+  if (aggregation instanceof LongTerms) {
+LongTerms longTerms = (LongTerms) aggregation;
+FieldType type = commonColumnMetadata.get(field);
+if (FieldType.IP.equals(type)) {
+  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(IpFieldMapper.longToIp((Long) bucket.getKey()), 
bucket.getDocCount()));
+} else if (FieldType.BOOLEAN.equals(type)) {
+  longTerms.getBuckets().stream().forEach(bucket -> {
+String key = (Long) bucket.getKey() == 1 ? "true" : "false";
+valueCounts.put(key, bucket.getDocCount());
+  });
+} else {
+  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+}
+  } else if (aggregation instanceof DoubleTerms) {
+DoubleTerms doubleTerms = (DoubleTerms) aggregation;
+doubleTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+  } else if (aggregation instanceof StringTerms) {
+StringTerms stringTerms = (StringTerms) aggregation;
+stringTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+  }
+  fieldCounts.put(field, valueCounts);
--- End diff --

The ES java api can be pretty awkward to use.  This is a good example of 
that.  The Aggregation object is abstract and has 3 subclasses related to term 
aggregations:  LongTerms, DoubleTerms and StringTerms.  All types fall into one 
of these (as far as I could tell) which is one reason we needed a function to 
get the fields types.  For example, an aggregation for a field of type "ip" is 
represented as a LongTerms object and the value returned is also a long.  To 
get it to display correct we need to convert it to a string representation of 
the ip address.  Same thing for booleans.  There are returned as LongTerms with 
a value of 1 or 0.

Hopefully I covered all the different types in the integration tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #675: METRON-379 STELLAR can differentiate between a value pass...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/675
  
Ok, wait, I think I might've misunderstood.  This `DefaultVariableResolver` 
is just being used for tests, validation and Lambda functions now, is that 
correct?  I think I'm ok with that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/667#discussion_r131148105
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
 ---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+  @Stellar(name = "FUZZY_SCORE",
+  description =
+  "Returns the Fuzzy Score which indicates the similarity score 
between two Strings "
+  +
+  "One point is given for every matched character. Subsequent 
matches yield two bonus "
+  +
+  "points. A higher score indicates a higher similarity",
+  params = {
+  "string - The full term that should be matched against",
+  "string - The query that will be matched against a term",
+  "string - The IETF BCP 47 language code to use"
+  },
+  returns = "integer representing the score")
+  /**
+   * FuzzyScoreFunction exposes the Apache Commons Text Similarity 
FuzzyScore through
+   * Stellar.
+   */
+  public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args: 
[string, string, string]");
+  }
+  String term = (String) list.get(0);
--- End diff --

Well, in this case, I don't think we want to use `ConversionUtils` since we 
probably dont' want to coerce a list into a string for a fuzzy score, but I 
might be wrong.

I think we probably want to return `0` in the case that someone passes in a 
wrong type too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/667#discussion_r131146245
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
 ---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+  @Stellar(name = "FUZZY_SCORE",
+  description =
+  "Returns the Fuzzy Score which indicates the similarity score 
between two Strings "
+  +
+  "One point is given for every matched character. Subsequent 
matches yield two bonus "
+  +
+  "points. A higher score indicates a higher similarity",
+  params = {
+  "string - The full term that should be matched against",
+  "string - The query that will be matched against a term",
+  "string - The IETF BCP 47 language code to use"
+  },
+  returns = "integer representing the score")
+  /**
+   * FuzzyScoreFunction exposes the Apache Commons Text Similarity 
FuzzyScore through
+   * Stellar.
+   */
+  public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args: 
[string, string, string]");
+  }
+  String term = (String) list.get(0);
--- End diff --

Also, the return should consider the common usage, I imagine something like:

IF (FUZZY_SCORE(fld,qry,'EN') > 4) THEN SET SOME FIELD




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/667#discussion_r131143225
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
 ---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+  @Stellar(name = "FUZZY_SCORE",
+  description =
+  "Returns the Fuzzy Score which indicates the similarity score 
between two Strings "
+  +
+  "One point is given for every matched character. Subsequent 
matches yield two bonus "
+  +
+  "points. A higher score indicates a higher similarity",
+  params = {
+  "string - The full term that should be matched against",
+  "string - The query that will be matched against a term",
+  "string - The IETF BCP 47 language code to use"
+  },
+  returns = "integer representing the score")
+  /**
+   * FuzzyScoreFunction exposes the Apache Commons Text Similarity 
FuzzyScore through
+   * Stellar.
+   */
+  public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args: 
[string, string, string]");
+  }
+  String term = (String) list.get(0);
--- End diff --

We really need a pattern or a helper class for stellar for variables, that 
everyone uses.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/667#discussion_r131142516
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
 ---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+  @Stellar(name = "FUZZY_SCORE",
+  description =
+  "Returns the Fuzzy Score which indicates the similarity score 
between two Strings "
+  +
+  "One point is given for every matched character. Subsequent 
matches yield two bonus "
+  +
+  "points. A higher score indicates a higher similarity",
+  params = {
+  "string - The full term that should be matched against",
+  "string - The query that will be matched against a term",
+  "string - The IETF BCP 47 language code to use"
+  },
+  returns = "integer representing the score")
+  /**
+   * FuzzyScoreFunction exposes the Apache Commons Text Similarity 
FuzzyScore through
+   * Stellar.
+   */
+  public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args: 
[string, string, string]");
+  }
+  String term = (String) list.get(0);
--- End diff --

1. The (CAST)list.get(0) is the common pattern in our stellar code.  I 
believe I have asked before in another pr maybe why we don't use the conversion 
utils. 
2. I think we want to return 0 for invalid args.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #678: METRON-1076: Management UI status check always reports th...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/metron/pull/678
  
+1 by inspection. Good stuff, much improved, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #679: METRON-1075: Add faceted search capabilities

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/679
  
+1 works as described.  Great Job!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Persisting user data

[Ryan Merriman]

Spring is JDBC-generic so I think we're good there.  Improving our docs on
this topic is being discussed in https://github.com/apache/metron/pull/646
so hopefully this will be clear once that's worked out.

Simon is correct, I found out the hard way that Hibernate is not an option
because of it's license.  I think EclipseLink would be a good alternative.
I've seen it used in other open source projects (Ambari for example) and I
was able to get it working in a POC without much effort.

On Thu, Aug 3, 2017 at 5:26 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Anything spring based is likely multi-db by definition as long as a we
> pick a good friendly ORM (not hibernate because licensing problems with
> apache, eclipselink?) But I suspect we should pick a good default and that
> that default should be postgres.
>
> > On 3 Aug 2017, at 10:24, Casey Stella  wrote:
> >
> > I'd vote for a DB-based solution, but I'd argue that any solution
> shouldn't
> > be database specific (i.e. postgres), but JDBC-generic.  People and
> > organizations have very strong views regarding databases and I'd prefer
> to
> > side-step those holy wars by being agnostic.
> >
> > On Wed, Aug 2, 2017 at 9:36 PM, Ryan Merriman 
> wrote:
> >
> >> Spring supports a variety of databases including Postgres.  I have no
> >> problem with using Postgres instead of MySQL.
> >>
> >> On Wed, Aug 2, 2017 at 3:32 PM, Simon Elliston Ball <
> >> si...@simonellistonball.com> wrote:
> >>
> >>> Agreed on Postgres. It's a lot easier to work with license-wise in
> apache
> >>> projects, and has a lot of the capability we need here, especially if
> we
> >>> can find a sensible ORM. Anyone got any thoughts on what would work
> >> there?
> >>>
> >>> Simon
> >>>
>  On 2 Aug 2017, at 21:21, Matt Foley  wrote:
> 
>  Hi Ryan,
>  Zookeeper has a default (and seldom changed) max znode size of 1MB,
> but
> >>> it is “designed to store data on the order of kilobytes in size.”[1]
> And
> >>> it’s not really intended for frequently-changing data, which is okay
> >> here.
> >>> But I just included it for completeness, I’m not advocating for its use
> >>> here.
> 
>  I agree with you that the problem, especially because it includes
> >> shared
> >>> config, would fit well in a db.  I’d suggest you consider PostgreSQL
> >> rather
> >>> than MySQL, as postgres is built into Redhat 6 and 7, and Ambari now
> uses
> >>> it by default, so an available server might be conveniently at hand in
> >> most
> >>> deployments.  Definitely assume the user will want to use an external
> db
> >>> instance, rather than one dedicated to this use.  Conveniently Postgres
> >>> also has a native REST interface, with the usual authorization options.
> 
>  Never mind about Ambari Views for now.  It’s just a way to get GUI
> >>> dashboards without writing all the infrastructure for it, which as you
> >> say
> >>> is somewhat water under the bridge.
>  Cheers,
>  --Matt
> 
>  [1] https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html
> 
> 
> 
>  On 8/2/17, 12:34 PM, "Ryan Merriman"  wrote:
> 
>    Matt,
> 
>    Thank you for the suggestions.  I forgot to include Zookeeper.  Are
> >>> there
>    any tradeoffs we should be aware of if we decide to use Zookeeper?
> >>> Are
>    there guidelines for how much data can be stored in Zookeeper?
> 
>    To answer your questions:
> 
>    1.  I think both use cases make sense so a combination of shared and
>    personal.
>    2.  I was planning on managing authorization in the REST layer.  For
> >>> now
>    viewer login auth (which is really REST auth) will suffice but we
> >>> might
>    consider other methods since authentication is pluggable here.
>    3.  I had not considered Ambari Views since this will support an
> >>> existing
>    UI.  How would Ambari Views help us here?
> 
>    I will proceed initially with a saved search POC using a relational
>    database unless you think that is a bad idea or there are other
> >> better
>    options.  Hopefully an example will further the discussion.
> 
>    Ryan
> 
> >   On Wed, Jul 26, 2017 at 6:31 PM, Matt Foley 
> >>> wrote:
> >
> > There’s a couple other places you could put config info (but maybe
> not
> > saved searches):
> > -  Zookeeper
> > -  metron-alerts-ui/config.xml or config.json  file
> > -  the Ambari database, whichever it happens to be
> >
> > Questions that influence the decision include:
> > 1. Should there be one configuration shared among users, or strictly
> > per-user config?  Or a combination of shared and personal?
> > 2. What security do you wish to maintain on changing those settings,
> >>> both
> > shared and personal?  What authentication/authorization 

[GitHub] metron pull request #677: METRON-1073: Add option to limit fields returned f...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/677#discussion_r131133901
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -81,8 +82,14 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
 .size(searchRequest.getSize())
 .from(searchRequest.getFrom())
 .query(new QueryStringQueryBuilder(searchRequest.getQuery()))
-.fetchSource(true)
+
 .trackScores(true);
+Optional fields = searchRequest.getFields();
+if (fields.isPresent()) {
+  searchSourceBuilder.fields(fields.get());
--- End diff --

bah, I retract.  Sorry, I totally misread this.  Nevermind me ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #677: METRON-1073: Add option to limit fields returned f...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/677#discussion_r131133808
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -81,8 +82,14 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
 .size(searchRequest.getSize())
 .from(searchRequest.getFrom())
 .query(new QueryStringQueryBuilder(searchRequest.getQuery()))
-.fetchSource(true)
+
 .trackScores(true);
+Optional fields = searchRequest.getFields();
+if (fields.isPresent()) {
+  searchSourceBuilder.fields(fields.get());
--- End diff --

sorry, I should say, if fields is empty.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #677: METRON-1073: Add option to limit fields returned f...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/677#discussion_r131133566
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -81,8 +82,14 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
 .size(searchRequest.getSize())
 .from(searchRequest.getFrom())
 .query(new QueryStringQueryBuilder(searchRequest.getQuery()))
-.fetchSource(true)
+
 .trackScores(true);
+Optional fields = searchRequest.getFields();
+if (fields.isPresent()) {
+  searchSourceBuilder.fields(fields.get());
--- End diff --

What happens if fields.get() is empty or null?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131129176
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,100 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
+byte[] source = JSONUtils.INSTANCE.toJSON(update.getDocument());
--- End diff --

I just checked this in the tests, it seems to not make a difference one way 
or another.  I removed the conversion in favor of letting ES do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131128726
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,100 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
+byte[] source = JSONUtils.INSTANCE.toJSON(update.getDocument());
--- End diff --

I am honestly not sure about the ES API subtleties, but I'm mimicking 
[ElasticsearchWriter](https://github.com/apache/metron/blob/ec959d20efa72e9868d87a02d9407bbaad34c4c8/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java#L88).

It seems to me that it's 6 of 1 and half a dozen of another considering it 
shouldn't matter terribly who converts to bytes, it's getting converted to 
bytes either way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #679: METRON-1075: Add faceted search capabilities

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/679#discussion_r131128135
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -179,4 +206,43 @@ public void init(Map globalConfig, 
AccessConfig config) {
 return latestIndices.values().toArray(new 
String[latestIndices.size()]);
   }
 
+  public void addFacetFields(SearchSourceBuilder searchSourceBuilder, 
List fields) {
+for(String field: fields) {
+  searchSourceBuilder = searchSourceBuilder.aggregation(new 
TermsBuilder(getAggregationName(field)).field(field));
+}
+  }
+
+  public Map> getFacetCounts(List 
fields, Aggregations aggregations, Map commonColumnMetadata) 
{
+Map> fieldCounts = new HashMap<>();
+for (String field: fields) {
+  Map valueCounts = new HashMap<>();
+  Aggregation aggregation = 
aggregations.get(getAggregationName(field));
+  if (aggregation instanceof LongTerms) {
+LongTerms longTerms = (LongTerms) aggregation;
+FieldType type = commonColumnMetadata.get(field);
+if (FieldType.IP.equals(type)) {
+  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(IpFieldMapper.longToIp((Long) bucket.getKey()), 
bucket.getDocCount()));
+} else if (FieldType.BOOLEAN.equals(type)) {
+  longTerms.getBuckets().stream().forEach(bucket -> {
+String key = (Long) bucket.getKey() == 1 ? "true" : "false";
+valueCounts.put(key, bucket.getDocCount());
+  });
+} else {
+  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+}
+  } else if (aggregation instanceof DoubleTerms) {
+DoubleTerms doubleTerms = (DoubleTerms) aggregation;
+doubleTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+  } else if (aggregation instanceof StringTerms) {
+StringTerms stringTerms = (StringTerms) aggregation;
+stringTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+  }
+  fieldCounts.put(field, valueCounts);
--- End diff --

Are there any other *Terms here or does this cover our "Other" type?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/666#discussion_r131127259
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -102,15 +128,100 @@ public SearchResponse search(SearchRequest 
searchRequest) throws InvalidSearchEx
   searchResult.setId(searchHit.getId());
   searchResult.setSource(searchHit.getSource());
   searchResult.setScore(searchHit.getScore());
+  searchResult.setIndex(searchHit.getIndex());
   return searchResult;
 }).collect(Collectors.toList()));
 return searchResponse;
   }
 
   @Override
-  public void init(Map globalConfig, AccessConfig config) {
-this.client = ElasticsearchUtils.getClient(globalConfig, 
config.getOptionalSettings());
-this.accessConfig = config;
+  public synchronized void init(AccessConfig config) {
+if(this.client == null) {
+  this.client = 
ElasticsearchUtils.getClient(config.getGlobalConfigSupplier().get(), 
config.getOptionalSettings());
+  this.accessConfig = config;
+}
+  }
+
+  @Override
+  public Document getLatest(final String guid, final String sensorType) 
throws IOException {
+Optional ret = searchByGuid(
+guid
+, sensorType
+, hit -> {
+  Long ts = 0L;
+  String doc = hit.getSourceAsString();
+  String sourceType = 
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+  try {
+return Optional.of(new Document(doc, guid, sourceType, 
ts));
+  } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve 
latest: " + e.getMessage(), e);
+  }
+}
+);
+return ret.orElse(null);
+  }
+
+  /**
+   * Return the search hit based on the UUID and sensor type.
+   * A callback can be specified to transform the hit into a type T.
+   * If more than one hit happens, the first one will be returned.
+   * @throws IOException
+   */
+   Optional searchByGuid(String guid, String sensorType, 
Function callback) throws IOException{
+QueryBuilder query =  QueryBuilders.matchQuery(Constants.GUID, guid);
+SearchRequestBuilder request = client.prepareSearch()
+ .setTypes(sensorType + "_doc")
+ .setQuery(query)
+ .setSource("message")
+ ;
+MultiSearchResponse response = client.prepareMultiSearch()
+ .add(request)
+ .get();
+for(MultiSearchResponse.Item i : response) {
+  org.elasticsearch.action.search.SearchResponse resp = 
i.getResponse();
+  SearchHits hits = resp.getHits();
+  for(SearchHit hit : hits) {
+Optional ret = callback.apply(hit);
+if(ret.isPresent()) {
+  return ret;
+}
+  }
+}
+return Optional.empty();
+
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+String indexPostfix = 
ElasticsearchUtils.getIndexFormat(accessConfig.getGlobalConfigSupplier().get()).format(new
 Date());
+String sensorType = update.getSensorType();
+String indexName = ElasticsearchUtils.getIndexName(sensorType, 
indexPostfix, null);
+
+String type = sensorType + "_doc";
+byte[] source = JSONUtils.INSTANCE.toJSON(update.getDocument());
--- End diff --

Why do we convert the update.getDocument() to a byte[], when the 
IndexRequest.source() method can take a Map?

Is there some correctness or typing concern?  I'm not particularly familiar 
with any subtleties of the ES apis like that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #666: METRON-1051: Enable the ability to update indexed message...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/666
  
@justinleet yeah, I'll add javadoc to `IndexDao`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #666: METRON-1051: Enable the ability to update indexed message...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/666
  
Can you add Javadoc to the various interfaces and interface methods you 
added?  I know we've been inconsistent (i.e. we usually don't), but I think 
it'd be helpful as more people dig into things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Persisting user data

[Simon Elliston Ball]

Anything spring based is likely multi-db by definition as long as a we pick a 
good friendly ORM (not hibernate because licensing problems with apache, 
eclipselink?) But I suspect we should pick a good default and that that default 
should be postgres. 

> On 3 Aug 2017, at 10:24, Casey Stella  wrote:
> 
> I'd vote for a DB-based solution, but I'd argue that any solution shouldn't
> be database specific (i.e. postgres), but JDBC-generic.  People and
> organizations have very strong views regarding databases and I'd prefer to
> side-step those holy wars by being agnostic.
> 
> On Wed, Aug 2, 2017 at 9:36 PM, Ryan Merriman  wrote:
> 
>> Spring supports a variety of databases including Postgres.  I have no
>> problem with using Postgres instead of MySQL.
>> 
>> On Wed, Aug 2, 2017 at 3:32 PM, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>> 
>>> Agreed on Postgres. It's a lot easier to work with license-wise in apache
>>> projects, and has a lot of the capability we need here, especially if we
>>> can find a sensible ORM. Anyone got any thoughts on what would work
>> there?
>>> 
>>> Simon
>>> 
 On 2 Aug 2017, at 21:21, Matt Foley  wrote:
 
 Hi Ryan,
 Zookeeper has a default (and seldom changed) max znode size of 1MB, but
>>> it is “designed to store data on the order of kilobytes in size.”[1]  And
>>> it’s not really intended for frequently-changing data, which is okay
>> here.
>>> But I just included it for completeness, I’m not advocating for its use
>>> here.
 
 I agree with you that the problem, especially because it includes
>> shared
>>> config, would fit well in a db.  I’d suggest you consider PostgreSQL
>> rather
>>> than MySQL, as postgres is built into Redhat 6 and 7, and Ambari now uses
>>> it by default, so an available server might be conveniently at hand in
>> most
>>> deployments.  Definitely assume the user will want to use an external db
>>> instance, rather than one dedicated to this use.  Conveniently Postgres
>>> also has a native REST interface, with the usual authorization options.
 
 Never mind about Ambari Views for now.  It’s just a way to get GUI
>>> dashboards without writing all the infrastructure for it, which as you
>> say
>>> is somewhat water under the bridge.
 Cheers,
 --Matt
 
 [1] https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html
 
 
 
 On 8/2/17, 12:34 PM, "Ryan Merriman"  wrote:
 
   Matt,
 
   Thank you for the suggestions.  I forgot to include Zookeeper.  Are
>>> there
   any tradeoffs we should be aware of if we decide to use Zookeeper?
>>> Are
   there guidelines for how much data can be stored in Zookeeper?
 
   To answer your questions:
 
   1.  I think both use cases make sense so a combination of shared and
   personal.
   2.  I was planning on managing authorization in the REST layer.  For
>>> now
   viewer login auth (which is really REST auth) will suffice but we
>>> might
   consider other methods since authentication is pluggable here.
   3.  I had not considered Ambari Views since this will support an
>>> existing
   UI.  How would Ambari Views help us here?
 
   I will proceed initially with a saved search POC using a relational
   database unless you think that is a bad idea or there are other
>> better
   options.  Hopefully an example will further the discussion.
 
   Ryan
 
>   On Wed, Jul 26, 2017 at 6:31 PM, Matt Foley 
>>> wrote:
> 
> There’s a couple other places you could put config info (but maybe not
> saved searches):
> -  Zookeeper
> -  metron-alerts-ui/config.xml or config.json  file
> -  the Ambari database, whichever it happens to be
> 
> Questions that influence the decision include:
> 1. Should there be one configuration shared among users, or strictly
> per-user config?  Or a combination of shared and personal?
> 2. What security do you wish to maintain on changing those settings,
>>> both
> shared and personal?  What authentication/authorization scheme will
>> you
> use?  Is viewer login auth sufficient for this?
> 3. Will you assume Ambari exists?  Did you consider using Ambari Views
>>> as
> the basis? (https://cwiki.apache.org/confluence/display/AMBARI/Views
>> )
> 
> On 7/26/17, 2:54 PM, "Ryan Merriman"  wrote:
> 
>   In anticipation of METRON-988 being merged into master, there will
>>> be a
>   need to persist user preferences such as UI layout, saved searches,
> search
>   history, etc.  I think where and how we persist this data should be
>   discussed in order to facilitate a design.  This data won't be
>> large
>>> in
>   scale and may or may not be relational.  The initial features I am
> aware of
>   don't require a 

[GitHub] metron issue #666: METRON-1051: Enable the ability to update indexed message...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/666
  
Just a brief note about the `TableProvider` business.  We had cut and 
pasted a mock HTableProvider through several of our projects.  In order to test 
the `HBaseDao`, I needed yet another HTableProvider cut and pasted. I 
considered this a bridge too far. Instead of doing that, I
* Took the effort to create it once, put the implementation in the test 
artifact created by `metron-hbase`
* Refactor existing tests to use that consolidated implementation

I apologize for increasing the complexity of this PR, but I chose to 
interpret this as cleanup that testing this PR strongly suggested be done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/667#discussion_r131099946
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
 ---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+  @Stellar(name = "FUZZY_SCORE",
+  description =
+  "Returns the Fuzzy Score which indicates the similarity score 
between two Strings "
+  +
+  "One point is given for every matched character. Subsequent 
matches yield two bonus "
+  +
+  "points. A higher score indicates a higher similarity",
+  params = {
+  "string - The full term that should be matched against",
+  "string - The query that will be matched against a term",
+  "string - The IETF BCP 47 language code to use"
+  },
+  returns = "integer representing the score")
+  /**
+   * FuzzyScoreFunction exposes the Apache Commons Text Similarity 
FuzzyScore through
+   * Stellar.
+   */
+  public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args: 
[string, string, string]");
+  }
+  String term = (String) list.get(0);
--- End diff --

If there's an exception here (specifically a classcastexception), we're 
going to exception.  Given the noise in the data that we have, I would expect 
this to happen.

I have a couple of questions:
* Do we want to return `NaN` or `Infinity` in that case and log at a warn 
level (or error level)?
* Do we want to throw an exception which could not possibly be caught in 
the stream?

I, personally, vote for the first, but I'd like to hear other people's 
impressions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #641: METRON-539: added HASH function for stellar.

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/641
  
Ok, I'm +1 on this by inspection as soon as we have the conflict resolved.  
Great job, @jjmeyer0 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Persisting user data

[Casey Stella]

I'd vote for a DB-based solution, but I'd argue that any solution shouldn't
be database specific (i.e. postgres), but JDBC-generic.  People and
organizations have very strong views regarding databases and I'd prefer to
side-step those holy wars by being agnostic.

On Wed, Aug 2, 2017 at 9:36 PM, Ryan Merriman  wrote:

> Spring supports a variety of databases including Postgres.  I have no
> problem with using Postgres instead of MySQL.
>
> On Wed, Aug 2, 2017 at 3:32 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > Agreed on Postgres. It's a lot easier to work with license-wise in apache
> > projects, and has a lot of the capability we need here, especially if we
> > can find a sensible ORM. Anyone got any thoughts on what would work
> there?
> >
> > Simon
> >
> > > On 2 Aug 2017, at 21:21, Matt Foley  wrote:
> > >
> > > Hi Ryan,
> > > Zookeeper has a default (and seldom changed) max znode size of 1MB, but
> > it is “designed to store data on the order of kilobytes in size.”[1]  And
> > it’s not really intended for frequently-changing data, which is okay
> here.
> > But I just included it for completeness, I’m not advocating for its use
> > here.
> > >
> > > I agree with you that the problem, especially because it includes
> shared
> > config, would fit well in a db.  I’d suggest you consider PostgreSQL
> rather
> > than MySQL, as postgres is built into Redhat 6 and 7, and Ambari now uses
> > it by default, so an available server might be conveniently at hand in
> most
> > deployments.  Definitely assume the user will want to use an external db
> > instance, rather than one dedicated to this use.  Conveniently Postgres
> > also has a native REST interface, with the usual authorization options.
> > >
> > > Never mind about Ambari Views for now.  It’s just a way to get GUI
> > dashboards without writing all the infrastructure for it, which as you
> say
> > is somewhat water under the bridge.
> > > Cheers,
> > > --Matt
> > >
> > > [1] https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html
> > >
> > >
> > >
> > > On 8/2/17, 12:34 PM, "Ryan Merriman"  wrote:
> > >
> > >Matt,
> > >
> > >Thank you for the suggestions.  I forgot to include Zookeeper.  Are
> > there
> > >any tradeoffs we should be aware of if we decide to use Zookeeper?
> > Are
> > >there guidelines for how much data can be stored in Zookeeper?
> > >
> > >To answer your questions:
> > >
> > >1.  I think both use cases make sense so a combination of shared and
> > >personal.
> > >2.  I was planning on managing authorization in the REST layer.  For
> > now
> > >viewer login auth (which is really REST auth) will suffice but we
> > might
> > >consider other methods since authentication is pluggable here.
> > >3.  I had not considered Ambari Views since this will support an
> > existing
> > >UI.  How would Ambari Views help us here?
> > >
> > >I will proceed initially with a saved search POC using a relational
> > >database unless you think that is a bad idea or there are other
> better
> > >options.  Hopefully an example will further the discussion.
> > >
> > >Ryan
> > >
> > >>On Wed, Jul 26, 2017 at 6:31 PM, Matt Foley 
> > wrote:
> > >>
> > >> There’s a couple other places you could put config info (but maybe not
> > >> saved searches):
> > >> -  Zookeeper
> > >> -  metron-alerts-ui/config.xml or config.json  file
> > >> -  the Ambari database, whichever it happens to be
> > >>
> > >> Questions that influence the decision include:
> > >> 1. Should there be one configuration shared among users, or strictly
> > >> per-user config?  Or a combination of shared and personal?
> > >> 2. What security do you wish to maintain on changing those settings,
> > both
> > >> shared and personal?  What authentication/authorization scheme will
> you
> > >> use?  Is viewer login auth sufficient for this?
> > >> 3. Will you assume Ambari exists?  Did you consider using Ambari Views
> > as
> > >> the basis? (https://cwiki.apache.org/confluence/display/AMBARI/Views
> )
> > >>
> > >> On 7/26/17, 2:54 PM, "Ryan Merriman"  wrote:
> > >>
> > >>In anticipation of METRON-988 being merged into master, there will
> > be a
> > >>need to persist user preferences such as UI layout, saved searches,
> > >> search
> > >>history, etc.  I think where and how we persist this data should be
> > >>discussed in order to facilitate a design.  This data won't be
> large
> > in
> > >>scale and may or may not be relational.  The initial features I am
> > >> aware of
> > >>don't require a relational model but I'm sure there will be some
> that
> > >> do in
> > >>the future.  I'm also assuming this code will live in the REST
> > >> application
> > >>but someone correct me if there is a reason to keep it somewhere
> > else.
> > >>
> > >>I think it would be preferable to 

[GitHub] metron issue #650: METRON-1038: Stellar should have a better collection of b...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/650
  
Committed, but I wanted to make a comment about the BiFunction.  Honestly, 
I like the notion of providing candy for 2-arg functions like we did for 
single-arg functions.  I think it'll make it super easy for @simonellistonball 
to add `POWER`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #666: METRON-1051: Enable the ability to update indexed message...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/666
  
Ok, architectural coverage is mentioned here, so I think this is ready for 
review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #678: METRON-1076: Management UI status check always reports th...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/678
  
+1 (non-binding)

Validated that with this fix, Ambari displays the correct status as opposed 
to the earlier behavior. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---