[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
+1 by inspection


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/946
  
I knew what you meant :)


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/946
  
ah, crap, looked at the wrong setting.  That's what I meant instead of 
`storm.library.path`


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/946
  
We could probably leverage this guy here


![image](https://user-images.githubusercontent.com/658443/36858092-6745d706-1d37-11e8-9cc3-fe9eec741ab0.png)



---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171624949
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
 
 
 org.elasticsearch.client
-transport
+x-pack-transport
--- End diff --

Yep, I concede the point, that is not appropriately licensed via apache and 
we cannot bundle that dependency.


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/946
  
@mmiklavc Yeah, I think that's the approach, however, there's a snag.  
Storm requires us to create uber jars, so probably what we want to do is have 
users actually put the xpath transport client on the storm.library.path.


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/946
  
@mmiklavc  I agree, as long as the user themselves is setting it up, I 
believe that would solve the license problem.  At least from my understanding 
of things.


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/946
  
@simonellistonball It looks like they are not explicitly bundling the 
X-Pack client. Rather, they're expecting the user to provide the jar file 
manually on the classpath and then dynamically instantiating with reflection. 
That would seem to solve our license issues.


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java#L71


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java#L225


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/946
  
Should we consider a dual client in the same project similar to the 
approach in 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ?

We can then switch based on the presence of the es.xpack.user setting in 
the ambari.


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171620565
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
 
 
 org.elasticsearch.client
-transport
+x-pack-transport
--- End diff --

That project does not appear to contain `PreBuiltXPackTransportClient`. ES 
points users to this - 
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/setup-xpack-client.html
 - which instructs users to add the ES Maven repo. The referenced jar's license 
file does not appear to dual license as Apache V2. 
https://artifacts.elastic.co/maven/org/elasticsearch/client/x-pack-transport/6.2.2/x-pack-transport-6.2.2.jar


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171619920
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
 
 
 org.elasticsearch.client
-transport
+x-pack-transport
--- End diff --

The main Elastic stuff is (e.g. the plain transport client if you pull it 
down has Apache license), but the x-pack stuff doesn't seem to. The 
elastic/elasticsearch repo (from a quick search) does not contain the x-pack 
code from what I can tell.

Elastic's blog post from the other day also seems to imply it's not in 
there (and not ASLv2), but I could be reading too far into it. Check out 
https://www.elastic.co/products/x-pack/open

>We are creating a new X-Pack folder in each of these repositories that 
will be licensed under the Elastic EULA, which allows for some derivative works 
and contribution.




---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171618015
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
 
 
 org.elasticsearch.client
-transport
+x-pack-transport
--- End diff --

I believe this is dual licensed ES commercial and ASLv2.  IT comes from 
[here](https://github.com/elastic/elasticsearch#license).


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171617250
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
 
 
 org.elasticsearch.client
-transport
+x-pack-transport
--- End diff --

For posterity, I pulled the jar with
```
{11:23}~ ➭ mvn org.apache.maven.plugins:maven-dependency-plugin:3.0.2:get 
\
-DremoteRepositories=https://artifacts.elastic.co/maven \
-Dartifact=org.elasticsearch.client:x-pack-api:5.6.7
```
and just opened up

`~/.m2/repository/org/elasticsearch/client/x-pack-transport/5.6.7/x-pack-transport-5.6.7.pom`

If someone knows more about how the ES licensing for this stuff works, let 
me know, because it' s definitely possible there's something I don't know about.


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171614360
  
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
 
 
 org.elasticsearch.client
-transport
+x-pack-transport
--- End diff --

Unlike the plain transport client, this jar is NOT Apache licensed.  I 
pulled it down and it has this
```
  x-pack-transport
  5.6.7
  ...
  

  Elastic Commercial Software End User License Agreement
  https://www.elastic.co/eula/
  repo

  
```


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[wardbekker]

Github user wardbekker commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171599970
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
 ---
@@ -111,19 +111,24 @@ public static TransportClient getClient(Map globalConfiguration,
 Settings.Builder settingsBuilder = Settings.builder();
 settingsBuilder.put("cluster.name", 
globalConfiguration.get("es.clustername"));
 settingsBuilder.put("client.transport.ping_timeout","500s");
+settingsBuilder.put("transport.type", "security4");
+Object xPackUser = globalConfiguration.get("es.xpackuser");
+if (xPackUser != null) {
+  settingsBuilder.put("xpack.security.user", xPackUser);
+}
 if (optionalSettings != null) {
   settingsBuilder.put(optionalSettings);
 }
 Settings settings = settingsBuilder.build();
-TransportClient client;
+PreBuiltXPackTransportClient client;
--- End diff --

yep. it might make sense to also check for a empty string for the 
"es.xpackuser" config, or does globalConfiguration.get return null also for 
empty strings?


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r171596212
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
 ---
@@ -111,19 +111,24 @@ public static TransportClient getClient(Map globalConfiguration,
 Settings.Builder settingsBuilder = Settings.builder();
 settingsBuilder.put("cluster.name", 
globalConfiguration.get("es.clustername"));
 settingsBuilder.put("client.transport.ping_timeout","500s");
+settingsBuilder.put("transport.type", "security4");
+Object xPackUser = globalConfiguration.get("es.xpackuser");
+if (xPackUser != null) {
+  settingsBuilder.put("xpack.security.user", xPackUser);
+}
 if (optionalSettings != null) {
   settingsBuilder.put(optionalSettings);
 }
 Settings settings = settingsBuilder.build();
-TransportClient client;
+PreBuiltXPackTransportClient client;
--- End diff --

First question I have is will this work with and without XPack enabled?


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[wardbekker]

GitHub user wardbekker opened a pull request:

https://github.com/apache/metron/pull/946

METRON-1465:Support for Elasticsearch X-pack

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wardbekker/metron xpack_support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/946.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #946


commit 9978ba7151fb155fbf000af17650ca2a9784d258
Author: Ward Bekker 
Date:   2018-02-26T14:21:59Z

Squashed commit of xpack commits




---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
Sorry, I'll try to get back to this today



---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
Any other feedback @ottobackwards or is this ready to go?


---

[GitHub] metron pull request #934: METRON-1423: Ambari work to handle Solr configurat...

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/934


---

[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/945
  
Are we losing anything by moving the scheme from Range to Trie types?, 
repeating my comment on 
 https://github.com/apache/metron/pull/922: 

Given that our use case is heavily dependant on sorting, I wonder why not 
the Trie based indices for numeric fields. I may be completely wrong on their 
advantages but would love to hear the logic behind the choice of Point indices.

If there is a good reason, maybe we should consider retaining those for 6.6 
in addition to the 5.5 clusters. Either way it would be be good to understand 
the basis for the type decision.


---