[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/946 thanks @mmiklavc ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/946 @ottobackwards - I have some open changes in a PR against this PR. Just to confirm, the expected result of this PR should be that users can choose 1. Which client they want to instantiate 2. Whether they want X-Pack enabled or not The default ootb functionality should be the same as what we currently do. X-Pack is a special case. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/946 @cestella " Yeah, I think that's the approach, however, there's a snag. Storm requires us to create uber jars, so probably what we want to do is have users actually put the xpath transport client on the storm.library.path." If only there was a way to load things into storm with classloader isolation and dependency inclusion. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/946 would this have any effect on people using x-pack alternatives? ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/946 @ottobackwards in it's current state, sort of, but you're not required to turn it on. In the desired (reflection based nifi style state) no, it should load it and use it if present, but otherwise just use the vanilla transport client. Of course someday they're going to force us to REST, which will bring all this up again no doubt. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/946 wait, does this PR mean we *require* x-pack from now on? ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user wardbekker commented on the issue: https://github.com/apache/metron/pull/946 @mmiklavc yes, correct, like with ES you need to install x-pack for Kibana. You will be prompted for username password after restart of Kibana. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/946 @wardbekker Did you test this with Kibana by chance? I'm seeing authentication exceptions in the Kibana UI. Looks like we need additional steps for it to work - https://www.elastic.co/guide/en/kibana/5.6/installing-xpack-kb.html ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user wardbekker commented on the issue: https://github.com/apache/metron/pull/946 Steps to test: - build centos vagrant dev build box - vagrant ssh - sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack - sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser - restart ES via ambari - verify 401 status: curl http://localhost:9200 - verify new documents are indexed in ES: http://node1:9200/_cat/indices?v ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/946 I knew what you meant :) ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user cestella commented on the issue: https://github.com/apache/metron/pull/946 ah, crap, looked at the wrong setting. That's what I meant instead of `storm.library.path` ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/946 We could probably leverage this guy here  ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user cestella commented on the issue: https://github.com/apache/metron/pull/946 @mmiklavc Yeah, I think that's the approach, however, there's a snag. Storm requires us to create uber jars, so probably what we want to do is have users actually put the xpath transport client on the storm.library.path. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/946 @mmiklavc I agree, as long as the user themselves is setting it up, I believe that would solve the license problem. At least from my understanding of things. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/946 @simonellistonball It looks like they are not explicitly bundling the X-Pack client. Rather, they're expecting the user to provide the jar file manually on the classpath and then dynamically instantiating with reflection. That would seem to solve our license issues. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java#L71 https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java#L225 ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/946 Should we consider a dual client in the same project similar to the approach in https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java ? We can then switch based on the presence of the es.xpack.user setting in the ambari. ---
