RE: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Charles Joynt
Regarding why I didn't choose to load data with the flatfile loader script... I want to be able to SEND enrichment data to Metron rather than have to set up cron jobs to PULL data. At the moment I'm trying to prove that the process works with a simple data source. In the future we will want enri

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Carolyn Duby
Agreed….Streaming enrichments is the right solution for DNS data. Do we have a web service for writing enrichments? Carolyn Duby Solutions Engineer, Northeast cd...@hortonworks.com +1.508.965.0584 Join my team! Enterprise Account Manager – Boston - http://grnh.se/wepchv1 Solutions Engineer – Bos

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Casey Stella
no, sadly we do not. On Wed, Jun 13, 2018 at 10:01 AM Carolyn Duby wrote: > Agreed….Streaming enrichments is the right solution for DNS data. > > Do we have a web service for writing enrichments? > > Carolyn Duby > Solutions Engineer, Northeast > cd...@hortonworks.com > +1.508.965.0584 > > Join

Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Ali Nazemian
Hi All, I have noticed that the recommendation from Elasticsearch team is changed to use Java Rest Client instead of Transport one. The rationale behind it looks convincing and it can also help Metron to be more decoupled from Elasticsearch roadmap, so Metron users can upgrade Elasticsearch with

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Simon Elliston Ball
That’s where something like the Nifi solution would come in... With the PutEnrichment processor and a ProcessHttpRequest processor, you do have a web service for loading enrichments. We could probably also create a rest service end point for it, which would make some sense, but there is a nice

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Otto Fowler
Do we even have a jira? If not maybe Carolyn et. al. can write one up that lays out some requirements and context. On June 13, 2018 at 10:04:27, Casey Stella (ceste...@gmail.com) wrote: no, sadly we do not. On Wed, Jun 13, 2018 at 10:01 AM Carolyn Duby wrote: > Agreed….Streaming enrichments

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Casey Stella
It was my understanding was that ES x-pack only supports the transport client (e.g. https://www.elastic.co/guide/en/x-pack/current/java-clients.html). I think that was a major reason why we chose to go that route. I might be wrong though. On Wed, Jun 13, 2018 at 10:30 AM Ali Nazemian wrote: >

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Otto Fowler
-jiras- On June 13, 2018 at 10:30:26, Simon Elliston Ball ( si...@simonellistonball.com) wrote: That’s where something like the Nifi solution would come in... With the PutEnrichment processor and a ProcessHttpRequest processor, you do have a web service for loading enrichments. We could probab

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Michael Miklavcic
I think there is some level of support for auth via their REST api, but I don't see anything specific to X-Pack as you mentioned. However, the major reason we did not adopt it at the time of upgrade was because a number of features were not available to REST yet and the effort to simultaneously upg

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

2018-06-13 Thread Simon Elliston Ball
Not convinced we should be writing Jiras against the metron project, or the nifi project if we don't know where it's actually going to end up to be honest. In any case, working code: https://github.com/simonellistonball/metron/tree/nifi/nifi-metron-bundle which is currently in a metron fork, for no

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Ali Nazemian
Hi Michael and Casey, It looks like ES believe Java Rest Client is mature enough to be pushed to different products at this stage. However, I haven't used it personally. I will share the question regarding x-pack support with Elasticsearch engineers to see if there is any issue regarding that. Ch