Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Thanks all for the feedback. From: Joe Percivall Sent: Monday, February 4, 2019 9:59 PM To: users@nifi.apache.org Subject: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL I believe also one of the reasons this was done is because PutElasticsearchHttp takes in batches of FlowFiles and does a bulk insert. In order to support FlowFile attribute expression on the URL, we would have to either only act on one FlowFile at a time or determine another mechanism for handling that ambiguity. PutElasticsearcHttpRecord on the other hand only takes in a single FlowFile with each onTrigger and could be more easily updated to support that use-case. Cheers, Joe On Mon, Feb 4, 2019 at 5:05 PM Matt Burgess mailto:mattyb...@apache.org>> wrote: The restriction to using the variable registry only has always been there AFAIK, but as of 1.6 we made the distinction in documentation on how expression language would be evaluated for each property. The choice was so that we weren't constantly recreating connections for each flow file, in fact all concurrent tasks share the same underlying OkHttpClient. We could probably do something fancier where we allow flowfile attributes to be evaluated as well, but have a modestly-sized least-recently-used (LRU) cache of clients, keeping them open until they are evicted (and closing them all when stopped). Please feel free to file an improvement Jira and we can discuss further there. Regards, Matt On Mon, Feb 4, 2019 at 4:16 PM Jean-Sebastien Vachon mailto:jsvac...@brizodata.com>> wrote: > > Hi all, > > I was just finishing modifying my flow to make it more reusable by having my > source document containing information about where to store the final > document (some Elasticsearch index) > Everything was fine until I found out that the PutElasticsearchHttp's > documentation was saying this... > > Supports Expression Language: true (will be evaluated using variable registry > only) > > > It looks like this restriction appeared around Nifi 1.6 (as per the > documentation)... is there a reason for such a limitation? > > My current flow was extracting the information from the input JSON document > and saving the information inside a Flow attribute. > > What can I do about this? I don't like monkey patching.. is there any other > way to get around this? > > Thanks -- Joe Percivall linkedin.com/in/Percivall<http://linkedin.com/in/Percivall> e: jperciv...@apache.com<mailto:jperciv...@apache.com>
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
I believe also one of the reasons this was done is because PutElasticsearchHttp takes in batches of FlowFiles and does a bulk insert. In order to support FlowFile attribute expression on the URL, we would have to either only act on one FlowFile at a time or determine another mechanism for handling that ambiguity. PutElasticsearcHttpRecord on the other hand only takes in a single FlowFile with each onTrigger and could be more easily updated to support that use-case. Cheers, Joe On Mon, Feb 4, 2019 at 5:05 PM Matt Burgess wrote: > The restriction to using the variable registry only has always been > there AFAIK, but as of 1.6 we made the distinction in documentation on > how expression language would be evaluated for each property. The > choice was so that we weren't constantly recreating connections for > each flow file, in fact all concurrent tasks share the same underlying > OkHttpClient. > > We could probably do something fancier where we allow flowfile > attributes to be evaluated as well, but have a modestly-sized > least-recently-used (LRU) cache of clients, keeping them open until > they are evicted (and closing them all when stopped). Please feel free > to file an improvement Jira and we can discuss further there. > > Regards, > Matt > > On Mon, Feb 4, 2019 at 4:16 PM Jean-Sebastien Vachon > wrote: > > > > Hi all, > > > > I was just finishing modifying my flow to make it more reusable by > having my source document containing information about where to store the > final document (some Elasticsearch index) > > Everything was fine until I found out that the PutElasticsearchHttp's > documentation was saying this... > > > > Supports Expression Language: true (will be evaluated using variable > registry only) > > > > > > It looks like this restriction appeared around Nifi 1.6 (as per the > documentation)... is there a reason for such a limitation? > > > > My current flow was extracting the information from the input JSON > document and saving the information inside a Flow attribute. > > > > What can I do about this? I don't like monkey patching.. is there any > other way to get around this? > > > > Thanks > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
The restriction to using the variable registry only has always been there AFAIK, but as of 1.6 we made the distinction in documentation on how expression language would be evaluated for each property. The choice was so that we weren't constantly recreating connections for each flow file, in fact all concurrent tasks share the same underlying OkHttpClient. We could probably do something fancier where we allow flowfile attributes to be evaluated as well, but have a modestly-sized least-recently-used (LRU) cache of clients, keeping them open until they are evicted (and closing them all when stopped). Please feel free to file an improvement Jira and we can discuss further there. Regards, Matt On Mon, Feb 4, 2019 at 4:16 PM Jean-Sebastien Vachon wrote: > > Hi all, > > I was just finishing modifying my flow to make it more reusable by having my > source document containing information about where to store the final > document (some Elasticsearch index) > Everything was fine until I found out that the PutElasticsearchHttp's > documentation was saying this... > > Supports Expression Language: true (will be evaluated using variable registry > only) > > > It looks like this restriction appeared around Nifi 1.6 (as per the > documentation)... is there a reason for such a limitation? > > My current flow was extracting the information from the input JSON document > and saving the information inside a Flow attribute. > > What can I do about this? I don't like monkey patching.. is there any other > way to get around this? > > Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
You are welcome. Good Luck ! LC De: "Jean-Sebastien Vachon" Para: "Luis Carmona" , "users" Enviados: Lunes, 4 de Febrero 2019 18:45:34 Asunto: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL Hi Luis, thanks for the hint... this is indeed a good work around. So simple that I should have thought about it \uD83D\uDE09 Regards From: Luis Carmona Sent: Monday, February 4, 2019 4:23 PM To: users Subject: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL HI Jean, I'm not even near to be an expert on NIFI, but I did accomplish to put to work an scenario similar to the one you describe. I was able to read some data, process it and store the Json payload in ES. I used HTTP invoke, instead of ES Processors. Hope it helps you. Regards, LC De: "Jean-Sebastien Vachon" Para: "users" Enviados: Lunes, 4 de Febrero 2019 18:16:23 Asunto: PutElasticsearchHttp can not use Flowfile attribute for ES_URL Hi all, I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index) Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this... Supports Expression Language: true (will be evaluated using variable registry only) It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation? My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute. What can I do about this? I don't like monkey patching.. is there any other way to get around this? Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi Luis, thanks for the hint... this is indeed a good work around. So simple that I should have thought about it Regards From: Luis Carmona Sent: Monday, February 4, 2019 4:23 PM To: users Subject: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL HI Jean, I'm not even near to be an expert on NIFI, but I did accomplish to put to work an scenario similar to the one you describe. I was able to read some data, process it and store the Json payload in ES. I used HTTP invoke, instead of ES Processors. Hope it helps you. Regards, LC De: "Jean-Sebastien Vachon" Para: "users" Enviados: Lunes, 4 de Febrero 2019 18:16:23 Asunto: PutElasticsearchHttp can not use Flowfile attribute for ES_URL Hi all, I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index) Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this... Supports Expression Language: true (will be evaluated using variable registry only) It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation? My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute. What can I do about this? I don't like monkey patching.. is there any other way to get around this? Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
HI Jean, I'm not even near to be an expert on NIFI, but I did accomplish to put to work an scenario similar to the one you describe. I was able to read some data, process it and store the Json payload in ES. I used HTTP invoke, instead of ES Processors. Hope it helps you. Regards, LC De: "Jean-Sebastien Vachon" Para: "users" Enviados: Lunes, 4 de Febrero 2019 18:16:23 Asunto: PutElasticsearchHttp can not use Flowfile attribute for ES_URL Hi all, I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index) Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this... Supports Expression Language: true (will be evaluated using variable registry only) It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation? My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute. What can I do about this? I don't like monkey patching.. is there any other way to get around this? Thanks
PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi all, I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index) Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this... Supports Expression Language: true (will be evaluated using variable registry only) It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation? My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute. What can I do about this? I don't like monkey patching.. is there any other way to get around this? Thanks
