Re: MergeContent resulting in corrupted JSON

2020-06-30 Thread Darren Govoni
Run the nifi jvm in a runtime profiler/analyzer like appdynamics and see if it 
detects any memory leaks or dangling unclosed file buffers/io. Throwing darts 
but the problem could be as deep as the Linux kernel or confined inside the jvm 
for your specific scenario.

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android


From: Jason Iannone 
Sent: Tuesday, June 30, 2020 10:36:02 PM
To: users@nifi.apache.org 
Subject: Re: MergeContent resulting in corrupted JSON

Previous spotting of the issue was a red herring. We removed our custom code 
and are still facing random "org.codehaus.jackson.JsonParseException: Illegal 
Character" during PutDatabaseRecord due to a flowfile containing malformed JSON 
post MergeContent. Error never occurs immediately and is usually once we've 
processed several million records. We did a NOOP run, which was ConsumeKafka -> 
UpdateCounter and everything seemed ok.

Here's the current form of the flow:

  1.  ConsumeKafka_2_0 - Encoding headers as ISO-8859-1 due to some containing 
binary data
 *   I have a fork of nifi with changes to allow base64 and hex encoding of 
select nifi headers.
 *   Next test will be without pulling any headers
  2.  RouteOnAttribute - Validate attributes
  3.  Base64EncodeContent - Content is binary, converting to a format we can 
store to later process
  4.  ExtractText - Copy Base64 encoded content to attribute
  5.  AttributesToJson - Provenance shows output as being fine
  6.  MergeContent - Provenance shows output of malformed JSON being written in 
the combined flowflle.
  7.  PutDatabaseRecord - Schema specified as Schema Text

Since we've removed all traces of custom code what are peoples thoughts on 
possible causes? Could this be an OS issue, or are there any known issues with 
specific versions of RHEL?

Logically I think it makes sense to remove JSON from the equation as a whole.

Thanks,
Jason

On Wed, Jun 24, 2020 at 2:54 PM Jason Iannone 
mailto:bread...@gmail.com>> wrote:
Exactly my thought, and we've been combing through the code but nothing 
significant has jumped out. Something that does are Nifi JIRA's, NIFI-6923, 
NIFI-6924, and NIFI-6846. Considering we're on 1.10.0 I've requested upgrading 
to 1.11.4.

Thanks,
Jason

On Tue, Jun 23, 2020 at 9:05 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
It should be okay to create a buffer like that. Assuming the FlowFile is small. 
Typically we try to avoid buffering the content of a FlowFile into memory. But 
if it’s a reasonable small FlowFile, that’s probably fine.

To be honest, if the issue is intermittent and doesn’t always happen on the 
same input, it sounds like a threading/concurrency bug. Do you have a buffer or 
anything like that as a member variable?

On Jun 22, 2020, at 10:02 PM, Jason Iannone 
mailto:bread...@gmail.com>> wrote:

I'm now thinking its due to how we handled reading the flowfile content into a 
buffer.

Previous:
session.read(flowFile, in -> {
  atomicVessel.set(ByteStreams.toByteArray(in));
});

Current:
final byte[] buffer = new byte[(int) flowFile.getSize()];
session.read(flowFile, in -> StreamUtils.fillBuffer(in, buffer, true));

Making this change reduced the occurrences of the data corruption, but we still 
saw it occur. What I'm now wondering is if sizing the byte array based on 
flowFile.getSize() is ideal? The contents of the file are raw bytes coming from 
ConsumeKafka_2_0.

Thanks,
Jason

On Mon, Jun 22, 2020 at 4:51 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Jason,

Glad to hear it. This is where the data provenance becomes absolutely 
invaluable. So now you should be able to trace the lineage of that FlowFile 
back to the start using data provenance. You can see exactly what it looked 
like when it was received. If it looks wrong there, the provenance shows 
exactly where it was received from so you know where to look next. If it looks 
good on receipt, you can trace the data through the flow and see exactly what 
the data looked like before and after each processor. And when you see which 
processor resulted in corruption, you can easily download the data as it looks 
when it went into the processor to make it easy to re-ingest and test.

Thanks
-Mark


On Jun 22, 2020, at 4:46 PM, Jason Iannone 
mailto:bread...@gmail.com>> wrote:

I spoke too soon, and must be the magic of sending an email! We found what 
appears to be corrupted content and captured the binary, hoping to play it 
through the code and see what's going on.

Thanks,
Jason

On Mon, Jun 22, 2020 at 4:35 PM Jason Iannone 
mailto:bread...@gmail.com>> wrote:
Hey Mark,

We hit the issue again, and when digging into the lineage we can see the 
content is fine coming into MergeContent but is corrupt on output of Join. Any 
other suggestions?

Thanks,
Jason

On Wed, Jun 10, 2020 at 2:26 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Jason,

Control characters should not cause any problem 

Re: MergeContent resulting in corrupted JSON

2020-06-30 Thread Jason Iannone
Previous spotting of the issue was a red herring. We removed our custom
code and are still facing random "org.codehaus.jackson.JsonParseException:
Illegal Character" during PutDatabaseRecord due to a flowfile containing
malformed JSON post MergeContent. Error never occurs immediately and is
usually once we've processed several million records. We did a NOOP run,
which was ConsumeKafka -> UpdateCounter and everything seemed ok.

Here's the current form of the flow:

   1. ConsumeKafka_2_0 - Encoding headers as ISO-8859-1 due to some
   containing binary data
  1. I have a fork of nifi with changes to allow base64 and hex
  encoding of select nifi headers.
  2. Next test will be without pulling any headers
   2. RouteOnAttribute - Validate attributes
   3. Base64EncodeContent - Content is binary, converting to a format we
   can store to later process
   4. ExtractText - Copy Base64 encoded content to attribute
   5. AttributesToJson - Provenance shows output as being fine
   6. MergeContent - Provenance shows output of malformed JSON being
   written in the combined flowflle.
   7. PutDatabaseRecord - Schema specified as Schema Text

Since we've removed all traces of custom code what are peoples thoughts on
possible causes? Could this be an OS issue, or are there any known issues
with specific versions of RHEL?

Logically I think it makes sense to remove JSON from the equation as a
whole.

Thanks,
Jason

On Wed, Jun 24, 2020 at 2:54 PM Jason Iannone  wrote:

> Exactly my thought, and we've been combing through the code but nothing
> significant has jumped out. Something that does are Nifi JIRA's, NIFI-6923,
> NIFI-6924, and NIFI-6846. Considering we're on 1.10.0 I've requested
> upgrading to 1.11.4.
>
> Thanks,
> Jason
>
> On Tue, Jun 23, 2020 at 9:05 AM Mark Payne  wrote:
>
>> It should be okay to create a buffer like that. Assuming the FlowFile is
>> small. Typically we try to avoid buffering the content of a FlowFile into
>> memory. But if it’s a reasonable small FlowFile, that’s probably fine.
>>
>> To be honest, if the issue is intermittent and doesn’t always happen on
>> the same input, it sounds like a threading/concurrency bug. Do you have a
>> buffer or anything like that as a member variable?
>>
>> On Jun 22, 2020, at 10:02 PM, Jason Iannone  wrote:
>>
>> I'm now thinking its due to how we handled reading the flowfile content
>> into a buffer.
>>
>> Previous:
>> session.read(flowFile, in -> {
>>   atomicVessel.set(ByteStreams.toByteArray(in));
>> });
>>
>> Current:
>> final byte[] buffer = new byte[(int) flowFile.getSize()];
>> session.read(flowFile, in -> StreamUtils.fillBuffer(in, buffer, true));
>>
>> Making this change reduced the occurrences of the data corruption, but we
>> still saw it occur. What I'm now wondering is if sizing the byte array
>> based on flowFile.getSize() is ideal? The contents of the file are raw
>> bytes coming from ConsumeKafka_2_0.
>>
>> Thanks,
>> Jason
>>
>> On Mon, Jun 22, 2020 at 4:51 PM Mark Payne  wrote:
>>
>>> Jason,
>>>
>>> Glad to hear it. This is where the data provenance becomes absolutely
>>> invaluable. So now you should be able to trace the lineage of that FlowFile
>>> back to the start using data provenance. You can see exactly what it looked
>>> like when it was received. If it looks wrong there, the provenance shows
>>> exactly where it was received from so you know where to look next. If it
>>> looks good on receipt, you can trace the data through the flow and see
>>> exactly what the data looked like before and after each processor. And when
>>> you see which processor resulted in corruption, you can easily download the
>>> data as it looks when it went into the processor to make it easy to
>>> re-ingest and test.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Jun 22, 2020, at 4:46 PM, Jason Iannone  wrote:
>>>
>>> I spoke too soon, and must be the magic of sending an email! We found
>>> what appears to be corrupted content and captured the binary, hoping to
>>> play it through the code and see what's going on.
>>>
>>> Thanks,
>>> Jason
>>>
>>> On Mon, Jun 22, 2020 at 4:35 PM Jason Iannone 
>>> wrote:
>>>
 Hey Mark,

 We hit the issue again, and when digging into the lineage we can see
 the content is fine coming into MergeContent but is corrupt on output of
 Join. Any other suggestions?

 Thanks,
 Jason

 On Wed, Jun 10, 2020 at 2:26 PM Mark Payne 
 wrote:

> Jason,
>
> Control characters should not cause any problem with MergeContent.
> MergeContent just copies bytes from one stream to another. It’s also worth
> noting that attributes don’t really come into play here. MergeContent is
> combining the FlowFile content, so even if it has some weird attributes,
> those won’t cause a problem in the output content. NiFi stores attributes
> as a mapping of String to String key/value pairs (i.e., Map String>). So the processor is assuming that if you want to convert a

Re: Enrichment of record data with a REST API

2020-06-30 Thread Mark Payne
Mike,

To really do a good job with Enrichment from an http endpoint, we need two 
transformations really. We need the ability to transform the input Record into 
what the web service wants/needs. And we also need the ability to take the 
response from that web service and join that response together with the input 
Record.

Ideally there would be script-based ways to do each of these, as well as other 
ways, such as using SQL to query for certain fields from the input record and 
certain fields from the lookup service response. I have some ideas floating 
around in my head but I’ve not really laid anything out very clearly. There are 
really two different approaches that I think could work here.

The first approach would be to have a “request transformation” property that 
would allow for taking an input record and transforming it into a request for 
the web service. There would then be another “response transformation” property 
that would merge the input and the response into a single Record. This is, in a 
way, simple, because it it means that the LookupRecord processor is all that is 
needed. But it would need a Controller Service for the Record Reader, one for 
the Record Writer, and a third for the Lookup Service. It would likely also 
require a Controller Service for the “request transformation” and another for 
the “response transformation.” So it ends up being a lot more complex to 
configure than it first seems. But perhaps this is okay...

The second approach would be to create a couple of different processors that 
would allow for the implementation of the Claim Check Pattern [1]. Here, we 
would have a “CheckFlowFile” processor, which could be used in conjunction with 
any existing processor for transforming the incoming Record into the needed 
request. The response would then be obtained, and then would need to be merged 
back together with some sort of “Merge Records” processor. This approach is 
probably cleaner in terms of flow design and easier to understand. It also 
results in better code re-use. Unfortunately, though, it does come with the 
complication of figuring out how to properly correlate the appropriate input 
Record with the appropriate “enrichment response” Record….

Just some thoughts :)

[1] 
https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html


On Jun 29, 2020, at 8:22 PM, Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:

Matt,

Yeah, I was thinking about that, but there are a lot of variables that have 
come up since I wrote that service. One of the big ones is how to take partial 
responses and merge them? There's no transformer API for that service. Nothing 
like a Groovy script, JOLT, etc.

What do you think? I think something JOLT-based similar to JoltTransformRecord 
could be a starting point.

On Mon, Jun 29, 2020 at 5:14 PM Matt Burgess 
mailto:mattyb...@apache.org>> wrote:
Mike,

I think you can use LookupRecord with a RestLookupService to do this.
If it's missing features or it otherwise doesn't work for your use
case, please let us know and/or write up whatever Jiras you feel are
appropriate.

Regards,
Matt

On Mon, Jun 29, 2020 at 4:56 PM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
>
> Does anyone know a good pattern using the Record API to enrich a data set 
> record by record with a REST API?
>
> Thanks,
>
> Mike



Re: In memoriam of Jeff Storck

2020-06-30 Thread Vijay Chhipa
I waited on pins and needles for the Java 11 support to come out. 
Little did I know that Jeff was the man behind it. 

Thanks for all of your efforts Jeff, because of you we were able to meet 
critical deadlines. 
RIP Jeff. 

Vijay


> On Jun 16, 2020, at 10:03 AM, Kevin Doran  wrote:
> 
> Jeff, you were a fantastic collaborator and friend. You will be dearly 
> missed. Thank you for all your contributions, and for all you’ve  shown and 
> taught me over the years. You’ve left behind a great legacy that will 
> continue to have a positive impact on the world for years to come, not just 
> your work but your way of working with others, and for that we are all 
> grateful. RIP.
> 
>> On Jun 15, 2020, at 3:30 PM, Pierre Villard > > wrote:
>> 
>> I can't say how much we will miss you Jeff. You were a great guy, always 
>> nice and helpful with everyone. You always went the extra mile to make 
>> things easier and more robust.
>> 
>> RIP Jeff
>> 
>> Le lun. 15 juin 2020 à 21:13, Jeremy Dyer > > a écrit :
>> This is shocking and heartbreaking news. Jeff was a great guy and will be 
>> deeply missed. 
>> 
>> The last time I saw Jeff in person was with Aldrin. We were eating at 
>> Bonchon chicken and he was mocking me for how little spice I could handle 
>> XD. I could always count on him for a good Dumb and Dumber reference and 
>> laugh. We also shared a common hatred for conference food.
>> 
>> RIP Jeff
>> 
>> On Mon, Jun 15, 2020 at 2:33 PM Joe Witt > > wrote:
>> You will be greatly missed.  Your impact to this community has been 
>> tremendous.  The items Andy summarizes were huge efforts that you drove over 
>> periods of many many months if not a year or more and they make NiFi so much 
>> more accessible than before.
>> 
>> RIP Jeff.
>> 
>> 
>> 
>> On Mon, Jun 15, 2020 at 11:24 AM Andy LoPresto > > wrote:
>> It is with a heavy heart that I write to the NiFi community today. Jeff 
>> Storck, a PMC member, committer, and genuine and helpful presence in the 
>> community, has passed away. 
>> 
>> I was lucky enough to know Jeff personally for many years, and his absence 
>> is a huge loss to all of us who did. Jeff was incredibly intelligent, but 
>> also kind and willing to share his experience with everyone. Whether playing 
>> volleyball (I am nowhere near as good but he humored me), discussing the 
>> best ramen and sushi spots, or evaluating a new exercise regime, Jeff 
>> brought passion to everything. A number of us are sharing stories of our 
>> favorite times with Jeff, and I am touched by how many people have a memory 
>> of Jeff reaching out and patiently helping them when they were new or 
>> struggling with a task. 
>> 
>> While other colleagues would happily transition to any topic _but_ work when 
>> we went to a nearby brewery at the end of a long day, Jeff would sit down 
>> next to me and say with a smile, "Ok Andy, work's done, now we can _really_ 
>> talk about Groovy unit testing." He never shied away from expressing his 
>> perspective and stood on conviction, but he was also open and genuinely 
>> wanted to hear other views to expand his mind. 
>> 
>> If you come across a Spock test in the NiFi codebase, that was most likely 
>> Jeff's work. He was intimately involved in much of the most challenging code 
>> - especially Kerberos integration, making the difficult but critical 
>> processes easier for our users. Anyone running NiFi on Java 11 should thank 
>> Jeff, as that was a labor of love, pushing against the headwinds of so many 
>> compatibility issues and language changes. The ease with which NiFi runs on 
>> multiple versions and platforms belies the immense amount of effort and 
>> dedication that he put into making this happen. 
>> 
>> There are so many aspects to Jeff that a note like this could never capture, 
>> but one that stands above the rest to me is Jeff's passion for learning and 
>> growth. He devoted himself to doing the best he could and constantly 
>> improving that. That is a noble philosophy that I know I will remember and 
>> admire moving forward. I’ve already started learning Kotlin because of 
>> Jeff’s enthusiasm and encouragement. 
>> 
>> Jeff’s family has created a GoFundMe page [1] and there they describe their 
>> intent to celebrate his life. I think that message is very positive and 
>> uplifting. To anyone wondering how they can honor Jeff's legacy, I suggest 
>> offering a helping hand to someone who needs it. Something as simple as 
>> responding to an extra "newbie" mailing list question at the end of a long 
>> day, or taking on a challenging task because your neighbor has their plate 
>> full. That's how Jeff lived, and he made the world a better place. 
>> 
>> 
>> 
>> Andy
>> 
>> [1] https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck 
>> 
>> 
>> Andy 

Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

2020-06-30 Thread Andy LoPresto
You should not need to explicitly set the additional module directory to cover 
that location. Is there a reason you can’t use the native Groovy JSON [1] 
parsing? That way you don’t have to download any additional libraries. 

[1] http://groovy-lang.org/json.html# 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jun 29, 2020, at 7:41 AM, Myklebust, Bjørn Magnar 
>  wrote:
> 
> Andy, just a quick followup on this.
>  
> I wanted to test a groovy-script with this code (not finished by far yet), 
> and the script is placed in the Script Body part of an 
> ExecuteGroovyScript-process in NiFi:
>  
>  
> import org.json.JSONObject
> import org.json.XML
> import org.apache.commons.io.IOUtils
> import java.nio.charset.*
>  
> def flowFile = session.get()
> if (!flowFile) return
>  
> flowFile = session.write(flowFile,
>   {inputStream, outputStream ->
>   def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
>   def xmlJSONObj = XML.toJSONObject(text);
>   def json = xmlJSONObj.toString();
>   outputStream.write(json.getBytes(StandardCharsets.UTF_8))
>   } as StreamCallback)
>  
> session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
>  
> But when trying to run this I get the message «unable to resolve class 
> org.json.JSONObject @ line 1»
> I have downloaded the jar file from this site:  
> https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar 
> 
> And placed it in my nifi/lib-directory.
> And the content of this jar you can see in the enclosed png-picture.
>  
> Do I need to set a value for the property Additional Classpath when the 
> jar-file is stored in the lib-directory?
>  
> Thanks,
> Bjørn
>  
>  
> Fra: Andy LoPresto  
> Sendt: torsdag 25. juni 2020 19:20
> Til: users@nifi.apache.org
> Emne: Re: Replacing a base64-encoded field in a JSON-document with its 
> decoded/converted value
>  
> Hi Bjørn,
>  
> No, XML to JSON conversion is not an Expression Language feature. You’ll need 
> to either get this data into a flowfile as the complete content to perform 
> the conversion with existing built-in tools, or add that step to your Groovy 
> script. 
>  
> With that additional requirement, I think using the Groovy script to perform 
> those steps in tandem is probably the most performant and logical approach 
> here. 
>  
>  
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar 
> mailto:bjorn.mykleb...@skatteetaten.no>> 
> wrote:
>  
> Thanks Andy.
> The XML-content is around 5 kB-ish.  But I also need to convert the XML to 
> JSON before replacing it back into the original JSON-file.  Can this be done 
> with e.g a ConvertAttribute before the ReplaceText?
>  
> Thanks,
> Bjørn
>  
>  
>  
> Fra: Andy LoPresto mailto:alopre...@apache.org>> 
> Sendt: onsdag 24. juni 2020 17:24
> Til: users@nifi.apache.org 
> Emne: Re: Replacing a base64-encoded field in a JSON-document with its 
> decoded/converted value
>  
> Hello Bjørn, 
>  
> If the size of the encoded XML document is small (under ~1 KB), you can 
> extract the Base64-encoded value to a flowfile attribute using 
> EvaluateJSONPath, perform the decoding using the base64Decode Expression 
> Language function [1], and then replace it into the flowfile JSON content 
> using ReplaceText (using some regex like "content": ".*" -> “content": 
> ”${decodedXML}” where decodedXML is the name of the attribute you are using). 
>  
> If the XML content could be very large, this will negatively affect your 
> performance, as attributes are stored directly in memory and handling large 
> amounts of data will impact the heap. In this case, I would recommend writing 
> a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly 
> JSON handling and extract the value, Base64 decode it, and replace it in a 
> couple lines. 
>  
> Hope this helps. 
>  
>  
> [1] 
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode
>  
> 
>  
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> 
> On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar 
> mailto:bjorn.mykleb...@skatteetaten.no>> 
> wrote:
>  
>  
> Hi.
> I have a set of Json-files which contain a base64-coded field (Jsonpath to 
> this field is $.data.content), and this field contains a XML-document.  
> Decoding the field 

Re: Session state in cluster HandleHttpRequest and HandleHttpReponse

2020-06-30 Thread Jeremy Pemberton-Pigott
Thanks Peter that makes sense. I'll try a wait/notify using an identifier for 
the node in the Spark messages being monitored so that the same node will 
receive the reply from Spark and respond to the client that initiated the 
connection. 

Regards,

Jeremy


On 30 Jun 2020, at 22:41, Peter Turcsanyi  wrote:


Hi Jeremy,

I don't think you can accept the request in one node and send back the response 
from another node. 
There is an open HTTP connection between the client and the NiFi node while the 
HandleHttpRequest -> ... -> HandleHttpResponse flow is running.
Even if we passed the request/response context object between the NiFi nodes 
via a distributed cache, it would not be possible to send back the HTTP 
response to a client that originally connected and sent the request to another 
node. At least I don't know any solution to it in/outside NiFi.

Best,
Peter

> On Tue, Jun 30, 2020 at 6:04 AM Jeremy Pemberton-Pigott 
>  wrote:
> Hi,
> 
> I have a cluster of 3 nodes and the incoming request on one node's 
> HandleHttpRequest may be replied to by a different node's HandleHttpResponse, 
> in between there is a Spark streaming job process.  Is there any example how 
> to do that, maybe with DistributeMapCacheService?  So that I can still reply 
> back to the client with the necessary data in the response.
> 
> Jeremy


NiFi 1.11.4 -- "Unable to access lib/bootstrap to create bootstrap classloader"

2020-06-30 Thread Joseph Wheeler
Hello!

I'm trying to deploy NiFi 1.11.4 to a new environment. After configuring all 
the necessary files and trying to start the service, I see the following 
message in the nifi-app.log file:

 INFO  [main]   org.apache.nifi.NiFi Launching 
NiFi...
 WARN   [main]   org.apache.nifi.NiFi Unable to access 
lib/bootstrap to create bootstrap classloader
Java.nio.file.NoSuchFileException: lib/bootstrap

 ERROR  [main]   org.apache.nifi.NiFi Failure to launch 
NiFi due to java.lang.IllegalArgumentException: Unable to access properties 
loader in the expected manner - apparent classpath or build issue


I found a bug report with this exact issue 
(https://issues.apache.org/jira/browse/NIFI-4685) but it was a few years ago 
and is still marked as Open/Unresolved.

Anybody seen this issue / have a solution?

I'm running this on RHEL7.

r/

JW


Re: Session state in cluster HandleHttpRequest and HandleHttpReponse

2020-06-30 Thread Peter Turcsanyi
Hi Jeremy,

I don't think you can accept the request in one node and send back the
response from another node.
There is an open HTTP connection between the client and the NiFi node while
the HandleHttpRequest -> ... -> HandleHttpResponse flow is running.
Even if we passed the request/response context object between the NiFi
nodes via a distributed cache, it would not be possible to send back the
HTTP response to a client that originally connected and sent the request to
another node. At least I don't know any solution to it in/outside NiFi.

Best,
Peter

On Tue, Jun 30, 2020 at 6:04 AM Jeremy Pemberton-Pigott <
fuzzych...@gmail.com> wrote:

> Hi,
>
> I have a cluster of 3 nodes and the incoming request on one node's
> HandleHttpRequest may be replied to by a different node's
> HandleHttpResponse, in between there is a Spark streaming job process.  Is
> there any example how to do that, maybe with DistributeMapCacheService?  So
> that I can still reply back to the client with the necessary data in the
> response.
>
> Jeremy
>


Re: Unsubscribe

2020-06-30 Thread Marton Szasz
Please send an email to users-unsubscr...@nifi.apache.org to unsubscribe.
Source: https://nifi.apache.org/mailing_lists.html

On Tue, 30 Jun 2020 at 14:58, Ravisankar Mani  wrote:

> Unsubscribe
>


Unsubscribe

2020-06-30 Thread Ravisankar Mani
Unsubscribe


Re: Unsubscribe

2020-06-30 Thread Marton Szasz
Please send an email to users-unsubscr...@nifi.apache.org to unsubscribe.
Source: https://nifi.apache.org/mailing_lists.html

On Tue, 30 Jun 2020 at 14:13, John W Von Holle  wrote:

> Unsubscribe
>
>
>
> John Von Holle
>
> Geospatial Engineer
>
> The MITRE Corporation
>
> 703-983-8374
>
> jvonho...@mitre.org
>
>
>


Unsubscribe

2020-06-30 Thread John W Von Holle
Unsubscribe

John Von Holle
Geospatial Engineer
The MITRE Corporation
703-983-8374
jvonho...@mitre.org



Re: Need help SSL LDAP Nifi Registry

2020-06-30 Thread Etienne Jouvin
Got it thanks to
https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-NiFi-to-Integrate-with-a-Secure-NiFi/ta-p/247765

Next steps would be to have NiFi and Registry on different hosts and see
how connections are made.



Le mar. 30 juin 2020 à 11:43, Etienne Jouvin  a
écrit :

> But now, I have NiFi and Registry with secure access (LDAP + SSL)
>
> I need to find out how to configure the Registry in NiFi, because for now
> I did not have to specify login.
> And even if my first bucket is Public, it is not accessible from NiFi.
>
>
> Le mar. 30 juin 2020 à 11:29, Etienne Jouvin  a
> écrit :
>
>> Hi Josef.
>>
>> No I did not try that.
>> And well done, with that I can access the UI, and can connect with LDAP
>> identity.
>>
>> Thanks a lot.
>>
>> Cheers
>>
>> Etienne
>>
>>
>>
>> Le mar. 30 juin 2020 à 11:15,  a écrit :
>>
>>> Hi Etienne
>>>
>>>
>>>
>>> Did you tried the following in «nifi-registry.properties»:
>>>
>>> nifi.registry.security.needClientAuth=false
>>>
>>>
>>>
>>> Cheers Josef
>>>
>>>
>>>
>>>
>>>
>>> *From: *Etienne Jouvin 
>>> *Reply to: *"users@nifi.apache.org" 
>>> *Date: *Tuesday, 30 June 2020 at 10:46
>>> *To: *"users@nifi.apache.org" 
>>> *Subject: *Need help SSL LDAP Nifi Registry
>>>
>>>
>>>
>>> Hello all.
>>>
>>>
>>>
>>> I am trying to setup LDAP authentication on NiFi Registry.
>>>
>>> I followed some links, like
>>> https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753
>>>
>>>
>>>
>>> But each time, it requires that a certificate is installed on client
>>> side. I had this "problem" for NiFi but because I did not provided
>>> the nifi.security.user.login.identity.provider
>>>
>>>
>>>
>>> But for the registry, I remember that and did it.
>>>
>>>
>>>
>>> For summary, what I have in nifi-registry.properties
>>>
>>> nifi.registry.security.keystore=./conf/keystore.jks
>>> nifi.registry.security.keystoreType=jks
>>> nifi.registry.security.keystorePasswd=password
>>> nifi.registry.security.keyPasswd=password
>>> nifi.registry.security.truststore=./conf/truststore.jks
>>> nifi.registry.security.truststoreType=jks
>>> nifi.registry.security.truststorePasswd=password
>>>
>>>
>>>
>>> (All of those informations were given by the tls-toolkit, when executed
>>> for NiFi)
>>>
>>> Then I put this
>>>
>>> #nifi.registry.security.identity.provider=
>>> nifi.registry.security.identity.provider=ldap-identity-provider
>>>
>>>
>>>
>>> In the file identity-providers.xml
>>>
>>> I setup the LDAP provider
>>>
>>> 
>>> ldap-identity-provider
>>>
>>> org.apache.nifi.registry.security.ldap.LdapIdentityProvider
>>> SIMPLE
>>>
>>> uid=admin,ou=system
>>> secret
>>>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>
>>> FOLLOW
>>> 10 secs
>>> 10 secs
>>>
>>> ldap://localhost:10389
>>> ou=users,dc=test,dc=ch
>>> uid={0}
>>>
>>> USE_DN
>>> 12 hours
>>> 
>>>
>>>
>>>
>>> And finally in authorizers.xml
>>>
>>> 
>>> file-user-group-provider
>>>
>>> org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
>>> ./conf/users.xml
>>> uid=firstuser,
>>> ou=users,dc=test,dc=ch
>>> 
>>>
>>>
>>>
>>> 
>>> file-access-policy-provider
>>>
>>> org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
>>> file-user-group-provider
>>> ./conf/authorizations.xml
>>>  uid=firstuser,
>>> ou=users,dc=test,dc=ch 
>>> 
>>>
>>> 
>>> 
>>>
>>>
>>>
>>>
>>>
>>> Starting Registry is OK.
>>>
>>>
>>>
>>> But when I want to access throw Chrome, I have a certificate error
>>> : ERR_BAD_SSL_CLIENT_AUTH_CERT
>>>
>>>
>>>
>>> How can I force the authentication to not request a client side
>>> certificate ?
>>>
>>>
>>>
>>> Thanks for any input.
>>>
>>>
>>>
>>> Etienne Jouvin
>>>
>>>
>>>
>>


Re: Need help SSL LDAP Nifi Registry

2020-06-30 Thread Etienne Jouvin
But now, I have NiFi and Registry with secure access (LDAP + SSL)

I need to find out how to configure the Registry in NiFi, because for now I
did not have to specify login.
And even if my first bucket is Public, it is not accessible from NiFi.


Le mar. 30 juin 2020 à 11:29, Etienne Jouvin  a
écrit :

> Hi Josef.
>
> No I did not try that.
> And well done, with that I can access the UI, and can connect with LDAP
> identity.
>
> Thanks a lot.
>
> Cheers
>
> Etienne
>
>
>
> Le mar. 30 juin 2020 à 11:15,  a écrit :
>
>> Hi Etienne
>>
>>
>>
>> Did you tried the following in «nifi-registry.properties»:
>>
>> nifi.registry.security.needClientAuth=false
>>
>>
>>
>> Cheers Josef
>>
>>
>>
>>
>>
>> *From: *Etienne Jouvin 
>> *Reply to: *"users@nifi.apache.org" 
>> *Date: *Tuesday, 30 June 2020 at 10:46
>> *To: *"users@nifi.apache.org" 
>> *Subject: *Need help SSL LDAP Nifi Registry
>>
>>
>>
>> Hello all.
>>
>>
>>
>> I am trying to setup LDAP authentication on NiFi Registry.
>>
>> I followed some links, like
>> https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753
>>
>>
>>
>> But each time, it requires that a certificate is installed on client
>> side. I had this "problem" for NiFi but because I did not provided
>> the nifi.security.user.login.identity.provider
>>
>>
>>
>> But for the registry, I remember that and did it.
>>
>>
>>
>> For summary, what I have in nifi-registry.properties
>>
>> nifi.registry.security.keystore=./conf/keystore.jks
>> nifi.registry.security.keystoreType=jks
>> nifi.registry.security.keystorePasswd=password
>> nifi.registry.security.keyPasswd=password
>> nifi.registry.security.truststore=./conf/truststore.jks
>> nifi.registry.security.truststoreType=jks
>> nifi.registry.security.truststorePasswd=password
>>
>>
>>
>> (All of those informations were given by the tls-toolkit, when executed
>> for NiFi)
>>
>> Then I put this
>>
>> #nifi.registry.security.identity.provider=
>> nifi.registry.security.identity.provider=ldap-identity-provider
>>
>>
>>
>> In the file identity-providers.xml
>>
>> I setup the LDAP provider
>>
>> 
>> ldap-identity-provider
>>
>> org.apache.nifi.registry.security.ldap.LdapIdentityProvider
>> SIMPLE
>>
>> uid=admin,ou=system
>> secret
>>
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> FOLLOW
>> 10 secs
>> 10 secs
>>
>> ldap://localhost:10389
>> ou=users,dc=test,dc=ch
>> uid={0}
>>
>> USE_DN
>> 12 hours
>> 
>>
>>
>>
>> And finally in authorizers.xml
>>
>> 
>> file-user-group-provider
>>
>> org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
>> ./conf/users.xml
>> uid=firstuser,
>> ou=users,dc=test,dc=ch
>> 
>>
>>
>>
>> 
>> file-access-policy-provider
>>
>> org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
>> file-user-group-provider
>> ./conf/authorizations.xml
>>  uid=firstuser,
>> ou=users,dc=test,dc=ch 
>> 
>>
>> 
>> 
>>
>>
>>
>>
>>
>> Starting Registry is OK.
>>
>>
>>
>> But when I want to access throw Chrome, I have a certificate error
>> : ERR_BAD_SSL_CLIENT_AUTH_CERT
>>
>>
>>
>> How can I force the authentication to not request a client side
>> certificate ?
>>
>>
>>
>> Thanks for any input.
>>
>>
>>
>> Etienne Jouvin
>>
>>
>>
>


Re: Need help SSL LDAP Nifi Registry

2020-06-30 Thread Etienne Jouvin
Hi Josef.

No I did not try that.
And well done, with that I can access the UI, and can connect with LDAP
identity.

Thanks a lot.

Cheers

Etienne



Le mar. 30 juin 2020 à 11:15,  a écrit :

> Hi Etienne
>
>
>
> Did you tried the following in «nifi-registry.properties»:
>
> nifi.registry.security.needClientAuth=false
>
>
>
> Cheers Josef
>
>
>
>
>
> *From: *Etienne Jouvin 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Tuesday, 30 June 2020 at 10:46
> *To: *"users@nifi.apache.org" 
> *Subject: *Need help SSL LDAP Nifi Registry
>
>
>
> Hello all.
>
>
>
> I am trying to setup LDAP authentication on NiFi Registry.
>
> I followed some links, like
> https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753
>
>
>
> But each time, it requires that a certificate is installed on client side.
> I had this "problem" for NiFi but because I did not provided
> the nifi.security.user.login.identity.provider
>
>
>
> But for the registry, I remember that and did it.
>
>
>
> For summary, what I have in nifi-registry.properties
>
> nifi.registry.security.keystore=./conf/keystore.jks
> nifi.registry.security.keystoreType=jks
> nifi.registry.security.keystorePasswd=password
> nifi.registry.security.keyPasswd=password
> nifi.registry.security.truststore=./conf/truststore.jks
> nifi.registry.security.truststoreType=jks
> nifi.registry.security.truststorePasswd=password
>
>
>
> (All of those informations were given by the tls-toolkit, when executed
> for NiFi)
>
> Then I put this
>
> #nifi.registry.security.identity.provider=
> nifi.registry.security.identity.provider=ldap-identity-provider
>
>
>
> In the file identity-providers.xml
>
> I setup the LDAP provider
>
> 
> ldap-identity-provider
>
> org.apache.nifi.registry.security.ldap.LdapIdentityProvider
> SIMPLE
>
> uid=admin,ou=system
> secret
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> FOLLOW
> 10 secs
> 10 secs
>
> ldap://localhost:10389
> ou=users,dc=test,dc=ch
> uid={0}
>
> USE_DN
> 12 hours
> 
>
>
>
> And finally in authorizers.xml
>
> 
> file-user-group-provider
>
> org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
> ./conf/users.xml
> uid=firstuser,
> ou=users,dc=test,dc=ch
> 
>
>
>
> 
> file-access-policy-provider
>
> org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
> file-user-group-provider
> ./conf/authorizations.xml
>  uid=firstuser,
> ou=users,dc=test,dc=ch 
> 
>
> 
> 
>
>
>
>
>
> Starting Registry is OK.
>
>
>
> But when I want to access throw Chrome, I have a certificate error
> : ERR_BAD_SSL_CLIENT_AUTH_CERT
>
>
>
> How can I force the authentication to not request a client side
> certificate ?
>
>
>
> Thanks for any input.
>
>
>
> Etienne Jouvin
>
>
>


Re: Need help SSL LDAP Nifi Registry

2020-06-30 Thread Josef.Zahner1
Hi Etienne

Did you tried the following in «nifi-registry.properties»:
nifi.registry.security.needClientAuth=false

Cheers Josef


From: Etienne Jouvin 
Reply to: "users@nifi.apache.org" 
Date: Tuesday, 30 June 2020 at 10:46
To: "users@nifi.apache.org" 
Subject: Need help SSL LDAP Nifi Registry

Hello all.

I am trying to setup LDAP authentication on NiFi Registry.
I followed some links, like 
https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753

But each time, it requires that a certificate is installed on client side. I 
had this "problem" for NiFi but because I did not provided the 
nifi.security.user.login.identity.provider

But for the registry, I remember that and did it.

For summary, what I have in nifi-registry.properties
nifi.registry.security.keystore=./conf/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=password
nifi.registry.security.keyPasswd=password
nifi.registry.security.truststore=./conf/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=password

(All of those informations were given by the tls-toolkit, when executed for 
NiFi)
Then I put this
#nifi.registry.security.identity.provider=
nifi.registry.security.identity.provider=ldap-identity-provider

In the file identity-providers.xml
I setup the LDAP provider

ldap-identity-provider

org.apache.nifi.registry.security.ldap.LdapIdentityProvider
SIMPLE

uid=admin,ou=system
secret











FOLLOW
10 secs
10 secs

ldap://localhost:10389
ou=users,dc=test,dc=ch
uid={0}

USE_DN
12 hours


And finally in authorizers.xml

file-user-group-provider

org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
./conf/users.xml
uid=firstuser, 
ou=users,dc=test,dc=ch



file-access-policy-provider

org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
file-user-group-provider
./conf/authorizations.xml
 uid=firstuser, 
ou=users,dc=test,dc=ch 






Starting Registry is OK.

But when I want to access throw Chrome, I have a certificate error : 
ERR_BAD_SSL_CLIENT_AUTH_CERT

How can I force the authentication to not request a client side certificate ?

Thanks for any input.

Etienne Jouvin



smime.p7s
Description: S/MIME Cryptographic Signature


Need help SSL LDAP Nifi Registry

2020-06-30 Thread Etienne Jouvin
Hello all.

I am trying to setup LDAP authentication on NiFi Registry.
I followed some links, like
https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753

But each time, it requires that a certificate is installed on client side.
I had this "problem" for NiFi but because I did not provided
the nifi.security.user.login.identity.provider

But for the registry, I remember that and did it.

For summary, what I have in nifi-registry.properties
nifi.registry.security.keystore=./conf/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=password
nifi.registry.security.keyPasswd=password
nifi.registry.security.truststore=./conf/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=password

(All of those informations were given by the tls-toolkit, when executed for
NiFi)
Then I put this
#nifi.registry.security.identity.provider=
nifi.registry.security.identity.provider=ldap-identity-provider

In the file identity-providers.xml
I setup the LDAP provider

ldap-identity-provider

org.apache.nifi.registry.security.ldap.LdapIdentityProvider
SIMPLE

uid=admin,ou=system
secret











FOLLOW
10 secs
10 secs

ldap://localhost:10389
ou=users,dc=test,dc=ch
uid={0}

USE_DN
12 hours


And finally in authorizers.xml

file-user-group-provider

org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
./conf/users.xml
uid=firstuser,
ou=users,dc=test,dc=ch



file-access-policy-provider

org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
file-user-group-provider
./conf/authorizations.xml
 uid=firstuser,
ou=users,dc=test,dc=ch 






Starting Registry is OK.

But when I want to access throw Chrome, I have a certificate error
: ERR_BAD_SSL_CLIENT_AUTH_CERT

How can I force the authentication to not request a client side certificate
?

Thanks for any input.

Etienne Jouvin