RE: Exception on Processor ConvertJSONToSQL

2016-08-15 Thread Peter Wicks (pwicks)
Carlos, I ran into this same error when querying Teradata. It looks like a lot of databases don't include this. I submitted a bug a couple weeks ago: https://issues.apache.org/jira/browse/NIFI-2356 I did something similar to your suggestion locally in a modified version of the code. Regards,

RE: PutSQL ERROR bulletin

2016-08-15 Thread Peter Wicks (pwicks)
Sven, I've run into this a couple times. In my case some records would insert and some would not. To find my issue: - I routed all failures back to PutSQL - Reduced the batch size down to about 10 - Changed the prioritization on the failure relationship so that hopefully failures will

RE: NiFi 1.1.0 stuck starting, no errors

2017-01-27 Thread Peter Wicks (pwicks)
ABG> Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 26, 2017, at 12:42 PM, Peter Wicks (pwicks) > <pwi...@micron.com<ma

NiFi 1.1.0 stuck starting, no errors

2017-01-26 Thread Peter Wicks (pwicks)
I'm looking for help in troubleshooting my NiFi 1.1.0 install. It's been running stably for some time, but I restarted it this morning when I deployed an updated custom NAR. Now it gets stuck at startup, see logs at the end. There are no error messages, and the processes don't die. The process

RE: adding dependencies like jdbc drivers to the build

2016-08-19 Thread Peter Wicks (pwicks)
of the same template or flow in different environments easier. On Aug 19, 2016 7:27 AM, "Peter Wicks (pwicks)" <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: Sumanth, If the driver is in your lib directory already then you should leave the path empty. All jar’s

v0.* QueryDatabaseTable vs v1 GenerateTableFetch

2016-08-15 Thread Peter Wicks (pwicks)
What is the future of QueryDatabaseTable? Unless I'm misunderstanding how it works it looks like GenerateTableFetch can do everything QueryDatabaseTable can do and then some. Is there a plan to phase out QueryDatabaseTable? Is there a reason for a new processor instead of an update to

RE: v0.* QueryDatabaseTable vs v1 GenerateTableFetch

2016-08-15 Thread Peter Wicks (pwicks)
Oh, disregard :). I misread GenerateTableFetch as being an actual data fetch vs a query builder. From: Peter Wicks (pwicks) Sent: Monday, August 15, 2016 9:11 PM To: 'users@nifi.apache.org' <users@nifi.apache.org> Subject: v0.* QueryDatabaseTable vs v1 GenerateTableFetch What is the

RE: I need help configuring Site-to-Site in Secure Mode.

2016-09-02 Thread Peter Wicks (pwicks)
you created/obtained support being used for both client and server auth. If they were created for server auth only, this could explain your issue. NiFi instances need to act as a client and a server at times. Thanks, Matt On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <

RE: I need help configuring Site-to-Site in Secure Mode.

2016-09-02 Thread Peter Wicks (pwicks)
erver=y,suspend=n,address=8000 -Bryan On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: Matt, That was the case on our first go round, when we only had SSL certs. We went back yesterday and got new certificates that support bot

RE: Kill-and-Fill Pattern?

2016-08-29 Thread Peter Wicks (pwicks)
at once. thanks toivo 2016-08-26 16:45 GMT+03:00 Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>>: I have a source SQL table that I’m reading with a SQL select statement. I want to kill and fill a destination SQL table with this source data on an interval. My non k

RE: I need help configuring Site-to-Site in Secure Mode.

2016-09-01 Thread Peter Wicks (pwicks)
Bryan, Paul and I have been working on this, and I think our issue is related to certificates. In your blog posting you used TLS-Toolkit in your example, but I think that is unrealistic for many environments. For example, this also creates the certificates for SSL right? But these will be

Processor scheduling resets on Nifi Restart

2016-08-31 Thread Peter Wicks (pwicks)
We've noticed that a job we have setup to run once a day will run again in the same day if we restart NiFi (and again, and again, depending on how often we restart NiFi). Is this by design?

RE: Erroneous Queue has No FlowFiles message

2016-09-09 Thread Peter Wicks (pwicks)
: Erroneous Queue has No FlowFiles message Peter, Thanks for the details! These will be very helpful investigating what's happening here. A couple follow-up questions... - Is this a cluster? - Is this instance secured? Thanks Matt On Fri, Sep 9, 2016 at 12:13 AM, Peter Wicks (pwicks) <

RE: Erroneous Queue has No FlowFiles message

2016-09-09 Thread Peter Wicks (pwicks)
narrow down the possible cause. Are you still able to replicate the issue? If so, can you enable debug level logging for org.apache.nifi.controller.StandardFlowFileQueue and see if there are any meaningful messages in the nifi-app.log? Thanks! Matt On Fri, Sep 9, 2016 at 9:52 AM, Peter Wicks (p

Interesting Site-to-Site quirk with nifi.security.identity.mapping.pattern.dn

2016-09-11 Thread Peter Wicks (pwicks)
I've been playing with site-to-site and found an interesting quirk. I had the full DN's from my certificates for my usernames, but decided to setup nifi.security.identity.mapping patterns for both the DN's and for Kerberos; which by the way works great for normal users. I renamed just my own

RE: Nifi 1.0.0 compatibility with Hive 1.1.0

2016-09-08 Thread Peter Wicks (pwicks)
Also, ORC File support was pulled out into its own library on the HIVE side. If you are willing to compile and run your own version you might need to include orc-core as a MVN dependency: https://mvnrepository.com/artifact/org.apache.orc/orc-core/1.2.0. From: Andre

RE: Erroneous Queue has No FlowFiles message

2016-09-08 Thread Peter Wicks (pwicks)
is sent. Once you delete the active request, you get the message you mentioned in the logs, this is not an issue. If you check the developer panel in chrome, you will see all 3 operations, post-get-delete in succession. On Fri, Sep 9, 2016, 8:48 AM Peter Wicks (pwicks) <pwi...@micron.com<mail

Erroneous Queue has No FlowFiles message

2016-09-08 Thread Peter Wicks (pwicks)
Running NiFI 1.0.0, I'm listing a queue that has 100k files queued. I've stopped both the incoming and outgoing processors, so the files are just hanging out in the queue, no possible motion. I get, "The queue has no FlowFiles" message. Here are the actual responses from the REST calls: POST

RE: Erroneous Queue has No FlowFiles message

2016-09-09 Thread Peter Wicks (pwicks)
] ] But the List FlowFiles command only looks at the Active queue… That looks like the root cause, what I don’t know is if this is by design. --Peter From: Peter Wicks (pwicks) Sent: Friday, September 09, 2016 3:28 PM To: 'users@nifi.apache.org' <users@nifi.apache.org> Subject: RE: Erroneous Que

RE: Erroneous Queue has No FlowFiles message

2016-09-09 Thread Peter Wicks (pwicks)
is following the PutSQL? Had any of those connections exceeded their configured back pressure threshold? Thanks again. Matt On Fri, Sep 9, 2016 at 11:18 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: PutSQL. The 100k FlowFiles are all SQL In

RE: Erroneous Queue has No FlowFiles message

2016-09-09 Thread Peter Wicks (pwicks)
-fd988c59e290] Finished listing FlowFiles for active queue with a total of 0 results 2016-09-09 21:27:29,656 INFO [NiFi Web Server-112] o.a.n.controller.StandardFlowFileQueue Canceling ListFlowFile Request with ID 10d92339-0157-1000-42f4-464c37340fdb Thanks, Peter From: Peter Wicks (pwicks) Sent

RE: Erroneous Queue has No FlowFiles message

2016-09-09 Thread Peter Wicks (pwicks)
. Haven't identified the case that's causing it yet but definitely have a better idea what's going on now. Thanks Matt Sent from my iPhone On Sep 9, 2016, at 5:42 PM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: Matt, I followed the swapping trai

RE: Download item from queue - what permission is required?

2016-09-20 Thread Peter Wicks (pwicks)
how to setup properly) or to fix in the next release. Have you already raised a JIRA? On Wed, Sep 21, 2016 at 12:30 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: No help here, except to share that I’ve also seen this error. I’ve been working

RE: Does NiFi support multiple queries

2016-09-21 Thread Peter Wicks (pwicks)
Karthik, PutSQL will handle both styles, and for multiple tables, without issue. Internally it creates a separate SQL Batch for each distinct SQL statement in the queue and then executes these batches separately. Feel free to mix as many Inserts/Updates as you wish for as many tables as you

PutHiveQL Multiple Ordered Statements

2016-09-23 Thread Peter Wicks (pwicks)
I have a PutHDFS processor drop a file, I then have a long chain of ReplaceText -> PutHiveQL processors that runs a series of steps. The below ~4 steps allow me to take the file generated by NiFi in one format and move it into the final table, which is ORC with several Timestamp columns (thus

RE: Nifi UI configured to be accessed over HTTPS not displayed in Internet Explorer

2016-09-22 Thread Peter Wicks (pwicks)
Nicolas, According to the NiFi users guide, you need to be running on Internet Explorer Edge browser to get NiFi working in iE. If you are running IE10 or IE11 it just isn't going to work because of the lack of

RE: Requesting Obscene FlowFile Batch Sizes

2016-09-20 Thread Peter Wicks (pwicks)
rks with a background thread, and I believe it still works this way: http://apache-nifi.1125220.n5.nabble.com/Nifi-amp-Spark-receiver-performance-configuration-td524.html -Bryan On Tue, Sep 20, 2016 at 10:22 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrot

RE: Download item from queue - what permission is required?

2016-09-20 Thread Peter Wicks (pwicks)
No help here, except to share that I’ve also seen this error. I’ve been working around it by downloading the FlowFile instead of viewing it. From: Andre [mailto:andre-li...@fucs.org] Sent: Monday, September 19, 2016 11:18 PM To: users@nifi.apache.org Subject: Download item from queue - what

Kill-and-Fill Pattern?

2016-08-26 Thread Peter Wicks (pwicks)
I have a source SQL table that I'm reading with a SQL select statement. I want to kill and fill a destination SQL table with this source data on an interval. My non kill-and-fill pattern is: ExecuteSQL -> Avro To JSON -> JSON To SQL -> PutSQL. I'm trying to come up with a good way to delete

Can't export Templates from NiFi 1.0.0 BETA

2016-08-24 Thread Peter Wicks (pwicks)
I've created a template in NiFi 1.0.0 BETA, but am unable to export it. Security is enabled, Kerberos. I get the following error: Unable to perform the desired action due to insufficient permissions. Contact the system administrator. I've tried giving myself and a group containing me both

RE: Access denied for kerberos users

2016-09-26 Thread Peter Wicks (pwicks)
Nicolas, If Bryan’s suggestion doesn’t work (and he’s probably correct), you may not have named your user correctly in NiFi. Go try to authenticate again, then go to {nifi install directory}/logs and look at the end of nif-user.log. You should see more details about your authentication

Enable Compression on Remote Port?

2016-11-10 Thread Peter Wicks (pwicks)
When I have a Remote Process Group and I view its Remote Ports I can see that all my ports show "Compressed" as No. How can I change this so that the ports use compression?

RE: How to increase the processing speed of the ExtractText and ReplaceText Processor?

2016-10-24 Thread Peter Wicks (pwicks)
Prabhu, Lee mentioned making sure you have good indexes, but I would caution you on this point. If you have a unique constraint then SQL Server will build an index on this automatically, but I would suggest dropping all other indexes that aren’t related to data integrity. Each time SQL Server

RE: Enable Compression on Remote Port?

2016-11-11 Thread Peter Wicks (pwicks)
release candidate should be created soon. We are in the process of closing down the remaining JIRAs assigned to the next release. Matt [1] https://issues.apache.org/jira/browse/NIFI-2687 On Fri, Nov 11, 2016 at 9:37 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com&g

Failing to Start NiFi 1.1.0, OverlappingFileLockException

2016-12-20 Thread Peter Wicks (pwicks)
I've successfully upgraded my DEV and TEST environments from NiFi 1.0.0 to NiFi 1.1.0. So I felt comfortable upgrading PROD until... I completed all the work and went to start the server, but am receiving the below stack trace. I dug through the code a bit and found that it's locking a file

RE: Failing to Start NiFi 1.1.0, OverlappingFileLockException

2016-12-21 Thread Peter Wicks (pwicks)
is running on the node? If possible would recommend running "ps -ef | grep nifi" as root (assuming that you're running on Linux). This would ensure that no other user has started the nifi process. Thanks -Mark On Dec 20, 2016, at 4:05 PM, Peter Wicks (pwicks) <pwi...@micron.c

RE: Visual Indicator for "Can't run because there are no threads"?

2017-03-06 Thread Peter Wicks (pwicks)
ather than just backpressure. Thanks Joe On Fri, Mar 3, 2017 at 4:57 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > I think everyone was really happy when backpressure finally got super > great indicators. Backpressure used to be my #1, “Why isn’t stuff moving?” >

RE: [EXT] NiFi 1.3: Simplest way possible of creating CSV files from SQL queries

2017-08-01 Thread Peter Wicks (pwicks)
I hate to respond with “me too”, but I haven’t seen a response and this kind of simplification is of interest to me. The PutDatabaseRecord processor already does something similar, and I have only needed the AvroReader processor without a schema registry. From: Márcio Faria

RE: [EXT] High-performance Nifi decoder

2017-08-23 Thread Peter Wicks (pwicks)
Hi Ali, How many FlowFile’s are you expecting per minute? My experience with NiFi has been that if you have 1 FlowFile that is several GB’s in size you can frequently process it faster than 100k FlowFile’s at a fraction the size. There is a lot of overhead in managing the life cycle of a

RE: [EXT] Parsing Email Attachments

2017-05-17 Thread Peter Wicks (pwicks)
Nick, Try escaping your \n’s, see if that helps. (?s)(.*\\n\\n${boundary}\\nContent-Type: text\/plain; charset="UTF-8"\\n\\n)(.*?)(\\n\\n${boundary}.*) From: Nick Carenza [mailto:nick.care...@thecontrolgroup.com] Sent: Thursday, May 18, 2017 11:27 AM To: users@nifi.apache.org Subject: [EXT]

RE: [EXT] New to Nifi - Failed to update database due to a failed batch update

2017-09-24 Thread Peter Wicks (pwicks)
Hi Aruna, Since you are using ReplaceText, you can view the contents of the FlowFile and check that you can copy/paste the SQL and execute it by hand in Postgres. If all that works try setting the batch size on PutSQL to 1 record. This will help check if it's all records that are having

RE: [EXT] New to Nifi - Failed to update database due to a failed batch update

2017-09-25 Thread Peter Wicks (pwicks)
update Hi, thank you for getting back. Could you please let me know how I can see the contents of the flow file ? The view option doesn't seem to work for me. Please see my last screenshot in my first email. Thanks Aruna On Sep 24, 2017, at 8:52 PM, Peter Wicks (pwicks) <pwi...@micron.

java.lang.OutOfMemoryError: unable to create new native thread

2017-11-14 Thread Peter Wicks (pwicks)
I've been getting the error: 2017-11-15 09:22:24,959 ERROR [NiFi Web Server-566674] org.apache.nifi.NiFi java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at

RE: [EXT] Re: java.lang.OutOfMemoryError: unable to create new native thread

2017-11-14 Thread Peter Wicks (pwicks)
it will be around some tcp socket handling thread in something like sftp for example. Can be a config issue or a bug. Thanks Joe On Tue, Nov 14, 2017 at 8:32 PM Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: I’ve been getting the error: 2017-11-15 09:22:24,959

Wait only if flagged?

2017-11-13 Thread Peter Wicks (pwicks)
I have a database flow, which is a sequence of 4 processors. For database performance reasons I need to make sure only one file is in this section of the flow at a time. Not just one file per queue/processor, but for the whole section. I feel like I should be able to use Wait/Notify to do

RE: [EXT] Re: Calling NIFI job from any Enterprise Scheduler

2017-11-13 Thread Peter Wicks (pwicks)
A lot of enterprise schedulers have an option to post to an HTTP endpoint. In the past I’ve used NiFi’s ListenHTTP processor to allow for remote triggering of a flow. But really any of the Listen/Consume processors could potentially be used for remote triggering: * ListenHTTP or

RE: [EXT] Re: Wait only if flagged?

2017-11-13 Thread Peter Wicks (pwicks)
;success" relationship which is routed back to wait. Regards, Matt On Mon, Nov 13, 2017 at 7:21 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > I have a database flow, which is a sequence of 4 processors. For database > performance reasons I need to make sure only one file

RE: [EXT] Re: Metrics on Lineage Duration

2017-11-07 Thread Peter Wicks (pwicks)
ather than getting an average. Does this help at all? Thanks -Mark On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a point in time so I can

RE: [EXT] Re: Metrics on Lineage Duration

2017-11-07 Thread Peter Wicks (pwicks)
2017-11-07 12:55 GMT+01:00 Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>>: Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a point in time so I can monitor my flow using this metric? Thanks, Peter

RE: [EXT] Re: Polling Processors impact on Latency

2017-11-07 Thread Peter Wicks (pwicks)
If you schedule the processor to run every 0 sec (the default) then in my experience you won’t notice latency from polling at all. But I guess this depends on your expectations, volume, and over all Flow processing time. Yes, event driven may help, but from what I’ve read it’s more about

Metrics on Lineage Duration

2017-11-07 Thread Peter Wicks (pwicks)
Is there a reporting task/processor that will let me aggregate Lineage Duration's for a point in time so I can monitor my flow using this metric? Thanks, Peter

RE: [EXT] CDC like updates on Nifi

2017-12-06 Thread Peter Wicks (pwicks)
To: users@nifi.apache.org Subject: Re: [EXT] CDC like updates on Nifi On Tue, Dec 5, 2017 at 11:55 PM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: Alberto, Hello Peter, Thanks for your answer. Since it sounds like you have control over the structure o

RE: [EXT] CDC like updates on Nifi

2017-12-05 Thread Peter Wicks (pwicks)
Alberto, Since it sounds like you have control over the structure of the tables, this should be doable. If you have a changelog table for each table this will probably be easier, and in your changelog table you’ll need to make sure you have a good transaction timestamp column and a change

Reading Email Message Body

2017-10-30 Thread Peter Wicks (pwicks)
A coworker and I were troubleshooting a bug in the ConsumeEWS processor where Unicode characters were being read as ASCII. I figured out there was a bug in my code for ConsumeEWS and plan to fix it, but as part of the research I found that the way Unicode text in the email is outputted to the

RE: [EXT] Issue with ResizeImage processor for PNG files

2018-06-29 Thread Peter Wicks (pwicks)
Raman, I created a ticket for this issue: https://issues.apache.org/jira/browse/NIFI-5355. I read through the links you provided, thanks for researching the problem so thoroughly. Found some others as well, such as

Hive w/ Kerberos Authentication starts failing after a week

2018-07-26 Thread Peter Wicks (pwicks)
We are seeing frequent failures of our Hive DBCP connections after a week of use when using Kerberos with Principal/Keytab. We've tried with both the Credential Service and without (though in looking at the code, there should be no difference). It looks like the tickets are expiring and

RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-27 Thread Peter Wicks (pwicks)
starts failing after a week Did you try executing 'klist' to see if the tickets are there and renewed? If expired, try manual kinit and see if that fixes. On Fri, Jul 27, 2018 at 1:51 AM Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: We are seeing frequent failures of our Hiv

RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after a week

2018-07-27 Thread Peter Wicks (pwicks)
the Hortonworks distribution it's fixed in the latest HDF 3.x release I think. Thanks Shawn From: Peter Wicks (pwicks) Sent: Friday, July 27, 2018 10:58 AM To: users@nifi.apache.org Subject: RE: [EXT] Re: Hive w/ Kerberos Authentication starts failing after

After 1.7.1 upgrade, no Provenance data is visible

2018-08-10 Thread Peter Wicks (pwicks)
After upgrading our NiFi instances to 1.7.1 we are not able to see Provenance data anymore in the UI. We see this across about a dozen instances. In the UI it tells me provenance is available for about the last 24 hours, and I can see that files have moved in and out of the processor in the last

RE: [EXT] Re: After 1.7.1 upgrade, no Provenance data is visible

2018-08-13 Thread Peter Wicks (pwicks)
. Check out the Migration Guide [1] for 1.7.0. It talks about what you'll need to do. [1] - https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance -- Mike On Fri, Aug 10, 2018 at 5:39 PM Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: After upgrading our NiFi ins

How many threads does Jetty use?

2018-07-18 Thread Peter Wicks (pwicks)
I know the default thread count for Jetty is 200, but is there a way to tell how many are actually being used and if I need to make adjustments? nifi.web.jetty.threads=200 Thanks, Peter

RE: [EXT] Adding a file to a zip file

2018-07-11 Thread Peter Wicks (pwicks)
Hi Kiran, In your flow, how do you avoid duplicate files going into MergeContent? For example: 1. file1.zip goes into Unpack zip file, it contains 5 files. 2. These 5 files are sent down both success paths (AttributeToJSON and increment fragment index and count) 3. 5 files show up at

Cluster Peer Lists

2018-09-27 Thread Peter Wicks (pwicks)
Hi NiFi team, We had one of the nodes in our cluster go offline today. We eventually resolved the issue, but it exposed some issues in our configuration across our edge NiFi instances. Right now we have non-clustered instances of NiFi distributed around the world, pushing data back to a three

Delay a FlowFile for a specific amount of time

2018-10-15 Thread Peter Wicks (pwicks)
A coworker and I were working on a problem where we needed to delay a group of FlowFile's for 60 minutes. Our first attempt of course used ControlRate, but with ControlRate the first file is let through immediately, and only after that are the rest of the files delayed. We got it working by

RE: [EXT] Re: Delay a FlowFile for a specific amount of time

2018-10-15 Thread Peter Wicks (pwicks)
with an expiration of 60 mins? If you never use a Notify processor then its basically just going to wait til the expiration. On Mon, Oct 15, 2018 at 4:10 PM Peter Wicks (pwicks) wrote: > > A coworker and I were working on a problem where we needed to delay a group > of FlowFile’s for 60 minutes. Our firs

RE: [EXT] ReplaceText cannot consume messages if Regex does not match

2018-10-18 Thread Peter Wicks (pwicks)
Hi Juan, What version of NiFi are you running on? What mode are you running ReplaceText in, all text or line by line? Other settings that might be important? What’s your RegEx look like (if your able to share). --Peter From: Juan Pablo Gardella [mailto:gardellajuanpa...@gmail.com] Sent:

Reverting position only versioned changes

2018-10-22 Thread Peter Wicks (pwicks)
In NiFi 1.7.1, I tried to revert some changes to a versioned processor group where the only changes were Processor position. The processor that was moved was running for a very long time, and the revert would not complete until it stopped the affected processor, even though a stop is not

Maximum Memory for NiFi?

2018-10-04 Thread Peter Wicks (pwicks)
We've had some more clustering issues, and found that some nodes are running out of memory when we have unexpected spikes in data, then we run into a GC stop-the-world event... We lowered our thread count, and that has allowed the cluster to stabilize for the time being. Our hardware is pretty

RE: [EXT] Re: Maximum Memory for NiFi?

2018-10-08 Thread Peter Wicks (pwicks)
you are using are not written in an optimal way to process the content in a streaming fashion. Did you initially start out lower than 70GB and head to increase it to that point? Just wondering what happens at lower levels like maybe 32GB. On Thu, Oct 4, 2018 at 4:20 PM Peter Wicks (pwicks

RE: [EXT] Re: Cluster Peer Lists

2018-09-28 Thread Peter Wicks (pwicks)
Alternatively, round robin DNS name or Reverse Proxy for the bootstrap node address can be used similarly. Thanks, Koji On Fri, Sep 28, 2018 at 4:30 AM Peter Wicks (pwicks) wrote: > > Hi NiFi team, > > > > We had one of the nodes in our cluster go offline today. We eventually &g

Problems with NiFi Registry Conflicts after Processor Upgrades

2018-11-29 Thread Peter Wicks (pwicks)
Ran into a NiFi Registry issue while upgrading our instances to NiFi 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so after upgrading our, our versioned processor groups show as having local changes, which is good. We went ahead and checked the changes into the

Cluster Response Times Intermittently Slow, CPU Spikes

2019-01-09 Thread Peter Wicks (pwicks)
For a few weeks, one of our clusters has been having large spikes in CPU across most or all nodes, at the same time. Beginning at the same time, our metrics collection job has intermittently started to see long response times when calling the following api end-points:

RE: [EXT] Telnet login and data capture

2019-04-23 Thread Peter Wicks (pwicks)
You should look at the ExecuteStreamCommand processor. Write up a script that does what you want in Bash or Python, pass any variables you need either through command line arguments or STIDIN (the contents of the FlowFile is passed as STDIN, I’ve written up whole Python scripts using

RE: [EXT] Re: FlowFile Repository can't checkpoint, out of heap space.

2019-08-15 Thread Peter Wicks (pwicks)
%7C0%7C0%7C637014811949909503=XDKoAcehZmYFqJ6JLVkVyhx0gsRGNyw7vDNfURhsn4E%3D=0> On Wed, Aug 14, 2019 at 12:12 PM Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: I have a node in a cluster whose FlowFile repository grew so fast that it exceeded the amount of available heap space an

RE: [EXT] Re: FlowFile Repository can't checkpoint, out of heap space.

2019-08-15 Thread Peter Wicks (pwicks)
tself should be evaluated/considered/determined. A full JIRA/description of the situation/logs/known state would be worthy of further resolution. Thanks On Thu, Aug 15, 2019 at 12:50 PM Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: We were able to recover this morning, in the end we deleted the

FlowFile Repository can't checkpoint, out of heap space.

2019-08-14 Thread Peter Wicks (pwicks)
I have a node in a cluster whose FlowFile repository grew so fast that it exceeded the amount of available heap space and now can't checkpoint. Or that is my interpretation of the error. "Cannot update journal file flowfile_repository/journals/.journal because this journal has already

RE: [EXT] Specifying formatters at a record field level

2019-08-14 Thread Peter Wicks (pwicks)
Not that I’m aware of. We implemented something custom that lets you specify it with attributes on the FlowFile (something like data.field.#.format=….), we do the same thing for binary/hex fields. But we didn’t contribute it as it’s part of a custom record processing processor that’s

RE: [EXT] Re: How to replace multi character delimiter with ASCII 001

2019-11-06 Thread Peter Wicks (pwicks)
Shawn, We had the same issue, and use special and multi character delimiters here. I have not been able to find a CSV library that supports multi-character delimiters, otherwise I would have updated the CSV Record Reader to support it. I created a special Record Reader that supports

Re: [EXT] Re: sslcontext certs

2020-10-14 Thread Peter Wicks (pwicks)
ntext service refers to the updated certificates and no NiFi configuration changed. Does this work for you? Nathan On Wed, Oct 14, 2020 at 3:29 PM Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: Micron Confidential I've found this annoying in the past as well. I would not be

Re: [EXT] Re: sslcontext certs

2020-10-14 Thread Peter Wicks (pwicks)
Micron Confidential I've found this annoying in the past as well. I would not be opposed to an additional implementation of the SSLContext that uses the NiFi certs by default, though... if it uses the client certificate as well you'd have to make it restricted, so as to prevent users from

Re: [EXT] AW: ExecuteSQL and Teradata

2021-01-12 Thread Peter Wicks (pwicks)
Micron Confidential Can you try setting `FINALIZE_AUTO_CLOSE=ON` in your connection string to Teradata? It’s not a best practice, but based on what the docs say, I think it might work. >From the docs: FINALIZE_AUTO_CLOSE values are OFF (default) or ON: · When set to OFF (the default), the