Re: [EXT] [ANNOUNCE] New Apache NiFi Committer Otto Fowler

Micron Confidential Congratulations Otto! From: Joe Witt Date: Monday, March 22, 2021 at 12:16 PM To: dev@nifi.apache.org Subject: [EXT] [ANNOUNCE] New Apache NiFi Committer Otto Fowler CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless you recognize the sender and were

Re: [EXT] Using github to merge PRs and just generally easier tracking of commits..

Should mergers be putting the “signed-off by: …” in when they Merge PR’s through Github? I haven’t merged anything in a while, but I was putting this in the comment box when I merged, and it showed up on the commit the same way it did when I did it through the CLI:

RE: [EXT] Re: Teradata and Nifi

Micron Confidential Harisa, We did this in the past (sorry, not on Teradata anymore). To streamline the process we ended up building a custom processor. Initially we used FASTLOAD CSV, and that worked OK, but it was hard to handle the error cases, since FASTLOAD errors don't work the same

RE: [EXT] Re: NiFi Standard Libraries Info

I know the NiFi 1.11 release process is already going on, but I thought the community decided back in September to split most of the libraries out into the new `NiFi Standard Libraries` Git repo/project because NiFi as a binary release to Apache Maven was getting too big? Was there further

RE: [EXT] Re: [DISCUSS] Time based release cycles

I feel like most users ask, "When is version x coming out" because they don't want to/or can't do a build themselves and they really want to use new features. I know it's a completely different direction from where I think your question was pointing Pierre, but I wonder how many users would be

RE: [EXT] [ANNOUNCE] New Apache NiFi Committer Kotaro Terada

おめでとう寺田さん！ -Original Message- From: Aldrin Piri Sent: Thursday, October 24, 2019 7:50 AM To: dev Subject: [EXT] [ANNOUNCE] New Apache NiFi Committer Kotaro Terada Apache NiFi community, On behalf of the Apache NiFI PMC, I am very pleased to announce that Kotaro has accepted the PMC's

Project access to Jira NiFi

When I was added to the PMC, it looks like my Jira permissions were not updated to reflect the change. Can someone on the PMC update my permissions in JIRA? Thanks, Peter

How does http://nifi.apache.org update?

How are updates pushed to http://nifi.apache.org? Are these done automatically by some Apache Infra job, or is there a developer out there pushing changes? --Peter

RE: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project

+1, binding -Original Message- From: Kevin Doran Sent: Tuesday, September 3, 2019 7:12 PM To: dev@nifi.apache.org; dev@nifi.apache.org Subject: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project +1, binding From: Tony Kurc Sent: Tuesday,

RE: [EXT] Re: OnPrimaryNodeStateChange vs Primary Only configuration

:07 AM Peter Wicks (pwicks) wrote: > > I'm working on a bug fix for HandleHttpRequest and need to check if a > processor is configured to run only on primary node (and not if a processor > has the attribute that ONLY allows it to run on primary node). > Here is the scenario

OnPrimaryNodeStateChange vs Primary Only configuration

I'm working on a bug fix for HandleHttpRequest and need to check if a processor is configured to run only on primary node (and not if a processor has the attribute that ONLY allows it to run on primary node). Here is the scenario for background: * NiFi cluster, but all nodes are on the same

RE: [EXT] Airflow and NiFi

User: "I want to change my production flow while it's running, you know, mid stream just route my content to a completely different flow, re-run data mid run through a new set of processors, fork it on the fly, you know, whatever I want anytime I want " NiFi: "Yeah, I can do that" Everyone

Should there be a sane limit on Strings in NiFI?

I noticed yesterday that automatic ellipsis was not working for relationship names in the Connection Creation window (PR submitted in NIFI-6512). As part of my test I started playing around with submitting fairly long names for relationships in RouteOnAttribute. This all worked with out issue

RE: [EXT] Re: Duplicate flow files *without* their content

Lars, If you are worried about it, using ReplaceText will have the same effect as your custom solution. When ReplaceText has it's `Replacement Strategy` set to `Always Replace` it doesn't read the contents of the FlowFile and simply writes out the replacement Value, which in your case could be

RE: [EXT] Re: Adding Color to Connections

d connection is > coloured in red). > > Pierre > > Le mar. 30 juil. 2019 à 18:19, Peter Wicks (pwicks) > a écrit : > > > You know when you have a crazy complex flow, and it's hard sometimes > > to even tell where things are going? Especially those failure > > co

Adding Color to Connections

You know when you have a crazy complex flow, and it's hard sometimes to even tell where things are going? Especially those failure conditions that are all going back to a central Funnel or Port? I thought it would be visually very helpful if you could add Color to your connections, using the

RE: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

flowfiles - beginning with the oldest first - until the back pressure limit is no longer in violation. Thanks, Mark On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) wrote: > I was thinking it would be nice to expire FlowFile's based on their > time in queue, in addition to th

RE: [EXT] Re: Auto cleanup custom Controller Service properties?

Controller Service properties? I believe you should still be able to delete the property with a delete icon next to it. This should be the same behavior for processors when a property is removed, regardless of dynamic properties. On Fri, Jul 19, 2019 at 11:01 AM Peter Wicks (pwicks) wrote: >

RE: [EXT] Re: Weird spring issue if I run a snapshot w/out internet access

As a final follow-up, this has been resolved with the fix merged to master and the Jira closed. --Peter -Original Message- From: Joe Witt Sent: Thursday, July 18, 2019 9:47 AM To: dev@nifi.apache.org Subject: [EXT] Re: Weird spring issue if I run a snapshot w/out internet access See

UI Bug driving me crazy (NIFI-6455)

I ran across a UI bug, but haven’t been able to figure it out on my own so I wrote up a ticket, NIFI-6455. The bug causes the last item on a scrollable Configure window to not be accessible. The scrollbar is the right size, but the last property is inaccessible. Showed up in 1.10 for both

RE: [EXT] Re: Thoughts on an internal "Terminate" handler for special processors

urns? Does this just eat up an additional thread and get reported as a "Terminated thread"? What thread pool, if any, would this thread come from? Just some thoughts that would have to be worked through... > On Jun 19, 2019, at 11:17 AM, Peter Wicks (pwicks) wrote: > > I'

Thoughts on an internal "Terminate" handler for special processors

I've run into a few cases where a processor that works with external resources (DBCP especially) will hang and can't be properly terminated. I was thinking it would make sense to have an interface or annotation that these processors could have that would flag them for "special" termination.

RE: [EXT] Re: NiFi 2.0 Roadmap

and all extensions come from a flow registry on demand. > Other obvious things like Java 11 as the base requirement and killing > off deprecated things come to mind. > > Thanks > > On Fri, Jun 14, 2019 at 11:45 AM Peter Wicks (pwicks) > > wrote: > > > I've seen a lot

NiFi 2.0 Roadmap

I've seen a lot of comments along the line of, "I don't think this will happen before NiFi 2.0". Do we have a roadmap/list somewhere of the big general changes planned for NiFi 2.0 or some kind of 2.0 roadmap? --Peter

RE: [EXT] Re: GitHub Stuff

I like having signed commits. I develop on both Windows and Linux, but have only had success getting signing working on Windows (which was a bit complicated as it was). You can see when I switched from mostly Windows to mostly Linux by when I stopped signing commits... Thanks, Peter

RE: [EXT] Sliding windows

Craig, If you have a timestamp set as an attribute on the processor, then this is kind of possible. Have a regular MergeContent processor, with "Maximum Group Size" set to 1 mb, set "Max Bin Age" to 3 min; you may need to tweak settings to get the right cadence, but these are generally the

RE: [EXT] Re: [ANNOUNCE] New Apache NiFi PMC member Peter Wicks

Thanks everyone. Looking forward to continuing to work with the community for a long time to come. -Original Message- From: Mike Thomsen Sent: Friday, May 31, 2019 5:50 AM To: dev@nifi.apache.org Subject: [EXT] Re: [ANNOUNCE] New Apache NiFi PMC member Peter Wicks Congratulations! On

RE: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

One more "not awesome" would be that core changes that affect extensions will be a little harder to test. If I make a core change that changes the signature of an interface/etc... I'll need to do some extra work to make sure I don't break extensions that use it. Still worth it, just one more

RE: [EXT] Re: Kicking the Kerberos out of org.apache.nifi.hadoop

you need to do with what is there already. I think that other Hadoop spnego stuff was added around the same time as part of a different effort, but likely a bit of overlap. On Tue, Apr 30, 2019 at 12:21 PM Peter Wicks (pwicks) wrote: > > I was thinking of moving the non-Hadoop specific Ke

Kicking the Kerberos out of org.apache.nifi.hadoop

I was thinking of moving the non-Hadoop specific Kerberos classes out of org.apache.nifi.hadoop into a more central location. There are some good utility classes, such as KerberosKeytabCredentials, that it would be nice to have centralized into a non-Hadoop specific library. Would it be

RE: [EXT] Re: Custom Javascript in UI Without Custom Build

25, 2019 at 3:26 PM Peter Wicks (pwicks) wrote: > Pinging to see if anyone has a work around for this? > > -Original Message- > From: Peter Wicks (pwicks) > Sent: Wednesday, April 17, 2019 8:22 AM > To: dev@nifi.apache.org > Subject: Custom Javascript in UI Without Cu

RE: Custom Javascript in UI Without Custom Build

Pinging to see if anyone has a work around for this? -Original Message- From: Peter Wicks (pwicks) Sent: Wednesday, April 17, 2019 8:22 AM To: dev@nifi.apache.org Subject: Custom Javascript in UI Without Custom Build I have some custom Javascript, used for application monitoring

RE: [EXT] Re: Latest NiFi customs?

One other thing, that seems to catch me every time I upgrade an old instance: you will need to go in and allow users to read provenance data again. Somewhere along the way (1.6?) provenance reading moved into a separate policy, and it does not get assigned to anyone after upgrade.

Custom Javascript in UI Without Custom Build

I have some custom Javascript, used for application monitoring, that I'd like to put in the UI. I know how to do this if I'm willing to run a custom build of NiFi. Is there anyway to do this without a custom build of NiFi? My first thought was creating a dummy processor with an Advanced UI,

RE: [EXT] MS SQL CDC Processor

Viresh, There are a couple of options for MS SQL CDC. The "simple" option is to create a view in MS SQL that joins the CDC table with the lsn_time_mapping table. Then use the QueryDatabaseTable processor to load this view incrementally using the "tran_end_time" column. I see you've also

LivySessionController - Unexpected Behavior

I've been working on a few enhancements to the LivySessionController as part of my ticket NIFI-6175 : Livy - Add Support for All Missing Session Startup Features. I've been running into a lot of... interesting? features in the code, and I was hoping someone (Matt?) who was involved with it

RE: [EXT] Re: Direct access to FlowFile Content Repository Path?

and length, would be read only, etc.. Another option is to just invoke the 3rd party lib on the files before pulling them into nifi. If data arrives in list/fetch run list/execute custom/fetch instead maybe. Thanks On Mon, Mar 4, 2019 at 11:43 PM Peter Wicks (pwicks) wrote: > I'm work

Direct access to FlowFile Content Repository Path?

I'm working on a custom processor. The processor calls a 3rd party library that needs the path to my FlowFile content (no streams/arrays, just file names). I could write the content out to a temp file, but the content is already right there in the content repository... and many of the files are

RE: [EXT] [discuss] release apache nifi 1.9.0

I'd like to see a review of this bug before we release 1.9.0, as it helps mature the Node Offload feature. NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled https://github.com/apache/nifi/pull/3255 Thanks, Peter -Original Message- From: Joe Witt

RE: [EXT] Re: [DISCUSS] Early, voluntary relocation to GitBox

://blogs.apache.org/infra/entry/improved_integration_between_apache_and On Fri, Dec 14, 2018 at 3:34 PM Peter Wicks (pwicks) wrote: > Does closing a PR through GitHub close the JIRA ticket correctly? I'm > assuming sign-off is automated this way, but wasn't sure about this step. > > Thank

Backwards Compatibility for Property with Default Value

I'm working on a ticket that tweaks the way the Minimum File Age property works in GetFile and ListFile (NIFI-5897). Right now users see "0 sec" and assumes that means the minimum age check does not happen, when in fact it does; and in some timezone scenarios you have future dated files, and

RE: [EXT] Re: [DISCUSS] Early, voluntary relocation to GitBox

Does closing a PR through GitHub close the JIRA ticket correctly? I'm assuming sign-off is automated this way, but wasn't sure about this step. Thanks, Peter -Original Message- From: Aldrin Piri [mailto:aldrinp...@gmail.com] Sent: Friday, December 14, 2018 6:23 AM To: dev Subject:

RE: [EXT] Custom UI for Custom Service Controller

I haven't done this before either, but you named your file: META-INF/nifi-controller-service-configuration? -Original Message- From: Ron Aday [mailto:ronaday@outlook.com] Sent: Friday, November 30, 2018 10:12 AM To: dev@nifi.apache.org Subject: [EXT] Custom UI for Custom Service

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

'there is nothing new to do now' then right? Do you agree? Thanks On Fri, Nov 9, 2018 at 3:59 PM Peter Wicks (pwicks) wrote: > > Joe, > > Several different opinions have been expressed by Matt Burgess, Mark Payne > and Bryan Bende about whether we should be storing exception information i

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

. So, with the above said can you clarify what you are precisely requesting and for 'who' - who is the actor. Thanks On Fri, Nov 9, 2018 at 2:12 PM Peter Wicks (pwicks) wrote: > > A one week bump on this thread. --Peter > > -Original Message----- > From: Peter Wicks (pwicks)

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

A one week bump on this thread. --Peter -Original Message- From: Peter Wicks (pwicks) Sent: Friday, November 2, 2018 11:54 AM To: dev@nifi.apache.org Subject: RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute Dev Team, I don’t think we've reached

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

to a race hazard in an external system) but I don't imagine we'd route on invokehttp.java.exception.* since (as others have mentioned) it looks pretty fragile. -- James On Tue, 30 Oct 2018 at 16:44, Peter Wicks (pwicks) wrote: > > Sorry for the delayed response, I've been traveling. > &

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

the errors out to >>> the original failure relationship, that could be quite confusing to >>> the user, especially as time goes on (how to suppress the "new" >>> errors, e.g.). >>> >>> IMHO I think we're between a rock and a hard place her

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

ation from in NiFi 2.0. Regards, Matt On Thu, Oct 25, 2018 at 10:12 PM Peter Wicks (pwicks) wrote: > > Mark, > > I agree with you that this is the best option in general terms. After > thinking about it some more I think the biggest use case is for > troubleshooting. If a file

RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute

/interpreting of the message with a bunch of routing > processors, which seems more brittle than just having the appropriate > relationships. > On Thu, Oct 25, 2018 at 1:36 AM Peter Wicks (pwicks) > wrote: >> >> When a FlowFile is routed to failure, frequently there is

New Standard Pattern - Put Exception that caused failure in an attribute

When a FlowFile is routed to failure, frequently there is no clear reason without looking into the actual error message. Some processors work around this by creating many different relationships, but even then frequently the generic Failure relationship also provides little guidance. I've seen

License of NLKBufferedReader

While digging into ReplaceText today, I noticed that org.apache.nifi.processors.standard.util.NLKBufferedReader is actually a direct copy of the JDK version of BufferedReader, with a few minor code edits made on top of it so that new line characters are kept. This code was in the original code

RE: [EXT] Calcite version used in Nifi 1.7+

In NiFi 1.7.0 it was Calcite v1.12.0. In NiFi 1.8 it will be Calcite v1.17.0 -Original Message- From: ruurd.schoonh...@dbiq.nl [mailto:ruurd.schoonh...@dbiq.nl] Sent: Tuesday, October 16, 2018 6:58 AM To: dev@nifi.apache.org Subject: [EXT] Calcite version used in Nifi 1.7+ Hi guys,

Future of HDF at Hortonworks/Cloudera

Normally, I wouldn’t ask a question like this on the NiFi Dev group. But I did a census and out of the top 10 committers to NiFi in the last year, 8 definitely work for Hortonworks (couldn’t figure out where MikeThomsen works). In the subsequent top 10 committers, there are at least four more

RE: [EXT] XKCD use case for NiFi

I don't think NiFi will run on his phone, MiNiFi maybe? -Original Message- From: Joe Gresock [mailto:jgres...@gmail.com] Sent: Wednesday, October 3, 2018 12:08 PM To: dev@nifi.apache.org Subject: [EXT] XKCD use case for NiFi https://xkcd.com/2054/ He needs NiFi!

Looking for reviewer for Unit Test only PR (NiFi-5381)

I try really hard not to ask for PR reviews... but I have a PR that only adds unit tests for GetSFTP and PutSFTP that's been open for about a month. I have a sizeable number of other code changes ready that affect these processors and wanted to get better unit tests in place before moving

RE: [EXT] Re: Moving UI objects on a parent you don't have access to

ow this due to time constraints. I do not object to >> your proposal to change this (maybe Matt Gilman feels differently?). >> If you open a Jira, it should be doable. >> >> Andy LoPresto >> alopre...@apache.org >> *alopresto.apa...@gmail.com * PGP >> Fing

Moving UI objects on a parent you don't have access to

While experimenting with permissions, I found that if I have no permissions to a process group, but do have permissions to a child that lives in that group, I can move that child around on the UI. I know that in the object model the x,y position values are part of the child, which I have

RE: [EXT] Re: Dark mode

I did something similar back in 1.2.0 using CSS overrides and a chrome plugin : https://userstyles.org/styles/142978/dark-nifi-1-2-0. -Original Message- From: Brandon DeVries [mailto:b...@jhu.edu] Sent: Friday, July 13, 2018 9:42 AM To: dev@nifi.apache.org Subject: [EXT] Re: Dark mode

RE: [EXT] RE: JIRA #NIFI-5327

Prashanth, Woops, I had a fix for that test error months ago, but dropped it while pushing a PR and never put in a ticket for it... I've submitted a PR with the fix. https://github.com/apache/nifi/pull/2819 --Peter

Preliminary work on SCP and SSH processors

I’ve been working on NIFI-539 and NIFI-3698, adding support for SSH based processors like SCP and an Execute SSH. I’ve submitted a PR, but it definitely needs some work still. Would appreciate some feedback on the approaches. Unlike most of my code changes, this one should be easy for most

Copying small portions of code form other Apache Projects

I'm working on some changes and found that the exact code I need, along with a lot of code I can make derivatives from, are buried in the Apache Ant project. I'm not going to include Ant as a dependency, just copy some code from it. Should there be some annotation on the code/license

RE: [EXT] Re: Primary Only Content Migration

. It needs a lot of thought/design/testing/etc.. I was just having a conversation about this yesterday. It is definitely a thing and will be a major effort. Will make a JIRA for this soon. Thanks On Thu, Jun 7, 2018 at 5:21 PM, Peter Wicks (pwicks) wrote: > Bryan, > > We see this with la

RE: [EXT] Re: Primary Only Content Migration

primary node will continue to be worked off by the downstream processors on the old node until they are all processed. -Bryan On Thu, Jun 7, 2018 at 1:55 PM, Peter Wicks (pwicks) wrote: > I'm sure many of you have the same situation, a flow that runs on a cluster, > and at some point

Primary Only Content Migration

I'm sure many of you have the same situation, a flow that runs on a cluster, and at some point merges back down to a primary only processor; your files sit there in the queue with nowhere to go... We've used the work around of having a remote processor group that loops the data back to the

RE: [EXT] [ANNOUNCE] New Apache NiFi Committer Sivaprasanna Sethuraman

Congratulations Sivaprasanna! -Original Message- From: Tony Kurc [mailto:tk...@apache.org] Sent: Tuesday, June 05, 2018 08:09 To: dev@nifi.apache.org Subject: [EXT] [ANNOUNCE] New Apache NiFi Committer Sivaprasanna Sethuraman On behalf of the Apache NiFI PMC, I am very pleased to

RE: [EXT] Re: ReplaceText Flow File Processing Count

pressure threshold to 1 on the queue > leading into ReplaceText. > > On Fri, May 4, 2018 at 3:50 AM, Peter Wicks (pwicks) <pwi...@micron.com> > wrote: >> Had a user notice today that a ReplaceText processor, scheduled to run every >> 20 minutes, had process

ReplaceText Flow File Processing Count

Had a user notice today that a ReplaceText processor, scheduled to run every 20 minutes, had processed all 14 files in queue at once. I looked at the code and see that ReplaceText does not do a standard session.get, but instead calls: final List flowFiles =

Should I postpone submitting PR's on seriously breaking changes until prep for 2.0?

I have a couple ideas for changes that would cause wide spread breakage (like moving the DatabaseAdapter selection into the DBCP Service). >From what I've seen, this level of non-backwards compatibility should be >postponed until a major version change? Thanks, Peter

RE: [EXT] Suggestion: Apache NiFi component enhancement

I think this is a good idea. But based on your example I think you would want to provide a primary Type along with a list of Alias types. If NiFi starts and it can no longer find a processor by the Type name it had in the flow.xml it can check he annotations/aliases to see if it's been renamed.

RE: (NiFi 1.6) Funnels with no outgoing relationship filling my app log

https://issues.apache.org/jira/browse/NIFI-5075 From: Peter Wicks (pwicks) Sent: Thursday, April 12, 2018 14:30 To: 'dev@nifi.apache.org' <dev@nifi.apache.org> Subject: (NiFi 1.6) Funnels with no outgoing relationship filling my app log I just upgraded one of my servers to NiFi 1.6.0.

(NiFi 1.6) Funnels with no outgoing relationship filling my app log

I just upgraded one of my servers to NiFi 1.6.0. I have a couple of funnels that just dead end, Flow File's come in but never go anywhere after that. Mostly I use this for troubleshooting/validation. The inbound relationships all have expiration times, and it's a quick way for me to inspect the

RE: [EXT] Re: NiFi Versioned Process Group Status Icons

cess Group is not configured to have any encapsulated components, we still render the counts as zero. Additionally, the presence of the icons helps drive awareness of the feature. Rob or Drew may have some additional insight to add. Hope this helps Matt On Fri, Mar 23, 2018 at 12:10 AM, Peter Wicks (p

NiFi Versioned Process Group Status Icons

Why does NiFi show status icons for Versioned Process Group's on servers that are not configured to connect to a NiFi Registry? Thanks, Peter

RE: [EXT] Re: Help with frontend-maven-plugin?

tomatically use the proxy for downloading node and npm, as well as passing the proxy to npm commands <https://docs.npmjs.com/misc/config#proxy>. -Scott On Tue, Mar 20, 2018 at 5:02 AM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: > The versi

Help with frontend-maven-plugin?

The version of "frontend-maven-plugin" used in NiFi is out of date, and the versions of NPM and Node referenced by it are also a bit old. I wouldn't bring this up, except there is a bug in this version of NPM that can cause issues when trying to download NPM packages via proxy server. This is

RE: [EXT] Re: Embedded Nifi

Vincent, Embedded NiFi still has a long ways to go to be really useful, in my opinion; and I don't know if anyone is actively working on those improvements. The PR Andy mentioned simply allows you to startup NiFi inside your process instead of running it directly from a startup script, but

RE: [EXT] Re: [DISCUSS] CI / Travis / Jenkins

Best news I've heard in a long time :) Thanks for all the hard work! -Original Message- From: af...@fucs.org [mailto:af...@fucs.org] On Behalf Of Andre Sent: Thursday, December 07, 2017 12:15 To: dev@nifi.apache.org Subject: [EXT] Re: [DISCUSS] CI / Travis / Jenkins Joe, Thanks for

CIFS/SMB Shares from Linux

We have a number of CIFS networks shares that I've been denied permission to mount on my NiFi Linux server. In the past I've accessed similar shares by running NiFi on a Windows server and then pushing the files using Site-to-Site, but I'd rather avoid that if I can. In the past those Windows

RE: [EXT] Re: Contrib Check Build - RAT and Unit Test Failure

be an actual windows problem and the test is possibly flagging it. Thanks On Wed, Nov 22, 2017 at 9:34 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > Joe, > > This did resolve the RAT issue, but the unit test I mentioned still fails > with the same error. > > -

RE: [EXT] Re: Contrib Check Build - RAT and Unit Test Failure

and Unit Test Failure Peter I dont' believe RAT works in parallel builds (-T2 for example). If I use RAT it is during a full clean build and activated via the 'contrib-check' profile. 'mvn clean install -Pcontrib-check' Thanks On Wed, Nov 22, 2017 at 9:19 PM, Peter Wicks (pwicks) <pwi...@micron.

Contrib Check Build - RAT and Unit Test Failure

I'm trying to successfully run Contrib Check on my dev box. Windows 10 x64 jdk1.8.0_91 MVN 3.3.9 I'm using IntelliJ, so my Run looks like: Command line: -T2 -Drat.skip=true clean install Profile: contrib-check If I don't disable RAT (-Drat.skip=true) then I get too many RAT failures for the

RE: [EXT] Re: Automated Testing of Flows

, Nov 15, 2017 at 8:07 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > I would like to find a way to automatically run integration/upgrade > tests on flows, rather than just one or two processors like the mock > framework supports. Preferably something Template driven, where I

Automated Testing of Flows

I would like to find a way to automatically run integration/upgrade tests on flows, rather than just one or two processors like the mock framework supports. Preferably something Template driven, where I can provide a template xml file, and an accompanying config file to setup the test, then run

Adding EL support to UpdateAttribute for the Attribute Name

I've run into a use case for adding EL support into the Attribute Name itself in UpdateAttribute. Looking for thoughts on other approaches, pros/cons of doing this. I'm generically extracting data from a database. Right now I have ~30 tables, but that number could be anything, just think big

RE: [EXT] Re: Architecting the MS SQL CDC Processor

essor were the first to support this, it could be the exemplar for past and future CDC processors :) Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-4491 On Mon, Oct 16, 2017 at 10:36 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > I've been working on a new processor th

Initial Admin Identity can't query provenance?

I just setup a new NiFi instance and setup myself as the Initial Admin Identity, I'm currently the only user on the box and I've made no security changes. I noticed today that I couldn't query provenance. I had to go in and create a new policy and add myself to it. This seems odd for an

Architecting the MS SQL CDC Processor

I've been working on a new processor that does Change Data Capture with Microsoft SQL Server. I followed Microsoft's documentation on how CDC works, and I've got some code that gets me the changes and is testing well. Right now, I don't actually have a processor, but a number of scripts that

RE: [EXT] Re: JAVA_HOME trouble in nifi.sh

n, JDK versions, etc)? That locateJava blurb is something we used from another ASF project and use heavily throughout our executables, so definitely need to track it down. Did you happen to see this substitution in any other files? On Thu, Oct 12, 2017 at 10:51 PM, Peter Wicks (pwicks) <pwi...@m

RE: [EXT] Re: Please refresh my memory on NAR dependencies

is the API JAR I think. If you need the implementations behind > it, you will probably need to declare that as a parent (not a > dependency) and perhaps still use the API JAR (though I'm guessing about the > latter). > > Regards, > Matt > > > On Sun, Oct 15, 2017 at 10:

RE: [EXT] Re: Funnel Queue Slowness

05 GMT+02:00 Peter Wicks (pwicks) <pwi...@micron.com>: > Joe, > > It really is about just forgetting that penalization is a thing. > Penalized files are fairly well marked when you do a List Queue. > > I think Funnel's need an overall re-examination. I noticed another >

RE: [EXT] Re: Please refresh my memory on NAR dependencies

(though I'm guessing about the latter). Regards, Matt On Sun, Oct 15, 2017 at 10:27 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > For NIFI-4465 I want the nifi-poi-bundle to include a Maven dependency on > nifi-record-serialization-services. So I start by adding th

RE: [EXT] Re: Funnel Queue Slowness

think? Thanks On Mon, Oct 9, 2017 at 9:01 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > Bryan, > > Yes, it was the penalty causing the issue. This feels like weird behavior for > Funnel’s, and I’m not sure if it makes sense for penalties to work this way. > >

Please refresh my memory on NAR dependencies

For NIFI-4465 I want the nifi-poi-bundle to include a Maven dependency on nifi-record-serialization-services. So I start by adding the dependency to the pom.xml. org.apache.nifi nifi-record-serialization-services I've tried several variations on this, with version numbers,

RE: [EXT] Re: Funnel Queue Slowness

, is it possible all the flow files going to failure are being penalized which would cause them to not be processed immediately? -Bryan On Oct 8, 2017, at 10:49 PM, Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: I’ve been running into an issue on 1.

Funnel Queue Slowness

I've been running into an issue on 1.4.0 where my Funnel sometimes runs slow. I haven't been able to create a nice reproducible test case to pass on. What I'm seeing is that my failure queue on the right will start to fill up, even though there is plenty of room for them in the next queue. You

RE: [EXT] Re: [VOTE] Release Apache NiFi 1.4.0 (RC2)

+1 (non-binding). Upgraded to 1.4.0, openJDK 1.8.0_102, RHEL. No issues. -Original Message- From: Joey Frazee [mailto:joey.fra...@icloud.com] Sent: Monday, October 02, 2017 3:50 AM To: dev@nifi.apache.org Subject: [EXT] Re: [VOTE] Release Apache NiFi 1.4.0 (RC2) +1 (non-binding) -

RE: [EXT] ConvertCSVToAvro vs CSVReader - Value Delimiter

Arun, I'm also using Ctrl+A as a delimiter and had the same problem. I haven't had time to write up a PR but it looked like a pretty easy fix to me too. I can't merge the change if you submit it, but I'd be happy to review it. --Peter -Original Message- From: Arun Manivannan

Lifecycle of a Processor instance

There had been a lot of internal discussions at work about the actual life cycle of the class instance instantiated by NiFi; and the variables scoped at the class level. When do Processors "reset", are new instances created for each run, or are instances recycled? What about concurrent threads

RE: [EXT] Re: NiFi Processors show 30 Second Execution time, 0 executions

est to go ahead and file a JIRA. In it you can reliably > post the attachments. There was a potentially related timezone > handling issue as I recall in this past release so perhaps there is > some relationship. > > Thanks > Joe > > On Sun, May 28, 2017 at 10:04 PM, Peter Wick

  1   2   >