Re: Refactoring

2019-07-31 Thread Andy LoPresto
Yes, to echo Joe, we welcome refactoring and refinements, burning down technical debt, etc. But for a project of this size (both in lines of code and number of community users/contributors/etc.), a formal issue tracking system is imperative. We do have existing tickets for generic refactoring

Re: Refactoring

2019-07-31 Thread Joe Witt
Understood. But we do need Jira tickets to cover, track, review changes. Thanks On Wed, Jul 31, 2019 at 4:42 PM Malthe wrote: > I should perhaps have clarified, what I mean by refactoring is that > code semantics are unchanged. If the road to contribution is one > squashed commit per ticket

Date conversion

2019-07-31 Thread Brom, Michael C.
NiFi devs I'm connecting to an oracle DB and pulling a date that is in the format dd-MMM-yy (ex: 19-JAN-05).January 19, 2005. But once I pull it into NiFi using an executeSQL processor, the converted date is 0005-01-19 00:00:00.0. Oddly, if the year is 2010 (just showing 10)it

Re: Refactoring

2019-07-31 Thread Malthe
I should perhaps have clarified, what I mean by refactoring is that code semantics are unchanged. If the road to contribution is one squashed commit per ticket then contributing meaningful changes becomes difficult when working with code that was originally written for a legacy Java runtime.

Re: Refactoring

2019-07-31 Thread Joe Witt
this is also useful. https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility On Wed, Jul 31, 2019 at 2:51 PM Andy LoPresto wrote: > I’d suggest it’s the same as the process around any other issue. Identify > a need (as you’ve done below), open a ticket for it, and

Re: Refactoring

2019-07-31 Thread Andy LoPresto
I’d suggest it’s the same as the process around any other issue. Identify a need (as you’ve done below), open a ticket for it, and contribute if you have the capabilities and time. If you need more discussion or direction, the mailing list is the right place for that. Once you have a PR,

Refactoring

2019-07-31 Thread Malthe
What's the policy or strategy towards refactoring code without having too much encumbering process around it? For example, there is code in StandardProcessSession.java [1] that is unpractical to work with without a refactoring (more reuse of shared logic, essentially). This applies to methods

RE: [EXT] Re: Duplicate flow files *without* their content

2019-07-31 Thread Peter Wicks (pwicks)
Lars, If you are worried about it, using ReplaceText will have the same effect as your custom solution. When ReplaceText has it's `Replacement Strategy` set to `Always Replace` it doesn't read the contents of the FlowFile and simply writes out the replacement Value, which in your case could be

Re: Duplicate flow files *without* their content

2019-07-31 Thread Lars Winderling
Hi Edward, thank you for your input. I didn't know about the cow-semantics, that's really useful. I'll check out the in-depth guide for sure!In my case, the content of the flow file does change heavily from one processor to the next one, so I doubt copy-on-write would help here. Best,Lars On

Re: Component State

2019-07-31 Thread Bryan Bende
Hello, I'm not sure specifically how to do it with nipyapi, but the REST end-point is /processors//{id}/state where id is the processor uuid. Thanks, Bryan On Wed, Jul 31, 2019 at 7:34 AM ashwinreddyc . wrote: > > Hello, > > I'm looking to get the component state of say QueryDatabaseTables

Re: [DISCUSS] Predictive Analytics for NiFi Metrics

2019-07-31 Thread Yolanda Davis
Hi Craig, Thanks for your feedback and insight on your use cases. What version of MiNiFi are you running? Concerning performing edge ML this may be possible for you with MiNiFi C++ version 0.6.0. That release supports the creation of python processors which can be added to your flow to execute

Re: [DISCUSS] Predictive Analytics for NiFi Metrics

2019-07-31 Thread Andy Christianson
As someone who operated a 24/7 mission-critical NiFi flow, this feature would have been a life saver. If I'm heading home on a Friday, it would be great to have some blinking red lights to let me know that the system predicts that it is going to experience backpressure sometime over the

Re: Nifi registry - extension-repository, extensions, bundles

2019-07-31 Thread Bryan Bende
Hi Mike, The ability to version control extension bundles is a new feature that the community has been working towards. The dream vision is to pull a flow flow registry and also be able to retrieve any extension bundles that go with it. Right now, the 0.4.0 release of registry has the ability to

Component State

2019-07-31 Thread ashwinreddyc .
Hello, I'm looking to get the component state of say QueryDatabaseTables using nipyapi module. How do I get it ? Here is the flow right click on the QueryDatabaseTables processor and select view State. We will find the key value pairs, which give the state of the component Can you please help?

Re: Duplicate flow files *without* their content

2019-07-31 Thread Edward Armes
HI Lars, In short. depending on the how a FlowFile is duplicated, the content shouldn't be duplicated as well. In general, content is only duplicated when it has been deemed to have been changed (copy-on-write semantics). For the most part (unless a FlowFIle has a large number of attributes) a

Re: [DISCUSS] Predictive Analytics for NiFi Metrics

2019-07-31 Thread Arpad Boda
If you could share a bit more details about your OPC and Modbus usage, that would be highly appreciated! On Wed, Jul 31, 2019 at 12:01 PM Craig Knell wrote: > Sounds. Great > > Let me know if you need some help > > Best regards > > Craig > > > > > On 31 Jul 2019, at 17:31, Arpad Boda wrote: >

Duplicate flow files *without* their content

2019-07-31 Thread Lars Winderling
Dear NiFi community, I often face the use-case where I import flow files with content of order O(1gb) or O(10gb) – already compressed. Let's day I need to branch off of a flow where the actual flow file should be processed further, and one some side branch I want just to do some kind of logging

Re: [DISCUSS] Predictive Analytics for NiFi Metrics

2019-07-31 Thread Craig Knell
Sounds. Great Let me know if you need some help Best regards Craig > On 31 Jul 2019, at 17:31, Arpad Boda wrote: > > Craig, > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and Modbus ( > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the way for > MiNiFi

Re: [DISCUSS] Predictive Analytics for NiFi Metrics

2019-07-31 Thread Arpad Boda
Craig, OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and Modbus ( https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the way for MiNiFi c++, hopefully both will be part of next release (0.7.0). It's gonna be legen... wait for it! :) Regards, Arpad On Wed, Jul 31, 2019 at