RE: 1.0 Run Status Colors

[Peter Wicks (pwicks)]

I created a ticket and put in a link to a mostly accurate screenshot, at least 
close enough to give you a feel for what I was thinking:
https://issues.apache.org/jira/browse/NIFI-2603

Direct screenshot link: https://goo.gl/photos/RZzz45RGeTAmWwiC7


-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com] 
Sent: Wednesday, August 17, 2016 3:46 PM
To: dev@nifi.apache.org
Subject: Re: 1.0 Run Status Colors

no big discussion really - just contributions.  If you could kick off a JIRA 
and attach a screenshot of what you have in mind that could help the discussion 
move along.

On Wed, Aug 17, 2016 at 2:33 PM, Peter Wicks (pwicks)  wrote:
> We've been playing with NiFi 1.0 and of course the new theme/layout is a big 
> change.
> One thing we don't particularly care for is the bland gray 
> Running/Stopped/Disabled colors. I would be willing to have them bland in the 
> header, but on the Processor/Process Group level I feel they really need to 
> be Green/Red/whatever color.
>
> I would be happy to submit the change, but I wanted to make sure there hadn't 
> been some kind heated discussion over colors or lack thereof before putting 
> in a PR.

Re: User-supplied attributes for processors

[Matt Burgess]

Kirk,

The processors have to explicitly know about dynamic properties (and
their intent) in order to use them appropriately. For a processor like
ListSFTP, it could be beneficial to have custom attributes (as
parameters to the SFTP session perhaps?) but the domain knowledge on
how they'd be used would have to be coded into the processor itself.
If you have a use case in mind, please feel free to file an
improvement Jira for ListSFTP to use dynamic properties.

Regards,
Matt

On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk
 wrote:
> Is there some reason why Processors, like ListSFTP, don’t allow custom 
> attributes to be added? Relatedly, why do these processors allow users to add 
> custom attributes & values in the UI even though it always throws the error:
> ‘[attribute]’ validated against ‘[value]’ is invalid because ‘[attribute]’ is 
> not a supported property
>
> - Kirk Tarou

Re: nifi.sensitive.props.key from external source

[Andy LoPresto]

Devin,

I am working on a PR to support this. I hope to have it completed by tomorrow. 

NIFI-1831 has a description of the task, PR 834 has my current changes. 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 18, 2016, at 10:33, Devin Fisher  
> wrote:
> 
> We are looking to deploy nifi as part of an appliance. Normally, we use
> etckeeper [1] to maintain, track and backup our configuration across the
> board. Etckeeper puts configuration files into a git repo. I would like to
> store nifi configuration in the same way but I'm concerned about storing
> the sensitive properties key there along with the flow.xml.gz. I would like
> to store that key somewhere else and load it in at start up time.  Any
> thoughts on how that could be done.  Ultimately, I just want is to not back
> up the key with the flow.xml.gz (which has the encrypted data). That way if
> someone gets a hold of the backup it would not trivial to decrypt the
> sensitive data in flow.xml.gz.
> 
> I thought I might be able to do this by adding a custom java.arg to the
> bootstrap.conf that would point to an environment variable.
> 
> Something like:
> java.arg.99=-Dnifi.sensitive.props.key=$NIFI_SENSITIVE_PROPS_KEY
> 
> But I'm not sure if System properties can stand in for nifi.perperies
> values and if the boot loader launches nifi in such a way to use
> environment variables.
> 
> 
> [1] https://github.com/joeyh/etckeeper

Re: Apache Yetus annotations : clarifying public vs private api

[Joe Witt]

Team,

I've gone ahead and focused more on ensuring that the
versioning/scheme and compatibility model we've been following is more
clearly articulated on [1] and also more specifically defined what is
considered part of the nifi-api.

I still think the yetus annotations or something inspired by them is
the right way to go but at this point wanted to avoid adding a
dependency to the nifi-api.  I'm actually inclined to think that if we
go further with this it should be done by bringing the yetus
annotations in as a source dependency rather than a binary one.  In
any event, the near term pain was to more clearly articulate the API
compatibility model and i believe that is resolved.  Also added some
comments about it on the JIRA i raised for it [2].

[1] 
https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
[2] https://issues.apache.org/jira/browse/NIFI-1373

On Sun, Jul 31, 2016 at 7:15 PM, Joe Witt  wrote:
> Team,
>
> The big ticket back-end changes for NiFi 1.x line seem to be closing
> down and Apache Yetus has pushed their artifacts to maven central so I
> am proceeding with NIFI-1373.  I plan to start with a limited rollout
> but specifically will annotate the nifi-api completely.  I also
> updated our wiki which talks about our version scheme [1].  We can see
> how this goes and then if we want to expand it to other parts of the
> codebase we can do so later.
>
> The specific text i added to that was:
> "
> Also, starting with the Apache NiFi 1.x codebase we have adopted
> Apache Yetus 'Audience Annotations' to explicitly mark code for things
> like interfaces, classes, and methods to indicate the intended
> audience and stability of those APIs.  If not otherwise marked please
> consider the interface, class, or method to be private/internal and
> unstable.  The vast majority of 'nifi-api' should be both public use
> and stable meaning we will treat any compatibility changes for these
> items as things which would likely require a major release.
> Otherwise, as a community we should be able to more easily navigate
> the necessary evolution and improvement of the codebase for these
> interfaces, classes, and methods for which we do not have to be so
> strictly adherent to backward compatibility.  A further consideration
> here though is as mentioned previously which is that even if we want
> to change classes, interfaces, and methods which are not public and
> stable we must also be mindful of things like configuration, user
> experience, REST API, etc... as these are also an important part of
> our 'interface' and compatibility which we must honor and account for.
> "
>
> The idea is that unless we have marked something it should be assumed
> to be private and unstable as far as implementation goes though this
> doesn't eliminate our need to honor the holistic interpretation of our
> "interface".
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme
>
> Thanks
> Joe
>
> On Tue, Jul 26, 2016 at 7:59 AM, Joe Skora  wrote:
>> +1
>>
>> This should help clarify things when learning NiFi and when diving in new
>> areas of the codebase.
>>
>> Thanks, I'm looking forward to it.
>>
>> On Mon, Jul 25, 2016 at 7:45 PM, Matt Burgess  wrote:
>>
>>> agreed, big +1 here, will make things much more clear re: stability
>>> and extensibility.
>>>
>>> On Mon, Jul 25, 2016 at 7:43 PM, Tony Kurc  wrote:
>>> > awesome!
>>> >
>>> > On Mon, Jul 25, 2016 at 7:32 PM, Joe Witt  wrote:
>>> >
>>> >> Team
>>> >>
>>> >> For those interested I am going to take a stab at working through the
>>> >> codebase [1] to apply Apache Yetus [2] annotations to make it more
>>> >> clear and well communicated what are the public vs private aspects of
>>> >> the API.
>>> >>
>>> >> We've had the 'nifi-api' module for some time now but unless you know
>>> >> that this was meant to be the public API for supported extension types
>>> >> you'd not really know that other parts of NiFi were not meant to be
>>> >> altered and could be changed.  Therefore, we've not been able to
>>> >> evolve the framework as directly as we'd like at times.  We had to
>>> >> wait until 1.0 to make changes which otherwise should have been ok to
>>> >> make.  Also, previously 'nifi-api' had a lot of extraneous bits in it
>>> >> that have now been refactored out to where they belong such as core
>>> >> framework itself or in a 'nifi-framework-api' module.  This means
>>> >> things in 'nifi-api' should be quite stable and public.
>>> >>
>>> >> Yetus gives us a chance to clearly articulate which parts of the API
>>> >> are meant for public use and which are meant for private use only and
>>> >> therefore are subject to change.  Furthermore, even for those public
>>> >> elements we can more effectively articulate stability in ways that map
>>> >> really nicely to our versioning scheme.
>>> >>
>>> >> My intent is to be very cautious 

nifi.sensitive.props.key from external source

[Devin Fisher]

We are looking to deploy nifi as part of an appliance. Normally, we use
etckeeper [1] to maintain, track and backup our configuration across the
board. Etckeeper puts configuration files into a git repo. I would like to
store nifi configuration in the same way but I'm concerned about storing
the sensitive properties key there along with the flow.xml.gz. I would like
to store that key somewhere else and load it in at start up time.  Any
thoughts on how that could be done.  Ultimately, I just want is to not back
up the key with the flow.xml.gz (which has the encrypted data). That way if
someone gets a hold of the backup it would not trivial to decrypt the
sensitive data in flow.xml.gz.

I thought I might be able to do this by adding a custom java.arg to the
bootstrap.conf that would point to an environment variable.

Something like:
java.arg.99=-Dnifi.sensitive.props.key=$NIFI_SENSITIVE_PROPS_KEY

But I'm not sure if System properties can stand in for nifi.perperies
values and if the boot loader launches nifi in such a way to use
environment variables.


[1] https://github.com/joeyh/etckeeper

Re: Apache NiFi Clarification

[Bryan Bende]

Hello,

NiFi can gather data from as many sources as needed. When data is brought
into NiFi it is written into NiFi's internal repositories and stored there
as it moves through the graph of processors, the data can then be delivered
to any desired system.

To bring data from a database to HDFS you would likely use the ExecuteSQL
or QueryDatabase processor to get the data out of the database, then some
intermediary processors if you need to convert the data, and then PutHDFS
to write the data to HDFS.

Thanks,

Bryan

On Thu, Aug 18, 2016 at 6:17 AM,  wrote:

> Hi,
> Good Day!
> I Saw your Blogs and Videos its Pretty Interesting and I am new in Apache
> Nifi.So i just want to know if the data will only to fetch into nifi or
> else data can export to any other databases or API. I need to your guides I
> am working on Nifi in My organization So i just to know how the flow to
> connect with Sql Server Or Any other Databases and How can i create a data
> flow to fetch the Server into Hadoop .but In my case its pretty different
> we need to connect multiple API Data into Nifi and Transfer Via Sqoop To
> Hive .If in this case we can choose Nifi  . please give some guides about
> the Apache Nifi and how the flow it will work . I am new to Apache Nifi So
> Please give Guides about Apache Nifi in my case  Thanks &
> Regards,Manikandan Kolanjinathan
> Junior Software Engineer
> Pinnacle Seven Technologies 0422-4208736, 4506535 
> www.pinnacleseven.comDelivering
> your business apps on cloud!
>
>

Re:Apache NiFi Clarification

[manikandan]

Hi,
Good Day! 
I Saw your Blogs and Videos its Pretty Interesting and I am new in Apache 
Nifi.So i just want to know if the data will only to fetch into nifi or else 
data can export to any other databases or API. I need to your guides I am 
working on Nifi in My organization So i just to know how the flow to connect 
with Sql Server Or Any other Databases and How can i create a data flow to 
fetch the Server into Hadoop .but In my case its pretty different we need to 
connect multiple API Data into Nifi and Transfer Via Sqoop To Hive .If in this 
case we can choose Nifi  . please give some guides about the Apache Nifi and 
how the flow it will work . I am new to Apache Nifi So Please give Guides about 
Apache Nifi in my case  Thanks & Regards,Manikandan Kolanjinathan
Junior Software Engineer
Pinnacle Seven Technologies 0422-4208736, 4506535 
www.pinnacleseven.comDelivering your business apps on cloud!