Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Joe, Thanks for the comments! Slightly deviating from code consistency discussion but I think you raised some important points. I may on the glass half empty side, but being a user who witnessed many time how modularity played on other communities i am not particular excited about the prospect.

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

more good points... We will need to think through/document the benefits that this extension bazar would provide and explain the risks it also brings to users. We should document for developers best practices in building new components or extending existing ones and we should allow users to

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

To be clear, I really love the idea of an extension registry and have at least one custom processor that would be a great fit for some of the reasons listed by Oleg, and its really cool thinking that user data can drive NiFi improvements. We're on the same page there. Let's go back to one of

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Just wanted to add one more point which IMHO just as important. . . Certain “artifacts” (i.e., NARs that depends on libraries which are not ASF friendly) may not fit the ASF licensing requirements of genuine Apache NiFi distribution, yet add a great value for greater community of NiFi users, so

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Adam I 100% agree with your comment on "official/sanctioned”. As an external artifact registry such as BinTray for example or GitHub, one can not control what is there, rather how to get it. The final decision is left to the end user. Artifacts could be rated and/or Apache NiFi (and/or

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Adam, Some great points there. I think what would be good here to keep in mind is 'who' will tame these things. For various patterns that are chosen and abstractions found and code written: - The developers do the taming. For the extension registry and which processors become popular or

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Hey all, I can understand Andre's perspective - when I was building the ListS3 processor, I mostly just copied the bits that made sense from ListHDFS and ListFile. That worked, but its a poor way to ensure consistency across List* processors. As a once-in-a-while contributor, I love the idea

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

I’ll second Pierre Yes with the current deployment model the amount of processors and the size of NiFi distribution is a concern simply because it’s growing with each release. But it should not be the driver to start jamming more functionality into existing processors which on the surface may

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

I tend to agree with a lot of the points made by James and Pierre... Given that the end user of NiFi is not always a developer, it seems more user-friendly to have the specific processors and not have users trying to come up with the right set of JARs and the right configuration properties

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

I am observing one assumption in this thread. For some reason we are implying all these will be hadoop compatible file systems. They don't always have an HDFS plugin, nor should they as a mandatory requirement. Untie completely from the Hadoop nar. This allows for effective minifi interaction

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Andrew, Thank you for contributing. On 22 Feb 2017 10:21 AM, "Andrew Grande" wrote: Andre, I came across multiple NiFi use cases where going through the HDFS layer and the fs plugin may not be possible. I.e. when no HDFS layer present at all, so no NN to connect to. Not

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

I agree with Andrew in the operations sense, and would like to add that the user experience around dynamic properties (and even "conditional" properties that are not dynamic but can be exposed when other properties are "Applied") can be less-than-ideal and IMHO should be used sparingly. Full

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

Andre, I came across multiple NiFi use cases where going through the HDFS layer and the fs plugin may not be possible. I.e. when no HDFS layer present at all, so no NN to connect to. Another important aspect is operations. Current PutHDFS model with additional jar location, well, it kinda works,

[DISCUSS] Scale-out/Object Storage - taming the diversity of processors

dev, I was having a chat with Pierre around PR#379 and we thought it would be worth sharing this with the wider group: I recently noticed that we merged a number of PRs and merges around scale-out/cloud based object store into the master. Would it make sense to start considering adopting a