Re: flow as code and minify scaling/isolation

[hunter morgan]

but what about filesystem access to processors? are those sandboxed?
On Feb 24, 2017, at 4:55 PM, Andrew Grande-2 [via Apache NiFi Developer List] 
<ml-node+s39713n14965...@n7.nabble.com<mailto:ml-node+s39713n14965...@n7.nabble.com>>
 wrote:

Hi,

I think all processors acting as clients do isolate Kerberos keytabs and
client certificates.

The Kafka situation is a current design limitation of Kafka, not NiFi. The
good news is there's an effort underway to have Kafka not rely on global
singleton config and specify those per connection instead. But this is more
in the Kafka 0.11.x line.

Andrew

On Fri, Feb 24, 2017, 4:23 PM hunter morgan <[hidden 
email]>
wrote:

> thanks for the links.
>
> i'm thinking that having the option of getting a template out of it or
> running in minifi would be good enough. i was sad to find that the rest api
> didn't seem to be included in minifi, so with it, accessible template
> export. i'm gonna look at that this weekend. glad to have more direction.
>
> yeah there is an impedance mismatch so far. but the minifi yaml config
> looks
> like the closest official completed work to such a workflow. i have mixed
> feelings about the flow repository stuff that's going on, but that's
> probably because i'm a dev that likes my existing tools (git, vi, cli
> goodness).
>
> it's hard to provide secure multitenant capability in nifi and isolate
> keytabs/jass/keystores between users, especially when processors use code
> (like kafka clients) that require or document using jvm opts to configure
> global jaas.
>
>
> also i think i wasn't joined to the list or something, so i should find out
> quicker next time there's a response.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/flow-as-code-and-minify-scaling-isolation-tp14564p14963.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com<http://Nabble.com>.
>



If you reply to this email, your message will be added to the discussion below:
http://apache-nifi-developer-list.39713.n7.nabble.com/flow-as-code-and-minify-scaling-isolation-tp14564p14965.html
To unsubscribe from flow as code and minify scaling/isolation, click 
here<http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=14564=SHVudGVyLk1vcmdhbkBjYXBpdGFsb25lLmNvbXwxNDU2NHwtMjQ1OTc3MDkx>.
NAML<http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/flow-as-code-and-minify-scaling-isolation-tp14564p14967.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: flow as code and minify scaling/isolation

[Andrew Grande]

Hi,

I think all processors acting as clients do isolate Kerberos keytabs and
client certificates.

The Kafka situation is a current design limitation of Kafka, not NiFi. The
good news is there's an effort underway to have Kafka not rely on global
singleton config and specify those per connection instead. But this is more
in the Kafka 0.11.x line.

Andrew

On Fri, Feb 24, 2017, 4:23 PM hunter morgan <hunter.mor...@capitalone.com>
wrote:

> thanks for the links.
>
> i'm thinking that having the option of getting a template out of it or
> running in minifi would be good enough. i was sad to find that the rest api
> didn't seem to be included in minifi, so with it, accessible template
> export. i'm gonna look at that this weekend. glad to have more direction.
>
> yeah there is an impedance mismatch so far. but the minifi yaml config
> looks
> like the closest official completed work to such a workflow. i have mixed
> feelings about the flow repository stuff that's going on, but that's
> probably because i'm a dev that likes my existing tools (git, vi, cli
> goodness).
>
> it's hard to provide secure multitenant capability in nifi and isolate
> keytabs/jass/keystores between users, especially when processors use code
> (like kafka clients) that require or document using jvm opts to configure
> global jaas.
>
>
> also i think i wasn't joined to the list or something, so i should find out
> quicker next time there's a response.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/flow-as-code-and-minify-scaling-isolation-tp14564p14963.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: flow as code and minify scaling/isolation

[Joe Witt]

Hunter

I believe you are still not registered to the list.  I did have to
moderate this through.

If you need help registering please let me know directly and I can help.

Thanks
Joe

On Fri, Feb 24, 2017 at 4:21 PM, hunter morgan
<hunter.mor...@capitalone.com> wrote:
> thanks for the links.
>
> i'm thinking that having the option of getting a template out of it or
> running in minifi would be good enough. i was sad to find that the rest api
> didn't seem to be included in minifi, so with it, accessible template
> export. i'm gonna look at that this weekend. glad to have more direction.
>
> yeah there is an impedance mismatch so far. but the minifi yaml config looks
> like the closest official completed work to such a workflow. i have mixed
> feelings about the flow repository stuff that's going on, but that's
> probably because i'm a dev that likes my existing tools (git, vi, cli
> goodness).
>
> it's hard to provide secure multitenant capability in nifi and isolate
> keytabs/jass/keystores between users, especially when processors use code
> (like kafka clients) that require or document using jvm opts to configure
> global jaas.
>
>
> also i think i wasn't joined to the list or something, so i should find out
> quicker next time there's a response.
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/flow-as-code-and-minify-scaling-isolation-tp14564p14963.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: flow as code and minify scaling/isolation

[hunter morgan]

thanks for the links.

i'm thinking that having the option of getting a template out of it or
running in minifi would be good enough. i was sad to find that the rest api
didn't seem to be included in minifi, so with it, accessible template
export. i'm gonna look at that this weekend. glad to have more direction.

yeah there is an impedance mismatch so far. but the minifi yaml config looks
like the closest official completed work to such a workflow. i have mixed
feelings about the flow repository stuff that's going on, but that's
probably because i'm a dev that likes my existing tools (git, vi, cli
goodness).

it's hard to provide secure multitenant capability in nifi and isolate
keytabs/jass/keystores between users, especially when processors use code
(like kafka clients) that require or document using jvm opts to configure
global jaas.


also i think i wasn't joined to the list or something, so i should find out
quicker next time there's a response.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/flow-as-code-and-minify-scaling-isolation-tp14564p14963.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: flow as code and minify scaling/isolation

[Bryan Rosander]

Hey Morgan,

As far as documentation on the yml schema, we've attempted to document both
its current state as well as the changes over time. [1]

In terms of converting the yml into a flow.xml (and nifi.properties), that
is done by the ConfigTransformer. [2]  I wouldn't think it would be very
difficult to wrap a main around this functionality and enable command line
yml -> flow.xml.

It's important to realize that the yml currently only supports a subset of
NiFi functionality.  Also, the conversion to yml is targeting a template
exported from NiFi, so it's not quite a perfect round trip for your
purposes.

I'm curious what kerberos isolation you're desiring.  I believe that the
processors that can talk to kerberized services do support specifying
credentials although I'm less familiar with that functionality.  If you
wanted completely separate jaas configs or something, it might be easier to
handle it the way you suggested, keeping functionality isolated by using
completely separate instances of (Mi)NiFi.

Thanks,
Bryan

[1] https://nifi.apache.org/minifi/system-admin-guide.html#config-file
[2]
https://github.com/apache/nifi-minifi/blob/master/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java

On Wed, Feb 1, 2017 at 2:37 PM, Morgan, Hunter  wrote:

> is there any internal documentation on the minifi config yaml to flow xml
> conversion process?
> i’d really like to leverage this code to go back and forth from yaml to
> xml flows/templates.
> one thing that i think is really missing from nifi is being able to
> develop and version control flows/templates with normal editors and vcs. i
> know the project is talking about building versioned flow repositories and
> things, but some developers want to develop flow as code, and it would be
> nice to have that as an option.
> any feedback on these concerns? i haven’t spent enough time diving into
> the minifi code to figure out how the conversion works, but i’m working on
> it. if anyone can get me up to speed, and/or point me at document that
> outlines it, that would be great.
>
> another topic is using minifi to deploy horizontally scalable and isolated
> flows. this is especially important as presently there is no kerberos
> credential isolation between flows in clustered nifi. as far as i can tell,
> using minifi to deploy/scale/isolate flows is not something that was
> intended, but it might be a useful alternative to the clustered model.
>
> thanks in advance
>
> Hunter Morgan
> Data Platform Engineering
> 540 391 0440
> hunter.mor...@capitalone.com
>
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

flow as code and minify scaling/isolation

[Morgan, Hunter]

is there any internal documentation on the minifi config yaml to flow xml 
conversion process?
i’d really like to leverage this code to go back and forth from yaml to xml 
flows/templates.
one thing that i think is really missing from nifi is being able to develop and 
version control flows/templates with normal editors and vcs. i know the project 
is talking about building versioned flow repositories and things, but some 
developers want to develop flow as code, and it would be nice to have that as 
an option.
any feedback on these concerns? i haven’t spent enough time diving into the 
minifi code to figure out how the conversion works, but i’m working on it. if 
anyone can get me up to speed, and/or point me at document that outlines it, 
that would be great.

another topic is using minifi to deploy horizontally scalable and isolated 
flows. this is especially important as presently there is no kerberos 
credential isolation between flows in clustered nifi. as far as i can tell, 
using minifi to deploy/scale/isolate flows is not something that was intended, 
but it might be a useful alternative to the clustered model.

thanks in advance

Hunter Morgan
Data Platform Engineering
540 391 0440
hunter.mor...@capitalone.com



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.