Re: Authorization for Configuration

[Justin Leet]

You're right, I didn't really think through the specific configs that each
sensor has across enrichment and indexing (because even indexing has the
various batch tuning configurations). I was thinking in terms of topology,
which isn't complete. Really it should (probably) be in terms of sensor all
the way through.

Here's some thoughts on that.

   - Users have permissions based on sensor throughout (parsing,
   enrichment, and indexing).
  - Ideally, this is fine-grained enough that administrators might be
  able to restrict at each level (e.g. I don't want a user to say the batch
  size is either 1 or MAX_INT, that could cause global performance issues
  even if it's just on a single sensor).
  - Do we need to be more fine-grained in the first cut (e.g. allow
  users to do indexing level configs, but
   - Global configs also need to be able to have permissions to apply to
   them (after all, I don't want a user to change something for everyone with
   no notice). I would expect a lot of these to be more admin-level things. If
   they aren't, I question if they should actually be in the global config.
   - I'm personally fine with either holding off or doing read perms. Right
   now, there aren't read perms, and it's obviously much harder for a user to
   cause larger scale issues with read perms on configuration itself. I guess
   I could potentially see users not wanting other tenants to see their
   transforms, but that seems fairly unlikely for most use cases. Read perms
   on data an obviously be applied outside of Metron on the data stores
   themselves as necessary.


On Mon, Dec 3, 2018 at 4:15 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> > @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
> pending this exact conversation winding down.
>
> Yup, understood.
>
> On Fri, Nov 30, 2018 at 7:36 AM Justin Leet  wrote:
>
> > To start with, I'm thinking just the configuration, in particular
> anything
> > that touches the ConfigurationsUtils.  I think for the first pass, we
> could
> > leave reading from configs out of it and focus on who can write configs
> > (since that's more disruptive to the system).
> >
> > To the best of my knowledge, this is:
> >
> >- Configuration through the Management UI
> >- Configuration through the CLI (e.g. the push script + Stellar + any
> >other scripts that have to do it).
> >
> > In terms of how fine-grained, I would essentially expect this to be at
> the
> > users+groups level and the individual sensor / topology level (e.g.
> parsers
> > can be authorized individually by users/group, indexing is on the whole).
> > Essentially, I don't propose trying to rearchitect things to try to have
> > multiple indexing topologies for complete separation, etc.  Just to take
> > the existing setup and be able to apply authorization on top of it.
> >
> > I would expect anything covered by the ZK configs to be done as part of
> > this effort (possibly in stages).  As noted, I would expect this to be a
> > feature branch rather than piecemeal replacement.
> >
> > @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
> > pending this exact conversation winding down.
> >
> > On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Justin, It probably makes sense to provide a list of these
> configuration
> > > items as subtasks on the FB Jira so that we can crosscheck what entry
> > > points have been implemented against the test scripts. Do you think
> this
> > > will impact streaming enrichments or the profiler at all? That is to
> say,
> > > as Ali asked, just how far are you looking to take the fine grained
> auth
> > > scope for this?
> > >
> > > M
> > >
> > > On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> > > wrote:
> > >
> > > > Hi Justin,
> > > >
> > > > By configuration do you mean the sensor related configurations only?
> > Are
> > > > you limiting the scope of this activity to the management-UI or also
> > > > Alert-UI as well? For example, defining different roles (pre-defined
> > > > or customizable) and the fine-grained integration with Ranger?
> > > >
> > > > Cheers,
> > > > Ali
> > > >
> > > > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Right now, our various configs can be modified by anyone with
> access
> > to
> > > > the
> > > > > various scripts. I'd like to start a discussion around building out
> > > some
> > > > > authorization to be able to add some more fine grained controls
> > around
> > > > > this.
> > > > >
> > > > > Other projects have some variants on how to accomplish this.
> > > Typically,
> > > > > this follows a pattern of calling out to a interface/class that
> takes
> > > in
> > > > > the operation and context (user and other info) and returns
> > true/false
> > > if
> > > > > something is authorized.
> > > > >
> > > > > In my mind, what we would need out 

Re: Authorization for Configuration

[Michael Miklavcic]

> @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
pending this exact conversation winding down.

Yup, understood.

On Fri, Nov 30, 2018 at 7:36 AM Justin Leet  wrote:

> To start with, I'm thinking just the configuration, in particular anything
> that touches the ConfigurationsUtils.  I think for the first pass, we could
> leave reading from configs out of it and focus on who can write configs
> (since that's more disruptive to the system).
>
> To the best of my knowledge, this is:
>
>- Configuration through the Management UI
>- Configuration through the CLI (e.g. the push script + Stellar + any
>other scripts that have to do it).
>
> In terms of how fine-grained, I would essentially expect this to be at the
> users+groups level and the individual sensor / topology level (e.g. parsers
> can be authorized individually by users/group, indexing is on the whole).
> Essentially, I don't propose trying to rearchitect things to try to have
> multiple indexing topologies for complete separation, etc.  Just to take
> the existing setup and be able to apply authorization on top of it.
>
> I would expect anything covered by the ZK configs to be done as part of
> this effort (possibly in stages).  As noted, I would expect this to be a
> feature branch rather than piecemeal replacement.
>
> @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
> pending this exact conversation winding down.
>
> On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Justin, It probably makes sense to provide a list of these configuration
> > items as subtasks on the FB Jira so that we can crosscheck what entry
> > points have been implemented against the test scripts. Do you think this
> > will impact streaming enrichments or the profiler at all? That is to say,
> > as Ali asked, just how far are you looking to take the fine grained auth
> > scope for this?
> >
> > M
> >
> > On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> > wrote:
> >
> > > Hi Justin,
> > >
> > > By configuration do you mean the sensor related configurations only?
> Are
> > > you limiting the scope of this activity to the management-UI or also
> > > Alert-UI as well? For example, defining different roles (pre-defined
> > > or customizable) and the fine-grained integration with Ranger?
> > >
> > > Cheers,
> > > Ali
> > >
> > > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Right now, our various configs can be modified by anyone with access
> to
> > > the
> > > > various scripts. I'd like to start a discussion around building out
> > some
> > > > authorization to be able to add some more fine grained controls
> around
> > > > this.
> > > >
> > > > Other projects have some variants on how to accomplish this.
> > Typically,
> > > > this follows a pattern of calling out to a interface/class that takes
> > in
> > > > the operation and context (user and other info) and returns
> true/false
> > if
> > > > something is authorized.
> > > >
> > > > In my mind, what we would need out of this is
> > > >
> > > >- Ability to apply fine grained permissions
> > > >- The various scripts and UI should flow this authorization
> > framework.
> > > >I believe most (if not all) of our configuration flows through
> > > >ConfigurationsUtils.  Anything that doesn't should either be
> hooked
> > in
> > > > or
> > > >refactored to flow through the same codepaths.
> > > >- Pluggability. We shouldn't force only one authorizer.
> > > >
> > > > In particular, I'm proposing we use Apache Ranger
> > > >  as a supported authorization framework,
> > > > implementing it alongside the authorization framework to validate
> what
> > we
> > > > build. In my mind, the main catch with Ranger is that, based on my
> > > > understanding, we won't be able to restrict users with direct access
> to
> > > > ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into
> > ZK's
> > > > ACLs).  I believe this is a reasonable restriction, especially as the
> > > > management UI gets improved to handle more of the configuration
> burden
> > > and
> > > > the number of users with access to ZK CLI begins to decrease.  Users
> > can
> > > > still add ZK ACLs separately to enforce that access there.
> > > >
> > > > For anyone not familiar with Ranger, essentially you build a plugin
> > that
> > > > hooks into the existing component's authorization framework (e.g. for
> > > > Storm, the plugin runs through the IAuthorizer
> > > > <
> > > >
> > >
> >
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> > > > >
> > > > interface, for Yarn it runs through YarnAuthorizationProvider
> > > > <
> > > >
> > >
> >
> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/security/YarnAuthorizationProvider.html
> > > > >).
> > > > Additionally, Ranger 

Re: Authorization for Configuration

[Ali Nazemian]

Would it be based on the operation as well? Like be able to read or modify.
So is this scenario valid? from the user experience perspective, a user may
be authorised to change the indexing/enrichment config (because there is
only one topology for them), but because he/she doesn't have sufficient
privilege to apply parser configs may not be able to modify that through
the management UI? I personally think it would be better to keep it simple
to be able to make it consistent across all the configs. For example, very
few roles (groups) to be able to either read configs or also modify them.
Having per sensor access management might be something that can be added
later on. I guess when it comes to CLI, it may become even more confusing
to have per sensor access for parsers and as a whole for the rest.

On Sat, Dec 1, 2018 at 1:36 AM Justin Leet  wrote:

> To start with, I'm thinking just the configuration, in particular anything
> that touches the ConfigurationsUtils.  I think for the first pass, we could
> leave reading from configs out of it and focus on who can write configs
> (since that's more disruptive to the system).
>
> To the best of my knowledge, this is:
>
>- Configuration through the Management UI
>- Configuration through the CLI (e.g. the push script + Stellar + any
>other scripts that have to do it).
>
> In terms of how fine-grained, I would essentially expect this to be at the
> users+groups level and the individual sensor / topology level (e.g. parsers
> can be authorized individually by users/group, indexing is on the whole).
> Essentially, I don't propose trying to rearchitect things to try to have
> multiple indexing topologies for complete separation, etc.  Just to take
> the existing setup and be able to apply authorization on top of it.
>
> I would expect anything covered by the ZK configs to be done as part of
> this effort (possibly in stages).  As noted, I would expect this to be a
> feature branch rather than piecemeal replacement.
>
> @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
> pending this exact conversation winding down.
>
> On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Justin, It probably makes sense to provide a list of these configuration
> > items as subtasks on the FB Jira so that we can crosscheck what entry
> > points have been implemented against the test scripts. Do you think this
> > will impact streaming enrichments or the profiler at all? That is to say,
> > as Ali asked, just how far are you looking to take the fine grained auth
> > scope for this?
> >
> > M
> >
> > On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> > wrote:
> >
> > > Hi Justin,
> > >
> > > By configuration do you mean the sensor related configurations only?
> Are
> > > you limiting the scope of this activity to the management-UI or also
> > > Alert-UI as well? For example, defining different roles (pre-defined
> > > or customizable) and the fine-grained integration with Ranger?
> > >
> > > Cheers,
> > > Ali
> > >
> > > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Right now, our various configs can be modified by anyone with access
> to
> > > the
> > > > various scripts. I'd like to start a discussion around building out
> > some
> > > > authorization to be able to add some more fine grained controls
> around
> > > > this.
> > > >
> > > > Other projects have some variants on how to accomplish this.
> > Typically,
> > > > this follows a pattern of calling out to a interface/class that takes
> > in
> > > > the operation and context (user and other info) and returns
> true/false
> > if
> > > > something is authorized.
> > > >
> > > > In my mind, what we would need out of this is
> > > >
> > > >- Ability to apply fine grained permissions
> > > >- The various scripts and UI should flow this authorization
> > framework.
> > > >I believe most (if not all) of our configuration flows through
> > > >ConfigurationsUtils.  Anything that doesn't should either be
> hooked
> > in
> > > > or
> > > >refactored to flow through the same codepaths.
> > > >- Pluggability. We shouldn't force only one authorizer.
> > > >
> > > > In particular, I'm proposing we use Apache Ranger
> > > >  as a supported authorization framework,
> > > > implementing it alongside the authorization framework to validate
> what
> > we
> > > > build. In my mind, the main catch with Ranger is that, based on my
> > > > understanding, we won't be able to restrict users with direct access
> to
> > > > ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into
> > ZK's
> > > > ACLs).  I believe this is a reasonable restriction, especially as the
> > > > management UI gets improved to handle more of the configuration
> burden
> > > and
> > > > the number of users with access to ZK CLI begins to decrease.  Users
> > can
> > > > still add ZK ACLs separately to

Re: Authorization for Configuration

[Justin Leet]

To start with, I'm thinking just the configuration, in particular anything
that touches the ConfigurationsUtils.  I think for the first pass, we could
leave reading from configs out of it and focus on who can write configs
(since that's more disruptive to the system).

To the best of my knowledge, this is:

   - Configuration through the Management UI
   - Configuration through the CLI (e.g. the push script + Stellar + any
   other scripts that have to do it).

In terms of how fine-grained, I would essentially expect this to be at the
users+groups level and the individual sensor / topology level (e.g. parsers
can be authorized individually by users/group, indexing is on the whole).
Essentially, I don't propose trying to rearchitect things to try to have
multiple indexing topologies for complete separation, etc.  Just to take
the existing setup and be able to apply authorization on top of it.

I would expect anything covered by the ZK configs to be done as part of
this effort (possibly in stages).  As noted, I would expect this to be a
feature branch rather than piecemeal replacement.

@Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
pending this exact conversation winding down.

On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Justin, It probably makes sense to provide a list of these configuration
> items as subtasks on the FB Jira so that we can crosscheck what entry
> points have been implemented against the test scripts. Do you think this
> will impact streaming enrichments or the profiler at all? That is to say,
> as Ali asked, just how far are you looking to take the fine grained auth
> scope for this?
>
> M
>
> On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> wrote:
>
> > Hi Justin,
> >
> > By configuration do you mean the sensor related configurations only? Are
> > you limiting the scope of this activity to the management-UI or also
> > Alert-UI as well? For example, defining different roles (pre-defined
> > or customizable) and the fine-grained integration with Ranger?
> >
> > Cheers,
> > Ali
> >
> > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> wrote:
> >
> > > Hi all,
> > >
> > > Right now, our various configs can be modified by anyone with access to
> > the
> > > various scripts. I'd like to start a discussion around building out
> some
> > > authorization to be able to add some more fine grained controls around
> > > this.
> > >
> > > Other projects have some variants on how to accomplish this.
> Typically,
> > > this follows a pattern of calling out to a interface/class that takes
> in
> > > the operation and context (user and other info) and returns true/false
> if
> > > something is authorized.
> > >
> > > In my mind, what we would need out of this is
> > >
> > >- Ability to apply fine grained permissions
> > >- The various scripts and UI should flow this authorization
> framework.
> > >I believe most (if not all) of our configuration flows through
> > >ConfigurationsUtils.  Anything that doesn't should either be hooked
> in
> > > or
> > >refactored to flow through the same codepaths.
> > >- Pluggability. We shouldn't force only one authorizer.
> > >
> > > In particular, I'm proposing we use Apache Ranger
> > >  as a supported authorization framework,
> > > implementing it alongside the authorization framework to validate what
> we
> > > build. In my mind, the main catch with Ranger is that, based on my
> > > understanding, we won't be able to restrict users with direct access to
> > > ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into
> ZK's
> > > ACLs).  I believe this is a reasonable restriction, especially as the
> > > management UI gets improved to handle more of the configuration burden
> > and
> > > the number of users with access to ZK CLI begins to decrease.  Users
> can
> > > still add ZK ACLs separately to enforce that access there.
> > >
> > > For anyone not familiar with Ranger, essentially you build a plugin
> that
> > > hooks into the existing component's authorization framework (e.g. for
> > > Storm, the plugin runs through the IAuthorizer
> > > <
> > >
> >
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> > > >
> > > interface, for Yarn it runs through YarnAuthorizationProvider
> > > <
> > >
> >
> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/security/YarnAuthorizationProvider.html
> > > >).
> > > Additionally, Ranger provides auditing capabilities for this
> > authorization
> > > and has plugins for a good portion of our stack already (so users can
> > > already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
> > >  for a list of the plugins they have
> > > built in.
> > >
> > > What this means for Metron is building out an authorization setup
> similar
> > > to Storm or Yarn or whatever we c

Re: Authorization for Configuration

[Michael Miklavcic]

Justin, It probably makes sense to provide a list of these configuration
items as subtasks on the FB Jira so that we can crosscheck what entry
points have been implemented against the test scripts. Do you think this
will impact streaming enrichments or the profiler at all? That is to say,
as Ali asked, just how far are you looking to take the fine grained auth
scope for this?

M

On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian  wrote:

> Hi Justin,
>
> By configuration do you mean the sensor related configurations only? Are
> you limiting the scope of this activity to the management-UI or also
> Alert-UI as well? For example, defining different roles (pre-defined
> or customizable) and the fine-grained integration with Ranger?
>
> Cheers,
> Ali
>
> On Wed, Nov 14, 2018 at 1:25 AM Justin Leet  wrote:
>
> > Hi all,
> >
> > Right now, our various configs can be modified by anyone with access to
> the
> > various scripts. I'd like to start a discussion around building out some
> > authorization to be able to add some more fine grained controls around
> > this.
> >
> > Other projects have some variants on how to accomplish this.  Typically,
> > this follows a pattern of calling out to a interface/class that takes in
> > the operation and context (user and other info) and returns true/false if
> > something is authorized.
> >
> > In my mind, what we would need out of this is
> >
> >- Ability to apply fine grained permissions
> >- The various scripts and UI should flow this authorization framework.
> >I believe most (if not all) of our configuration flows through
> >ConfigurationsUtils.  Anything that doesn't should either be hooked in
> > or
> >refactored to flow through the same codepaths.
> >- Pluggability. We shouldn't force only one authorizer.
> >
> > In particular, I'm proposing we use Apache Ranger
> >  as a supported authorization framework,
> > implementing it alongside the authorization framework to validate what we
> > build. In my mind, the main catch with Ranger is that, based on my
> > understanding, we won't be able to restrict users with direct access to
> > ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
> > ACLs).  I believe this is a reasonable restriction, especially as the
> > management UI gets improved to handle more of the configuration burden
> and
> > the number of users with access to ZK CLI begins to decrease.  Users can
> > still add ZK ACLs separately to enforce that access there.
> >
> > For anyone not familiar with Ranger, essentially you build a plugin that
> > hooks into the existing component's authorization framework (e.g. for
> > Storm, the plugin runs through the IAuthorizer
> > <
> >
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> > >
> > interface, for Yarn it runs through YarnAuthorizationProvider
> > <
> >
> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/security/YarnAuthorizationProvider.html
> > >).
> > Additionally, Ranger provides auditing capabilities for this
> authorization
> > and has plugins for a good portion of our stack already (so users can
> > already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
> >  for a list of the plugins they have
> > built in.
> >
> > What this means for Metron is building out an authorization setup similar
> > to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
> > our solution to be pluggable.  At that point, we build an implementation
> of
> > the authorizer compatible with Ranger along with the plugin.
> >
> > I think this could probably be a fairly small feature branch, which I'm
> > suggesting primarily to do the Ranger implementation alongside the
> general
> > authorization work to validate what's being built.  I think the main
> > tasking would be something similar to:
> >
> >- Build out pluggable authorization for our configs.
> >- This includes testing (and possibly doing something similar to
> Storm,
> >where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
> >DenyAuthorizer, etc.)
> >- Ensure that all the code paths consistently flow through this
> >Authorization.
> >- Build a Ranger compatible version of this.
> >- Define the Ranger plugin for this.
> >- Make sure auditing is defined.
> >- Integration testing (particularly with Kerberos. After all, if they
> >want to do authorization and auditing, they're almost certainly using
> >Kerberos).
> >
> > Is there anything missing that we'd need or want for this?  Are there any
> > other concerns we'd want to make sure are taken care of?
> >
>
>
> --
> A.Nazemian
>

Re: [DISCUSS] Authorization for Configuration

[Ali Nazemian]

Hi Justin,

By configuration do you mean the sensor related configurations only? Are
you limiting the scope of this activity to the management-UI or also
Alert-UI as well? For example, defining different roles (pre-defined
or customizable) and the fine-grained integration with Ranger?

Cheers,
Ali

On Thu, Nov 15, 2018 at 1:16 AM Justin Leet  wrote:

> Hi all,
>
> Sorry for the second copy of this email, I forgot the  [DISCUSS] tag on the
> original.  Otherwise, this has the same content.
>
> Right now, our various configs can be modified by anyone with access to the
> various scripts. I'd like to start a discussion around building out some
> authorization to be able to add some more fine grained controls around
> this.
>
> Other projects have some variants on how to accomplish this.  Typically,
> this follows a pattern of calling out to a interface/class that takes in
> the operation and context (user and other info) and returns true/false if
> something is authorized.
>
> In my mind, what we would need out of this is
>
>- Ability to apply fine grained permissions
>- The various scripts and UI should flow this authorization framework.
>I believe most (if not all) of our configuration flows through
>ConfigurationsUtils.  Anything that doesn't should either be hooked in
> or
>refactored to flow through the same codepaths.
>- Pluggability. We shouldn't force only one authorizer.
>
> In particular, I'm proposing we use Apache Ranger
>  as a supported authorization framework,
> implementing it alongside the authorization framework to validate what we
> build. In my mind, the main catch with Ranger is that, based on my
> understanding, we won't be able to restrict users with direct access to
> ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
> ACLs).  I believe this is a reasonable restriction, especially as the
> management UI gets improved to handle more of the configuration burden and
> the number of users with access to ZK CLI begins to decrease.  Users can
> still add ZK ACLs separately to enforce that access there.
>
> For anyone not familiar with Ranger, essentially you build a plugin that
> hooks into the existing component's authorization framework (e.g. for
> Storm, the plugin runs through the IAuthorizer
> <
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> >
> interface,
> for Yarn it runs through YarnAuthorizationProvider
> <
> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/security/YarnAuthorizationProvider.html
> >).
> Additionally, Ranger provides auditing capabilities for this authorization
> and has plugins for a good portion of our stack already (so users can
> already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
>  for a list of the plugins they have
> built in.
>
> What this means for Metron is building out an authorization setup similar
> to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
> our solution to be pluggable.  At that point, we build an implementation of
> the authorizer compatible with Ranger along with the plugin.
>
> I think this could probably be a fairly small feature branch, which I'm
> suggesting primarily to do the Ranger implementation alongside the general
> authorization work to validate what's being built.  I think the main
> tasking would be something similar to:
>
>- Build out pluggable authorization for our configs.
>- This includes testing (and possibly doing something similar to Storm,
>where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
>DenyAuthorizer, etc.)
>- Ensure that all the code paths consistently flow through this
>Authorization.
>- Build a Ranger compatible version of this.
>- Define the Ranger plugin for this.
>- Make sure auditing is defined.
>- Integration testing (particularly with Kerberos. After all, if they
>want to do authorization and auditing, they're almost certainly using
>Kerberos).
>
> Is there anything missing that we'd need or want for this?  Are there any
> other concerns we'd want to make sure are taken care of?
>


-- 
A.Nazemian

Re: Authorization for Configuration

[Ali Nazemian]

Hi Justin,

By configuration do you mean the sensor related configurations only? Are
you limiting the scope of this activity to the management-UI or also
Alert-UI as well? For example, defining different roles (pre-defined
or customizable) and the fine-grained integration with Ranger?

Cheers,
Ali

On Wed, Nov 14, 2018 at 1:25 AM Justin Leet  wrote:

> Hi all,
>
> Right now, our various configs can be modified by anyone with access to the
> various scripts. I'd like to start a discussion around building out some
> authorization to be able to add some more fine grained controls around
> this.
>
> Other projects have some variants on how to accomplish this.  Typically,
> this follows a pattern of calling out to a interface/class that takes in
> the operation and context (user and other info) and returns true/false if
> something is authorized.
>
> In my mind, what we would need out of this is
>
>- Ability to apply fine grained permissions
>- The various scripts and UI should flow this authorization framework.
>I believe most (if not all) of our configuration flows through
>ConfigurationsUtils.  Anything that doesn't should either be hooked in
> or
>refactored to flow through the same codepaths.
>- Pluggability. We shouldn't force only one authorizer.
>
> In particular, I'm proposing we use Apache Ranger
>  as a supported authorization framework,
> implementing it alongside the authorization framework to validate what we
> build. In my mind, the main catch with Ranger is that, based on my
> understanding, we won't be able to restrict users with direct access to
> ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
> ACLs).  I believe this is a reasonable restriction, especially as the
> management UI gets improved to handle more of the configuration burden and
> the number of users with access to ZK CLI begins to decrease.  Users can
> still add ZK ACLs separately to enforce that access there.
>
> For anyone not familiar with Ranger, essentially you build a plugin that
> hooks into the existing component's authorization framework (e.g. for
> Storm, the plugin runs through the IAuthorizer
> <
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> >
> interface, for Yarn it runs through YarnAuthorizationProvider
> <
> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/security/YarnAuthorizationProvider.html
> >).
> Additionally, Ranger provides auditing capabilities for this authorization
> and has plugins for a good portion of our stack already (so users can
> already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
>  for a list of the plugins they have
> built in.
>
> What this means for Metron is building out an authorization setup similar
> to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
> our solution to be pluggable.  At that point, we build an implementation of
> the authorizer compatible with Ranger along with the plugin.
>
> I think this could probably be a fairly small feature branch, which I'm
> suggesting primarily to do the Ranger implementation alongside the general
> authorization work to validate what's being built.  I think the main
> tasking would be something similar to:
>
>- Build out pluggable authorization for our configs.
>- This includes testing (and possibly doing something similar to Storm,
>where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
>DenyAuthorizer, etc.)
>- Ensure that all the code paths consistently flow through this
>Authorization.
>- Build a Ranger compatible version of this.
>- Define the Ranger plugin for this.
>- Make sure auditing is defined.
>- Integration testing (particularly with Kerberos. After all, if they
>want to do authorization and auditing, they're almost certainly using
>Kerberos).
>
> Is there anything missing that we'd need or want for this?  Are there any
> other concerns we'd want to make sure are taken care of?
>


-- 
A.Nazemian

[DISCUSS] Authorization for Configuration

[Justin Leet]

Hi all,

Sorry for the second copy of this email, I forgot the  [DISCUSS] tag on the
original.  Otherwise, this has the same content.

Right now, our various configs can be modified by anyone with access to the
various scripts. I'd like to start a discussion around building out some
authorization to be able to add some more fine grained controls around this.

Other projects have some variants on how to accomplish this.  Typically,
this follows a pattern of calling out to a interface/class that takes in
the operation and context (user and other info) and returns true/false if
something is authorized.

In my mind, what we would need out of this is

   - Ability to apply fine grained permissions
   - The various scripts and UI should flow this authorization framework.
   I believe most (if not all) of our configuration flows through
   ConfigurationsUtils.  Anything that doesn't should either be hooked in or
   refactored to flow through the same codepaths.
   - Pluggability. We shouldn't force only one authorizer.

In particular, I'm proposing we use Apache Ranger
 as a supported authorization framework,
implementing it alongside the authorization framework to validate what we
build. In my mind, the main catch with Ranger is that, based on my
understanding, we won't be able to restrict users with direct access to
ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
ACLs).  I believe this is a reasonable restriction, especially as the
management UI gets improved to handle more of the configuration burden and
the number of users with access to ZK CLI begins to decrease.  Users can
still add ZK ACLs separately to enforce that access there.

For anyone not familiar with Ranger, essentially you build a plugin that
hooks into the existing component's authorization framework (e.g. for
Storm, the plugin runs through the IAuthorizer

interface,
for Yarn it runs through YarnAuthorizationProvider
).
Additionally, Ranger provides auditing capabilities for this authorization
and has plugins for a good portion of our stack already (so users can
already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
 for a list of the plugins they have
built in.

What this means for Metron is building out an authorization setup similar
to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
our solution to be pluggable.  At that point, we build an implementation of
the authorizer compatible with Ranger along with the plugin.

I think this could probably be a fairly small feature branch, which I'm
suggesting primarily to do the Ranger implementation alongside the general
authorization work to validate what's being built.  I think the main
tasking would be something similar to:

   - Build out pluggable authorization for our configs.
   - This includes testing (and possibly doing something similar to Storm,
   where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
   DenyAuthorizer, etc.)
   - Ensure that all the code paths consistently flow through this
   Authorization.
   - Build a Ranger compatible version of this.
   - Define the Ranger plugin for this.
   - Make sure auditing is defined.
   - Integration testing (particularly with Kerberos. After all, if they
   want to do authorization and auditing, they're almost certainly using
   Kerberos).

Is there anything missing that we'd need or want for this?  Are there any
other concerns we'd want to make sure are taken care of?

Authorization for Configuration

[Justin Leet]

Hi all,

Right now, our various configs can be modified by anyone with access to the
various scripts. I'd like to start a discussion around building out some
authorization to be able to add some more fine grained controls around this.

Other projects have some variants on how to accomplish this.  Typically,
this follows a pattern of calling out to a interface/class that takes in
the operation and context (user and other info) and returns true/false if
something is authorized.

In my mind, what we would need out of this is

   - Ability to apply fine grained permissions
   - The various scripts and UI should flow this authorization framework.
   I believe most (if not all) of our configuration flows through
   ConfigurationsUtils.  Anything that doesn't should either be hooked in or
   refactored to flow through the same codepaths.
   - Pluggability. We shouldn't force only one authorizer.

In particular, I'm proposing we use Apache Ranger
 as a supported authorization framework,
implementing it alongside the authorization framework to validate what we
build. In my mind, the main catch with Ranger is that, based on my
understanding, we won't be able to restrict users with direct access to
ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
ACLs).  I believe this is a reasonable restriction, especially as the
management UI gets improved to handle more of the configuration burden and
the number of users with access to ZK CLI begins to decrease.  Users can
still add ZK ACLs separately to enforce that access there.

For anyone not familiar with Ranger, essentially you build a plugin that
hooks into the existing component's authorization framework (e.g. for
Storm, the plugin runs through the IAuthorizer

interface, for Yarn it runs through YarnAuthorizationProvider
).
Additionally, Ranger provides auditing capabilities for this authorization
and has plugins for a good portion of our stack already (so users can
already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
 for a list of the plugins they have
built in.

What this means for Metron is building out an authorization setup similar
to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
our solution to be pluggable.  At that point, we build an implementation of
the authorizer compatible with Ranger along with the plugin.

I think this could probably be a fairly small feature branch, which I'm
suggesting primarily to do the Ranger implementation alongside the general
authorization work to validate what's being built.  I think the main
tasking would be something similar to:

   - Build out pluggable authorization for our configs.
   - This includes testing (and possibly doing something similar to Storm,
   where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
   DenyAuthorizer, etc.)
   - Ensure that all the code paths consistently flow through this
   Authorization.
   - Build a Ranger compatible version of this.
   - Define the Ranger plugin for this.
   - Make sure auditing is defined.
   - Integration testing (particularly with Kerberos. After all, if they
   want to do authorization and auditing, they're almost certainly using
   Kerberos).

Is there anything missing that we'd need or want for this?  Are there any
other concerns we'd want to make sure are taken care of?