Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[Justin Leet]

The vote has failed. The voting was:
1 -1’s (binding)

A new RC will be created once this issue is resolved.

On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com  wrote:

> -1
>
> In my testing it appears that an issue was introduced in 0.2 which is
> causing a segfault on the destructor (
>
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> ).
> I've opened METRON-1910 and am testing a fix now.
>
> On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
>
> > This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> > The release candidate is available at:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> >
> > Full list of changes in this release:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> >
> > The tag to be voted upon is:
> > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > <
> >
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > >
> >
> > The source archives being voted upon can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> >
> > The release artifacts are signed with the following key:
> > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > 
> > Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> > 0.3.0
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> > This is only for the plugin, so the VM validation is not required.
> >
> > This vote will be open for until 3pm EDT on Friday November 27 2018.
> >
> > [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> >
> > [ ] 0 No opinion
> >
> > [ ] -1 Do not release this package because...
> >
> --
>
> Jon Zeolla
>

Re: Authorization for Configuration

[Justin Leet]

To start with, I'm thinking just the configuration, in particular anything
that touches the ConfigurationsUtils.  I think for the first pass, we could
leave reading from configs out of it and focus on who can write configs
(since that's more disruptive to the system).

To the best of my knowledge, this is:

   - Configuration through the Management UI
   - Configuration through the CLI (e.g. the push script + Stellar + any
   other scripts that have to do it).

In terms of how fine-grained, I would essentially expect this to be at the
users+groups level and the individual sensor / topology level (e.g. parsers
can be authorized individually by users/group, indexing is on the whole).
Essentially, I don't propose trying to rearchitect things to try to have
multiple indexing topologies for complete separation, etc.  Just to take
the existing setup and be able to apply authorization on top of it.

I would expect anything covered by the ZK configs to be done as part of
this effort (possibly in stages).  As noted, I would expect this to be a
feature branch rather than piecemeal replacement.

@Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
pending this exact conversation winding down.

On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Justin, It probably makes sense to provide a list of these configuration
> items as subtasks on the FB Jira so that we can crosscheck what entry
> points have been implemented against the test scripts. Do you think this
> will impact streaming enrichments or the profiler at all? That is to say,
> as Ali asked, just how far are you looking to take the fine grained auth
> scope for this?
>
> M
>
> On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> wrote:
>
> > Hi Justin,
> >
> > By configuration do you mean the sensor related configurations only? Are
> > you limiting the scope of this activity to the management-UI or also
> > Alert-UI as well? For example, defining different roles (pre-defined
> > or customizable) and the fine-grained integration with Ranger?
> >
> > Cheers,
> > Ali
> >
> > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> wrote:
> >
> > > Hi all,
> > >
> > > Right now, our various configs can be modified by anyone with access to
> > the
> > > various scripts. I'd like to start a discussion around building out
> some
> > > authorization to be able to add some more fine grained controls around
> > > this.
> > >
> > > Other projects have some variants on how to accomplish this.
> Typically,
> > > this follows a pattern of calling out to a interface/class that takes
> in
> > > the operation and context (user and other info) and returns true/false
> if
> > > something is authorized.
> > >
> > > In my mind, what we would need out of this is
> > >
> > >- Ability to apply fine grained permissions
> > >- The various scripts and UI should flow this authorization
> framework.
> > >I believe most (if not all) of our configuration flows through
> > >ConfigurationsUtils.  Anything that doesn't should either be hooked
> in
> > > or
> > >refactored to flow through the same codepaths.
> > >- Pluggability. We shouldn't force only one authorizer.
> > >
> > > In particular, I'm proposing we use Apache Ranger
> > >  as a supported authorization framework,
> > > implementing it alongside the authorization framework to validate what
> we
> > > build. In my mind, the main catch with Ranger is that, based on my
> > > understanding, we won't be able to restrict users with direct access to
> > > ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into
> ZK's
> > > ACLs).  I believe this is a reasonable restriction, especially as the
> > > management UI gets improved to handle more of the configuration burden
> > and
> > > the number of users with access to ZK CLI begins to decrease.  Users
> can
> > > still add ZK ACLs separately to enforce that access there.
> > >
> > > For anyone not familiar with Ranger, essentially you build a plugin
> that
> > > hooks into the existing component's authorization framework (e.g. for
> > > Storm, the plugin runs through the IAuthorizer
> > > <
> > >
> >
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> > > >
> > > interface, for Yarn it runs through YarnAuthorizationProvider
> > > <
> > >
> >
> http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/security/YarnAuthorizationProvider.html
> > > >).
> > > Additionally, Ranger provides auditing capabilities for this
> > authorization
> > > and has plugins for a good portion of our stack already (so users can
> > > already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
> > >  for a list of the plugins they have
> > > built in.
> > >
> > > What this means for Metron is building out an authorization setup
> similar
> > > to Storm or Yarn or whatever we