Re: [DISCUSS] Storm topology.classpath setting

[Justin Leet]

topology.classpath is a topology level config, so Mike's potential
alternative is to migrate that down from Ambari into the actual topologies
we launch (assuming it works as expected).

Basically we're trading ease of use in Ambari vs. additional setup for each
topology that's run.

I'm in favor of pushing the config down to each topology (again, assuming
it works), despite the additional topology work, because it seems more
likely that users are going to see the issues from not restarting Storm and
Metron immediately after install/start than they are from missing configs
at the Flux level.  The topology config just becomes a bullet point in
instructions on creating a new topology, rather than a surprising and
nonintuitive issue resulting from install.

Once Metron installs and starts through Ambari, I would expect it to work
immediately.

On Tue, Apr 25, 2017 at 5:14 PM, Casey Stella  wrote:

> I don't know that I'm offended by either approach, but the way we have it
> now isn't offending me overly, honestly.
>
> Casey
>
> On Tue, Apr 25, 2017 at 5:11 PM, David Lyle  wrote:
>
> > I think you have one more restart than is strictly required but I get
> your
> > point. IIRC, when I did the original implementation there was a bit of a
> > wrinkle getting the right configs on the right classpath so I punted in
> > favor of topology.classpath.
> >
> > If you've got a more efficient way, I'm all for it.
> >
> > -D...
> >
> >
> > On Tue, Apr 25, 2017 at 4:55 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > We currently define topology.classpath at the global level for Storm.
> The
> > > reason for this feature was to enable setting the classpath to include
> > > things like hbase-site and core-site without modifying the topology jar
> > > files on the cluster. That made things simpler, but this also has the
> > > consequence that Metron's installation process via Ambari looks
> something
> > > like this:
> > >
> > >1. Install Metron
> > >2. Start Metron - part of install process
> > >3. Restart Storm  - bc of the new property that has been added that
> > >needs to be distributed
> > >4. Restart Metron - the topologies won't get the new classpath
> > otherwise
> > >
> > > We're in effect restarting Metron a couple times, which can take some
> > time.
> > >
> > > A possible alternative here is to set the classpath on the individual
> > > topologies. By doing so, we remove the need to restart Storm and Metron
> > > during the install. The down side is that this requires us to duplicate
> > > this setting for every flux file or topology, including newly-added
> > > topologies. What do folks think about which would be better?
> > >
> > > Best,
> > > Mike
> > >
> >
>

Re: [DISCUSS] Modify bylaws to allow speculative branches

[Justin Leet]

I'd prefer to still keep "Review then Commit" but I'd be interested in
lowering the review bar for it as long as the branch gets a solid final
review before merge.  I don't think a feature branch needs to be
micromanaged, but people should still take a quick glance at the commits
that come through. My stance on this may change if we find feature commits
stagnating for periods of time.

Basically, what I'd be looking for out of a feature branch is something
that's:
* Implementing a feature too large or with too much experimentation for a
single PR.
* Is at least looked at semi-regularly to have the opportunity for the
community to point out issues or scalability concerns without minor
concerns being blockers to progress. Otherwise, it's no different from a
giant PR that doesn't get examined until the PR is made.
* Reviews should only very rarely be blockers to progress continuing along
the branch. The branch shouldn't grind to a halt periodically or even
significantly slow progress to cleanup syntax or similar things.  The
tradeoff for this is I'd expect refactoring in a lot of cases before it
hits master to catch up on any of these things that we do care about, but
that aren't as important while the branch gets built up initially.
* Kept reasonably up to date with master. I really don't want a feature
branch sitting out for three months without being synced and then suddenly
everything explodes when we try to bring it back. It just causes all sorts
of review and testing issues.
* Ideally managed so that squashing multiple people's commits appropriately
is not a horrible nightmare.
* (Hopefully) able to be merged into master more smoothly as it's gotten
the major concerns cleaned out along the way.

Ideally, we also use this sort of framework to avoid some of the large PRs
we've seen around different features that felt very hard to get a grasp of
and review.

I'm pretty ambivalent on branch committers, and I'd be curious to get some
opinions (or possibly experiences if anyone has some) before I form a solid
opinion.


On Mon, Apr 24, 2017 at 12:01 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> "Branch committers ... do not cast binding votes or vetoes in the project."
> It sounds similar to what we do for other PRs except in this case it's 1..n
> contributors instead of just one. The step for getting the feature branch
> merged into trunk sounds reasonable and clear enough, but as Casey
> mentioned I'm also curious what the commit/review lifecycle looks like
> while the feature branch is active.
>
> On Fri, Apr 21, 2017 at 4:00 PM, Casey Stella  wrote:
>
> > So, what would that look like from a practical perspective?
> > * I presume commits would still associate to a JIRA, right?
> > * Are you proposing changing the strategy from Review then Commit to
> Commit
> > then Review for these branches?
> >
> > I know that we have some people who are active in the Hadoop project on
> the
> > PMC as mentors or are in more established projects with this kind of
> thing,
> > can you guys chime in about how that looks like and make some suggestions
> > as to some gotchas?
> >
> > Casey
> >
> > On Fri, Apr 21, 2017 at 5:19 PM, James Sirota 
> wrote:
> >
> > > Looking at Hadoop's bylaws
> > >
> > > https://hadoop.apache.org/bylaws.html
> > >
> > > They have this:
> > >
> > > Significant, pervasive features are often developed in a speculative
> > > branch of the repository. The PMC may grant commit rights on the branch
> > to
> > > its consistent contributors, while the initiative is active. Branch
> > > committers are responsible for shepherding their feature into an active
> > > release and do not cast binding votes or vetoes in the project.
> > >
> > > I would like to have something similar in our bylaws where we can have
> > > feature branches with their own set of committers.  We may have
> > > prototype-grade features that need to be rapidly evolved with the
> > feedback
> > > from the community.  So we would need a lighter process to evolve these
> > > features rapidly, while still keeping the community involved.  A more
> > > rigorous review process can then be applied when the feature is more
> > mature
> > > and is ready to be committed into master.
> > >
> > > What do you think?
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> >
>

Re: Reducing Warnings in Build

[Justin Leet]

First off, I would absolutely love to see the warnings reduced and the
quality of our code improved.  I'm in favor of all four of the points, and
I think it's a good start towards weeding out a lot of issues.

So regarding question 1:

That awkwardness comes about because org.simple.json is all untyped Maps
under the hood (
https://github.com/fangyidong/json-simple/blob/master/src/main/java/org/json/simple/JSONObject.java#L19).
So anything that touches those classes starts requiring casts everywhere. A
couple months ago a PR was made against the library that seems to have
added a lot of this stuff (
https://github.com/fangyidong/json-simple/pull/113).  Basically, it was
forked (https://github.com/cliftonlabs/json-simple and
https://cliftonlabs.github.io/json-simple/) and they tried to contribute it
back and it's been sitting there.

I strongly think we should consider swapping over to the fork.  We'd have
to do a bit of research, but it seems the intent was to be completely
backwards compatible while updating things like generics and some various
issues they presumably saw with the original.

For 2, and I'm not an expert in this by any means, but I would be in favor
of defaulting to UTF-8.  It's most likely what's happening under the hood
anyway (even though it's technically platform dependent), so defaulting
probably makes it both more explicit and removes potential for issues if a
given system has a different underlying default.

Justin

On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:

> Hello everybody,
>
> Currently our build has a significant amount of warnings (most from the
> error prone plugin). A lot of this has been documented here:
> https://issues.apache.org/jira/browse/METRON-617
>
> I want to continue to work on this Jira. I really want to reduce the number
> of warnings in our build. As the Jira points out, a lot of them are
> unchecked casts or the implicit use of default charset.
>
> When starting this, I want to start with a specific module. I plan on
> starting with `metron-interface` as it's fairly self contained and is
> pretty new. Below I want to layout what I plan on doing. Please let me know
> what you all think:
>
> 1. Suppress warnings where generics are not supported or checking type is
> not possible.
> 2. For all unit tests that require supplying a charset I'll supply the
> UTF-8 charset.
> 3. Update the API to have a charset configuration for each resource that
> needs one.
> 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> prone doesn't support this level of suppression. Which is probably a good
> thing. We will need to explicitly state what we want to suppress instead.
>
> Two big questions came up right away when I was thinking about the above:
>
> 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> Map. I know it extends Map, but is it really safe to do
> something like the following? Should we have a utility that truly builds a
> map that uses generics? I plan on doing some testing around this, but if
> anyone has any experience with this it would be greatly appreciated. Here
> is an example of what I am talking about:
>
> JSONObject message = ...;
> @SuppressWarnings("unchecked")
> Map state = (Map) message;
>
>
> 2. This one is about configuring charset (#3 above). Specifically in
> metron-rest. Right now, I believe there are 3 sensor resources (index,
> enrichment, and parser) that throw warnings due to implicitly using the
> default charset. I propose that we have 3 configs (listed below). These
> configs would take any valid charset, default, or not set. If not set then
> UTF-8 would be default. Does this seem fair?
>
> sensor:
>   index.encoding: UTF-8
>   enrichment.encoding: UTF-8
>   parser.encoding: UTF-8
>
>
> Does anyone see any problems that may occur if we go about it this way? Any
> help on this would be very much appreciated.
>
> Thanks,
> JJ
>

Re: [DISCUSS] next release proposal

[Justin Leet]

METRON-799 is in master.  I know at least METRON-859 now has a conflict on
Kerberos-setup.md, so if anyone has anything similar, please make sure to
update your branch.

On Thu, Apr 20, 2017 at 9:25 AM, Otto Fowler 
wrote:

> Maybe in the next release we should have theme labels for the release in
> jira
>
>
> On April 20, 2017 at 09:24:14, Casey Stella (ceste...@gmail.com) wrote:
>
> +1, I agree. We should see this through. I will point out also that while
> not strictly required, METRON-861 (
> https://github.com/apache/incubator-metron/pull/534) would be required to
> use the CLI utilities if people are planning on setting acl's on the znodes
> for our config in their installation in a kerberized environment. I
> mention it only because it doesn't appear to be kerberos related, but it
> was inspired by a kerberos-related problem.
>
> Casey
>
> On Thu, Apr 20, 2017 at 9:17 AM, Otto Fowler 
> wrote:
>
> > +1 on the proposed changes.
> >
> > I think that it is important that if it is in reach, we ‘complete’ any
> > major theme of the release.
> > In this case kerberos. I think that given where pycapa and rest are, we
> > should make the effort
> > to get them in.
> >
> >
> >
> > On April 20, 2017 at 08:52:11, Ryan Merriman (merrim...@gmail.com)
> wrote:
> >
> > Matt,
> >
> > I started a discussion around Kerberos support as a prerequisite for
> MPack
> > work and the consensus was that a service should support Kerberos before
> > it's included in the MPack. There is a PR out there for Kerberos support
> > for REST (https://github.com/apache/incubator-metron/pull/535) but it
> has
> > not been reviewed yet. There is also likely a small amount of work to be
> > done once METRON-799 is accepted. Because of these 2 dependencies I think
> > it's probably best to wait on METRON-795.
> >
> > Ryan
> >
> > On Thu, Apr 20, 2017 at 1:13 AM, Matt Foley  wrote:
> >
> > > Hi all,
> > > I’ve put together RC1 for the 0.4.0 release of Metron, along with its
> > > book-site. It is available for your review at
> > > https://dist.apache.org/repos/dist/dev/incubator/metron/0.4.
> > > 0-RC1-incubating/
> > >
> > > I’m not putting it to VOTE yet, because I think some additional fixes
> are
> > > probably necessary:
> > > • We should add documentation for the remaining backward-incompatible
> > > changes
> > > • We should add these important bug fixes that have been committed to
> > > master since the 0.4.0 branch was cut:
> > > o METRON-634 fixes for Mpack for Centos7
> > > o METRON-856 Ansible rpm build wipes out prior binary build
> > > o METRON-821 Minor fixes in full dev kerberos setup instructions
> > > o Please give me your +1 for these additions
> > > • These PRs are currently open, and seem important to complete the
> > > Kerberos picture:
> > > o METRON-799: The MPack should function in a kerberized cluster
> > > o METRON-835 Use Profiler with Kerberos
> > > o METRON-836 Use Pycapa with Kerberos
> > > o METRON-859 Use REST application with Kerberos
> > > o Please give me your evaluation of whether these can be committed
> > > Real Soon Now, or we should not wait for them.
> > > • This PR is open and seems important to complete the REST picture:
> > > o METRON-795: Install Metron REST with Ambari MPack
> > > o Please give me your evaluation of whether this can be committed Real
> > > Soon Now, or we should not wait for it.
> > > • Anything else? I’ve deliberately left out the commits on master that
> > > represent new functionality not already in (or mostly in) the 0.4.0
> > branch.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 4/18/17, 5:30 PM, "Matt Foley"  wrote:
> > >
> > > Thanks, we’re now up to 4 backward-incompatible issues. Any others
> > > should be so marked?
> > >
> > > On 4/17/17, 4:43 PM, "Matt Foley"  wrote:
> > >
> > > Hi all,
> > > Out of the 58 Jiras resolved, completely or partially, between
> > > 0.3.1 and 0.4.0, only one is labeled “backward-incompatible” and has
> text
> > > in the “Docs Text” field. And it’s super minor (METRON-771).
> > >
> > > Is this really true? If so, great, but if not, please help people
> > > upgrade without glitches: Fix these fields in your jiras, so they can
> be
> > > included in the Release Notes.
> > > a) In the “Labels” field, add “backward-incompatible”. (It will
> > > autocomplete for you.)
> > > b) In the “Doc Text” field, say what the issue is and what a
> > > person upgrading should do about it, if anything.
> > >
> > > As usual, non-response will be considered positive confirmation
> > > that no response is necessary :-)
> > > Please try to address in the next day or so.
> > >
> > > Thanks,
> > > Your humble Release Manager
> > >
> > >
> > > On 4/12/17, 10:59 AM, "zeo...@gmail.com"  wrote:
> > >
> > > I agree conceptually but haven't looked at them each
> > > individually to see
> > > how much they impact and if a short timeline for 

Re: Having difficulty setting up both QuickDev and FullDev

[Justin Leet]

If it didn't resolve the issue, then yeah, we might as well just undo it,
and I can update the PR with that instead.

Justin

On Thu, Apr 20, 2017 at 2:11 PM, David Lyle <dlyle65...@gmail.com> wrote:

> Being as the dependency additon didn't actually resolve the out-of-order
> build issue, would it make sense to undo it?
>
> -D...
>
>
> On Thu, Apr 20, 2017 at 1:26 PM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> > Our instructions on building the RPMs are slightly out of date.  The pom
> > for metron-deployment changed slightly to have a dependency on a couple
> > other repos. This means during the project build before the RPM build it
> > needs to be mvn clean install, instead of mvn clean package.
> >
> > Basically from the root pom, do a mvn clean install -DskipTests
> >
> > I'd create a ticket and PR on the doc, but it looks like Apache Jira just
> > went down.
> >
> > Justin
> >
> > On Thu, Apr 20, 2017 at 12:53 PM, Anand Subramanian <
> > asubraman...@hortonworks.com> wrote:
> >
> > > I am seeing the same issue that Rita mentioned with full-dev on latest
> > > master. I have docker up and running, but the RPMs are still not built.
> > >
> > > Here is the output from mvn build-rpms:
> > > --
> > > ➜  metron-deployment git:(master) mvn clean package -Pbuild-rpms
> > > -DskipTests
> > > [INFO] Scanning for projects...
> > > [INFO] 
> > > 
> > > [INFO] Reactor Build Order:
> > > [INFO]
> > > [INFO] metron-deployment
> > > [INFO] metron-rpm
> > > [INFO]
> > > [INFO] 
> > > 
> > > [INFO] Building metron-deployment 0.4.0
> > > [INFO] 
> > > 
> > > [INFO]
> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
> > > metron-deployment ---
> > > [INFO]
> > > [INFO] 
> > > 
> > > [INFO] Building metron-rpm 0.4.0
> > > [INFO] 
> > > 
> > > Downloading: http://repo.hortonworks.com/
> content/repositories/releases/
> > > org/apache/metron/metron-platform/0.4.0/metron-platform-0.4.0.pom
> > > [WARNING] The POM for org.apache.metron:metron-platform:pom:0.4.0 is
> > > missing, no dependency information available
> > > Downloading: http://repo.hortonworks.com/
> content/repositories/releases/
> > > org/apache/metron/metron-analytics/0.4.0/metron-analytics-0.4.0.pom
> > > [WARNING] The POM for org.apache.metron:metron-analytics:pom:0.4.0 is
> > > missing, no dependency information available
> > > [INFO] 
> > > 
> > > [INFO] Reactor Summary:
> > > [INFO]
> > > [INFO] metron-deployment .. SUCCESS [
> > > 0.113 s]
> > > [INFO] metron-rpm . FAILURE [
> > > 2.353 s]
> > > [INFO] 
> > > 
> > > [INFO] BUILD FAILURE
> > > [INFO] 
> > > 
> > > [INFO] Total time: 2.569 s
> > > [INFO] Finished at: 2017-04-20T22:17:37+05:30
> > > [INFO] Final Memory: 12M/309M
> > > [INFO] 
> > > 
> > > [ERROR] Failed to execute goal on project metron-rpm: Could not resolve
> > > dependencies for project org.apache.metron:metron-rpm:pom:0.4.0: The
> > > following artifacts could not be resolved: org.apache.metron:metron-
> > platform:pom:0.4.0,
> > > org.apache.metron:metron-analytics:pom:0.4.0: Failure to find
> > > org.apache.metron:metron-platform:pom:0.4.0 in http://clojars.org/repo
> > > was cached in the local repository, resolution will not be reattempted
> > > until the update interval of clojars.org has elapsed or updates are
> > > forced -> [Help 1]
> > > [ERROR]
> > > [ERROR] To see the full stack trace of the errors, re-run Maven with
> the
> > > -e switch.
> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > > [ERROR]
> > > [ER