[ANNOUNCE] Metron Apache Community Demo for Metron_0.3.1

Lets hold the demo of the latest Metron build from 9-AM to 10AM PST on Feb.10, 2017. Please respond to this thread with the set of features that you would like to demo Topic: Metron Community Demo Join from PC, Mac, Linux, iOS or Android: https://hortonworks.zoom.us/j/263433204 Or join by

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2

Casey, the below vote call message has several inconsistencies that invalidate it. Please search for “RC1” or “rc1”. I count three, starting with the first line :-) There is also an instance of “0.3.0”. Thanks, --Matt On 2/7/17, 8:18 AM, "Casey Stella" wrote: This

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 #14.1 passed Elapsed time 23 min 28 sec On my second try around 5pm --- If your project is set up for it, you can reply to this email and have your reply appear on

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2

+1 binding Valid checksums Build successful Integration tests successful Deploy of "Quick Dev" successful Deploy of "Full Dev" successful On Tue, Feb 7, 2017 at 11:18 AM, Casey Stella wrote: > This is a call to vote on releasing Apache Metron 0.3.1-RC1 incubating > > >

Re: [GitHub] incubator-metron issue #439: add stellar external functions feature (work in...

Sorry for late reply work has been crazy around here too. Changed name and added a few comments and I am not sure if I mentioned it but devopsec is just my dev handle / dev name, wasn't sure if that was apparent. I will push up another revision later this week, add in your comments and i'll revise

[GitHub] incubator-metron pull request #445: METRON-706: Add Stellar transformations ...

GitHub user mmiklavc opened a pull request: https://github.com/apache/incubator-metron/pull/445 METRON-706: Add Stellar transformations and filters to enrichment and threat intel loaders This PR completes work in https://issues.apache.org/jira/browse/METRON-706 (Note:

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

Github user devopsec commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/439#discussion_r99967075 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java --- @@ -0,0 +1,292 @@ +/** + *

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/444 Also, with only 30 parallel builds available across all apache projects, asking for 2 separate containers per build might be greedy. ;) --- If your project is set up for it, you can

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/444 Ok, this was just too volatile. I also got 43 minutes for the integration test run. It's probably worth investigating in the future, but for now I'm going to revert to just one

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/444 @ottobackwards Yeah, we have to use the apache [job queue](https://blogs.apache.org/infra/entry/apache_gains_additional_travis_ci) (which has 30 parallel builds) on travis rather than the

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 you bet btw -the build on this pr isn't even going at all --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/444 Looking at those logs, it appears that the first phase, the build (not the actual integration tests) is 4 minutes 39 seconds in my integration-test phase

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 integration tests took 43 minutes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 #14 passed Elapsed time 43 min 10 sec Total time 1 hr 6 min 21 sec --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/444 Yep, that's how I did it. You can see a build of it on my account [here](https://travis-ci.org/cestella/incubator-metron/builds/199349340) --- If your project is set up for it, you can

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/444 @ottobackwards Yes, that should do it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 So to test this - if you have travis set up for your git account, I think you just push it to a branch in your repo and let travis build it? --- If your project is set up for it,

Re: [DISCUSS] Build Times are getting out of hand

haha there was some desperation there, I'll admit. ;) On Tue, Feb 7, 2017 at 3:12 PM, Otto Fowler wrote: > This PR gets a star just for the commit messages, it isn’t even Friday > Casey > > > On February 7, 2017 at 14:49:22, Casey Stella (ceste...@gmail.com) wrote: > >

Re: [DISCUSS] Build Times are getting out of hand

This PR gets a star just for the commit messages, it isn’t even Friday Casey On February 7, 2017 at 14:49:22, Casey Stella (ceste...@gmail.com) wrote: I spent a minute or two looking at how we might use travis configuration-alone to drop the wall-clock time of the build and put it up for review

Re: [DISCUSS] Build Times are getting out of hand

Down to 24 minutes? Nice job. On Tue, Feb 7, 2017 at 1:49 PM, Casey Stella wrote: > I spent a minute or two looking at how we might use travis > configuration-alone to drop the wall-clock time of the build and put it up > for review at

Re: [DISCUSS] Build Times are getting out of hand

I spent a minute or two looking at how we might use travis configuration-alone to drop the wall-clock time of the build and put it up for review at https://github.com/apache/incubator-metron/pull/444 It does 2 things: - Separates the build, the unit tests and the integration tests -

[GitHub] incubator-metron pull request #444: METRON-705: Parallelize the build in tra...

GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/444 METRON-705: Parallelize the build in travis to the extent that is obvious Travis suggests [here](https://blog.travis-ci.com/2012-11-28-speeding-up-your-tests-by-parallelizing-them/)

Re: BulkMessageWriterBolt and MessageGetters

You are correct, the BulkMessageWriterBolt/MessageGetters combination is not flexible enough. You would have to modify BulkMessageWriterBolt. I have addressed this in METRON-695 which will be submitted as a PR shortly. It will be easy to do what you want after that is merged in. Ryan On Tue,

BulkMessageWriterBolt and MessageGetters

I am trying to use the `BulkMessageWriterBolt` to write a specific tuple field named "measurement" to a Kafka topic. - id: "kafkaBolt" className: "org.apache.metron.writer.bolt.BulkMessageWriterBolt" constructorArgs: - "${kafka.zk}" configMethods:

Re: [Discuss] Direction of metron-docker

​Is having a goal of replacing Vagrant/Virtualbox for Docker in "Quick Dev" and "Full Dev" mutually exclusive of the goals you outlined above? We could have both, no? I am unsure if you are objecting to this specific goal or not.​ On Mon, Feb 6, 2017 at 1:03 PM, Ryan Merriman

Re: [DISCUSS] Build Times are getting out of hand

FYI, found this for Docker - https://docs.travis-ci.com/user/docker/ On Tue, Feb 7, 2017 at 9:09 AM, David Lyle wrote: > Absolutely agree. I also think we'd want both once we've done that. Travis > is good for smoke testing PRs and Commits. Jenkins is good for nightly runs

[VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2

This is a call to vote on releasing Apache Metron 0.3.1-RC1 incubating Full list of changes in this release: https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.1-RC2-incubating/CHANGES The tag/commit to be voted upon is apache-metron-0.3.0-rc1-incubating:

Re: [DISCUSS] Build Times are getting out of hand

Absolutely agree. I also think we'd want both once we've done that. Travis is good for smoke testing PRs and Commits. Jenkins is good for nightly runs of medium duration tests and would be great for automating our distributed testing if we found infrastructure to support it. I've seen them used in

[GitHub] incubator-metron pull request #443: METRON-703: Rev the version from 0.3.0 t...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/443 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-metron issue #443: METRON-703: Rev the version from 0.3.0 to 0.3.1

Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/443 Built and installed the mpack. Versioning looks good where it shows up, and everything installed and started up correctly. --- If your project is set up for it, you can reply to this

Re: [DISCUSS] Build Times are getting out of hand

I agree with this. I don't think we should switch to an alternate system until we find that we are absolutely incapable of eking out any further efficiency from the current setup. On Tue, Feb 7, 2017 at 8:04 AM, Casey Stella wrote: > I believe that some people use travis and

Re: [DISCUSS] Build Times are getting out of hand

Mike, unfortunately something changed recently, and I can't run `mvn clean install -T 2C` locally anymore. I'd like to echo that I think working on fixing the dependency issue is a very good idea. We've actually faced issues with this on the REST API PR. Working to fix this and having a standard

Re: [DISCUSS] Build Times are getting out of hand

Debugging integration tests in an IDE uses the same approach with our current infrastructure or with docker: start up the topology with LocalRunner. I've had mixed success with our current infrastructure. As Mike alluded to, some tests work fine (most of the parser topologies and enrichment

Re: [DISCUSS] Build Times are getting out of hand

I believe that some people use travis and some people request Jenkins from Apache Infra. That being said, personally, I think we should take the opportunity to correct the underlying issues. 50 minutes for a build seems excessive to me. On Mon, Feb 6, 2017 at 10:07 PM, Otto Fowler

Re: [DISCUSS] Build Times are getting out of hand

Mike, I can verify that the integration tests do not run in parallel via mvn -T 1C clean install At a minimum the integration test infrastructure will need to hunt for an open port to bind to rather than assuming one. On Tue, Feb 7, 2017 at 9:26 AM, Michael Miklavcic <

[GitHub] incubator-metron issue #443: METRON-703: Rev the version from 0.3.0 to 0.3.1

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/443 I verified that this works in vagrant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] incubator-metron issue #443: METRON-703: Rev the version from 0.3.0 to 0.3.1

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/443 I'm in the process of spinning this up in vagrant and I believe @justinleet will be testing out the mpack just to make sure nothing is borked. Please hold off until we report in to

[GitHub] incubator-metron issue #443: METRON-703: Rev the version from 0.3.0 to 0.3.1

Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/443 +1 Did a quick find-search and did not find any out-of-place 0.3.0 tags left. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[RESULT][VOTE] Releasing Apache Metron (incubating) 0.3.1-RC1

The vote fails; a new release candidate will be cut when METRON-703 is accepted. Results: +1 Nick Allen James Sirota Casey Stella -1 David Lyle

[GitHub] incubator-metron pull request #443: METRON-703: Rev the version from 0.3.0 t...

GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/443 METRON-703: Rev the version from 0.3.0 to 0.3.1 In order to release, we need to up the version to 0.3.1 so that the artifacts produced continue to function. You can merge this pull

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC1

Whoops, you're absolutely right. We forgot to rev the version in the artifacts. I'm going to cancel the vote and rerelease when that JIRA gets in. On Tue, Feb 7, 2017 at 7:56 AM, David Lyle wrote: > -1 Unless I'm mistaken, the artifacts are versioned 0.3.0. > > -D... > >

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC1

-1 Unless I'm mistaken, the artifacts are versioned 0.3.0. -D... On Mon, Feb 6, 2017 at 10:46 PM, James Sirota wrote: > +1 deployed on AWS > > 06.02.2017, 15:39, "Nick Allen" : > > +1 > > > > Valid checksums > > Build successful > > Integration tests

Re: [Discuss] Direction of metron-docker

>From a user perspective, We used Vagrant when we first encountered Metron to see and learn about it, it was quick and easy to deploy - handy. Now we switched to Ambari Mpack for internal use. We mostly write and deploy Parsers for Metron, so having just Ambari Mpack is enough for us. I haven't