date:20170421

Re: Ansible Docker fails to build latest

2017-04-21 Thread Otto Fowler

OK,
the issue is the old old maven that is in the ansible image.
Changing it to 3.3.9 resolved.



On April 21, 2017 at 17:21:02, Otto Fowler (ottobackwa...@gmail.com) wrote:

Yes.   I am following the readme in there and starting the image, then
just running mvn package -DskipTests



On April 21, 2017 at 15:28:15, Casey Stella (ceste...@gmail.com) wrote:

Sorry, can you go through how you're getting to this error? I'm not super
familiar with this part of the stack..is this compiling Metron inside of a
docker image?

On Tue, Apr 18, 2017 at 1:45 PM, Otto Fowler 
wrote:

> Is it something to do with the relocation of google.common that happens in
> the other shaded jars but not in the storm kafka client?
>
>
>
> On April 18, 2017 at 12:56:12, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I’m trying to build latest in ansible docker, but I’m seeing the following
> error. Note that metron-common is a dependency in this module.
>
> [INFO] 
> 
> [INFO] Building metron-storm-kafka 0.4.0
> [INFO] 
> 
> Downloading: http://clojars.org/repo/org/apache/storm/storm-kafka-
> client/1.0.1.2.5.0.0-1245/storm-kafka-client-1.0.1.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.pom (4 KB at 31.5 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/storm/storm/1.0.1.2.5.
> 0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm/1.0.1.2.5.0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm/1.0.1.2.5.0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> (49 KB at 433.0 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/kafka/kafka-clients/0.
> 10.0.2.5.0.0-1245/kafka-clients-0.10.0.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/kafka/kafka-clients/0.10.0.2.5.0.0-1245/
> kafka-clients-0.10.0.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/kafka/kafka-clients/0.10.0.2.5.0.0-1245/
> kafka-clients-0.10.0.2.5.0.0-1245.pom (2 KB at 21.1 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/storm/storm-kafka-
> client/1.0.1.2.5.0.0-1245/storm-kafka-client-1.0.1.2.5.0.0-1245.jar
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.jar
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.jar (64 KB at 417.1 KB/sec)
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> metron-storm-kafka ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory /root/incubator-metron/metron-
> platform/metron-storm-kafka/src/main/resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @
> metron-storm-kafka ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 7 source files to /root/incubator-metron/metron-
> platform/metron-storm-kafka/target/classes
> /root/incubator-metron/metron-platform/metron-storm-kafka/
> src/main/java/org/apache/metron/storm/kafka/flux/
> SimpleStormKafkaBuilder.java:21: error: package com.google.common.base
> does not exist
> import com.google.common.base.Joiner;
>
> Has anyone tried the ansible docker image recently? Or seen this error?
>
>
>

Re: Ansible Docker fails to build latest

2017-04-21 Thread Casey Stella

Sorry, can you go through how you're getting to this error?  I'm not super
familiar with this part of the stack..is this compiling Metron inside of a
docker image?

On Tue, Apr 18, 2017 at 1:45 PM, Otto Fowler 
wrote:

> Is it something to do with the relocation of google.common that happens in
> the other shaded jars but not in the storm kafka client?
>
>
>
> On April 18, 2017 at 12:56:12, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I’m trying to build latest in ansible docker, but I’m seeing the following
> error.  Note that metron-common is a dependency in this module.
>
> [INFO] 
> 
> [INFO] Building metron-storm-kafka 0.4.0
> [INFO] 
> 
> Downloading: http://clojars.org/repo/org/apache/storm/storm-kafka-
> client/1.0.1.2.5.0.0-1245/storm-kafka-client-1.0.1.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.pom (4 KB at 31.5 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/storm/storm/1.0.1.2.5.
> 0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm/1.0.1.2.5.0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm/1.0.1.2.5.0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> (49 KB at 433.0 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/kafka/kafka-clients/0.
> 10.0.2.5.0.0-1245/kafka-clients-0.10.0.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/kafka/kafka-clients/0.10.0.2.5.0.0-1245/
> kafka-clients-0.10.0.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/kafka/kafka-clients/0.10.0.2.5.0.0-1245/
> kafka-clients-0.10.0.2.5.0.0-1245.pom (2 KB at 21.1 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/storm/storm-kafka-
> client/1.0.1.2.5.0.0-1245/storm-kafka-client-1.0.1.2.5.0.0-1245.jar
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.jar
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.jar (64 KB at 417.1 KB/sec)
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> metron-storm-kafka ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory /root/incubator-metron/metron-
> platform/metron-storm-kafka/src/main/resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @
> metron-storm-kafka ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 7 source files to /root/incubator-metron/metron-
> platform/metron-storm-kafka/target/classes
> /root/incubator-metron/metron-platform/metron-storm-kafka/
> src/main/java/org/apache/metron/storm/kafka/flux/
> SimpleStormKafkaBuilder.java:21: error: package com.google.common.base
> does not exist
> import com.google.common.base.Joiner;
>
> Has anyone tried the ansible docker image recently?  Or seen this error?
>
>
>

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/542
  
@ottobackwards you're totally right.  We desperately are in need of better 
stellar documentation:
* A language reference
* A set of introductory lessons in Stellar


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

2017-04-21 Thread cestella

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/542

METRON-873: Stellar string literals do not support quote escaping

## Contributor Comments
Right now, in stellar, we cannot represent a string literal that contains 
`'foo'` if the string is quoted with `'` or `"foo"` if the string is quoted 
with `"`.  This is unfortunate and should be corrected.

To test this out, start up the stellar REPL in fulldev *or* run it locally 
by running `mvn exec:java 
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell"` from 
`metron-platform/metron-common` and try the following strings:
* `'\'foo\''` should yield `'foo'`
* `"\"foo\""` should yield `"foo"`
* `TO_UPPER('\'foo\'')` should yield `'FOO'`
* `TO_UPPER("\"foo\"")` should yield `"FOO"`

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
stellar_quoted_strings

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/542.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #542


commit cde9211b5bf7aa3ed4b91477605bbe6685540c71
Author: cstella 
Date:   2017-04-21T16:13:18Z

Add quote escaping to Stellar string literals.

commit 2920faaf24385330dab2a8490d7a599ab9aae822
Author: cstella 
Date:   2017-04-21T16:34:29Z

Documentationc hange.

commit 88338f110a5db945cb6ea1a99221cb2c6d49633c
Author: cstella 
Date:   2017-04-21T17:41:11Z

Cleaned up grammar a bit and added proper support for backslash.

commit f123ddf711e237f60a9bff140a6c47c4dc39fa53
Author: cstella 
Date:   2017-04-21T17:45:03Z

missed newlines

commit 0a2284c97c15ec0c1243156f3214fc3beadabea3
Author: cstella 
Date:   2017-04-21T18:06:11Z

Updated for bug and to add \n, \t and \r




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

2017-04-21 Thread cestella

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/542


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/542
  
We need a stellar guide, more suited to learning stellar, than just 
documenting the facts of it.  Something that gets you in the shell and gives 
you some exercises.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/542
  
We're not using SCHAR anymore.  The current commit should address the 
backslash issue.  Give it a try and let me know what you think. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #534: METRON-861: Allow JVM args to be passed to CLI ...

2017-04-21 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/534
  
I could get behind that.  Anyone else have other ideas?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reducing Warnings in Build

2017-04-21 Thread Otto Fowler

Until there is a use case supporting changing to other then the 80% plus
case, this should not be changed.
or a customer requirement etc.  This would need some design and discussion
time as well to map out all the
implications for the complete pipeline.

I think configurable charset should be separated from the cleanup.


On April 21, 2017 at 12:26:36, JJ Meyer (jjmey...@gmail.com) wrote:

Nick,

You're not off base at all. This is exactly the push back I wanted to hear.
I wasn't really sure if what I was proposing was the proper way of going
about it. My understanding of charsets is limited, and I've usually just
defaulted to UTF-8. That being said, right or wrong, my thought process for
including multiple configs was potential use of multiple charsets that may
be compatible (if that's possible?). But as I'm typing this out, I'm
probably over doing it. Especially since currently we usually take system
default, which a lot of times is UTF-8. So, you are right, it is probably
best to start with the path of least resistance and include one
configuration that is set to UTF-8. Then expand if we ever see a need to do
so.

Thanks,
JJ

On Fri, Apr 21, 2017 at 10:45 AM, Ryan Merriman 
wrote:

> I think Nick brings up some good points. Would there ever be a reason to
> not use UTF8 as the default from parsing a message on? All the tools we
> use for analytics work with UTF8 (am I wrong?).
>
> The only case I can see needing a configurable charset would be if a
> message coming from a sensor were encoded in a charset other than UTF8.
In
> that case there would need to be a configurable charset per parser (in
> parser config?) for decoding but the message could be encoded/decoded
with
> UTF8 after that. Or we could just make UTF encoding a prerequisite for
> sending messages to Metron.
>
> On Fri, Apr 21, 2017 at 10:32 AM, Nick Allen  wrote:
>
> > Per (2), I think it makes sense to make the charset configurable, but
> with
> > the proposal of 3 separate settings, wouldn't things blow up horribly
if
> > the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?
> They
> > are not even speaking the same language, no?
> >
> > This makes me think that we need to understand the scenarios under
which
> a
> > user would want to change the charset, before we know what kinds of
> levers
> > to bake-in. What sort of use case would drive someone to change the
> > charset? Would there be a particular sensor producing telemetry with a
> > different charset from others? If one source of telemetry is different
> > than the others, would the entire system even work with varying
charsets?
> >
> > Without a good understanding of use cases, I think the only mildly safe
> > thing to do right now, is to have a single, global charset setting. The
> > user would have to ensure that their entire environment and all the
JVMs
> > driving it are set to use that charset.
> >
> > Perhaps my questioning comes from a lack of understanding of charsets.
I
> > can't remember ever having to deviate from the default. Please chime in
> > and educate me, if I am off base.
> >
> >
> >
> >
> >
> >
> > On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:
> >
> > > Hello everybody,
> > >
> > > Currently our build has a significant amount of warnings (most from
the
> > > error prone plugin). A lot of this has been documented here:
> > > https://issues.apache.org/jira/browse/METRON-617
> > >
> > > I want to continue to work on this Jira. I really want to reduce the
> > number
> > > of warnings in our build. As the Jira points out, a lot of them are
> > > unchecked casts or the implicit use of default charset.
> > >
> > > When starting this, I want to start with a specific module. I plan on
> > > starting with `metron-interface` as it's fairly self contained and is
> > > pretty new. Below I want to layout what I plan on doing. Please let
me
> > know
> > > what you all think:
> > >
> > > 1. Suppress warnings where generics are not supported or checking
type
> is
> > > not possible.
> > > 2. For all unit tests that require supplying a charset I'll supply
the
> > > UTF-8 charset.
> > > 3. Update the API to have a charset configuration for each resource
> that
> > > needs one.
> > > 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out
error
> > > prone doesn't support this level of suppression. Which is probably a
> good
> > > thing. We will need to explicitly state what we want to suppress
> instead.
> > >
> > > Two big questions came up right away when I was thinking about the
> above:
> > >
> > > 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> > > Map. I know it extends Map, but is it really safe to
do
> > > something like the following? Should we have a utility that truly
> builds
> > a
> > > map that uses generics? I plan on doing some testing around this, but
> if
> > > anyone has any experience with this it would be greatly appreciated.
>

Re: Reducing Warnings in Build

2017-04-21 Thread JJ Meyer

Nick,

You're not off base at all. This is exactly the push back I wanted to hear.
I wasn't really sure if what I was proposing was the proper way of going
about it. My understanding of charsets is limited, and I've usually just
defaulted to UTF-8. That being said, right or wrong, my thought process for
including multiple configs was potential use of multiple charsets that may
be compatible (if that's possible?). But as I'm typing this out, I'm
probably over doing it. Especially since currently we usually take system
default, which a lot of times is UTF-8. So, you are right, it is probably
best to start with the path of least resistance and include one
configuration that is set to UTF-8. Then expand if we ever see a need to do
so.

Thanks,
JJ

On Fri, Apr 21, 2017 at 10:45 AM, Ryan Merriman  wrote:

> I think Nick brings up some good points.  Would there ever be a reason to
> not use UTF8 as the default from parsing a message on?  All the tools we
> use for analytics work with UTF8 (am I wrong?).
>
> The only case I can see needing a configurable charset would be if a
> message coming from a sensor were encoded in a charset other than UTF8.  In
> that case there would need to be a configurable charset per parser (in
> parser config?) for decoding but the message could be encoded/decoded with
> UTF8 after that.  Or we could just make UTF encoding a prerequisite for
> sending messages to Metron.
>
> On Fri, Apr 21, 2017 at 10:32 AM, Nick Allen  wrote:
>
> > Per (2), I think it makes sense to make the charset configurable, but
> with
> > the proposal of 3 separate settings, wouldn't things blow up horribly if
> > the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?
> They
> > are not even speaking the same language, no?
> >
> > This makes me think that we need to understand the scenarios under which
> a
> > user would want to change the charset, before we know what kinds of
> levers
> > to bake-in.  What sort of use case would drive someone to change the
> > charset?  Would there be a particular sensor producing telemetry with a
> > different charset from others?  If one source of telemetry is different
> > than the others, would the entire system even work with varying charsets?
> >
> > Without a good understanding of use cases, I think the only mildly safe
> > thing to do right now, is to have a single, global charset setting.  The
> > user would have to ensure that their entire environment and all the JVMs
> > driving it are set to use that charset.
> >
> > Perhaps my questioning comes from a lack of understanding of charsets.  I
> > can't remember ever having to deviate from the default.  Please chime in
> > and educate me, if I am off base.
> >
> >
> >
> >
> >
> >
> > On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:
> >
> > > Hello everybody,
> > >
> > > Currently our build has a significant amount of warnings (most from the
> > > error prone plugin). A lot of this has been documented here:
> > > https://issues.apache.org/jira/browse/METRON-617
> > >
> > > I want to continue to work on this Jira. I really want to reduce the
> > number
> > > of warnings in our build. As the Jira points out, a lot of them are
> > > unchecked casts or the implicit use of default charset.
> > >
> > > When starting this, I want to start with a specific module. I plan on
> > > starting with `metron-interface` as it's fairly self contained and is
> > > pretty new. Below I want to layout what I plan on doing. Please let me
> > know
> > > what you all think:
> > >
> > > 1. Suppress warnings where generics are not supported or checking type
> is
> > > not possible.
> > > 2. For all unit tests that require supplying a charset I'll supply the
> > > UTF-8 charset.
> > > 3. Update the API to have a charset configuration for each resource
> that
> > > needs one.
> > > 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> > > prone doesn't support this level of suppression. Which is probably a
> good
> > > thing. We will need to explicitly state what we want to suppress
> instead.
> > >
> > > Two big questions came up right away when I was thinking about the
> above:
> > >
> > > 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> > > Map. I know it extends Map, but is it really safe to do
> > > something like the following? Should we have a utility that truly
> builds
> > a
> > > map that uses generics? I plan on doing some testing around this, but
> if
> > > anyone has any experience with this it would be greatly appreciated.
> Here
> > > is an example of what I am talking about:
> > >
> > > JSONObject message = ...;
> > > @SuppressWarnings("unchecked")
> > > Map state = (Map) message;
> > >
> > >
> > > 2. This one is about configuring charset (#3 above). Specifically in
> > > metron-rest. Right now, I believe there are 3 sensor resources (index,
> > > enrichment, and

Re: Reducing Warnings in Build

2017-04-21 Thread Ryan Merriman

I think Nick brings up some good points.  Would there ever be a reason to
not use UTF8 as the default from parsing a message on?  All the tools we
use for analytics work with UTF8 (am I wrong?).

The only case I can see needing a configurable charset would be if a
message coming from a sensor were encoded in a charset other than UTF8.  In
that case there would need to be a configurable charset per parser (in
parser config?) for decoding but the message could be encoded/decoded with
UTF8 after that.  Or we could just make UTF encoding a prerequisite for
sending messages to Metron.

On Fri, Apr 21, 2017 at 10:32 AM, Nick Allen  wrote:

> Per (2), I think it makes sense to make the charset configurable, but with
> the proposal of 3 separate settings, wouldn't things blow up horribly if
> the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?  They
> are not even speaking the same language, no?
>
> This makes me think that we need to understand the scenarios under which a
> user would want to change the charset, before we know what kinds of levers
> to bake-in.  What sort of use case would drive someone to change the
> charset?  Would there be a particular sensor producing telemetry with a
> different charset from others?  If one source of telemetry is different
> than the others, would the entire system even work with varying charsets?
>
> Without a good understanding of use cases, I think the only mildly safe
> thing to do right now, is to have a single, global charset setting.  The
> user would have to ensure that their entire environment and all the JVMs
> driving it are set to use that charset.
>
> Perhaps my questioning comes from a lack of understanding of charsets.  I
> can't remember ever having to deviate from the default.  Please chime in
> and educate me, if I am off base.
>
>
>
>
>
>
> On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:
>
> > Hello everybody,
> >
> > Currently our build has a significant amount of warnings (most from the
> > error prone plugin). A lot of this has been documented here:
> > https://issues.apache.org/jira/browse/METRON-617
> >
> > I want to continue to work on this Jira. I really want to reduce the
> number
> > of warnings in our build. As the Jira points out, a lot of them are
> > unchecked casts or the implicit use of default charset.
> >
> > When starting this, I want to start with a specific module. I plan on
> > starting with `metron-interface` as it's fairly self contained and is
> > pretty new. Below I want to layout what I plan on doing. Please let me
> know
> > what you all think:
> >
> > 1. Suppress warnings where generics are not supported or checking type is
> > not possible.
> > 2. For all unit tests that require supplying a charset I'll supply the
> > UTF-8 charset.
> > 3. Update the API to have a charset configuration for each resource that
> > needs one.
> > 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> > prone doesn't support this level of suppression. Which is probably a good
> > thing. We will need to explicitly state what we want to suppress instead.
> >
> > Two big questions came up right away when I was thinking about the above:
> >
> > 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> > Map. I know it extends Map, but is it really safe to do
> > something like the following? Should we have a utility that truly builds
> a
> > map that uses generics? I plan on doing some testing around this, but if
> > anyone has any experience with this it would be greatly appreciated. Here
> > is an example of what I am talking about:
> >
> > JSONObject message = ...;
> > @SuppressWarnings("unchecked")
> > Map state = (Map) message;
> >
> >
> > 2. This one is about configuring charset (#3 above). Specifically in
> > metron-rest. Right now, I believe there are 3 sensor resources (index,
> > enrichment, and parser) that throw warnings due to implicitly using the
> > default charset. I propose that we have 3 configs (listed below). These
> > configs would take any valid charset, default, or not set. If not set
> then
> > UTF-8 would be default. Does this seem fair?
> >
> > sensor:
> >   index.encoding: UTF-8
> >   enrichment.encoding: UTF-8
> >   parser.encoding: UTF-8
> >
> >
> > Does anyone see any problems that may occur if we go about it this way?
> Any
> > help on this would be very much appreciated.
> >
> > Thanks,
> > JJ
> >
>

Re: Reducing Warnings in Build

2017-04-21 Thread Michael Miklavcic

Personally, I think it makes sense to have an inbound/outbound charset
setting for the parsers. We should generally standardize on UTF-8 across
Metron and its topologies, but accept potentially different charsets from
the inbound sensors. That is to say, standardize the defaults for the
things we do control, and handle edge cases for the things we don't. And
UTF-8 is fully compatible with ASCII without requiring a mapping, so this
should generally work.
http://stackoverflow.com/questions/15965811/why-utf8-is-compatible-with-ascii.
Even with defaults, I think we should still let users configure this to
whatever they want, but offer the warning of what will happen if the
expected charsets don't match up.

On Fri, Apr 21, 2017 at 9:32 AM, Nick Allen  wrote:

> Per (2), I think it makes sense to make the charset configurable, but with
> the proposal of 3 separate settings, wouldn't things blow up horribly if
> the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?  They
> are not even speaking the same language, no?
>
> This makes me think that we need to understand the scenarios under which a
> user would want to change the charset, before we know what kinds of levers
> to bake-in.  What sort of use case would drive someone to change the
> charset?  Would there be a particular sensor producing telemetry with a
> different charset from others?  If one source of telemetry is different
> than the others, would the entire system even work with varying charsets?
>
> Without a good understanding of use cases, I think the only mildly safe
> thing to do right now, is to have a single, global charset setting.  The
> user would have to ensure that their entire environment and all the JVMs
> driving it are set to use that charset.
>
> Perhaps my questioning comes from a lack of understanding of charsets.  I
> can't remember ever having to deviate from the default.  Please chime in
> and educate me, if I am off base.
>
>
>
>
>
>
> On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:
>
> > Hello everybody,
> >
> > Currently our build has a significant amount of warnings (most from the
> > error prone plugin). A lot of this has been documented here:
> > https://issues.apache.org/jira/browse/METRON-617
> >
> > I want to continue to work on this Jira. I really want to reduce the
> number
> > of warnings in our build. As the Jira points out, a lot of them are
> > unchecked casts or the implicit use of default charset.
> >
> > When starting this, I want to start with a specific module. I plan on
> > starting with `metron-interface` as it's fairly self contained and is
> > pretty new. Below I want to layout what I plan on doing. Please let me
> know
> > what you all think:
> >
> > 1. Suppress warnings where generics are not supported or checking type is
> > not possible.
> > 2. For all unit tests that require supplying a charset I'll supply the
> > UTF-8 charset.
> > 3. Update the API to have a charset configuration for each resource that
> > needs one.
> > 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> > prone doesn't support this level of suppression. Which is probably a good
> > thing. We will need to explicitly state what we want to suppress instead.
> >
> > Two big questions came up right away when I was thinking about the above:
> >
> > 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> > Map. I know it extends Map, but is it really safe to do
> > something like the following? Should we have a utility that truly builds
> a
> > map that uses generics? I plan on doing some testing around this, but if
> > anyone has any experience with this it would be greatly appreciated. Here
> > is an example of what I am talking about:
> >
> > JSONObject message = ...;
> > @SuppressWarnings("unchecked")
> > Map state = (Map) message;
> >
> >
> > 2. This one is about configuring charset (#3 above). Specifically in
> > metron-rest. Right now, I believe there are 3 sensor resources (index,
> > enrichment, and parser) that throw warnings due to implicitly using the
> > default charset. I propose that we have 3 configs (listed below). These
> > configs would take any valid charset, default, or not set. If not set
> then
> > UTF-8 would be default. Does this seem fair?
> >
> > sensor:
> >   index.encoding: UTF-8
> >   enrichment.encoding: UTF-8
> >   parser.encoding: UTF-8
> >
> >
> > Does anyone see any problems that may occur if we go about it this way?
> Any
> > help on this would be very much appreciated.
> >
> > Thanks,
> > JJ
> >
>

Re: Reducing Warnings in Build

2017-04-21 Thread Nick Allen

Per (2), I think it makes sense to make the charset configurable, but with
the proposal of 3 separate settings, wouldn't things blow up horribly if
the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?  They
are not even speaking the same language, no?

This makes me think that we need to understand the scenarios under which a
user would want to change the charset, before we know what kinds of levers
to bake-in.  What sort of use case would drive someone to change the
charset?  Would there be a particular sensor producing telemetry with a
different charset from others?  If one source of telemetry is different
than the others, would the entire system even work with varying charsets?

Without a good understanding of use cases, I think the only mildly safe
thing to do right now, is to have a single, global charset setting.  The
user would have to ensure that their entire environment and all the JVMs
driving it are set to use that charset.

Perhaps my questioning comes from a lack of understanding of charsets.  I
can't remember ever having to deviate from the default.  Please chime in
and educate me, if I am off base.

On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:

> Hello everybody,
>
> Currently our build has a significant amount of warnings (most from the
> error prone plugin). A lot of this has been documented here:
> https://issues.apache.org/jira/browse/METRON-617
>
> I want to continue to work on this Jira. I really want to reduce the number
> of warnings in our build. As the Jira points out, a lot of them are
> unchecked casts or the implicit use of default charset.
>
> When starting this, I want to start with a specific module. I plan on
> starting with `metron-interface` as it's fairly self contained and is
> pretty new. Below I want to layout what I plan on doing. Please let me know
> what you all think:
>
> 1. Suppress warnings where generics are not supported or checking type is
> not possible.
> 2. For all unit tests that require supplying a charset I'll supply the
> UTF-8 charset.
> 3. Update the API to have a charset configuration for each resource that
> needs one.
> 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> prone doesn't support this level of suppression. Which is probably a good
> thing. We will need to explicitly state what we want to suppress instead.
>
> Two big questions came up right away when I was thinking about the above:
>
> 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> Map. I know it extends Map, but is it really safe to do
> something like the following? Should we have a utility that truly builds a
> map that uses generics? I plan on doing some testing around this, but if
> anyone has any experience with this it would be greatly appreciated. Here
> is an example of what I am talking about:
>
> JSONObject message = ...;
> @SuppressWarnings("unchecked")
> Map state = (Map) message;
>
>
> 2. This one is about configuring charset (#3 above). Specifically in
> metron-rest. Right now, I believe there are 3 sensor resources (index,
> enrichment, and parser) that throw warnings due to implicitly using the
> default charset. I propose that we have 3 configs (listed below). These
> configs would take any valid charset, default, or not set. If not set then
> UTF-8 would be default. Does this seem fair?
>
> sensor:
>   index.encoding: UTF-8
>   enrichment.encoding: UTF-8
>   parser.encoding: UTF-8
>
>
> Does anyone see any problems that may occur if we go about it this way? Any
> help on this would be very much appreciated.
>
> Thanks,
> JJ
>

Re: Failure to Deploy "Quick Dev"

2017-04-21 Thread Casey Stella

I'm betting we need to regenerate the quickdev image after all the mpack
changes.

On Fri, Apr 21, 2017 at 10:45 AM, Otto Fowler 
wrote:

> From lira:
>
> I 'think' that quickdev is actually build from full_dev, with metron
> installed already. So it may be that we need a new image built to make this
> not an upgrade situation?
>
>
> On April 21, 2017 at 10:37:39, Nick Allen (n...@nickallen.org) wrote:
>
> Not sure if I am doing something stupid, but I cannot deploy "Quick Dev"
> from master currently. Full details in
> https://issues.apache.org/jira/browse/METRON-872.
>

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer

Github user nishihatapalmer commented on the issue:

https://github.com/apache/incubator-metron/pull/541
  
Correct, there's no NFA or DFA under the hood of the SequenceMatcher.  

You can create sequences using the regex syntax using the 
SequenceMatcherCompiler, as long as only syntax which creates fixed length 
sequences is used.  So you can match bytes (hex values), sets of bytes [01 02 
03], any bytes ., bitmasks, strings and case insensitive strings, but not 
wildcards or optional bytes.  For example:

01 ^02 'a string' [f0-ff] 'another string' [0a 0d]

The RegexCompiler can accept the full regex syntax including *, +, ?, and 
it does create NFAs - but this isn't tested.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #539: METRON-867: In the event that we graduate, remo...

2017-04-21 Thread justinleet

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/539
  
I'm fine with looking at it Monday.  I may take a quick look if there's any 
JIRAs from other projects we can steal as a starting template, but if not we 
can start running through the graduation guides and creating tickets as needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/541
  
Currently, I'm using the SequenceMatcher to compile a matching expression 
and then using a searcher to search in the byte array for that expression (code 
is 
[here](https://github.com/cestella/incubator-metron/blob/c50a50d230ae1d71a7a512fc199e26264b17ca60/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/pattern/ByteArrayMatchingUtil.java)
 ).  From what I can tell, this isn't using the NFA or DFA under the hood, is 
that wrong?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer

Github user nishihatapalmer commented on the issue:

https://github.com/apache/incubator-metron/pull/541
  
When you say use regexes, do you mean use the regex syntax to create fixed 
length sequences, or do you mean use full regex functionality?  Full regex 
exists using NFAs and DFAs, but needs testing, as I haven't looked at that part 
of byteseek for quite some time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer

Github user nishihatapalmer commented on the issue:

https://github.com/apache/incubator-metron/pull/541

There is a slightly out of date (note to self: update this!) syntax
document at:

https://github.com/nishihatapalmer/byteseek/blob/master/src/main/java/net/byteseek/parser/regex/Regular%20Expression%20syntax.txt

It gives an overview of most of the syntax, but some of it is only usable
by full regexes, not sequence matchers. In particular it can only accept
syntax which leads to a fixed length expression, so these are **excluded**:

```
* zero to many
+ one to many
() groups
{n,n} n to m copies.
X | Y alternatives.
```

Shorthands defined in this document also do not currently function properly
(e.g. [ascii].

Finally note that inversion ^ functions differently to most regular
expression syntaxes. The token being inverted is the following token, not the
entire set. So most regex would say something like [^ 01 02 03] meaning every
byte except 01, 02 and 03. In byteseek this would be ^[ 01 02 03], as you are
inverting the set. [ ^01 02 03] is also valid - except you are now specifying
a set containing everything but 01 (which already covers 02 and 03).

It's fairly easy to create a different parser if necessary, but most of
byteseek regex syntax is fairly standard - but oriented towards bytes rather
than strings as the default atomic unit.

Any questions please feel free to ask (and I really must update the syntax
document!).

Regards,

Matt.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

2017-04-21 Thread cestella

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/541


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reducing Warnings in Build

2017-04-21 Thread Justin Leet

First off, I would absolutely love to see the warnings reduced and the
quality of our code improved.  I'm in favor of all four of the points, and
I think it's a good start towards weeding out a lot of issues.

So regarding question 1:

That awkwardness comes about because org.simple.json is all untyped Maps
under the hood (
https://github.com/fangyidong/json-simple/blob/master/src/main/java/org/json/simple/JSONObject.java#L19).
So anything that touches those classes starts requiring casts everywhere. A
couple months ago a PR was made against the library that seems to have
added a lot of this stuff (
https://github.com/fangyidong/json-simple/pull/113).  Basically, it was
forked (https://github.com/cliftonlabs/json-simple and
https://cliftonlabs.github.io/json-simple/) and they tried to contribute it
back and it's been sitting there.

I strongly think we should consider swapping over to the fork.  We'd have
to do a bit of research, but it seems the intent was to be completely
backwards compatible while updating things like generics and some various
issues they presumably saw with the original.

For 2, and I'm not an expert in this by any means, but I would be in favor
of defaulting to UTF-8.  It's most likely what's happening under the hood
anyway (even though it's technically platform dependent), so defaulting
probably makes it both more explicit and removes potential for issues if a
given system has a different underlying default.

Justin

On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:

> Hello everybody,
>
> Currently our build has a significant amount of warnings (most from the
> error prone plugin). A lot of this has been documented here:
> https://issues.apache.org/jira/browse/METRON-617
>
> I want to continue to work on this Jira. I really want to reduce the number
> of warnings in our build. As the Jira points out, a lot of them are
> unchecked casts or the implicit use of default charset.
>
> When starting this, I want to start with a specific module. I plan on
> starting with `metron-interface` as it's fairly self contained and is
> pretty new. Below I want to layout what I plan on doing. Please let me know
> what you all think:
>
> 1. Suppress warnings where generics are not supported or checking type is
> not possible.
> 2. For all unit tests that require supplying a charset I'll supply the
> UTF-8 charset.
> 3. Update the API to have a charset configuration for each resource that
> needs one.
> 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> prone doesn't support this level of suppression. Which is probably a good
> thing. We will need to explicitly state what we want to suppress instead.
>
> Two big questions came up right away when I was thinking about the above:
>
> 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> Map. I know it extends Map, but is it really safe to do
> something like the following? Should we have a utility that truly builds a
> map that uses generics? I plan on doing some testing around this, but if
> anyone has any experience with this it would be greatly appreciated. Here
> is an example of what I am talking about:
>
> JSONObject message = ...;
> @SuppressWarnings("unchecked")
> Map state = (Map) message;
>
>
> 2. This one is about configuring charset (#3 above). Specifically in
> metron-rest. Right now, I believe there are 3 sensor resources (index,
> enrichment, and parser) that throw warnings due to implicitly using the
> default charset. I propose that we have 3 configs (listed below). These
> configs would take any valid charset, default, or not set. If not set then
> UTF-8 would be default. Does this seem fair?
>
> sensor:
>   index.encoding: UTF-8
>   enrichment.encoding: UTF-8
>   parser.encoding: UTF-8
>
>
> Does anyone see any problems that may occur if we go about it this way? Any
> help on this would be very much appreciated.
>
> Thanks,
> JJ
>

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/541
  
Hey, thanks for that feedback @nishihatapalmer !  I adjusted to use the 
suggested searcher.  I did have one more question, I'm looking to document the 
possible regex's available for binary search, is there any documentation on the 
restrictions or capabilities of the compiled regex's?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: So we graduated...

2017-04-21 Thread Ali Nazemian

That's great! Congratulation everybody.

On Fri, Apr 21, 2017 at 12:54 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Congrats all
>
> On Apr 20, 2017 8:38 PM, "zeo...@gmail.com"  wrote:
>
> > Well done everybody!  Congrats
> >
> > Jon
> >
> > On Thu, Apr 20, 2017 at 8:55 PM Matt Foley  wrote:
> >
> > > Really exciting!  Congrats to the founding team!
> > > --Matt
> > >
> > >
> > > On 4/20/17, 4:02 PM, "Houshang Livian" 
> wrote:
> > >
> > > Congratulations Team. Great work!
> > >
> > >
> > >
> > >
> > > On 4/20/17, 2:55 PM, "larry mccay"  wrote:
> > >
> > > >Wonderful news and well deserved!
> > > >This community has embraced and committed to the Apache way so
> > > quickly.
> > > >
> > > >
> > > >On Thu, Apr 20, 2017 at 5:39 PM, Kyle Richardson <
> > > kylerichards...@gmail.com>
> > > >wrote:
> > > >
> > > >> That's awesome! Congratulations everyone. Looking forward to the
> > > official
> > > >> announcement on Monday.
> > > >>
> > > >> -Kyle
> > > >>
> > > >> > On Apr 20, 2017, at 5:15 PM, David Lyle  >
> > > wrote:
> > > >> >
> > > >> > Outstanding! Great work everyone. Building a TLP worthy
> > community
> > > is
> > > >> > difficult and worthy work, congratulations all!
> > > >> >
> > > >> > -D...
> > > >> >
> > > >> >> On Thu, Apr 20, 2017 at 5:12 PM, Casey Stella <
> > > ceste...@gmail.com>
> > > >> wrote:
> > > >> >>
> > > >> >> For anyone paying attention to incubator-general, it will
> come
> > > as no
> > > >> >> surprise that we graduated as of last night's board meeting.
> > We
> > > have a
> > > >> >> press released queued up and planned for monday along with a
> PR
> > > >> (METRON-687
> > > >> >> at https://github.com/apache/incubator-metron/pull/539).
> > > >> >>
> > > >> >> It escaped my notice that the graduation was talked about on
> > > >> >> incubator-general; otherwise I'd have sent this email earlier
> > > and been
> > > >> less
> > > >> >> cagey in 687's description.  Even so, I'd like to ask that
> > > everyone
> > > >> keep it
> > > >> >> to themselves until monday morning after the press release
> gets
> > > out the
> > > >> >> door.  I know the cat is out of the bag, but it'd be nice to
> > > have a bit
> > > >> of
> > > >> >> an embargo.
> > > >> >>
> > > >> >> Thanks!
> > > >> >>
> > > >> >> Casey
> > > >> >>
> > > >>
> > >
> > >
> > >
> > > --
> >
> > Jon
> >
>



-- 
A.Nazemian

[GitHub] incubator-metron pull request #540: METRON-869 Include build instructions fo...

2017-04-21 Thread anandsubbu

Github user anandsubbu closed the pull request at:

https://github.com/apache/incubator-metron/pull/540


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #538: METRON-868 Fix documentation on building RPMs

2017-04-21 Thread dlyle65535

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/538
  
Also @mmiklavc, please remember if the intended effect was to make the rpm 
build happen after everything else completed then install + pom dependencies 
didn't work. If you take a close look during the multi-threaded build, you'll 
see docker-build output interleaved with the output from the other module 
builds.

e.g. (from a mvn clean install -Pbuild-rpms run)
```
INFO]
[INFO] --- maven-shade-plugin:2.4.3:shade (default) @ elasticsearch-shaded 
---
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources 
(copy-wait-for-it-to-hbase) @ metron-docker ---
[INFO] Executing tasks
 [echo]  Displaying value of property 
 [echo] ../../../..
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-rpm-sources) @ 
metron-rpm ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources 
(copy-wait-for-it-to-kafkazk) @ metron-docker ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources 
(copy-wait-for-it-to-elasticsearch) @ metron-docker ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
```
In this example, you'll see that metron-rpm and metron-docker are happening 
at the same time at the beginning of the build even though they're later in the 
build order.

```
[INFO] metron-writer
[INFO] metron-storm-kafka
[INFO] metron-hbase
[INFO] metron-profiler-common
[INFO] metron-profiler-client
[INFO] metron-profiler
[INFO] metron-enrichment
[INFO] metron-indexing
[INFO] metron-solr
[INFO] metron-pcap
[INFO] metron-parsers
[INFO] metron-pcap-backend
[INFO] metron-data-management
[INFO] metron-api
[INFO] metron-management
[INFO] elasticsearch-shaded
[INFO] metron-elasticsearch
[INFO] metron-deployment
[INFO] metron-rpm
[INFO] metron-docker
```
Here's the output from the build that shows the actual build executed order:

```
[INFO] Building metron-analytics 0.4.0
[INFO] Building metron-docker 0.4.0
[INFO] Building metron-interface 0.4.0
[INFO] Building metron-platform 0.4.0
[INFO] Building metron-maas-common 0.4.0
[INFO] Building metron-test-utilities 0.4.0
[INFO] Building metron-deployment 0.4.0
[INFO] Building metron-config 0.4.0
[INFO] Building metron-rpm 0.4.0
[INFO] Building metron-integration-test 0.4.0
[INFO] Building metron-maas-service 0.4.0
[INFO] Building metron-common 0.4.0
[INFO] Building metron-rest-client 0.4.0
[INFO] Building metron-statistics 0.4.0
[INFO] Building metron-writer 0.4.0
[INFO] Building metron-hbase 0.4.0
[INFO] Building metron-storm-kafka 0.4.0
[INFO] Building metron-profiler-common 0.4.0
[INFO] Building metron-pcap 0.4.0
[INFO] Building metron-profiler-client 0.4.0
[INFO] Building metron-pcap-backend 0.4.0
[INFO] Building metron-api 0.4.0
[INFO] Building metron-profiler 0.4.0
[INFO] Building metron-enrichment 0.4.0
[INFO] Building metron-indexing 0.4.0
[INFO] Building metron-parsers 0.4.0
[INFO] Building metron-data-management 0.4.0
[INFO] Building metron-elasticsearch 0.4.0
[INFO] Building metron-solr 0.4.0
[INFO] Building metron-management 0.4.0
[INFO] Building metron-rest 0.4.0
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ansible Docker fails to build latest

Re: Ansible Docker fails to build latest

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

[GitHub] incubator-metron issue #534: METRON-861: Allow JVM args to be passed to CLI ...

Re: Reducing Warnings in Build

Re: Reducing Warnings in Build

Re: Reducing Warnings in Build

Re: Reducing Warnings in Build

Re: Reducing Warnings in Build

Re: Failure to Deploy "Quick Dev"

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

[GitHub] incubator-metron issue #539: METRON-867: In the event that we graduate, remo...

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

Re: Reducing Warnings in Build

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

Re: So we graduated...

[GitHub] incubator-metron pull request #540: METRON-869 Include build instructions fo...

[GitHub] incubator-metron issue #538: METRON-868 Fix documentation on building RPMs

25 matches

Site Navigation

Mail list logo

Footer information