Re: [MENTORS] Licensing

[zeo...@gmail.com]

Okay great, thanks you.  I've sent an email out to general@ to discuss.

It looks like things are all set up on asf

for
that new repo - would somebody be willing to setup a read-only git mirror
to github ?  My
understanding is that I'm unable to make this request myself, as I am not a
PMC member.  The documentation that I've been able to find says:

Please file an INFRA  issue
(component: Git) to request a new codebase to be mirrored or to change the
settings of an existing mirror. When requesting a new mirror, please
include the following information:

   -

   Name of the codebase, for example "Apache Tika"
   -

   Name of the requested Git mirror, for example "tika.git"
   -

   Subversion path of the codebase, for example "/lucene/tika"
   -

   Subversion layout, in case it is different from the standard "trunk,
   branches, tags" structure.


If this is something that I should be able to do I would be more than happy
to, just let me know.  Thanks!

Jon

On Mon, Apr 10, 2017 at 4:19 PM P. Taylor Goetz  wrote:

> In my opinion, I think you are probably okay including it. You would want
> to retain any license headers, and also make sure your NOTICE file is
> updated to include the copyright information.
>
> I’m not by any means an expert in these matters, so I would recommend
> getting a second opinion by leveraging the collective intelligence of the
> general@incubator (while you are a podling or even afterward) or
> legal-discuss@. The latter list is often home to some important legal
> discussions, and I would recommend PMC members read and/or subscribe to
> that list.
>
> -Taylor
>
> > On Apr 5, 2017, at 4:50 PM, zeo...@gmail.com  wrote:
> >
> > *Background*
> > We have a situation where a portion of code
> > <
> https://github.com/apache/incubator-metron/tree/master/metron-sensors/bro-plugin-kafka
> >
> > was created for Metron, which is a plugin for a separate open source
> > project, bro.  The code was separately pushed out by the initial author
> to
> > the bro community under the 3-Clause BSD license
> > , and
> > some important
> > enhancements
> > <
> https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a8353b4a0a8
> >
> > have been made to it in that separate community, which we would like to
> > include, in addition to some recent changes
> > <
> https://github.com/apache/incubator-metron/commit/a2452a25caffdd8c35fd9efe0ed49ce0dd2e3781
> >
> > that have been made in the Metron code base as well (i.e. we are not
> simply
> > pulling the code down from the external project).  This was discussed
> > recently
> > <
> https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E
> >
> > on the dev mailing list, and we wanted to get some clarification before
> > moving forward.
> >
> > *Questions*
> > 1. Is it valid to assume that, as Casey mentioned
> > <
> https://lists.apache.org/thread.html/7468692c96ed0cb012ac9014229694ba8edf3a3b3b55d346eec57019@%3Cdev.metron.apache.org%3E
> >,
> > these are two separate plugins and we can simply make modifications to
> our
> > code base to resolve the current multithreading issue?
> >
> > 2. If we do 1, will this require a line in our LICENSE file as a
> derivative
> > work?
> >
> > Anyone, feel free to correct my historical understanding here - I wasn't
> > closely involved with the project at the time this all occurred.  Thanks,
> >
> > Jon
> > --
> >
> > Jon
>
> --

Jon

[GitHub] incubator-metron issue #500: METRON-795: Install Metron REST with Ambari MPa...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/500
  
Just merged in master and pushed changes out to address the feedback so 
far.  

One question I have:  which account should this service run as?  My 
assumption would be the "metron" user but then we would likely need to update 
the permissions in HDFS which may be out of scope for this PR.  As it stands 
now it runs as root but I don't think this is correct.  Thoughts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #500: METRON-795: Install Metron REST with Amb...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/500#discussion_r110771928
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py
 ---
@@ -0,0 +1,83 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.core.exceptions import ComponentIsNotRunning
+from resource_management.core.exceptions import ExecutionFailed
+from resource_management.core.resources.system import File
+from resource_management.core.source import Template
+from resource_management.libraries.functions.format import format
+from resource_management.libraries.script import Script
+from resource_management.core.resources.system import Execute
+
+from rest_commands import RestCommands
+
+
+class RestMaster(Script):
+def get_component_name(self):
+# TODO add this at some point - currently will cause problems with 
hdp-select
+# return "parser-master"
+pass
+
+def install(self, env):
+from params import params
+env.set_params(params)
+commands = RestCommands(params)
+commands.setup_repo()
+self.install_packages(env)
+
+def configure(self, env, upgrade_type=None, config_dir=None):
+from params import params
+env.set_params(params)
+
+File(format("{metron_config_path}/application.yml"),
--- End diff --

Good idea.  I changed the name to rest_application.yml.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [MENTORS] Licensing

[P. Taylor Goetz]

In my opinion, I think you are probably okay including it. You would want to 
retain any license headers, and also make sure your NOTICE file is updated to 
include the copyright information.

I’m not by any means an expert in these matters, so I would recommend getting a 
second opinion by leveraging the collective intelligence of the 
general@incubator (while you are a podling or even afterward) or 
legal-discuss@. The latter list is often home to some important legal 
discussions, and I would recommend PMC members read and/or subscribe to that 
list.

-Taylor

> On Apr 5, 2017, at 4:50 PM, zeo...@gmail.com  wrote:
> 
> *Background*
> We have a situation where a portion of code
> 
> was created for Metron, which is a plugin for a separate open source
> project, bro.  The code was separately pushed out by the initial author to
> the bro community under the 3-Clause BSD license
> , and
> some important
> enhancements
> 
> have been made to it in that separate community, which we would like to
> include, in addition to some recent changes
> 
> that have been made in the Metron code base as well (i.e. we are not simply
> pulling the code down from the external project).  This was discussed
> recently
> 
> on the dev mailing list, and we wanted to get some clarification before
> moving forward.
> 
> *Questions*
> 1. Is it valid to assume that, as Casey mentioned
> ,
> these are two separate plugins and we can simply make modifications to our
> code base to resolve the current multithreading issue?
> 
> 2. If we do 1, will this require a line in our LICENSE file as a derivative
> work?
> 
> Anyone, feel free to correct my historical understanding here - I wasn't
> closely involved with the project at the time this all occurred.  Thanks,
> 
> Jon
> -- 
> 
> Jon

[GitHub] incubator-metron issue #522: METRON-839: RPM build should happen after archi...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/522
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] New Stellar Functions

[Matt Foley]

Hi Kyle,
For now at least, it seems to me that something with as much core usefulness, 
and presumably non-customer-specific, as string concat should go into 
metron-common.  That said, I suspect Mike is right that JOIN would be 
sufficient.  But if that fits your use case poorly, just say so.

--Matt

On 4/10/17, 10:13 AM, "Michael Miklavcic"  wrote:

Hey Kyle,

It probably belongs here -

https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
There is an existing JOIN function for strings - might this suit your
needs? I didn't see a unit test for it, so it would probably be good for us
to backfill with a test here as well. I can submit a PR for it, or if
you're already in that code, you're welcome to also -

https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java

e.g.
Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
System.out.println(joined);
Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
System.out.println(joined2);

Output I get is:
A:B:C:D
ABCD

AFA where to put these functions, I believe we have a number of options now
that we have the ability to add to the storm topology classpath and
sideload jars.
- https://github.com/apache/incubator-metron/pull/204
- https://github.com/apache/incubator-metron/pull/468

I think that if the functions are unique to a customer, they should
probably be built as a stand-alone Maven project. I believe Otto is working
on this ATM if I'm not mistaken. If there is universal (across all
functions in the system, whether parsing, analytics or otherwise) then they
should probably go in with the dsl package in metron-common. At some point
we might want to make Stellar its own module, but there is some work there.

Best,
Mike


On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson 
wrote:

> I have the need for a new Stellar function to perform string 
concatenation.
> I have it implemented but am curious about where new functions should live
> given the new capabilities around 3rd party Stellar function libraries.
>
> So, I guess my question is, should this function live in:
> 1) metron-common with the other string functions
> 2) another metron project
> 3) as a standalone project and not part of the metron source tree
>
> While I'm specifically asking about this case, I think it's also 
worthwhile
> that we think about where other new functions should live in the long 
term.
>
> Thanks!
>
> -Kyle
>

[GitHub] incubator-metron pull request #514: METRON-829 Use Fastcapa with Kerberos

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/514


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #522: METRON-839: RPM build should happen after archi...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/522
  
+1 - looks good, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #489: METRON-623: Management UI

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/489
  
Master has been merged in.  I tested this on full dev but ran into an issue 
with the RPMs being created before the archives were ready, resulting in the 
error you see.  If I include the fix from 
https://github.com/apache/incubator-metron/pull/522 it works.

I'm having trouble bringing down the latest image for quick dev but I will 
test it there once my internet behaves better.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #522: METRON-839: RPM build should happen afte...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/incubator-metron/pull/522

METRON-839: RPM build should happen after archives are built

## Contributor Comments
This PR separates the build process into 2 sequential tasks:  Archive build 
and RPM build.  This ensures the archives have been built before RPMs are 
created from them.

I tested this against full dev and verified the RPMs were properly created.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ x If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/522.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #522


commit 4a9bae4fb22e449fb4f1395faed9779ef53db537
Author: merrimanr 
Date:   2017-04-10T18:25:03Z

Archive and RPM builds are now separate




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [MENTORS] Create Additional Repo for Apache Metron Incubating

[zeo...@gmail.com]

Thanks!  Would you mind taking a look at my other thread, here
?
It is related to the use of this new repo.  Much appreciated,

Jon

On Mon, Apr 10, 2017 at 1:54 PM P. Taylor Goetz  wrote:

> Done. It may take ~1hr. before the repo is created.
>
> Once you are a TLP you will be able to do this yourself. But for now
> because you are still an incubator podling it has to be done by an IPMC
> member.
>
> -Taylor
>
> > On Apr 10, 2017, at 11:25 AM, Nick Allen  wrote:
> >
> > Hi Mentors -
> >
> > We are in need of an additional Git repository for Metron.  You can view
> > the discussion thread here [1], if you have not followed the discussion.
> I
> > created a ticket INFRA-13828 [2], to request the repository and was
> > notified that
> > we need to use https://reporeq.apache.org/.  The ticket also suggests
> that
> > a mentor may need to do this.
> >
> > In trying to do this myself, there is a drop-down field "PMCs" where I
> need
> > to select "Metron".  But it does not allow me to select anything.  It is
> > stuck on "Your PMCs" which then screws up the "Generated Name" field.
> >
> > Can you try going to https://reporeq.apache.org/, login with your Apache
> > ID, then fill-in the form to create the repo?  Here are all the fields
> that
> > need populated.
> >
> >   - Repository Name: incubator-metron-bro-plugin-kafka
> >   - Repository Description: Apache Metron (incubating) - Bro/Kafka
> >   Integration
> >   - Commit Notification List: comm...@metron.incubator.apache.org
> >   - GitHub Notification List: dev@metron.incubator.apache.org
> >
> > Let me know, if we need to take another approach.
> >
> > Thanks
> >
> >
> > [1]
> >
> https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E
> >
> > [2] https://issues.apache.org/jira/browse/INFRA-13828
>
> --

Jon

Re: [MENTORS] Create Additional Repo for Apache Metron Incubating

[P. Taylor Goetz]

Done. It may take ~1hr. before the repo is created.

Once you are a TLP you will be able to do this yourself. But for now because 
you are still an incubator podling it has to be done by an IPMC member.

-Taylor

> On Apr 10, 2017, at 11:25 AM, Nick Allen  wrote:
> 
> Hi Mentors -
> 
> We are in need of an additional Git repository for Metron.  You can view
> the discussion thread here [1], if you have not followed the discussion. I
> created a ticket INFRA-13828 [2], to request the repository and was
> notified that
> we need to use https://reporeq.apache.org/.  The ticket also suggests that
> a mentor may need to do this.
> 
> In trying to do this myself, there is a drop-down field "PMCs" where I need
> to select "Metron".  But it does not allow me to select anything.  It is
> stuck on "Your PMCs" which then screws up the "Generated Name" field.
> 
> Can you try going to https://reporeq.apache.org/, login with your Apache
> ID, then fill-in the form to create the repo?  Here are all the fields that
> need populated.
> 
>   - Repository Name: incubator-metron-bro-plugin-kafka
>   - Repository Description: Apache Metron (incubating) - Bro/Kafka
>   Integration
>   - Commit Notification List: comm...@metron.incubator.apache.org
>   - GitHub Notification List: dev@metron.incubator.apache.org
> 
> Let me know, if we need to take another approach.
> 
> Thanks
> 
> 
> [1]
> https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E
> 
> [2] https://issues.apache.org/jira/browse/INFRA-13828

[GitHub] incubator-metron issue #514: METRON-829 Use Fastcapa with Kerberos

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/514
  
Kicking Travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #514: METRON-829 Use Fastcapa with Kerberos

[nickwallen]

GitHub user nickwallen reopened a pull request:

https://github.com/apache/incubator-metron/pull/514

METRON-829 Use Fastcapa with Kerberos

Yes, you can use Fastcapa with Kerberos with no code change.  I describe 
the entire process in the README.  

This PR is dependent on #509 which explains the extra commits that are 
currently visible.

## Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-829

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #514


commit a062c440751088bb5b3ea9df1cfa90136e820fce
Author: Nick Allen 
Date:   2017-04-05T20:55:42Z

METRON-829 Use Fastcapa with Kerberos

commit 3165e5f4c5aad51560be966a26b97aae65ef1c7d
Author: Nick Allen 
Date:   2017-04-05T21:08:53Z

Grammar is king

commit d9575a3481cb5cda4217a8b8d68590109e9c6c6b
Author: Nick Allen 
Date:   2017-04-10T14:37:01Z

Removed JAAS configuration steps which are not needed with Fastcapa.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #514: METRON-829 Use Fastcapa with Kerberos

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/incubator-metron/pull/514


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #514: METRON-829 Use Fastcapa with Kerberos

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/514
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #514: METRON-829 Use Fastcapa with Kerberos

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/514
  
@nickwallen kick travis; that's an intermittent error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] New Stellar Functions

[Otto Fowler]

I am working on that.  What i’m shooting for is having archetypes for
parsers, stellar function libraries, and   - metron extensions.

The stellar work will be a follow on however.


On April 10, 2017 at 13:13:46, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hey Kyle,

It probably belongs here -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
There is an existing JOIN function for strings - might this suit your
needs? I didn't see a unit test for it, so it would probably be good for us
to backfill with a test here as well. I can submit a PR for it, or if
you're already in that code, you're welcome to also -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java

e.g.
Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
System.out.println(joined);
Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
System.out.println(joined2);

Output I get is:
A:B:C:D
ABCD

AFA where to put these functions, I believe we have a number of options now
that we have the ability to add to the storm topology classpath and
sideload jars.
- https://github.com/apache/incubator-metron/pull/204
- https://github.com/apache/incubator-metron/pull/468

I think that if the functions are unique to a customer, they should
probably be built as a stand-alone Maven project. I believe Otto is working
on this ATM if I'm not mistaken. If there is universal (across all
functions in the system, whether parsing, analytics or otherwise) then they
should probably go in with the dsl package in metron-common. At some point
we might want to make Stellar its own module, but there is some work there.

Best,
Mike


On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson 
wrote:

> I have the need for a new Stellar function to perform string
concatenation.
> I have it implemented but am curious about where new functions should
live
> given the new capabilities around 3rd party Stellar function libraries.
>
> So, I guess my question is, should this function live in:
> 1) metron-common with the other string functions
> 2) another metron project
> 3) as a standalone project and not part of the metron source tree
>
> While I'm specifically asking about this case, I think it's also
worthwhile
> that we think about where other new functions should live in the long
term.
>
> Thanks!
>
> -Kyle
>

Re: [DISCUSS] New Stellar Functions

[Michael Miklavcic]

Hey Kyle,

It probably belongs here -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
There is an existing JOIN function for strings - might this suit your
needs? I didn't see a unit test for it, so it would probably be good for us
to backfill with a test here as well. I can submit a PR for it, or if
you're already in that code, you're welcome to also -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java

e.g.
Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
System.out.println(joined);
Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
System.out.println(joined2);

Output I get is:
A:B:C:D
ABCD

AFA where to put these functions, I believe we have a number of options now
that we have the ability to add to the storm topology classpath and
sideload jars.
- https://github.com/apache/incubator-metron/pull/204
- https://github.com/apache/incubator-metron/pull/468

I think that if the functions are unique to a customer, they should
probably be built as a stand-alone Maven project. I believe Otto is working
on this ATM if I'm not mistaken. If there is universal (across all
functions in the system, whether parsing, analytics or otherwise) then they
should probably go in with the dsl package in metron-common. At some point
we might want to make Stellar its own module, but there is some work there.

Best,
Mike


On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson 
wrote:

> I have the need for a new Stellar function to perform string concatenation.
> I have it implemented but am curious about where new functions should live
> given the new capabilities around 3rd party Stellar function libraries.
>
> So, I guess my question is, should this function live in:
> 1) metron-common with the other string functions
> 2) another metron project
> 3) as a standalone project and not part of the metron source tree
>
> While I'm specifically asking about this case, I think it's also worthwhile
> that we think about where other new functions should live in the long term.
>
> Thanks!
>
> -Kyle
>

[GitHub] incubator-metron issue #519: METRON-832 Fixed CEF parser for Palo Alto FITW

[kylerichardson]

Github user kylerichardson commented on the issue:

https://github.com/apache/incubator-metron/pull/519
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[MENTORS] Create Additional Repo for Apache Metron Incubating

[Nick Allen]

Hi Mentors -

We are in need of an additional Git repository for Metron.  You can view
the discussion thread here [1], if you have not followed the discussion. I
created a ticket INFRA-13828 [2], to request the repository and was
notified that
we need to use https://reporeq.apache.org/.  The ticket also suggests that
a mentor may need to do this.

In trying to do this myself, there is a drop-down field "PMCs" where I need
to select "Metron".  But it does not allow me to select anything.  It is
stuck on "Your PMCs" which then screws up the "Generated Name" field.

Can you try going to https://reporeq.apache.org/, login with your Apache
ID, then fill-in the form to create the repo?  Here are all the fields that
need populated.

   - Repository Name: incubator-metron-bro-plugin-kafka
   - Repository Description: Apache Metron (incubating) - Bro/Kafka
   Integration
   - Commit Notification List: comm...@metron.incubator.apache.org
   - GitHub Notification List: dev@metron.incubator.apache.org

Let me know, if we need to take another approach.

Thanks


[1]
https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E

[2] https://issues.apache.org/jira/browse/INFRA-13828

[GitHub] incubator-metron issue #514: METRON-829 Use Fastcapa with Kerberos

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/514
  
Ok.  The instructions should be ready-to-go now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110672618
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
--- End diff --

Okay, thanks. I stand corrected about the URI/URL.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #500: METRON-795: Install Metron REST with Amb...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/500#discussion_r110668381
  
--- Diff: metron-interface/metron-rest/src/main/scripts/metron-rest ---
@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# metron rest application service
+# description: rest application
+# processname: metron-rest
+#
+if [ -f /etc/rc.d/init.d/functions ]; then
+. /etc/rc.d/init.d/functions
+fi
+
+NAME=metron-rest
+DESC="Metron REST Application"
+SCRIPTNAME=/etc/init.d/$NAME
+LOGFILE="/var/log/metron-rest.log"
+METRON_VERSION=0.3.1
+METRON_HOME=/usr/metron/$METRON_VERSION
+METRON_REST_PORT=${METRON_REST_PORT:-8082}
+DAEMON="java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar 
--spring.config.location=$METRON_HOME/config/application.yml 
--spring.profiles.active=dev --server.port=$METRON_REST_PORT"
--- End diff --

The dev profile shouldn't be activated by default in a production setup.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #500: METRON-795: Install Metron REST with Amb...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/500#discussion_r110668033
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py
 ---
@@ -0,0 +1,83 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.core.exceptions import ComponentIsNotRunning
+from resource_management.core.exceptions import ExecutionFailed
+from resource_management.core.resources.system import File
+from resource_management.core.source import Template
+from resource_management.libraries.functions.format import format
+from resource_management.libraries.script import Script
+from resource_management.core.resources.system import Execute
+
+from rest_commands import RestCommands
+
+
+class RestMaster(Script):
+def get_component_name(self):
+# TODO add this at some point - currently will cause problems with 
hdp-select
+# return "parser-master"
+pass
+
+def install(self, env):
+from params import params
+env.set_params(params)
+commands = RestCommands(params)
+commands.setup_repo()
+self.install_packages(env)
+
+def configure(self, env, upgrade_type=None, config_dir=None):
+from params import params
+env.set_params(params)
+
+File(format("{metron_config_path}/application.yml"),
--- End diff --

Is this a good name for the config file? There should be some reference to 
it being the application.yml for the rest server.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/521

METRON-835 Use Profiler with Kerberos

## Contributor Comments

* Enhanced the Kerberos documentation to outline additional steps needed to 
use the Profiler with Kerberos.  
* Enhanced some of the steps to help others avoid stupid user mistakes that 
I made when setting up a Kerberized-environment.

## Pull Request Checklist

- [x] Is there a JIRA ticket associated with this PR?
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-835

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #521


commit de24191976c7d7fbcda5a72c5064c006bbaacf00
Author: Nick Allen 
Date:   2017-04-10T13:58:10Z

METRON-835 Use Profiler with Kerberos




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/507
  
FYI I was able to get this working.  Mike's docs are 100% correct, there 
were just a few minor steps that tripped me up (like using relative paths 
instead of absolute paths.)  I updated those just to help others avoid the same 
stupid user mistakes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #519: METRON-832 Fixed CEF parser for Palo Alt...

[simonellistonball]

GitHub user simonellistonball reopened a pull request:

https://github.com/apache/incubator-metron/pull/519

METRON-832 Fixed CEF parser for Palo Alto FITW

## Contributor Comments

This is a minor fix to the pattern based on some data found in the wild.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/incubator-metron METRON-832

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #519


commit 3d87488cc3de21675008d156d51addc8274d82ae
Author: Simon Elliston Ball 
Date:   2017-04-07T12:33:41Z

Added test for Palo data found in the wild

commit 616318b5ab2472bd7779402a55a89893f27bf63a
Author: Simon Elliston Ball 
Date:   2017-04-07T12:34:08Z

Fixed matching pattern to handle Palo data and match syslog RFC




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #519: METRON-832 Fixed CEF parser for Palo Alt...

[simonellistonball]

Github user simonellistonball closed the pull request at:

https://github.com/apache/incubator-metron/pull/519


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #514: METRON-829 Use Fastcapa with Kerberos

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/514
  
@cestella Thanks.  Duh. That makes sense.  

Thinking back, I probably just needed the JAAS stuff since I was trying to 
test with the Kafka Console Producer before landing data with Fastcapa.  I'll 
work through the steps again and remove the JAAS stuff so as not confuse the 
instructions.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #514: METRON-829 Use Fastcapa with Kerberos

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/514
  
That shouldn't be required as librdkafka doesn't know anything about JAAS 
since JAAS is a Java thing and librdkafka is decidedly not.  It does ticket 
renewal literally by having you specify the keytab and the kinit command (the 
default one works).  I validated that by botching the kinit command in the 
librdkafka config and having it scream bloody murder in the logs periodically 
and stop working.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #505: METRON-817: Customise output file path p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/505


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #515: METRON-826: Ambari MPack should utilize ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/515


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110629490
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
--- End diff --

That is not to say the CHOP cannot be useful however


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110629309
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
--- End diff --

If we are going to do URL/URI work, we should create functions off of the 
URI / URL classes and use their specific parsing for uri named component parts


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110584819
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
--- End diff --

IMHO the `chop` function would be helpful when fine-tuning string 
extraction operations through stellar (E.g. when extracting specific variations 
of hostnames or URLs). 

Aside of that, I went ahead and added the `chomp` string function as well 
since I agree that it would be of use.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110582764
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
+  , description = "Remove the last character from a String"
+  , params = { "the String to chop last character from, may be 
null"}
+  , returns = "String without last character, null if null String 
input"
+  )
+  public static class chop extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings.size() == 0) {
+throw new IllegalArgumentException("[CHOP] missing argument: 
string to be chopped");
+  }
+
+  String chop = StringUtils.chop((String) strings.get(0));
+  return chop;
+}
+  }
+
+  @Stellar( name = "PREPENDIFMISSING"
--- End diff --

Sure, changed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110582728
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
+  , description = "Remove the last character from a String"
+  , params = { "the String to chop last character from, may be 
null"}
+  , returns = "String without last character, null if null String 
input"
+  )
+  public static class chop extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings.size() == 0) {
+throw new IllegalArgumentException("[CHOP] missing argument: 
string to be chopped");
+  }
+
+  String chop = StringUtils.chop((String) strings.get(0));
+  return chop;
+}
+  }
+
+  @Stellar( name = "PREPENDIFMISSING"
+  , description = "Prepends the prefix to the start of the string 
if the string does not already start with any of the prefixes"
+  , params = { "str - The string.", "prefix - The prefix to 
prepend to the start of the string", "prefixes - Additional prefixes that are 
valid" }
+  , returns = "A new String if prefix was prepended, the same 
string otherwise."
+  )
+  public static class prependifmissing extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  String prefixed;
+  switch (strings.size()) {
+case 2: prefixed = 
org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), 
(String) strings.get(1));
--- End diff --

Changed the import statement to use lang3.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110581823
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
+  , description = "Remove the last character from a String"
+  , params = { "the String to chop last character from, may be 
null"}
+  , returns = "String without last character, null if null String 
input"
+  )
+  public static class chop extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings.size() == 0) {
+throw new IllegalArgumentException("[CHOP] missing argument: 
string to be chopped");
+  }
+
+  String chop = StringUtils.chop((String) strings.get(0));
+  return chop;
+}
+  }
+
+  @Stellar( name = "PREPENDIFMISSING"
+  , description = "Prepends the prefix to the start of the string 
if the string does not already start with any of the prefixes"
+  , params = { "str - The string.", "prefix - The prefix to 
prepend to the start of the string", "prefixes - Additional prefixes that are 
valid" }
+  , returns = "A new String if prefix was prepended, the same 
string otherwise."
+  )
+  public static class prependifmissing extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  String prefixed;
+  switch (strings.size()) {
+case 2: prefixed = 
org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), 
(String) strings.get(1));
+  break;
+case 3: prefixed = 
org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), 
(String) strings.get(1), (String) strings.get(2));
+  break;
+default: throw new IllegalArgumentException("[PREPENDIFMISSING] 
incorrect arguments. Usage: PREPENDIFMISSING   [...]");
--- End diff --

Now included the arguments provided by user as well, apart from Usage info.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110581688
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
 ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws 
Exception {
   public void testFormatWithMissingArguments() throws Exception {
 run("FORMAT('missing arg: %d')", Collections.emptyMap());
   }
+
+
+  /**
+  * CHOP StringFunction
+  * @throws Exception
+  */
+  @Test
+  public void testChop() throws Exception {
+Assert.assertEquals("ab",   run("CHOP('abc')", new HashedMap()));
+Assert.assertEquals(null,   run("CHOP(null)", new HashedMap()));
+Assert.assertEquals("abc",  run("CHOP(msg)", ImmutableMap.of("msg", 
"abc\r\n")));
+Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", 
"\n")));
+  }
+
+  @Test(expected = ParseException.class)
+  public void testChopWithMissingArguments() throws Exception {
+run("CHOP()", Collections.emptyMap());
+  }
+
+  /**
+   * PREPENDIFMISSING StringFunction
+   * @throws Exception
+   */
+  @Test
+  public void testPrependIfMissing() throws Exception {
+Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 
'xyz')", new HashedMap()));
+Assert.assertEquals("xyzXYZabc",  run("PREPENDIFMISSING('XYZabc', 
'xyz', 'mno')", new HashedMap()));
--- End diff --

Done, added.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110581713
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
 ---
@@ -343,4 +343,89 @@ public Object apply(List args) {
   return String.format(format, formatArgs);
 }
   }
+
+  @Stellar( name="CHOP"
+  , description = "Remove the last character from a String"
+  , params = { "the String to chop last character from, may be 
null"}
+  , returns = "String without last character, null if null String 
input"
+  )
+  public static class chop extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings.size() == 0) {
+throw new IllegalArgumentException("[CHOP] missing argument: 
string to be chopped");
+  }
+
+  String chop = StringUtils.chop((String) strings.get(0));
+  return chop;
+}
+  }
+
+  @Stellar( name = "PREPENDIFMISSING"
+  , description = "Prepends the prefix to the start of the string 
if the string does not already start with any of the prefixes"
+  , params = { "str - The string.", "prefix - The prefix to 
prepend to the start of the string", "prefixes - Additional prefixes that are 
valid" }
+  , returns = "A new String if prefix was prepended, the same 
string otherwise."
+  )
+  public static class prependifmissing extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  String prefixed;
+  switch (strings.size()) {
+case 2: prefixed = 
org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), 
(String) strings.get(1));
+  break;
+case 3: prefixed = 
org.apache.commons.lang3.StringUtils.prependIfMissing((String) strings.get(0), 
(String) strings.get(1), (String) strings.get(2));
+  break;
+default: throw new IllegalArgumentException("[PREPENDIFMISSING] 
incorrect arguments. Usage: PREPENDIFMISSING   [...]");
+  }
+  return prefixed;
+}
+  }
+
+  @Stellar( name = "APPENDIFMISSING"
+  , description = "Appends the suffix to the end of the string if 
the string does not already end with any of the suffixes"
+  , params = { "str - The string.", "suffix - The suffix to append 
to the end of the string", "suffixes - Additional suffixes that are valid 
terminators" }
+  , returns = "A new String if suffix was appended, the same 
string otherwise."
+  )
+  public static class appendifmissing extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  String suffixed;
+  switch (strings.size()) {
+case 2:
+  suffixed = 
org.apache.commons.lang3.StringUtils.appendIfMissing((String) strings.get(0), 
(String) strings.get(1));
+  break;
+case 3:
+  suffixed = 
org.apache.commons.lang3.StringUtils.appendIfMissing((String) strings.get(0), 
(String) strings.get(1), (String) strings.get(2));
+  break;
+default:
+  throw new IllegalArgumentException("[APPENDIFMISSING] incorrect 
arguments. Usage: APPENDIFMISSING   [...]");
+  }
+  return suffixed;
+}
+  }
+
+  @Stellar( name = "COUNTMATCHES"
+  , description = "Counts how many times the substring appears in 
the larger string"
+  , params = { "str - the CharSequence to check, may be null", 
"sub - the substring to count, may be null"}
+  , returns = "the number of occurrences, 0 if either CharSequence 
is null"
--- End diff --

Corrected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #516: METRON-830 Adding StringFunctions to Ste...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/516#discussion_r110581665
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
 ---
@@ -207,4 +207,78 @@ public void testFormatWithNoArguments() throws 
Exception {
   public void testFormatWithMissingArguments() throws Exception {
 run("FORMAT('missing arg: %d')", Collections.emptyMap());
   }
+
+
+  /**
+  * CHOP StringFunction
+  * @throws Exception
+  */
+  @Test
+  public void testChop() throws Exception {
+Assert.assertEquals("ab",   run("CHOP('abc')", new HashedMap()));
+Assert.assertEquals(null,   run("CHOP(null)", new HashedMap()));
+Assert.assertEquals("abc",  run("CHOP(msg)", ImmutableMap.of("msg", 
"abc\r\n")));
+Assert.assertEquals("", run("CHOP(msg)", ImmutableMap.of("msg", 
"\n")));
+  }
+
+  @Test(expected = ParseException.class)
+  public void testChopWithMissingArguments() throws Exception {
+run("CHOP()", Collections.emptyMap());
+  }
+
+  /**
+   * PREPENDIFMISSING StringFunction
+   * @throws Exception
+   */
+  @Test
+  public void testPrependIfMissing() throws Exception {
+Assert.assertEquals("xyzabc", run("PREPENDIFMISSING('abc', 
'xyz')", new HashedMap()));
+Assert.assertEquals("xyzXYZabc",  run("PREPENDIFMISSING('XYZabc', 
'xyz', 'mno')", new HashedMap()));
+Assert.assertEquals(null, run("PREPENDIFMISSING(null, null, 
null)", new HashedMap()));
+Assert.assertEquals("xyz",run("PREPENDIFMISSING('', 'xyz', 
null)", new HashedMap()));
+  }
+
+  @Test(expected = ParseException.class)
+  public void testPrependIfMissingWithIncorrectArgs() throws Exception {
--- End diff --

Yup, you're right. Added try/catch blocks now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #516: METRON-830 Adding StringFunctions to Stellar - ...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/incubator-metron/pull/516
  
Hi @mattf-horton  and @ottobackwards , thank you for the review. Please see 
review comments addressed as a part of my latest commit, and also responses 
inline.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---