[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/656


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r129431043
  
--- Diff: 
metron-interface/metron-alerts/src/app/service/elasticsearch-localstorage-impl.ts
 ---
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Observable} from 'rxjs/Rx';
+import {Headers, RequestOptions} from '@angular/http';
+
+import {HttpUtil} from '../utils/httpUtil';
+import {DataSource} from './data-source';
+import {Alert} from '../model/alert';
+import {ColumnMetadata} from '../model/column-metadata';
+import {ElasticsearchUtils} from '../utils/elasticsearch-utils';
+import {
+  ALERTS_COLUMN_NAMES, ALERTS_TABLE_METADATA, ALERTS_RECENT_SEARCH,
+  ALERTS_SAVED_SEARCH, NUM_SAVED_SEARCH
+} from '../utils/constants';
+import {ColumnNames} from '../model/column-names';
+import {ColumnNamesService} from './column-names.service';
+import {TableMetadata} from '../model/table-metadata';
+import {SaveSearch} from '../model/save-search';
+import {AlertsSearchResponse} from '../model/alerts-search-response';
+import {SearchRequest} from '../model/search-request';
+
+export class ElasticSearchLocalstorageImpl extends DataSource {
+
+  private defaultColumnMetadata = [
+new ColumnMetadata('_id', 'string'),
+new ColumnMetadata('timestamp', 'date'),
+new ColumnMetadata('source:type', 'string'),
+new ColumnMetadata('ip_src_addr', 'ip'),
+new ColumnMetadata('enrichments:geo:ip_dst_addr:country', 'string'),
+new ColumnMetadata('ip_dst_addr', 'ip'),
+new ColumnMetadata('host', 'string'),
+new ColumnMetadata('alert_status', 'string')
+  ];
+
+  getAlerts(searchRequest: SearchRequest): 
Observable {
+let url = '/search/*,-*' + ElasticsearchUtils.excludeIndexName + 
'/_search';
+let request: any  = JSON.parse(JSON.stringify(searchRequest));
+request.query = { query_string: { query: searchRequest.query } };
--- End diff --

I noticed that the Alerts List is showing all telemetry sources, not only 
the ones that are marked as alerts (`is_alert: true`).  Should that be the 
case?  

Or is this something that would get fixed once we port it to the REST API?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #656: METRON-1050 Improve Docs of 'profile.period.duration'

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/656
  
+1 Thanks, @nickwallen , looks great!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[BlackHatJohnDoe]

Github user BlackHatJohnDoe commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r129420943
  
--- Diff: 
metron-maven-archetypes/metron-maven-parser-extension-archetype/src/main/resources/archetype-resources/metron-parser-__parserName__/README.md
 ---
@@ -0,0 +1,5 @@
+#${parserClassName} Parser
--- End diff --

`s/#${parserClassName}/# ${parserClassName}/`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I really like what you've done.  There is a ton of 
functionality here and it works great.  I did not find any functional problems 
beyond what you've already called out.  I think we should get this into master 
and start iterating.

Also, thank you so much for summarizing the discussions that have already 
occurred in the PR description.  That was a HUGE help to me in reviewing.

I need to hold off on the +1 until we address a couple items.  
* [ ] 
[Licensing](https://github.com/apache/metron/pull/620#issuecomment-317834046)
* [ ] 
[Tests](https://github.com/apache/metron/pull/620#issuecomment-317788710) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #669: METRON-1064: Make init script OS-agnostic

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/669

METRON-1064: Make init script OS-agnostic

## Contributor Comments
This PR makes the init scripts for metron-rest and metron-config 
OS-agnostic by:
- moving the metron env file to "/etc/default" instead of "/etc/sysconfig"
- using "su -c ..." to start the process instead of "runuser ..."
- using the appropriate kill logic instead of "killproc"

During testing I also uncovered a bug where the metron env file is not 
installed for the management UI if it and the REST application are installed on 
different hosts.  This PR fixes that bug as well. 

I have tested this for regressions in full dev and am in the process of 
testing in an Ubuntu cluster.  I will update this thread once that testing is 
completely.  Until then, please don't merge this PR.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron ubuntu-init-scripts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/669.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #669


commit 8954ad198f86236cddc2862ba0d305457ec19483
Author: merrimanr 
Date:   2017-07-12T17:34:06Z

changed location of env file

commit 7cd13e251b1a5fc0e87e89bcad10fb39ad4fbd5a
Author: Nick Allen 
Date:   2017-07-12T18:29:32Z

Altered init scripts to work on both Ubuntu and CentOS

commit 3fd5367aff42037c9f19fd5acf7eb137bbeb52b0
Author: merrimanr 
Date:   2017-07-12T18:33:13Z

Merge remote-tracking branch 'mirror/master' into ubuntu-init-scripts

commit 1551d5a65b7906ec1031e925aeb6cb5ff5ca8430
Author: merrimanr 
Date:   2017-07-12T18:36:23Z

Merge pull request #29 from nickwallen/BUG-81912

Altered init scripts to work on both Ubuntu and CentOS

commit d591ce3bf5f1688858299047e86ab05994352acc
Author: merrimanr 
Date:   2017-07-13T17:53:37Z

changed kill command, added metron env file 

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/620
  
Ran a license check on all dependencies, including transitive.

```
$ npm -g license-checker
$ cd metron-interface/metron-alerts
$ license-checker | grep licenses | sort | uniq -c | sort -n
   1├─ licenses: MIT
   1 │  ├─ licenses
   1 │  ├─ licenses: (BSD-2-Clause OR MIT OR Apache-2.0)
   1 │  ├─ licenses: (MIT AND CC-BY-3.0)
   1 │  ├─ licenses: (OFL-1.1 AND MIT)
   1 │  ├─ licenses: (WTFPL OR MIT)
   1 │  ├─ licenses: BSD-3-Clause OR MIT
   1 │  ├─ licenses: CC-BY-4.0
   1 │  ├─ licenses: Custom: 
https://github.com/substack/node-browserify
   1 │  ├─ licenses: LGPL
   1 │  ├─ licenses: LGPL-2.1+
   1 │  ├─ licenses: Public Domain
   1 │  ├─ licenses: UNLICENSED
   2 │  ├─ licenses: Apache License, Version 2.0
   2 │  ├─ licenses: Unlicense
   3 │  ├─ licenses: BSD
   7 │  ├─ licenses: BSD-2-Clause
  16 │  ├─ licenses: BSD*
  19 │  ├─ licenses: MIT*
  20 │  ├─ licenses: Apache-2.0
  24 │  ├─ licenses: BSD-3-Clause
  88 │  ├─ licenses: ISC
 762 │  ├─ licenses: MIT
```

The following licenses we need to follow-up on to see if they are 
permissible.

* `CC-BY-4.0`: https://www.apache.org/legal/resolved.html#category-b
```
├─ caniuse-db@1.0.3706
│  ├─ licenses: CC-BY-4.0
│  ├─ repository: https://github.com/Fyrd/caniuse
│  ├─ publisher: Alexis Deveria
│  ├─ email: adeve...@gmail.com
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/node_modules/caniuse-db/LICENSE
```

* `LGPL`, `LGPL-2.1+`: https://www.apache.org/legal/resolved.html#category-x
```
├─ xmldom@0.1.27
│  ├─ licenses: LGPL
│  ├─ repository: https://github.com/jindw/xmldom
│  ├─ publisher: jindw
│  ├─ email: ji...@xidea.org
│  ├─ url: http://www.xidea.org
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/node_modules/xmldom/LICENSE
```
```
├─ jschardet@1.5.0
│  ├─ licenses: LGPL-2.1+
│  ├─ repository: https://github.com/aadsm/jschardet
│  ├─ publisher: António Afonso
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/node_modules/jschardet/LICENSE
```

* `Custom` @ https://github.com/substack/node-browserify
```
├─ weak-map@1.0.5
│  ├─ licenses: Custom: https://github.com/substack/node-browserify
│  ├─ repository: https://github.com/drses/weak-map
│  ├─ publisher: Mark Miller
│  ├─ email: erig...@gmail.com
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/node_modules/weak-map/README.md
```

* `UNLICENSED`, `Unlicense`: ?
```
├─ tweetnacl@0.14.5
│  ├─ licenses: Unlicense
│  ├─ repository: https://github.com/dchest/tweetnacl-js
│  ├─ publisher: TweetNaCl-js contributors
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/node_modules/tweetnacl/LICENSE
```
```
├─ spdx-license-ids@1.2.2
│  ├─ licenses: Unlicense
│  ├─ repository: https://github.com/shinnn/spdx-license-ids
│  ├─ publisher: Shinnosuke Watanabe
│  ├─ url: https://github.com/shinnn
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/node_modules/spdx-license-ids/LICENSE
```

* Something must be wrong with our package definition.  We should indicate 
Apache for `metron-alerts`.
```
├─ metron-alerts@0.4.0
│  ├─ licenses: UNLICENSED
│  ├─ private: true
│  └─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/README.md
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r129347174
  
--- Diff: metron-interface/metron-alerts/README.md ---
@@ -0,0 +1,80 @@
+- [Caveats](#caveats)
+- [Prerequisites](#prerequisites)
+- [Development Setup](#development-setup)
+- [E2E Tests](#e2e-tests)
+- [Mpack Integration](#mpack-integration)
+- [Installing on an existing Cluster](#installing-on-an-existing-cluster)
+
+## Caveats
+* UI doesn't have an authentication module yet
+* UI uses local storage to save all the data.  A middleware needs to be 
designed and developed for persisting the data
+
+## Prerequisites
+* Elastic search should be up and running and should have some alerts 
populated by metron topologies
+* The alerts can be populated using Quick Dev, Full Dev  or any other setup
+* UI is developed using angular4 and uses angular-cli
+* node.JS >= 7.8.0
+
+## Development Setup
+
+Install all the dependent node_modules using the following command
+```
+cd incubator-metron/metron-interface/metron-alerts
+npm install
+```
+UI can be run by using the following command
+```
+./scripts/start-dev.sh
+```
+**NOTE**: *In the development mode ui by default connects to ES at 
http://node1:9200 for fetching data. If you wish to change it you can change 
the ES url at incubator-metron/metron-interface/metron-alerts/proxy.conf.json*
+
+## E2E Tests
+
+An expressjs server is available for mocking the elastic search api.
+
+1. Run e2e webserver :
+```
+cd incubator-metron/metron-interface/metron-alerts
--- End diff --

Quick hit: Remove references to "incubator-metron".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r129382584
  
--- Diff: 
metron-interface/metron-alerts/e2e/alerts-list/alerts-list.e2e-spec.ts ---
@@ -0,0 +1,119 @@
+/// 
--- End diff --

We are missing the Apache license header in this Typescript file and some 
others.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129375098
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -581,6 +634,24 @@ In the core language functions, we support basic 
functional programming primitiv
 * group - The integer that selects what group to select, starting at 1
   * Returns: The value of the group, or null if not matched or no group at 
index.
 
+### `ROUND`
+  * Description: Rounds a number to the nearest integer
+  * Input:
+* number - The number to round
+  * Returns: The nearest integer.
+
+### `SIN`
+  * Description: Returns the `sin` of a number.
--- End diff --

Returns the sine of a number.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129384031
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/math/MathOperation.java
 ---
@@ -0,0 +1,39 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+package org.apache.metron.stellar.common.utils.math;
+
+import java.util.function.Function;
+
+public class MathOperation {
+  private int maxArgs;
+  private Function operation;
+  public MathOperation(Function operation, int maxArgs) {
+this.operation = operation;
+this.maxArgs = maxArgs;
+  }
--- End diff --

If you give equal billing to binary ops in MathOperations.java, as 
requested below, then this also needs an additional constructor with BiFunction 
arg, I think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129375196
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -581,6 +634,24 @@ In the core language functions, we support basic 
functional programming primitiv
 * group - The integer that selects what group to select, starting at 1
   * Returns: The value of the group, or null if not matched or no group at 
index.
 
+### `ROUND`
+  * Description: Rounds a number to the nearest integer
+  * Input:
+* number - The number to round
+  * Returns: The nearest integer.
+
+### `SIN`
+  * Description: Returns the `sin` of a number.
+  * Input:
+* number - The number to take the `sin` of
+  * Returns: The `sin` of the number passed in.
+
+### `SQRT`
+  * Description: Returns the `sqrt` of a number.
--- End diff --

Returns the square root of a number.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129379794
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/math/StellarMathFunction.java
 ---
@@ -0,0 +1,64 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+package org.apache.metron.stellar.common.utils.math;
+
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.StellarFunction;
+
+import java.util.List;
+
+public class StellarMathFunction implements StellarFunction {
+  MathOperation _func;
+  public StellarMathFunction(MathOperations _func) {
+this._func = _func.op;
+  }
+
+  public StellarMathFunction(MathOperation _func) {
+this._func = _func;
+  }
+
+  @Override
+  public Object apply(List args, Context context) throws 
ParseException {
+if(args.size() < _func.getMaxArgs()) {
--- End diff --

If this is the logical use of getMaxArgs() then it should be called 
minRequiredArgs(), shouldn't it?  Or perhaps you need another member for that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129374680
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -238,6 +249,18 @@ In the core language functions, we support basic 
functional programming primitiv
 * bloomfilters - A list of bloom filters to merge
   * Returns: Bloom Filter or null if the list is empty
 
+### `CEIL`
+  * Description: Returns the ceiling of a number.
+  * Input:
+* number - The number to take the ceiling of
+  * Returns: The ceiling of the number passed in.
+
+### `COS`
+  * Description: Returns the `cos` of a number.
--- End diff --

Returns the cosine of a number.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129375260
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -613,6 +684,12 @@ In the core language functions, we support basic 
functional programming primitiv
 * key - Property to get the value for
   * Returns: String
 
+### `TAN`
+  * Description: Returns the `tan` of a number.
--- End diff --

You know what I want :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129384644
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MathFunctionsTest.java
 ---
@@ -38,10 +46,156 @@ public static Object run(String rule, Map variables) {
 
   @Test
   public void testAbs() {
-Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 
0)), 0, 1e-7);
-Assert.assertTrue(Double.isNaN((Double)run("ABS(value)", 
ImmutableMap.of("value", Double.NaN;
-Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 
10.5)), 10.5, 1e-7);
-Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 
-10.5)), 10.5, 1e-7);
+assertValues("ABS",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(10.5d, 10.5d);
+ put(-10.5d, 10.5d);
+   }}
+);
+  }
+
+  @Test
+  public void testSqrt() {
+assertValues("SQRT",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(25d, 5d);
+ put(-10.5d, Double.NaN);
+   }}
+);
+  }
+
+  @Test
+  public void testCeil() {
+assertValues("CEIL",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(10.5d, 11d);
+ put(-10.5d, -10d);
+   }}
+);
+  }
+
+  @Test
+  public void testFloor() {
+assertValues("FLOOR",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(10.5d, 10d);
+ put(-10.5d, -11d);
+   }}
+);
+  }
+
+  @Test
+  public void testSin() {
+assertValues("SIN",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(Math.PI/6, 0.5);
+ put(Math.PI/4, Math.sqrt(2)/2.0);
+ put(Math.PI/3, Math.sqrt(3)/2.0);
+ put(Math.PI/2, 1d);
+   }}
+);
+  }
+
+  @Test
+  public void testCos() {
+assertValues("COS",
+   new HashMap(baseExpectations) {{
+ put(0d, 1d);
+ put(Math.PI/6, Math.sqrt(3)/2.0);
+ put(Math.PI/4, Math.sqrt(2)/2.0);
+ put(Math.PI/3, 0.5d);
+ put(Math.PI/2, 0d);
+   }}
+);
+  }
+
+  @Test
+  public void testTan() {
+assertValues("TAN",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(Math.PI/6, Math.sqrt(3)/3.0);
+ put(Math.PI/4, 1d);
+ put(Math.PI/3, Math.sqrt(3));
+ put(Math.PI/2, Math.sin(Math.PI/2)/Math.cos(Math.PI/2));
+   }}
+);
+  }
+
+  @Test
+  public void testExp() {
+assertValues("EXP",
+   new HashMap(baseExpectations) {{
+ put(0d, 1d);
+ put(0.5d, Math.sqrt(Math.E));
+ put(-0.5d, 1/Math.sqrt(Math.E));
+ put(1d, Math.E);
+ put(2d, Math.E*Math.E);
+   }}
+);
+  }
+
+  @Test
+  public void testRound() {
+assertValues("ROUND",
+   new HashMap(baseExpectations) {{
+ put(0d, 0d);
+ put(0.5d, 1d);
+ put(0.4d, 0d);
+ put(-0.5d, 0d);
+   }}
+);
+  }
+
+  @Test
+  public void testNaturalLog() {
+testLog("LN", Math.E);
+  }
+
+  @Test
+  public void testLog2() {
+testLog("LOG2", 2);
+  }
+
+  @Test
+  public void testLog10() {
+testLog("LOG10", 10);
+  }
+
+  public void assertValues(String func, Map expected) {
+for(Map.Entry test : expected.entrySet()) {
+  for(String expr : ImmutableList.of(func + "(value)"
+,func + "(" + test.getKey() + ")"
+)
+ )
+  {
+if (Double.isNaN(test.getValue())) {
+  Assert.assertTrue(expr + " != NaN, where value == " + 
test.getKey(), Double.isNaN(toDouble(run(expr, ImmutableMap.of("value", 
test.getKey());
+} else {
+  Assert.assertEquals(expr + " != " + test.getValue() + " (where 
value == " + test.getKey() + ")", test.getValue(), toDouble(run(expr, 
ImmutableMap.of("value", test.getKey(, EPSILON);
+}
+  }
+}
+  }

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129374250
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -110,6 +110,8 @@ In the core language functions, we support basic 
functional programming primitiv
 | [ `BLOOM_EXISTS`](#bloom_exists) 
  |
 | [ `BLOOM_INIT`](#bloom_init) 
  |
 | [ `BLOOM_MERGE`](#bloom_merge)   
  |
+| [ `CEIL`](#ceil) 
  |
+| [ `COS`](#ceil)  
 |
--- End diff --

wrong link


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129374210
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -110,6 +110,8 @@ In the core language functions, we support basic 
functional programming primitiv
 | [ `BLOOM_EXISTS`](#bloom_exists) 
  |
 | [ `BLOOM_INIT`](#bloom_init) 
  |
 | [ `BLOOM_MERGE`](#bloom_merge)   
  |
+| [ `CEIL`](#ceil) 
  |
--- End diff --

If this means CEILING, let's call it that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r129378858
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/math/MathOperations.java
 ---
@@ -0,0 +1,66 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+package org.apache.metron.stellar.common.utils.math;
+
+import java.util.function.Function;
+
+public enum MathOperations implements Function {
+  ABS(d -> Math.abs(d)),
+  CEIL(d -> Math.ceil(d)),
+  COS(d -> Math.cos(d)),
+  FLOOR(d -> Math.floor(d)),
+  LOG10(d -> Math.log10(d)),
+  LOG2(d -> Math.log(d)/Math.log(2)),
+  LN(d -> Math.log(d)),
+  SIN(d -> Math.sin(d)),
+  SQRT(d -> Math.sqrt(d)),
+  TAN(d -> Math.tan(d)),
+  EXP(d -> Math.exp(d)),
+  ROUND(new MathOperation(d -> {
+Double val = d[0].doubleValue();
+return Double.isNaN(val)?Double.NaN:Math.round(d[0].doubleValue());
+  }, 1)),
+  ;
+
+  private static class SingleOpFunc implements Function {
+Function f;
+public SingleOpFunc(Function f) {
+  this.f = f;
+}
+@Override
+public Number apply(Number[] numbers) {
+  return f.apply(numbers[0].doubleValue());
+}
+  }
--- End diff --

Wouldn't it make sense to give BinaryOp equal billing?
Or are you going to make @simonellistonball work it out? :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129370546
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/622
  
@cestella , 
> Would this approach require scans on read in the critical path?

I don't perceive that decoding rowkeys is on any critical path.  You only 
need to look up Profile by serial number (or hash) in the case of decoding 
rowkeys.  No? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #655: METRON-1043: Profiler Should be Less Dramatic When...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/655


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/622
  
@nickwallen brought up the issue of wildcard queries on our rowkeys.  It 
has always bothered me that we can't do wildcard queries on groups.  If you 
have, for example, a single groupBy based on day of week, that's just 7 
possible values, and if you want them all you could just do 7 queries and 
combine them.  But if you have three groupBy's, and they have 7, 31, and 256 
possible values, then to simulate a wildcard query you would have to do over 
55,000 individual queries!  Of course you would just do an hbase scan, but it 
would require a full table scan to select the time range desired.

I propose that we re-order the rowkey elements to support prefix queries on 
Profile and time range, with wildcarding for primarily groups, and secondarily 
entities, ie:
\\\\\\

So if I want the results for all rows in a time range regarding entity 
"192.168.222.123" regardless of group, I can query it, and if I want all rows 
in a time range regardless of entity value or group, I can query that too, as 
efficiently as an ordinary time range query.  What do you think?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #659: METRON-1053 Relocate Metron Docker

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/659


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #659: METRON-1053 Relocate Metron Docker

[kylerichardson]

Github user kylerichardson commented on the issue:

https://github.com/apache/metron/pull/659
  
+1, thanks for taking this on


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/622
  
@mattf-horton Would this approach require scans on read in the critical 
path?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/622
  
@cestella , tl;dr: The discussion of serial numbers is a distraction.  
Let's just use the profileHash and forget the serial number.  It was a 
micro-optimization.

Answer to your question:  Two cases:
- If you have the profileHash, then you can look up the Profile using an 
hbase wildcard query for rowkey \\* , and since the profileHash 
is unique, it will be essentially as efficient as using the full rowkey.
- If you are trying to decode a rowkey and only have the serial number then 
I stated some assumptions: "The expectation is that we will seldom (almost 
never) need to reference back to the Profile specification, and the total usage 
of Profile specs will be human-scale finite, **so it is okay to "scan" the 
ProfileSpecs table to find the full Profile spec referenced by a profileSN.** 
If this is not true, use the full hash as both the rowkey in the PeriodSpecs 
table, and as the reference element in the Profile rowkeys."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[BlackHatJohnDoe]

Github user BlackHatJohnDoe commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r129361903
  
--- Diff: metron-interface/metron-alerts/scripts/start-dev.sh ---
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+SCRIPTS_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+$SCRIPTS_ROOT/../node_modules/@angular/cli/bin/ng serve --proxy-config 
proxy.conf.json
--- End diff --

I was running into port conflicts between this and the management UI.  
Also, I had to specify `--host` in order to bind to the right interface(s) so 
it came through my mapping to my host machine (I was running this in full-dev).

I don't think any additional params need to be added by default, it's just 
confusing if there's, for instance, a port conflict and `ng` asks you to 
specify `--port` but when you specify `--port` to the bash script it doesn't 
get passed to `ng`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/622
  
@mattf-horton Wouldn't you have to use the serial number to retrieve 
profiles?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #664: METRON-1059 address AvoidStarImport checkstyle warning in...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/664
  
Please resolve conflicts in this PR, then we should be on track to commit


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #663: METRON-1058 address checkstyle warnings for UnusedImports...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/663
  
@dbist please be sure to take care of the jira


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129356740
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(1000, 15, TimeUnit.MINUTES);
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to retrieve profile measurements 
over
+   * a time horizon.
+   *
+   * @param profile The name of the profile.
+   * @param entity The name of the entity.
+   * @param groups The group(s) used to sort the profile data.
+   * @param start When the time horizon starts in epoch milliseconds.
+   * @param end When the time horizon ends in epoch milliseconds.
+   * @return All of the row keys necessary to retrieve the profile 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129356631
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #663: METRON-1058 address checkstyle warnings for Unused...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/663


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/622
  
@cestella , we would not need to keep an index resident in memory.  Most of 
the time we would just have the active Profiles in memory, exactly as we do 
today.  You only need to retrieve the Profile by serial number on the rare 
occasions that you have to decode rowkeys.  That said, it's fine with me to 
just use the profileHash.  I agree it decreases complexity.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/620
  
It seems the end-to-end tests are failing for me.  This is what I did to 
run them.  The README is not totally clear to me on how these should be run.

1. `cd incubator-metron/metron-interface/metron-alerts`
1. `npm install`
1. `./scripts/start-dev.sh`
1. Open new console...
1. `cd incubator-metron/metron-interface/metron-alerts
1. `sh ./scripts/start-server-for-e2e.sh`
1. Open new console...
1. `cd incubator-metron/metron-interface/metron-alerts`
1. `npm run e2e`

And here is the output.
```
Spec started

  metron-alerts App
✗ should have all the UI elements
  - Expected 'Alerts (0 of )' to equal 'Alerts (25 of 169)' for alerts 
title
✗ should have all pagination controls and they should be working
  - Failed: No element found using locator: By(css selector, 
metron-table-pagination .fa.fa-chevron-left)
✓ should have all settings controls and they should be working
✓ play pause should start polling and stop polling
✗ should select columns from table configuration
  - Expected [ 'score' ] to equal [ 'score', '_id', 'timestamp', 
'source:type', 'ip_src_addr', 'enrichments:geo:ip_dst_addr:country', 
'ip_dst_addr', 'host', 'alert_status' ].
  - Failed: No element found using locator: By(css selector, 
app-configure-table label[for="select-deselect-_id"])

A Jasmine spec timed out. Resetting the WebDriver Control Flow.
  metron-alerts configure table
✗ should select columns from table configuration
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Expected [ 'score' ] to equal [ 'score', '_id', 'timestamp', 
'source:type', 'ip_src_addr', 'enrichments:geo:ip_dst_addr:country', 
'ip_dst_addr', 'host', 'alert_status' ].
  - Failed: No element found using locator: By(css selector, 
app-configure-table label[for="select-deselect-_id"])

  metron-alerts Search
✓ should display all the default values for saved searches
✓ should have all save search controls and they save search should be 
working
✗ should populate search items when selected on table
  - Failed: Index out of bound. Trying to access element at index: 0, 
but there are only 0 elements that match locator By(link text, US)
✗ should delete search items from search box
  - Failed: Index out of bound. Trying to access element at index: 0, 
but there are only 0 elements that match locator By(link text, US)
✗ should delete first search items from search box having multiple 
search fields
  - Failed: Index out of bound. Trying to access element at index: 0, 
but there are only 0 elements that match locator By(link text, US)
✗ manually entering search queries to search box and pressing enter 
key should search
  - Failed: No element found using locator: By(css selector, 
metron-table-pagination span)

**
*Failures*
**

1) metron-alerts App should have all the UI elements
  - Expected 'Alerts (0 of )' to equal 'Alerts (25 of 169)' for alerts title

2) metron-alerts App should have all pagination controls and they should be 
working
  - Failed: No element found using locator: By(css selector, 
metron-table-pagination .fa.fa-chevron-left)

3) metron-alerts App should select columns from table configuration
  - Expected [ 'score' ] to equal [ 'score', '_id', 'timestamp', 
'source:type', 'ip_src_addr', 'enrichments:geo:ip_dst_addr:country', 
'ip_dst_addr', 'host', 'alert_status' ].
  - Failed: No element found using locator: By(css selector, 
app-configure-table label[for="select-deselect-_id"])

4) metron-alerts configure table should select columns from table 
configuration
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Expected [ 'score' ] to equal [ 'score', '_id', 'timestamp', 
'source:type', 'ip_src_addr', 'enrichments:geo:ip_dst_addr:country', 
'ip_dst_addr', 'host', 'alert_status' ].
  - Failed: No element found using locator: By(css selector, 
app-configure-table label[for="select-deselect-_id"])

5) metron-alerts Search should populate search items when selected on table
  - Failed: Index out of bound. Trying to access element at index: 0, but 
there are only 0 elements that match locator By(link text, US)

6) metron-alerts Search should delete search items from search box
  - Failed: Index out of bound. Trying to access element at index: 0, but 
there are only 0 elements that match locator By(link 

[GitHub] metron pull request #643: METRON-1026: threatintel_taxii_load.sh throws exce...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/643#discussion_r129352107
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/taxii/TaxiiLoader.java
 ---
@@ -165,6 +167,19 @@ public static Options getOptions() {
   public static final long DEFAULT_TIME_BETWEEN_POLLS = ONE_HR_IN_MS;
 
 
+  public static boolean isStixExtractor(Extractor e) {
+if(e instanceof StixExtractor) {
+  return true;
+}
+else if(e instanceof ExtractorDecorator) {
+  ExtractorDecorator decorator = (ExtractorDecorator)e;
+  if(decorator.getUnderlyingExtractor() != null && 
decorator.getUnderlyingExtractor() instanceof StixExtractor) {
+return true;
+  }
+}
+return false;
+  }
+
--- End diff --

why not just?
`e instanceof StixExtractor || 
(ExtractorDecorator)e.getUnderlyingExtractor() instanceof StixExtractor`
this is java after all and has shortcutting... even stellar does that now 
too :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #663: METRON-1058 address checkstyle warnings for UnusedImports...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/663
  
+1, Thanks for the contribution


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[BlackHatJohnDoe]

Github user BlackHatJohnDoe commented on the issue:

https://github.com/apache/metron/pull/620
  
When doing an `npm install` I got:
```
node-pre-gyp info check checked for 
"/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/fsevents/lib/binding/Release/node-v57-darwin-x64/fse.node"
 (not found)
node-pre-gyp http GET 
https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
node-pre-gyp http 404 
https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
node-pre-gyp ERR! Tried to download(404): 
https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for fsevents@1.1.1 and 
node@8.2.1 (node-v57 ABI) (falling back to source compile with node-gyp)
node-pre-gyp http 404 status code downloading tarball 
https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
```

After the install, an `npm ls` showed:
```
npm ERR! invalid: @angular/cli@1.0.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/@angular/cli
npm ERR! extraneous: zone.js@0.7.8 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/@angular/cli/node_modules/zone.js
npm ERR! extraneous: @ngtools/json-schema@1.0.5 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/@ngtools/json-schema
npm ERR! extraneous: @ngtools/webpack@1.3.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/@ngtools/webpack
npm ERR! extraneous: autoprefixer@6.7.7 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/autoprefixer
npm ERR! extraneous: common-tags@1.4.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/common-tags
npm ERR! extraneous: css-loader@0.26.4 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/css-loader
npm ERR! extraneous: cssnano@3.10.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/cssnano
npm ERR! extraneous: denodeify@1.2.1 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/denodeify
npm ERR! extraneous: ember-cli-normalize-entity-name@1.0.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/ember-cli-normalize-entity-name
npm ERR! extraneous: ember-cli-string-utils@1.1.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/ember-cli-string-utils
npm ERR! extraneous: exports-loader@0.6.4 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/exports-loader
npm ERR! extraneous: extract-text-webpack-plugin@2.0.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/extract-text-webpack-plugin
npm ERR! extraneous: file-loader@0.10.1 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/file-loader
npm ERR! extraneous: fs-extra@2.1.2 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/fs-extra
npm ERR! extraneous: html-webpack-plugin@2.28.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/html-webpack-plugin
npm ERR! extraneous: inflection@1.12.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/inflection
npm ERR! extraneous: inquirer@3.1.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/inquirer
npm ERR! extraneous: istanbul-instrumenter-loader@2.0.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/istanbul-instrumenter-loader
npm ERR! extraneous: json-loader@0.5.4 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/json-loader
npm ERR! extraneous: karma-sourcemap-loader@0.3.7 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/karma-sourcemap-loader
npm ERR! extraneous: karma-webpack@2.0.3 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/karma-webpack
npm ERR! extraneous: less@2.7.2 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/less
npm ERR! extraneous: less-loader@2.2.3 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/less-loader
npm ERR! extraneous: node-modules-path@1.0.1 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/node-modules-path
npm ERR! extraneous: nopt@4.0.1 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/nopt
npm ERR! extraneous: opn@4.0.2 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/opn
npm ERR! extraneous: portfinder@1.0.13 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/portfinder
npm ERR! extraneous: postcss-loader@0.13.0 
/Users/johndoe/dev/metron/metron-interface/metron-alerts/node_modules/postcss-loader
npm ERR! extraneous: postcss-url@5.1.2 

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/620
  
Thanks @iraghumitra .  That fixed the build.  Will continue taking a look 
at it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #577: METRON-746: Build Custom Checkstyle and IDE formatting se...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/577
  
Can I +1 this a second time?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #659: METRON-1053 Relocate Metron Docker

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/659
  
+1, I think this will help cut down what we are seeing on the user list.  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #607: METRON-982 add new rest api for storm supervisor status

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/607
  
Hey @ottobackwards, list has been on my todo list for a while.  I'll spin 
it up today or tomorrow.  Sorry for the delay.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #655: METRON-1043: Profiler Should be Less Dramatic When Missin...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/655
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #659: METRON-1053 Relocate Metron Docker

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/659
  
Merged in @kylerichardson 's changes.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[iraghumitra]

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/620
  
@nickwallen I fixed the build issue. Looks like the package-lock.json 
didn't do its job I will check it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/667
  
Oh ProfilerIntegrationTest you've done it again!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

GitHub user ottobackwards reopened a pull request:

https://github.com/apache/metron/pull/667

METRON-1061 Add FUZZY_SCORE STELLAR function

[Apache Commons Text 
Similarity](https://commons.apache.org/proper/commons-text/javadocs/api-release/org/apache/commons/text/similarity/FuzzyScore.html)

FuzzyScore :
"A matching algorithm that is similar to the searching algorithms 
implemented in editors such as Sublime Text, TextMate, Atom and others.
One point is given for every matched character. Subsequent matches yield 
two bonus points. A higher score indicates a higher similarity."

This PR adds FUZZY_SCORE to STELLAR, exposing the Apache Commons FuzzyScore 
class functionality.

This allows for fuzzy matching of strings to queries, and composing 
statements based on thresholds related to those scores.

For example:

term = "metron"
query = "metron"

```java
16 == FUZZY_SCORE(term,query,'en')
```
+1 for each match and +2 bonus for the 2nd, 3rd, 4th, 5th, and 6th matches.



This is related to 
[METRON-1052](https://issues.apache.org/jira/browse/METRON-1052)

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
-  [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron fuzzy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/667.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #667


commit 34c9ffa97b47005ac10063a43ff1890f51c9a3cf
Author: Otto Fowler 
Date:   2017-07-25T12:31:18Z

add FUZZY_SCORE stellar function




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #668: Metron 1063 address javadoc warnings in metron-ste...

[dbist]

GitHub user dbist opened a pull request:

https://github.com/apache/metron/pull/668

Metron 1063 address javadoc warnings in metron-stellar

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dbist/metron METRON-1063

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #668


commit 55c71de5033959387c3fb7399c80ca8b433f7f21
Author: Artem Ervits 
Date:   2017-07-24T17:14:58Z

METRON-1059

commit f30e80e10260e46d0df39475bbff4fe14de5ae21
Author: Artem Ervits 
Date:   2017-07-24T18:12:35Z

METRON-1059-2

commit 07a187ae88147234384ba627bbf061620d94b58d
Author: Artem Ervits 
Date:   2017-07-24T19:03:11Z

METRON-1059-3

commit db0d61526429a86939b6a119aeffc3787d1c888d
Author: Artem Ervits 
Date:   2017-07-25T14:36:58Z

METRON-1063-0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129319336
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129318110
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(1000, 15, TimeUnit.MINUTES);
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to retrieve profile measurements 
over
+   * a time horizon.
+   *
+   * @param profile The name of the profile.
+   * @param entity The name of the entity.
+   * @param groups The group(s) used to sort the profile data.
+   * @param start When the time horizon starts in epoch milliseconds.
+   * @param end When the time horizon ends in epoch milliseconds.
+   * @return All of the row keys necessary to retrieve the profile 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129309663
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
 ---
@@ -81,20 +99,19 @@ public SaltyRowKeyBuilder(int saltDivisor, long 
duration, TimeUnit units) {
* @return All of the row keys necessary to retrieve the profile 
measurements.
*/
   @Override
-  public List rowKeys(String profile, String entity, List 
groups, long start, long end) {
+  public List encode(String profile, String entity, List 
groups, long start, long end) {
 // be forgiving of out-of-order start and end times; order is critical 
to this algorithm
 end = Math.max(start, end);
 start = Math.min(start, end);
--- End diff --

This does look fishy.  I will open a separate JIRA and track this 
separately.  Thanks, @mattf-horton!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #665: METRON-1060: Add performance timing logging to enr...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/665#discussion_r129308721
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
 ---
@@ -179,6 +180,7 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
   @SuppressWarnings("unchecked")
   @Override
   public void execute(Tuple tuple) {
+long texecute1 = System.currentTimeMillis();
--- End diff --

I wondered about this also @justinleet and the light searching I did wasn't 
all that conclusive. Do you by chance have any references?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129306225
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron issue #642: METRON-984 Stellar functions to decode encoded fields or ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/642
  
@cestella  are we still OK with this?  I'd like to get it in


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/667

METRON-1061 Add FUZZY_SCORE STELLAR function

[Apache Commons Text 
Similarity](https://commons.apache.org/proper/commons-text/javadocs/api-release/org/apache/commons/text/similarity/FuzzyScore.html)

FuzzyScore :
"A matching algorithm that is similar to the searching algorithms 
implemented in editors such as Sublime Text, TextMate, Atom and others.
One point is given for every matched character. Subsequent matches yield 
two bonus points. A higher score indicates a higher similarity."

This PR adds FUZZY_SCORE to STELLAR, exposing the Apache Commons FuzzyScore 
class functionality.

This allows for fuzzy matching of strings to queries, and composing 
statements based on thresholds related to those scores.

For example:

term = "metron"
query = "metron"

```java
16 == FUZZY_SCORE(term,query,'en')
```
+1 for each match and +2 bonus for the 2nd, 3rd, 4th, 5th, and 6th matches.



This is related to 
[METRON-1052](https://issues.apache.org/jira/browse/METRON-1052)

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
-  [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron fuzzy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/667.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #667


commit 34c9ffa97b47005ac10063a43ff1890f51c9a3cf
Author: Otto Fowler 
Date:   2017-07-25T12:31:18Z

add FUZZY_SCORE stellar function




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #650: METRON-1038: Stellar should have a better collection of b...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/650
  
I would like it to be committed to make it much easier for me to port my PR 
to it so +1 (non-binding)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #451: METRON-157: Added CEF Parser

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/451
  
Hey @Ravi0204 you might have better luck sending an email to the Metron 
user@ list with that question.  This PR is closed and the range of people 
paying attention to it is constrained.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #665: METRON-1060: Add performance timing logging to enr...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/665#discussion_r129279181
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
 ---
@@ -179,6 +180,7 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
   @SuppressWarnings("unchecked")
   @Override
   public void execute(Tuple tuple) {
+long texecute1 = System.currentTimeMillis();
--- End diff --

I don't know if we actually care given that it's logging, but iirc 
`currentTimeMillis()` returns can differ based on ntp syncs and so on (end time 
can be before start time, for example).  `nanotime()` is generally more 
reliable for this, but it can also be more expensive (on the order of millis).

Even with `currentTimeMillis()`, I think the actual call can still end up 
being slow (milli+ depending on the system).  It might be worthwhile to wrap 
the timing calls in a debug guard given that these run in the `execute()`, but 
I'm not sure if it'll make a difference in practice.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #635: METRON-1018: Integration tests should reference flux yaml...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/635
  
I'm +1 by inspection. Thanks for the contribution, it's really nice to not 
have the potential source of error from these being different.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #649: METRON-1035 Added SUM to the rules triage aggregat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/649


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #650: METRON-1038: Stellar should have a better collection of b...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/650
  
Any comments or review here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/652


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129252984
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129252678
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129252112
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129232374
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -275,27 +280,105 @@ The Profiler runs as an independent Storm topology.  
The configuration for the P
 The values can be changed on disk and then the Profiler topology must be 
restarted.
 
 
-| Setting   | Description
-|---|---
-| profiler.workers  | The number of worker processes 
to create for the topology.
-| profiler.executors| The number of executors to spawn 
per component.
-| profiler.input.topic  | The name of the Kafka topic from 
which to consume data.
-| profiler.output.topic | The name of the Kafka topic to 
which profile data is written.  Only used with profiles that use the [`triage` 
result field](#result).
-| profiler.period.duration  | The duration of each profile 
period.  This value should be defined along with 
`profiler.period.duration.units`.
-| profiler.period.duration.units| The units used to specify the 
`profiler.period.duration`.
-| profiler.ttl  | If a message has not been 
applied to a Profile in this period of time, the Profile will be forgotten and 
its resources will be cleaned up. This value should be defined along with 
`profiler.ttl.units`.
-| profiler.ttl.units| The units used to specify the 
`profiler.ttl`.
-| profiler.hbase.salt.divisor   | A salt is prepended to the row 
key to help prevent hotspotting.  This constant is used to generate the salt.  
Ideally, this constant should be roughly equal to the number of nodes in the 
Hbase cluster.
-| profiler.hbase.table  | The name of the HBase table that 
profiles are written to.
-| profiler.hbase.column.family  | The column family used to store 
profiles.
-| profiler.hbase.batch  | The number of puts that are 
written in a single batch.
-| profiler.hbase.flush.interval.seconds | The maximum number of seconds 
between batch writes to HBase.
-
-After altering the configuration, start the Profiler.
+| Setting  
 | Description
+|---   
 |---
+| [`profiler.input.topic`](#profilerinputtopic)
 | The name of the Kafka topic from which to consume data.
+| [`profiler.output.topic`](#profileroutputtopic)  
 | The name of the Kafka topic to which profile data is written.  Only used 
with profiles that define the [`triage` result field](#result).
+| [`profiler.period.duration`](#profilerperiodduration)
 | The duration of each profile period.  
+| [`profiler.period.duration.units`](#profilerperioddurationunits) 
 | The units used to specify the 
[`profiler.period.duration`](#profilerperiodduration).  
+| [`profiler.workers`](#profilerworkers)   
 | The number of worker processes for the topology.
+| [`profiler.executors`](#profilerexecutors)   
 | The number of executors to spawn per component.
+| [`profiler.ttl`](#profilerttl)   
 | If a message has not been applied to a Profile in this period of time, 
the Profile will be forgotten and its resources will be cleaned up. 
+| [`profiler.ttl.units`](#profilerttlunits)
 | The units used to specify the `profiler.ttl`.
+| [`profiler.hbase.salt.divisor`](#profilerhbasesaltdivisor)   
 | A salt is prepended to the row key to help prevent hotspotting. 
+| [`profiler.hbase.table`](#profilerhbasetable)
 | The name of the HBase table that profiles are written to.
+| [`profiler.hbase.column.family`](#profilerhbasecolumnfamily) 
 | The column family used to store profiles.
+| [`profiler.hbase.batch`](#profilerhbasebatch)
 | The number of puts that are written to HBase in a single batch.
+| 
[`profiler.hbase.flush.interval.seconds`](#profilerhbaseflushintervalseconds) | 
The maximum number of seconds between batch writes to HBase.
 
-```
-$ $METRON_HOME/start_profiler_topology.sh
-```
+### `profiler.input.topic`
+
+*Default*: indexing
+
+The name of the Kafka topic from which to consume data.  By default, the 
Profiler consumes data from the `indexing` topic so that it has access to fully 
enriched telemetry.
+
+### `profiler.output.topic`
+
+*Default*: enrichments
+
+The name of the 

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129232188
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -275,27 +280,105 @@ The Profiler runs as an independent Storm topology.  
The configuration for the P
 The values can be changed on disk and then the Profiler topology must be 
restarted.
 
 
-| Setting   | Description
-|---|---
-| profiler.workers  | The number of worker processes 
to create for the topology.
-| profiler.executors| The number of executors to spawn 
per component.
-| profiler.input.topic  | The name of the Kafka topic from 
which to consume data.
-| profiler.output.topic | The name of the Kafka topic to 
which profile data is written.  Only used with profiles that use the [`triage` 
result field](#result).
-| profiler.period.duration  | The duration of each profile 
period.  This value should be defined along with 
`profiler.period.duration.units`.
-| profiler.period.duration.units| The units used to specify the 
`profiler.period.duration`.
-| profiler.ttl  | If a message has not been 
applied to a Profile in this period of time, the Profile will be forgotten and 
its resources will be cleaned up. This value should be defined along with 
`profiler.ttl.units`.
-| profiler.ttl.units| The units used to specify the 
`profiler.ttl`.
-| profiler.hbase.salt.divisor   | A salt is prepended to the row 
key to help prevent hotspotting.  This constant is used to generate the salt.  
Ideally, this constant should be roughly equal to the number of nodes in the 
Hbase cluster.
-| profiler.hbase.table  | The name of the HBase table that 
profiles are written to.
-| profiler.hbase.column.family  | The column family used to store 
profiles.
-| profiler.hbase.batch  | The number of puts that are 
written in a single batch.
-| profiler.hbase.flush.interval.seconds | The maximum number of seconds 
between batch writes to HBase.
-
-After altering the configuration, start the Profiler.
+| Setting  
 | Description
+|---   
 |---
+| [`profiler.input.topic`](#profilerinputtopic)
 | The name of the Kafka topic from which to consume data.
+| [`profiler.output.topic`](#profileroutputtopic)  
 | The name of the Kafka topic to which profile data is written.  Only used 
with profiles that define the [`triage` result field](#result).
+| [`profiler.period.duration`](#profilerperiodduration)
 | The duration of each profile period.  
+| [`profiler.period.duration.units`](#profilerperioddurationunits) 
 | The units used to specify the 
[`profiler.period.duration`](#profilerperiodduration).  
+| [`profiler.workers`](#profilerworkers)   
 | The number of worker processes for the topology.
+| [`profiler.executors`](#profilerexecutors)   
 | The number of executors to spawn per component.
+| [`profiler.ttl`](#profilerttl)   
 | If a message has not been applied to a Profile in this period of time, 
the Profile will be forgotten and its resources will be cleaned up. 
+| [`profiler.ttl.units`](#profilerttlunits)
 | The units used to specify the `profiler.ttl`.
+| [`profiler.hbase.salt.divisor`](#profilerhbasesaltdivisor)   
 | A salt is prepended to the row key to help prevent hotspotting. 
+| [`profiler.hbase.table`](#profilerhbasetable)
 | The name of the HBase table that profiles are written to.
+| [`profiler.hbase.column.family`](#profilerhbasecolumnfamily) 
 | The column family used to store profiles.
+| [`profiler.hbase.batch`](#profilerhbasebatch)
 | The number of puts that are written to HBase in a single batch.
+| 
[`profiler.hbase.flush.interval.seconds`](#profilerhbaseflushintervalseconds) | 
The maximum number of seconds between batch writes to HBase.
 
-```
-$ $METRON_HOME/start_profiler_topology.sh
-```
+### `profiler.input.topic`
+
+*Default*: indexing
+
+The name of the Kafka topic from which to consume data.  By default, the 
Profiler consumes data from the `indexing` topic so that it has access to fully 
enriched telemetry.
+
+### `profiler.output.topic`
+
+*Default*: enrichments
+
+The name of the 

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129229395
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -129,23 +129,28 @@ This section will describe the steps required to get 
your first "Hello, World!""
 }
 ```
 
-1. Ensure that test messages are being sent to the Profiler's input topic 
in Kafka.  The Profiler will consume messages from the `inputTopic` defined in 
the [Profiler's configuration](#configuring-the-profiler).  By default this is 
the `indexing` topic.
+1. Ensure that test messages are being sent to the Profiler's input topic 
in Kafka.  The Profiler will consume messages from the `inputTopic` defined in 
the Profiler's configuration (see [Configuring the 
Profiler](#configuring-the-profiler)).  By default this is the `indexing` topic.
 
 1. Check the HBase table to validate that the Profiler is writing the 
profile.  Remember that the Profiler is flushing the profile every 15 minutes.  
You will need to wait at least this long to start seeing profile data in HBase.
 ```
 $ /usr/hdp/current/hbase-client/bin/hbase shell
 hbase(main):001:0> count 'profiler'
 ```
 
-1. Use the Profiler Client to read the profile data.  The below example 
`PROFILE_GET` command will read data written by the sample profile given above, 
if 10.0.0.1 is one of the input values for `ip_src_addr`.
-More information on configuring and using the client can be found 
[here](../metron-profiler-client).
-It is assumed that the `PROFILE_GET` client is correctly configured before 
using it.
+1. Use the [Profiler Client]((../metron-profiler-client)) to read the 
profile data.  The following `PROFILE_GET` command will read the data written 
by the `hello-world` profile. This assumes that `10.0.0.1` is one of the values 
for `ip_src_addr` contained within the telemetry consumed by the Profiler.
+
 ```
 $ bin/stellar -z node1:2181
 [Stellar]>>> PROFILE_GET( "hello-world", "10.0.0.1", PROFILE_FIXED(30, 
"MINUTES"))
 [451, 448]
 ```
 
+This result indicates that over the past 30 minutes, the Profiler 
stored two values.  In the first 15 minute period, the IP `10.0.0.1` was seen 
in 451 telemetry messages.  In the second 15 minute period, the same IP was 
seen in 448 telemetry messages.
--- End diff --

I think it would be more correct to say, `over the past 30 minutes, the 
"hello-world" Profiler stored two values related to the ip_src_addr 
"10.0.0.1".` 

(because it may have stored additional values for other ip_src_addr 
addresses that aren't mentioned here)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129219973
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -17,7 +17,7 @@ Any field contained within a message can be used to 
generate a profile.  A profi
 
 Follow these instructions to install the Profiler.  This assumes that core 
Metron has already been installed and validated.  
 
-1. Build the Metron RPMs by [following these 
instructions](../../metron-deployment#rpm).  
+1. Build the Metron RPMs (see Building the 
[RPM](../../metron-deployment#rpm)s).  
--- End diff --

Suggest: please consider changing the referenced h2 title in 
metron-deployment/README.md to "RPMs", so that you can then include the entire 
actual header anchor text in the square brackets and corresponding link name in 
the parentheses; ie, \[RPMs\]\(../../metron-deployment#rpms\)  Chunking it like 
this patch makes the fix-up semantics much harder to reason about (and gives 
ugly links that aren't whole words). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129220852
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -83,9 +83,9 @@ At this point the Profiler is running and consuming 
telemetry messages.  We have
 
 ## Getting Started
 
-This section will describe the steps required to get your first "Hello, 
World!"" profile running.  This assumes that you have successfully [installed 
the Profiler](#installation) and have it running.
+This section will describe the steps required to get your first "Hello, 
World!"" profile running.  This assumes that you have a successful Profiler 
[installation](#installation) and have it running.
--- End diff --

The anchor text in square brackets must be capitalized to match the header 
text being referenced: \[Installation\].  As is it will cause a broken link in 
Doxia.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129230420
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -275,27 +280,105 @@ The Profiler runs as an independent Storm topology.  
The configuration for the P
 The values can be changed on disk and then the Profiler topology must be 
restarted.
 
 
-| Setting   | Description
-|---|---
-| profiler.workers  | The number of worker processes 
to create for the topology.
-| profiler.executors| The number of executors to spawn 
per component.
-| profiler.input.topic  | The name of the Kafka topic from 
which to consume data.
-| profiler.output.topic | The name of the Kafka topic to 
which profile data is written.  Only used with profiles that use the [`triage` 
result field](#result).
-| profiler.period.duration  | The duration of each profile 
period.  This value should be defined along with 
`profiler.period.duration.units`.
-| profiler.period.duration.units| The units used to specify the 
`profiler.period.duration`.
-| profiler.ttl  | If a message has not been 
applied to a Profile in this period of time, the Profile will be forgotten and 
its resources will be cleaned up. This value should be defined along with 
`profiler.ttl.units`.
-| profiler.ttl.units| The units used to specify the 
`profiler.ttl`.
-| profiler.hbase.salt.divisor   | A salt is prepended to the row 
key to help prevent hotspotting.  This constant is used to generate the salt.  
Ideally, this constant should be roughly equal to the number of nodes in the 
Hbase cluster.
-| profiler.hbase.table  | The name of the HBase table that 
profiles are written to.
-| profiler.hbase.column.family  | The column family used to store 
profiles.
-| profiler.hbase.batch  | The number of puts that are 
written in a single batch.
-| profiler.hbase.flush.interval.seconds | The maximum number of seconds 
between batch writes to HBase.
-
-After altering the configuration, start the Profiler.
+| Setting  
 | Description
+|---   
 |---
+| [`profiler.input.topic`](#profilerinputtopic)
 | The name of the Kafka topic from which to consume data.
+| [`profiler.output.topic`](#profileroutputtopic)  
 | The name of the Kafka topic to which profile data is written.  Only used 
with profiles that define the [`triage` result field](#result).
+| [`profiler.period.duration`](#profilerperiodduration)
 | The duration of each profile period.  
+| [`profiler.period.duration.units`](#profilerperioddurationunits) 
 | The units used to specify the 
[`profiler.period.duration`](#profilerperiodduration).  
+| [`profiler.workers`](#profilerworkers)   
 | The number of worker processes for the topology.
+| [`profiler.executors`](#profilerexecutors)   
 | The number of executors to spawn per component.
+| [`profiler.ttl`](#profilerttl)   
 | If a message has not been applied to a Profile in this period of time, 
the Profile will be forgotten and its resources will be cleaned up. 
+| [`profiler.ttl.units`](#profilerttlunits)
 | The units used to specify the `profiler.ttl`.
+| [`profiler.hbase.salt.divisor`](#profilerhbasesaltdivisor)   
 | A salt is prepended to the row key to help prevent hotspotting. 
+| [`profiler.hbase.table`](#profilerhbasetable)
 | The name of the HBase table that profiles are written to.
+| [`profiler.hbase.column.family`](#profilerhbasecolumnfamily) 
 | The column family used to store profiles.
+| [`profiler.hbase.batch`](#profilerhbasebatch)
 | The number of puts that are written to HBase in a single batch.
+| 
[`profiler.hbase.flush.interval.seconds`](#profilerhbaseflushintervalseconds) | 
The maximum number of seconds between batch writes to HBase.
--- End diff --

All these links now work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #656: METRON-1050 Improve Docs of 'profile.period.durati...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/656#discussion_r129230326
  
--- Diff: metron-analytics/metron-profiler/README.md ---
@@ -129,23 +129,28 @@ This section will describe the steps required to get 
your first "Hello, World!""
 }
 ```
 
-1. Ensure that test messages are being sent to the Profiler's input topic 
in Kafka.  The Profiler will consume messages from the `inputTopic` defined in 
the [Profiler's configuration](#configuring-the-profiler).  By default this is 
the `indexing` topic.
+1. Ensure that test messages are being sent to the Profiler's input topic 
in Kafka.  The Profiler will consume messages from the `inputTopic` defined in 
the Profiler's configuration (see [Configuring the 
Profiler](#configuring-the-profiler)).  By default this is the `indexing` topic.
 
 1. Check the HBase table to validate that the Profiler is writing the 
profile.  Remember that the Profiler is flushing the profile every 15 minutes.  
You will need to wait at least this long to start seeing profile data in HBase.
 ```
 $ /usr/hdp/current/hbase-client/bin/hbase shell
 hbase(main):001:0> count 'profiler'
 ```
 
-1. Use the Profiler Client to read the profile data.  The below example 
`PROFILE_GET` command will read data written by the sample profile given above, 
if 10.0.0.1 is one of the input values for `ip_src_addr`.
-More information on configuring and using the client can be found 
[here](../metron-profiler-client).
-It is assumed that the `PROFILE_GET` client is correctly configured before 
using it.
+1. Use the [Profiler Client]((../metron-profiler-client)) to read the 
profile data.  The following `PROFILE_GET` command will read the data written 
by the `hello-world` profile. This assumes that `10.0.0.1` is one of the values 
for `ip_src_addr` contained within the telemetry consumed by the Profiler.
+
 ```
 $ bin/stellar -z node1:2181
 [Stellar]>>> PROFILE_GET( "hello-world", "10.0.0.1", PROFILE_FIXED(30, 
"MINUTES"))
 [451, 448]
 ```
 
+This result indicates that over the past 30 minutes, the Profiler 
stored two values.  In the first 15 minute period, the IP `10.0.0.1` was seen 
in 451 telemetry messages.  In the second 15 minute period, the same IP was 
seen in 448 telemetry messages.
+
+More information on configuring and using the Profiler client can be 
found [here](../metron-profiler-client).
+It is assumed that the `PROFILE_GET` client is correctly configured before 
using it.
--- End diff --

Suggest: It is assumed that the `PROFILE_GET` client is correctly 
configured to match the Profile configuration before using it to read that 
Profile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129227588
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to