[GitHub] metron pull request #702: METRON-1114: Add group by capabilities to search R...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/702#discussion_r137762973
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -330,43 +334,112 @@ public void update(Document update, Optional 
index) throws IOException {
 return latestIndices.values().toArray(new 
String[latestIndices.size()]);
   }
 
-  public void addFacetFields(SearchSourceBuilder searchSourceBuilder, 
List fields) {
-for(String field: fields) {
-  searchSourceBuilder = searchSourceBuilder.aggregation(new 
TermsBuilder(getAggregationName(field)).field(field));
+  private org.elasticsearch.search.sort.SortOrder 
getElasticsearchSortOrder(
+  org.apache.metron.indexing.dao.search.SortOrder sortOrder) {
+return sortOrder == 
org.apache.metron.indexing.dao.search.SortOrder.DESC ?
+org.elasticsearch.search.sort.SortOrder.DESC : 
org.elasticsearch.search.sort.SortOrder.ASC;
+  }
+
+  private Order getElasticsearchGroupOrder(GroupOrder groupOrder) {
+if (groupOrder.getGroupOrderType() == GroupOrderType.TERM) {
+  return groupOrder.getSortOrder() == SortOrder.ASC ? Order.term(true) 
: Order.term(false);
+} else {
+  return groupOrder.getSortOrder() == SortOrder.ASC ? 
Order.count(true) : Order.count(false);
 }
   }
 
   public Map> getFacetCounts(List 
fields, Aggregations aggregations, Map commonColumnMetadata) 
{
 Map> fieldCounts = new HashMap<>();
 for (String field: fields) {
   Map valueCounts = new HashMap<>();
-  Aggregation aggregation = 
aggregations.get(getAggregationName(field));
-  if (aggregation instanceof LongTerms) {
-LongTerms longTerms = (LongTerms) aggregation;
-FieldType type = commonColumnMetadata.get(field);
-if (FieldType.IP.equals(type)) {
-  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(IpFieldMapper.longToIp((Long) bucket.getKey()), 
bucket.getDocCount()));
-} else if (FieldType.BOOLEAN.equals(type)) {
-  longTerms.getBuckets().stream().forEach(bucket -> {
-String key = (Long) bucket.getKey() == 1 ? "true" : "false";
-valueCounts.put(key, bucket.getDocCount());
-  });
-} else {
-  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
-}
-  } else if (aggregation instanceof DoubleTerms) {
-DoubleTerms doubleTerms = (DoubleTerms) aggregation;
-doubleTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
-  } else if (aggregation instanceof StringTerms) {
-StringTerms stringTerms = (StringTerms) aggregation;
-stringTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+  Aggregation aggregation = 
aggregations.get(getFacentAggregationName(field));
+  if (aggregation instanceof Terms) {
+Terms terms = (Terms) aggregation;
+terms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(formatKey(bucket.getKey(), commonColumnMetadata.get(field)), 
bucket.getDocCount()));
   }
   fieldCounts.put(field, valueCounts);
 }
 return fieldCounts;
   }
 
-  private String getAggregationName(String field) {
+  private String formatKey(Object key, FieldType type) {
+if (FieldType.IP.equals(type)) {
+  return IpFieldMapper.longToIp((Long) key);
+} else if (FieldType.BOOLEAN.equals(type)) {
+  return (Long) key == 1 ? "true" : "false";
+} else {
+  return key.toString();
+}
+  }
+
+  private TermsBuilder getGroupsTermBuilder(GroupRequest groupRequest, int 
index) {
+List groups = groupRequest.getGroups();
+Group group = groups.get(index);
--- End diff --

If the groups field is empty, this will end up throwing an exception:
```
{
  "timestamp": "2017-09-08 10:44:34",
  "status": 500,
  "error": "Internal Server Error",
  "exception": "java.lang.IndexOutOfBoundsException",
  "message": "Index: 0, Size: 0",
  "path": "/api/v1/search/group"
}
```
It seems like we should either return no results (after all, if there's 
nothing requested in the group by, then no results seems at least vaguely 
reasonable) or throw a more informative exception


---

[GitHub] metron pull request #737: METRON-1161: Add ability to edit parser command li...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/737

METRON-1161: Add ability to edit parser command line options in the 
management UI

## Contributor Comments
This PR is a follow up to https://github.com/apache/metron/pull/717 and 
exposes the new Storm parser topology options in the Management UI.  This can 
be verified in full dev by navigating to the Management UI and editing any 
parser.  There should be a new "Storm Settings" pane in the primary panel that 
can be expanded and exposes input controls for all the new parser topology 
properties.  Editing these should successfully change the parser configs in 
zookeeper.

I also performed a slight refactor of the Raw JSON editor to make it easier 
to maintain as we add more parser config properties.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1161

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/737.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #737


commit 3b2a7a36443ab8c5599223951dc9d25f587d3ca0
Author: merrimanr 
Date:   2017-09-06T21:20:49Z

initial commit

commit dd3d44cdece6d553444f838891b33010491f99fa
Author: merrimanr 
Date:   2017-09-07T19:24:29Z

added new properties to readme panel and updated tests




---

[GitHub] metron issue #702: METRON-1114: Add group by capabilities to search REST end...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/702
  
This is really good, thanks for the contribution!  Are we intending (as a 
follow-on activity), to enable something like top hits aggregation in here so 
we can get the documents in the buckets back, or are we just expecting follow 
on queries to be the norm?

At least for using the API, it seems like it would be nice to at least have 
something rudimentary, but again, that's not a this PR thing.


---

[GitHub] metron pull request #730: METRON-1151: Grok patterns in HDFS should be owned...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/730


---

[GitHub] metron issue #724: METRON-1142: Add Geo Hashing functions to stellar

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/724
  
@cestella That absolutely makes sense.  Thanks a lot for adding it.

+1


---

[GitHub] metron issue #735: METRON-1160 Blueprint configuration validation failed: Mi...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/735
  
+1 by inspection.  Thanks for the contribution


---

[GitHub] metron pull request #738: METRON-1166: Stellar short circuiting fails when a...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/738

METRON-1166: Stellar short circuiting fails when a complex condition using 
a boolean op is followed by the opposite boolean op

## Contributor Comments
Rather subtle one here. Stellar short circuiting fails when a parenthetical 
boolean expression involving one boolean op is followed by the other boolean op.
For instance:
* `(false && true) || true` would yield `false`, which is wrong because 
we're mixing `&&` and `||`.
* `(false || false) || true` would yield `true`, which is correct because 
both ops are `||`

Test this in the REPL by throwing some statements at it.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron short_circuit_bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/738.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #738


commit 2c4d2cf68700dc2aceed146c66a4c74439a630bd
Author: cstella 
Date:   2017-09-08T14:21:30Z

fixed bug




---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137826193
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

I don't understand why we check for capabilities here, but not when 
setting.  It looks like I can define as much as I want, but only see if the 
capability is there?  Why do we only guard here?


---

[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/736#discussion_r137808595
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
 ---
@@ -131,50 +133,99 @@ public boolean isInitialized() {
 @Override
 public Object apply(List args, Context context) throws 
ParseException {
 
-  // user must provide the message as a string
-  String arg0 = Util.getArg(0, String.class, args);
-  if(arg0 == null) {
-throw new IllegalArgumentException(format("expected string, found 
null"));
+  // the use can pass in one or more messages in a few different forms
+  Object arg0 = Util.getArg(0, Object.class, args);
+  List messages = getMessages(arg0);
+
+  // user must provide the stand alone profiler
+  StandAloneProfiler profiler = Util.getArg(1, 
StandAloneProfiler.class, args);
+  try {
+for (JSONObject message : messages) {
+  profiler.apply(message);
+}
+
+  } catch (ExecutionException e) {
+throw new IllegalArgumentException(format("Failed to apply 
message; error=%s", e.getMessage()), e);
+  }
+
+  return profiler;
+}
+
+/**
+ * Gets a message or messages from the function arguments.
+ *
+ * @param arg The function argument containing the message(s).
+ * @return A list of messages
+ */
+private List getMessages(Object arg) {
+  List messages;
+
+  if (arg instanceof String) {
+messages = getMessagesFromString((String) arg);
+
+  } else if (arg instanceof List) {
+messages = getMessagesFromList((List) arg);
+
+  } else if (arg instanceof JSONObject) {
+messages = Collections.singletonList((JSONObject) arg);
+
+  } else {
+throw new IllegalArgumentException(format("invalid message: found 
'%s', expected String, List, or JSONObject",
+ClassUtils.getShortClassName(arg, "null")));
   }
 
-  // there could be one or more messages
+  return messages;
+}
+
+/**
+ * Gets a message or messages from a List
+ *
+ * @param listOfStrings The function argument is a List of Strings.
+ * @return A list of messages.
+ */
+private List getMessagesFromList(List 
listOfStrings) {
--- End diff --

Just a very small nit, but could we make that an iterable instead of a List 
since that's all you really need?  We now have SETs in stellar too and I think 
we may end up creating lazy iterable transformation functions too. (e.g. 
`PROFILE_APPLY( LAZY_MAP( [ msg1, msg2 ], m -> REGEX_APPLY(, m))`


---

[GitHub] metron pull request #729: METRON-1150: REST parseMessage endpoint fails with...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/729


---

[GitHub] metron pull request #739: METRON-1165 [FEATURE-BRANCH] Add ability for Bundl...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/739

METRON-1165 [FEATURE-BRANCH] Add ability for BundleSystem to add bundles 
after initialization

As part of supporting parser extensions and the bundle system hosted in 
other applications than storm, and specifically for hosting in the rest 
service, we need the ability to add new bundles to the system, preferably 
without requiring an application restart or reloading the entire system.

The use case:

1. Install a parser extension assembly from the extension ui
2. switch to the sensor ui
3. create new instance (+) and be able to use the parser(s) from the 
assembly you just installed.

This PR lays down the ground work for that capability, and will be 
integrated with the extension ui branch if or when taken to the Feature Branch.

In this PR I further refactor the original NIFI Nar concept of Initial 
Context, which is used in the BundleClassLoaders and the ExtensionManager.  I 
have moved the context to full classes, and moved their creation to those 
classes in a Builder implementation.

This allows the creation of multiple 'contexts' with different parameters 
and scopes.
I have also added the ability to merge one context into another.

Thus the idea is that:
- these classes have their initial context from the full scan of the bundle 
system library directories.
- adding a new bundle means on demand creation of a new context, for just 
that bundle
- we merge the bundles to make the new content available

The add capability is limited availability ( it is ```java protected 
``` ).
It's only public facing exposure is through the BundleSystem class.

BundleSystem is the 'easy button', simplified interface to the bundles, and 
should be in most cases the only interface required.  Exposing it this way 
allows us to be sure that the ADD calls are only called in a context where we 
can do the type of locking we want to do.

## Testing

This is a preparation PR.  The add functionality is not utilized outside 
the tests.
So this is a code review, no regressions check.

- Build and Tests should run
- Full Dev should start and have data in kibana ( no regression )


- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [na] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [na] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron 
- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [na] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron ext-bundle-add

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/739.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #739


commit 4d294b9b5ea646e38f2ae6072e6f7f83277d65aa
Author: Otto Fowler 
Date:   2017-09-08T05:35:05Z

implement ability to add new bundles to system




---

[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/736#discussion_r137822891
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
 ---
@@ -131,50 +133,99 @@ public boolean isInitialized() {
 @Override
 public Object apply(List args, Context context) throws 
ParseException {
 
-  // user must provide the message as a string
-  String arg0 = Util.getArg(0, String.class, args);
-  if(arg0 == null) {
-throw new IllegalArgumentException(format("expected string, found 
null"));
+  // the use can pass in one or more messages in a few different forms
+  Object arg0 = Util.getArg(0, Object.class, args);
+  List messages = getMessages(arg0);
+
+  // user must provide the stand alone profiler
+  StandAloneProfiler profiler = Util.getArg(1, 
StandAloneProfiler.class, args);
+  try {
+for (JSONObject message : messages) {
+  profiler.apply(message);
+}
+
+  } catch (ExecutionException e) {
+throw new IllegalArgumentException(format("Failed to apply 
message; error=%s", e.getMessage()), e);
+  }
+
+  return profiler;
+}
+
+/**
+ * Gets a message or messages from the function arguments.
+ *
+ * @param arg The function argument containing the message(s).
+ * @return A list of messages
+ */
+private List getMessages(Object arg) {
+  List messages;
+
+  if (arg instanceof String) {
+messages = getMessagesFromString((String) arg);
+
+  } else if (arg instanceof List) {
+messages = getMessagesFromList((List) arg);
+
+  } else if (arg instanceof JSONObject) {
+messages = Collections.singletonList((JSONObject) arg);
+
+  } else {
+throw new IllegalArgumentException(format("invalid message: found 
'%s', expected String, List, or JSONObject",
+ClassUtils.getShortClassName(arg, "null")));
   }
 
-  // there could be one or more messages
+  return messages;
+}
+
+/**
+ * Gets a message or messages from a List
+ *
+ * @param listOfStrings The function argument is a List of Strings.
+ * @return A list of messages.
+ */
+private List getMessagesFromList(List 
listOfStrings) {
--- End diff --

Yes, that's better.  Will do.


---

[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/736#discussion_r137825517
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
 ---
@@ -131,50 +133,99 @@ public boolean isInitialized() {
 @Override
 public Object apply(List args, Context context) throws 
ParseException {
 
-  // user must provide the message as a string
-  String arg0 = Util.getArg(0, String.class, args);
-  if(arg0 == null) {
-throw new IllegalArgumentException(format("expected string, found 
null"));
+  // the use can pass in one or more messages in a few different forms
+  Object arg0 = Util.getArg(0, Object.class, args);
+  List messages = getMessages(arg0);
+
+  // user must provide the stand alone profiler
+  StandAloneProfiler profiler = Util.getArg(1, 
StandAloneProfiler.class, args);
+  try {
+for (JSONObject message : messages) {
+  profiler.apply(message);
+}
+
+  } catch (ExecutionException e) {
+throw new IllegalArgumentException(format("Failed to apply 
message; error=%s", e.getMessage()), e);
+  }
+
+  return profiler;
+}
+
+/**
+ * Gets a message or messages from the function arguments.
+ *
+ * @param arg The function argument containing the message(s).
+ * @return A list of messages
+ */
+private List getMessages(Object arg) {
+  List messages;
+
+  if (arg instanceof String) {
+messages = getMessagesFromString((String) arg);
+
+  } else if (arg instanceof List) {
+messages = getMessagesFromList((List) arg);
+
+  } else if (arg instanceof JSONObject) {
+messages = Collections.singletonList((JSONObject) arg);
+
+  } else {
+throw new IllegalArgumentException(format("invalid message: found 
'%s', expected String, List, or JSONObject",
+ClassUtils.getShortClassName(arg, "null")));
   }
 
-  // there could be one or more messages
+  return messages;
+}
+
+/**
+ * Gets a message or messages from a List
+ *
+ * @param listOfStrings The function argument is a List of Strings.
+ * @return A list of messages.
+ */
+private List getMessagesFromList(List 
listOfStrings) {
--- End diff --

Done and done.


---

[GitHub] metron issue #738: METRON-1166: Stellar short circuiting fails when a comple...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/738
  
+1, pending Travis.  Good catch, thanks a lot for the contribution.


---

[GitHub] metron issue #729: METRON-1150: REST parseMessage endpoint fails with Unable...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/729
  
+1.  The issues I was seeing where a problem in my own code ( surprise 
right? ). This fix is correct and appropriate for the scope it seeks to 
address.  It is really good work.  


---

[GitHub] metron issue #739: METRON-1165 [FEATURE-BRANCH] Add ability for BundleSystem...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/739
  
@mattf-horton 


---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137841078
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

When would you not get an empty map?  It looks like if you are not using ZK 
you always get an empty map.  I must be missing something


---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137841201
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

Never mind, calling getOrCreate will put the capability in. I see. Sorry


---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137841605
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

Yep, exactly.  No worries.  I'm glad you're looking at it closely. 


---

[GitHub] metron issue #736: METRON-1162 Apply Live Messages to the Profile Debugger

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/736
  
+1 by inspection, lgtm!


---

[GitHub] metron pull request #738: METRON-1166: Stellar short circuiting fails when a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/738


---

[GitHub] metron issue #741: METRON-1153 HDFS HdfsWriter never recovers from exception...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/741
  
@ottobackwards From the ticket description, it looks like you thought there 
was a more involved refactoring for doing something like this.  Are you good 
with this sort of solution (at least as an initial step)?


---

[GitHub] metron pull request #741: METRON-1153 HDFS HdfsWriter never recovers from ex...

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/metron/pull/741

METRON-1153 HDFS HdfsWriter never recovers from exceptions

## Contributor Comments
Added a try-catch around the actual write that will rotate the file and try 
again if there's a stream closed underneath it.  Added two unit tests, one that 
ensures things flow through nicely to a single file with a double write and one 
that ensures that things flow to two files if the channel is closed underneath 
for whatever reason (done by just calling `closeOutputFile()` when outside of 
the normal flow).

It's not a perfect solution, but it should alleviate any transient pain and 
let us know if the problem is deeper if it keeps showing up.

Also added a missing set to a field from the constructor which probably 
wasn't helping things since I happened to notice it.  It's a one line change, 
so it seemed excessive to create a separate PR.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron METRON-1153

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #741


commit e5a2f3c6114f26b2091640a57bf4ca02e74addfa
Author: justinjleet 
Date:   2017-09-08T19:34:08Z

adding attempt to get new file on channel closed exception




---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/740
  
How would we, say, specify a global config value as a map (e.g. foo={'bar' 
: 'blah'})?  global config is also exposed as variables stellar in the parser 
and enrichments, so this could very well be a use-case.  complex global config 
is also used in the stellar config IIRC.


---

[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/734#discussion_r137874476
  
--- Diff: metron-interface/metron-rest/README.md ---
@@ -361,6 +363,21 @@ Request and Response objects are JSON formatted.  The 
JSON schemas are available
 * 200 - Returns sample message
 * 404 - Either Kafka topic is missing or contains no messages
 
+### `POST /api/v1/metaalert/searchByAlert`
+  * Description: Searches meta alerts to find any containing an alert for 
the provided GUID
+  * Input:
+* guid - GUID of the alert
+  * Returns:
+* 200 - Returns the meta alerts associated with this alert
+* 404 - Either Kafka topic is missing or contains no messages
--- End diff --

Is this a mistake?


---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137838874
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

It is just not strictly needed since if I start out with an empty map and 
there is no global config in Zk, then we are just left with the empty map, so 
we're good.  So functionally getOrCreate is not needed here.

That being said, I can see how it would bring a nice sense of symmetry to 
just use getOrCreate in this case also.  Let me try that out real quick.




---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137864851
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

Nick. If I understand correctly, if there is no ZK, then this turns on the 
GLOBAL_CONFIG flag, and sets whatever properties.

Is there any risk that other functions that check for GLOBAL_CONFIG can see 
the flag, but assuming the full config is present, when in this case only what 
the user has set is present?




---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/740
  
I think I'd be ok without that capability if we had an ability to pass in a 
global config as a JSON blob, like we do the initial set of variables.  What do 
you think?


---

[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/734#discussion_r137874898
  
--- Diff: metron-interface/metron-rest/README.md ---
@@ -361,6 +363,21 @@ Request and Response objects are JSON formatted.  The 
JSON schemas are available
 * 200 - Returns sample message
 * 404 - Either Kafka topic is missing or contains no messages
 
+### `POST /api/v1/metaalert/searchByAlert`
+  * Description: Searches meta alerts to find any containing an alert for 
the provided GUID
+  * Input:
+* guid - GUID of the alert
+  * Returns:
+* 200 - Returns the meta alerts associated with this alert
+* 404 - Either Kafka topic is missing or contains no messages
--- End diff --

Absolutely a mistake.  Thought I'd fixed that, but apparently not.


---

[GitHub] metron pull request #740: METRON-1167 Define Session Specific Global Configu...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/740#discussion_r137840091
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -308,17 +315,68 @@ private void handleMagic( String rawExpression) {
   .collect(Collectors.joining(", "));
   writeLine(functions);
 
-} else if(MAGIC_VARS.equals(command)) {
+} else if (MAGIC_VARS.equals(command)) {
 
-  // list all variables
+  // '%vars' -> list all variables in scope
   executor.getVariables()
-  .forEach((k,v) -> writeLine(String.format("%s = %s", k, v)));
+  .forEach((k, v) -> writeLine(String.format("%s = %s", k, 
v)));
+
+} else if (MAGIC_GLOBALS.equals(command)) {
+
+  // '%globals' -> list all globals in scope
+  Map globals = Collections.emptyMap();
--- End diff --

Yep, it is prettier this way.  Thanks  @ottobackwards 


---

[GitHub] metron issue #741: METRON-1153 HDFS HdfsWriter never recovers from exception...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/741
  
This looks good to me as a stopgap.  I'd like to understand eventually why 
this happens, but it appears very intermittent, so this isn't a bad solution at 
least as a stopgap.

+1 by inspection.


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137883199
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/StringFunctionsTest.java
 ---
@@ -408,6 +408,19 @@ public void testAppendIfMissing() throws Exception {
 
   }
 
+  @Test
+  public void testSubstring() throws Exception {
+Map variables = ImmutableMap.of("s", "apache metron");
+Assert.assertEquals("metron", run("SUBSTRING(s, 7)", variables));
+Assert.assertEquals("me", run("SUBSTRING(s, 7, 9)", variables));
+Assert.assertNull(run("SUBSTRING(null, 6, 9)", new HashMap<>()));
--- End diff --

Can you add a test for start being explicitly null, and another one for end 
being explicitly null?  I mostly care about start being null, since that's a 
case not present in Java, given that int can't be null.


---

[GitHub] metron issue #742: METRON-1168: Add SUBSTRING method to stellar

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/742
  
+1, thanks a lot for the contribution.  And for catching that we never 
actually had this


---

[GitHub] metron pull request #734: METRON-1158: Build backend for grouping alerts int...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/734#discussion_r137889738
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
 ---
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.integration;
+
+import static 
org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.File;
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
+import org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao;
+import org.apache.metron.elasticsearch.dao.MetaAlertStatus;
+import 
org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.MetaAlertDao;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class ElasticsearchMetaAlertIntegrationTest {
+
+  private static final int MAX_RETRIES = 10;
+  private static final int SLEEP_MS = 500;
+  private static final String SENSOR_NAME = "test";
+  private static final String INDEX_DIR = "target/elasticsearch_meta";
+  private static final String DATE_FORMAT = ".MM.dd.HH";
+  private static final String INDEX =
+  SENSOR_NAME + "_index_" + new 
SimpleDateFormat(DATE_FORMAT).format(new Date());
+  private static final String NEW_FIELD = "new-field";
+
+  private static String metaMappingSource;
+  private static IndexDao esDao;
+  private static IndexDao metaDao;
+  private static ElasticSearchComponent es;
+
+  @BeforeClass
+  public static void setup() throws Exception {
+buildMetaMappingSource();
+// setup the client
+es = new ElasticSearchComponent.Builder()
--- End diff --

You might want to consider moving this to TestConfig instead.  The REST 
layer depends heavily on the InMemory components and reuses them across all the 
integration tests.  You are breaking the pattern although I understand why 
because setting up InMemory components is usually done in @BeforeClass in other 
modules.  This would allow other controller integration tests to leverage this 
infrastructure without having to do anything.

The SearchController uses a mock ES backend but, in my humble opinion, it's 
hard to maintain and is unnecessary.  Would be nice to switch that to what 
you've done here someday.


---

[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/736


---

[GitHub] metron pull request #732: METRON-632: Added validation of "shew.enrichmentTy...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/732#discussion_r137895036
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/writer/SimpleHbaseEnrichmentWriter.java
 ---
@@ -134,6 +136,39 @@ public void configure(String sensorName, 
WriterConfiguration configuration) {
 LOG.debug("Sensor: '{}': {Provider: '{}', Converter: '{}'}", 
sensorName, getClassName(provider), getClassName(converter));
   }
 
+
+  private void validateEnrichmentType(String sensorName, 
WriterConfiguration configuration) {
+Map sensorConfig = 
configuration.getSensorConfig(sensorName);
+Object enrichmentTypeObj = 
Configurations.ENRICHMENT_TYPE.get(sensorConfig);
+if (enrichmentTypeObj == null) {
+  throw new IllegalArgumentException(String.format("%s must be 
provided", Configurations.ENRICHMENT_TYPE.getKey()));
+}
+
+if (!(enrichmentTypeObj instanceof String)) {
+  throw new IllegalArgumentException(String.format("%s must be a 
string", Configurations.ENRICHMENT_TYPE.getKey()));
+}
+
+String enrichmentType = enrichmentTypeObj.toString();
+if (enrichmentType.trim().isEmpty()) {
+  throw new IllegalArgumentException(String.format("%s must not be an 
empty string",
+  Configurations.ENRICHMENT_TYPE.getKey()));
+}
+  }
+
+  private void validateKeyColumns(String sensorName, WriterConfiguration 
configuration) {
+Map sensorConfig = 
configuration.getSensorConfig(sensorName);
+Object keyColumnsObj = Configurations.KEY_COLUMNS.get(sensorConfig);
+
+try {
+  List keyColumns = getColumns(keyColumnsObj, true);
+  if (keyColumns == null || keyColumns.isEmpty()) {
+throw new IllegalArgumentException(String.format("%s must be 
provided", Configurations.KEY_COLUMNS.getKey()));
+  }
+} catch (RuntimeException ex) {
+  throw new IllegalArgumentException(ex.getMessage());
--- End diff --

can we pass in `ex` as the second arg?


---

[GitHub] metron issue #700: METRON-1077: Use column meta data end point in alerts ui

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/700
  
Thanks @iraghumitra.  I tested this in full dev and everything worked as 
expected.  +1


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/740
  
so, what happens if we modify a field via `%define` from a stellar shell 
started with `-z` ?


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/740
  
Those options seem to add a bit more complexity than I'd like.  How about 
we just restrict %define to strings only for now.  If you need to use complex 
types, then you can still do the "CONFIG_GET, CONFIG_PUT, restart REPL" dance 
that I described in the PR description.


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/740
  
"a bit more complexity than I'd like" may just need to be read as "Nick is 
lazy", but I'll let you be the judge


---

Re: [DISCUSS] Metron release 0.4.1

[Matt Foley]

Looks like everything’s in now.  Thanks very much everyone for the efforts!  
I’m spinning the RC and will have it out for vote shortly.
--Matt


On 9/7/17, 11:41 PM, "Matt Foley"  wrote:

METRON-1163 turns out to be invalid, due to a bad maven settings file.  It 
is no longer an issue.
Just waiting for the last couple items.

On 9/7/17, 9:26 AM, "Matt Foley"  wrote:

Okay.  Please ping when committed.
Also, any input on https://issues.apache.org/jira/browse/METRON-1163 ?

On 9/7/17, 7:39 AM, "Ryan Merriman"  wrote:

Matt,

We recently found a bug that's breaking certain features in the 
management
UI.  The fixes are still in review (
https://github.com/apache/metron/pull/729 and
https://github.com/apache/metron/pull/730) and should make it in 
soon.  It
would be good to include these if possible.

Ryan

On Thu, Sep 7, 2017 at 12:23 AM, Matt Foley  
wrote:

> I’ve got a blocker bug, 
https://issues.apache.org/jira/browse/METRON-1163
> , RAT failures for metron-interface/metron-alerts.  A comment in 
the jira
> suggests a way to address it, but someone familiar with the code 
should
> look at it.
>
>
>
> Raghu, would you be able to take a look?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Matt Foley  on behalf of Matt Foley 
<
> ma...@apache.org>
> Date: Tuesday, September 5, 2017 at 10:01 AM
> To: "dev@metron.apache.org" 
> Subject: Re: [DISCUSS] Metron release 0.4.1
>
>
>
> Great, working on it!
>
>
>
> From: Nick Allen 
> Date: Tuesday, September 5, 2017 at 8:00 AM
> To: Casey Stella , "zeo...@gmail.com" <
> zeo...@gmail.com>
> Cc: Anand Subramanian , Matt Foley <
> mfo...@hortonworks.com>, "dev@metron.apache.org" 

> Subject: Re: [DISCUSS] Metron release 0.4.1
>
>
>
> All set here.  Let's get this shipped!
>
>
>
>
>
>
>
> On Tue, Sep 5, 2017 at 10:44 AM Casey Stella  
wrote:
>
> me too
>
>
>
> On Tue, Sep 5, 2017 at 10:43 AM, zeo...@gmail.com 

> wrote:
>
> All set from my perspective.
>
>
>
> Jon
>
>
>
> On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> Hello Matt,
>
> Simon's pull request supersedes mine (METRON-1139 /
> https://github.com/apache/metron/pull/722) and is already merged 
into
> master.
>
> Thanks,
> Anand
>
>
>
> On 9/1/17, 12:41 AM, "Matt Foley"  wrote:
>
> >Please mark them 0.4.1, as that’s what the community says we 
want to call
> the upcoming release, and everything that’s there when I throw 
the switch
> will be included.
> >
> >Jon and Anand, will they be in by end/day Friday?
> >Thanks,
> >--Matt
> >
> >On 8/31/17, 7:45 AM, "Nick Allen"  wrote:
> >
> >Matt, et al - For JIRAs that are going into master, should 
we be
> marking
> >these as "Next + 1" or "0.4.1" ?
> >
> >On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com 

> wrote:
> >
> >> Can I advocate to get METRON-1129 in the RC, and throw in 
a second
> vote for
> >> METRON-1134?  Both in an attempt to better support of 
prod/offline
> use.
> >> Happy to provide testing cycles for the former.
> >>
> >> Jon
> >>
> >> On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
> >> asubraman...@hortonworks.com> wrote:
> >>
   

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/740
  
Yes, I think it would be useful to pass in a global config as a command 
line argument.  Maybe something like...
```
bin/stellar -g config/zookeeper/global.json
```

I think we could tackle that as a follow-on.



---

[GitHub] metron issue #734: METRON-1158: Build backend for grouping alerts into meta ...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/734
  
Great job on this.  I spent a lot of time testing and had trouble finding 
anything wrong.  The one thing I did notice was that the "timestamp" field was 
missing from documents in the metaalerts index.  This is the field we commonly 
use for sorting so we might want to consider adding it or switching our default 
sort field to "_timestamp" instead.  


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/740
  
And in terms of the complex types I can see the use case definitely.  In 
terms of the REPL and the changes in this PR, a `%define` does the same as when 
GLOBAL_CONFIG gets created with a `-z` for complex types.  In either case, the 
complex type gets loaded as a string. 

For example, I launched Full Dev and created a GLOBAL_CONFIG that 
looks-like this.
```
{
...
"foo": "{'bar' : 'blah'}"
}
```
I then launched the REPL in a debug session with a -z.  The value for the 
key "foo" is a string.



---

[GitHub] metron pull request #702: METRON-1114: Add group by capabilities to search R...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/702#discussion_r137893211
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/IndexConfig.java
 ---
@@ -56,8 +52,10 @@ public IndexDao indexDao() {
   String hbaseProviderImpl = 
environment.getProperty(MetronRestConstants.INDEX_HBASE_TABLE_PROVIDER_IMPL, 
String.class, null);
   String indexDaoImpl = 
environment.getProperty(MetronRestConstants.INDEX_DAO_IMPL, String.class, null);
   int searchMaxResults = 
environment.getProperty(MetronRestConstants.SEARCH_MAX_RESULTS, Integer.class, 
-1);
+  int searchMaxGroups = 
environment.getProperty(MetronRestConstants.SEARCH_MAX_GROUPS, Integer.class, 
1000);
--- End diff --

Changed the SEARCH_MAX_RESULTS default to 1000.


---

[GitHub] metron pull request #702: METRON-1114: Add group by capabilities to search R...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/702#discussion_r137893133
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/GroupOrderType.java
 ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum GroupOrderType {
+
+  @JsonProperty("count")
--- End diff --

Added some documentation to the README.


---

[GitHub] metron pull request #702: METRON-1114: Add group by capabilities to search R...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/702#discussion_r137893095
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
 ---
@@ -330,43 +334,112 @@ public void update(Document update, Optional 
index) throws IOException {
 return latestIndices.values().toArray(new 
String[latestIndices.size()]);
   }
 
-  public void addFacetFields(SearchSourceBuilder searchSourceBuilder, 
List fields) {
-for(String field: fields) {
-  searchSourceBuilder = searchSourceBuilder.aggregation(new 
TermsBuilder(getAggregationName(field)).field(field));
+  private org.elasticsearch.search.sort.SortOrder 
getElasticsearchSortOrder(
+  org.apache.metron.indexing.dao.search.SortOrder sortOrder) {
+return sortOrder == 
org.apache.metron.indexing.dao.search.SortOrder.DESC ?
+org.elasticsearch.search.sort.SortOrder.DESC : 
org.elasticsearch.search.sort.SortOrder.ASC;
+  }
+
+  private Order getElasticsearchGroupOrder(GroupOrder groupOrder) {
+if (groupOrder.getGroupOrderType() == GroupOrderType.TERM) {
+  return groupOrder.getSortOrder() == SortOrder.ASC ? Order.term(true) 
: Order.term(false);
+} else {
+  return groupOrder.getSortOrder() == SortOrder.ASC ? 
Order.count(true) : Order.count(false);
 }
   }
 
   public Map> getFacetCounts(List 
fields, Aggregations aggregations, Map commonColumnMetadata) 
{
 Map> fieldCounts = new HashMap<>();
 for (String field: fields) {
   Map valueCounts = new HashMap<>();
-  Aggregation aggregation = 
aggregations.get(getAggregationName(field));
-  if (aggregation instanceof LongTerms) {
-LongTerms longTerms = (LongTerms) aggregation;
-FieldType type = commonColumnMetadata.get(field);
-if (FieldType.IP.equals(type)) {
-  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(IpFieldMapper.longToIp((Long) bucket.getKey()), 
bucket.getDocCount()));
-} else if (FieldType.BOOLEAN.equals(type)) {
-  longTerms.getBuckets().stream().forEach(bucket -> {
-String key = (Long) bucket.getKey() == 1 ? "true" : "false";
-valueCounts.put(key, bucket.getDocCount());
-  });
-} else {
-  longTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
-}
-  } else if (aggregation instanceof DoubleTerms) {
-DoubleTerms doubleTerms = (DoubleTerms) aggregation;
-doubleTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
-  } else if (aggregation instanceof StringTerms) {
-StringTerms stringTerms = (StringTerms) aggregation;
-stringTerms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(bucket.getKeyAsString(), bucket.getDocCount()));
+  Aggregation aggregation = 
aggregations.get(getFacentAggregationName(field));
+  if (aggregation instanceof Terms) {
+Terms terms = (Terms) aggregation;
+terms.getBuckets().stream().forEach(bucket -> 
valueCounts.put(formatKey(bucket.getKey(), commonColumnMetadata.get(field)), 
bucket.getDocCount()));
   }
   fieldCounts.put(field, valueCounts);
 }
 return fieldCounts;
   }
 
-  private String getAggregationName(String field) {
+  private String formatKey(Object key, FieldType type) {
+if (FieldType.IP.equals(type)) {
+  return IpFieldMapper.longToIp((Long) key);
+} else if (FieldType.BOOLEAN.equals(type)) {
+  return (Long) key == 1 ? "true" : "false";
+} else {
+  return key.toString();
+}
+  }
+
+  private TermsBuilder getGroupsTermBuilder(GroupRequest groupRequest, int 
index) {
+List groups = groupRequest.getGroups();
+Group group = groups.get(index);
--- End diff --

Added a better exception


---

[GitHub] metron pull request #700: METRON-1077: Use column meta data end point in ale...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/700


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/742

METRON-1168: Add SUBSTRING method to stellar

## Contributor Comments

Add `SUBSTRING` function to stellar.

Testing:
Open up the REPL and play with `SUBSTRING`

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron substring

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/742.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #742


commit 93f88f0095bba80edf55f9b808787686f13a76a3
Author: cstella 
Date:   2017-09-08T20:13:10Z

Add substring method

commit 10db1804276b32a8a6ed0ddf93c79401dafba829
Author: cstella 
Date:   2017-09-08T20:17:52Z

updating readme.




---

Re: [DISCUSS] Metron release 0.4.1

[Ryan Merriman]

Matt,

This was committed a few hours. I think you saw it but just wanted to make sure.

Ryan

> On Sep 7, 2017, at 11:26 AM, Matt Foley  wrote:
> 
> Okay.  Please ping when committed.
> Also, any input on https://issues.apache.org/jira/browse/METRON-1163 ?
> 
> On 9/7/17, 7:39 AM, "Ryan Merriman"  wrote:
> 
>Matt,
> 
>We recently found a bug that's breaking certain features in the management
>UI.  The fixes are still in review (
>https://github.com/apache/metron/pull/729 and
>https://github.com/apache/metron/pull/730) and should make it in soon.  It
>would be good to include these if possible.
> 
>Ryan
> 
>>On Thu, Sep 7, 2017 at 12:23 AM, Matt Foley  wrote:
>> 
>> I’ve got a blocker bug, https://issues.apache.org/jira/browse/METRON-1163
>> , RAT failures for metron-interface/metron-alerts.  A comment in the jira
>> suggests a way to address it, but someone familiar with the code should
>> look at it.
>> 
>> 
>> 
>> Raghu, would you be able to take a look?
>> 
>> 
>> 
>> Thanks,
>> 
>> --Matt
>> 
>> 
>> 
>> From: Matt Foley  on behalf of Matt Foley <
>> ma...@apache.org>
>> Date: Tuesday, September 5, 2017 at 10:01 AM
>> To: "dev@metron.apache.org" 
>> Subject: Re: [DISCUSS] Metron release 0.4.1
>> 
>> 
>> 
>> Great, working on it!
>> 
>> 
>> 
>> From: Nick Allen 
>> Date: Tuesday, September 5, 2017 at 8:00 AM
>> To: Casey Stella , "zeo...@gmail.com" <
>> zeo...@gmail.com>
>> Cc: Anand Subramanian , Matt Foley <
>> mfo...@hortonworks.com>, "dev@metron.apache.org" 
>> Subject: Re: [DISCUSS] Metron release 0.4.1
>> 
>> 
>> 
>> All set here.  Let's get this shipped!
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Tue, Sep 5, 2017 at 10:44 AM Casey Stella  wrote:
>> 
>> me too
>> 
>> 
>> 
>> On Tue, Sep 5, 2017 at 10:43 AM, zeo...@gmail.com 
>> wrote:
>> 
>> All set from my perspective.
>> 
>> 
>> 
>> Jon
>> 
>> 
>> 
>> On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
>> asubraman...@hortonworks.com> wrote:
>> 
>> Hello Matt,
>> 
>> Simon's pull request supersedes mine (METRON-1139 /
>> https://github.com/apache/metron/pull/722) and is already merged into
>> master.
>> 
>> Thanks,
>> Anand
>> 
>> 
>> 
>>> On 9/1/17, 12:41 AM, "Matt Foley"  wrote:
>>> 
>>> Please mark them 0.4.1, as that’s what the community says we want to call
>> the upcoming release, and everything that’s there when I throw the switch
>> will be included.
>>> 
>>> Jon and Anand, will they be in by end/day Friday?
>>> Thanks,
>>> --Matt
>>> 
>>> On 8/31/17, 7:45 AM, "Nick Allen"  wrote:
>>> 
>>>   Matt, et al - For JIRAs that are going into master, should we be
>> marking
>>>   these as "Next + 1" or "0.4.1" ?
>>> 
   On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com 
>>> wrote:
>>> 
 Can I advocate to get METRON-1129 in the RC, and throw in a second
>> vote for
 METRON-1134?  Both in an attempt to better support of prod/offline
>> use.
 Happy to provide testing cycles for the former.
 
 Jon
 
 On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
 asubraman...@hortonworks.com> wrote:
 
> Hi Matt,
> 
> This defect needs to be included as a follow-on to METRON-1122:
> * METRON-1141 (https://github.com/apache/metron/pull/723)
> 
> 
> Thanks,
> Anand
> 
> 
> 
> On 8/30/17, 8:57 PM, "Michael Miklavcic" <
>> michael.miklav...@gmail.com>
> wrote:
> 
>> I have some work around fixing how we handle config with Ambari
>> that I'd
>> like to see go in. No PR yet, but coming soon. I expect to have
>> this by
> the
>> RC deadline.
>> 
>> Mike
>> 
>> On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen 
>> wrote:
>> 
>>> The following PRs are usability enhancements for the
>> Profiler.  They
 are
>>> fairly simple and I think are very helpful for
>> troubleshooting.  I
 don't
>>> want to hold up the release, but it would be a "nice to have"
>> to get
> these
>>> in.
>>> 
>>> If anyone has cycles, I would appreciate some reviews of these
>> PRs.
>>> https://github.com/apache/metron/pull/721
>>> https://github.com/apache/metron/pull/716
>>> 
>>> 
>>> 
>>> On Tue, Aug 29, 2017 at 8:59 PM Matt Foley 
>> wrote:
>>> 
 Okay, just to be clear, you’re requesting we wait until
>> Friday, if
 necessary, to cut an RC with 717 in it?
 Thanks,
 --Matt
 
 On 8/29/17, 11:45 AM, "Casey Stella" 
>> wrote:
 
709 is in and 717 is under concerted review by Otto.
>> I'd like
 to
> see
 it in
by Friday.
 
On Tue, 

Re: [DISCUSS] Metron release 0.4.1

[Matt Foley]

Hi Ryan, yup, got it.  It’s in.

On 9/8/17, 1:25 PM, "Ryan Merriman"  wrote:

Matt,

This was committed a few hours. I think you saw it but just wanted to make 
sure.

Ryan

> On Sep 7, 2017, at 11:26 AM, Matt Foley  wrote:
> 
> Okay.  Please ping when committed.
> Also, any input on https://issues.apache.org/jira/browse/METRON-1163 ?
> 
> On 9/7/17, 7:39 AM, "Ryan Merriman"  wrote:
> 
>Matt,
> 
>We recently found a bug that's breaking certain features in the 
management
>UI.  The fixes are still in review (
>https://github.com/apache/metron/pull/729 and
>https://github.com/apache/metron/pull/730) and should make it in soon. 
 It
>would be good to include these if possible.
> 
>Ryan
> 
>>On Thu, Sep 7, 2017 at 12:23 AM, Matt Foley  wrote:
>> 
>> I’ve got a blocker bug, https://issues.apache.org/jira/browse/METRON-1163
>> , RAT failures for metron-interface/metron-alerts.  A comment in the jira
>> suggests a way to address it, but someone familiar with the code should
>> look at it.
>> 
>> 
>> 
>> Raghu, would you be able to take a look?
>> 
>> 
>> 
>> Thanks,
>> 
>> --Matt
>> 
>> 
>> 
>> From: Matt Foley  on behalf of Matt Foley <
>> ma...@apache.org>
>> Date: Tuesday, September 5, 2017 at 10:01 AM
>> To: "dev@metron.apache.org" 
>> Subject: Re: [DISCUSS] Metron release 0.4.1
>> 
>> 
>> 
>> Great, working on it!
>> 
>> 
>> 
>> From: Nick Allen 
>> Date: Tuesday, September 5, 2017 at 8:00 AM
>> To: Casey Stella , "zeo...@gmail.com" <
>> zeo...@gmail.com>
>> Cc: Anand Subramanian , Matt Foley <
>> mfo...@hortonworks.com>, "dev@metron.apache.org" 
>> Subject: Re: [DISCUSS] Metron release 0.4.1
>> 
>> 
>> 
>> All set here.  Let's get this shipped!
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Tue, Sep 5, 2017 at 10:44 AM Casey Stella  wrote:
>> 
>> me too
>> 
>> 
>> 
>> On Tue, Sep 5, 2017 at 10:43 AM, zeo...@gmail.com 
>> wrote:
>> 
>> All set from my perspective.
>> 
>> 
>> 
>> Jon
>> 
>> 
>> 
>> On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
>> asubraman...@hortonworks.com> wrote:
>> 
>> Hello Matt,
>> 
>> Simon's pull request supersedes mine (METRON-1139 /
>> https://github.com/apache/metron/pull/722) and is already merged into
>> master.
>> 
>> Thanks,
>> Anand
>> 
>> 
>> 
>>> On 9/1/17, 12:41 AM, "Matt Foley"  wrote:
>>> 
>>> Please mark them 0.4.1, as that’s what the community says we want to 
call
>> the upcoming release, and everything that’s there when I throw the switch
>> will be included.
>>> 
>>> Jon and Anand, will they be in by end/day Friday?
>>> Thanks,
>>> --Matt
>>> 
>>> On 8/31/17, 7:45 AM, "Nick Allen"  wrote:
>>> 
>>>   Matt, et al - For JIRAs that are going into master, should we be
>> marking
>>>   these as "Next + 1" or "0.4.1" ?
>>> 
   On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com 
>>> wrote:
>>> 
 Can I advocate to get METRON-1129 in the RC, and throw in a second
>> vote for
 METRON-1134?  Both in an attempt to better support of prod/offline
>> use.
 Happy to provide testing cycles for the former.
 
 Jon
 
 On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
 asubraman...@hortonworks.com> wrote:
 
> Hi Matt,
> 
> This defect needs to be included as a follow-on to METRON-1122:
> * METRON-1141 (https://github.com/apache/metron/pull/723)
> 
> 
> Thanks,
> Anand
> 
> 
> 
> On 8/30/17, 8:57 PM, "Michael Miklavcic" <
>> michael.miklav...@gmail.com>
> wrote:
> 
>> I have some work around fixing how we handle config with Ambari
>> that I'd
>> like to see go in. No PR yet, but coming soon. I expect to have
>> this by
> the
>> RC deadline.
>> 
>> Mike
>> 
>> On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen 
>> wrote:
>> 
>>> The following PRs are usability enhancements for the
>> Profiler.  They
 are
>>> fairly simple and I think are very helpful for
>> troubleshooting.  I
 don't
>>> want to hold up the release, but it would be a 

[GitHub] metron issue #702: METRON-1114: Add group by capabilities to search REST end...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/702
  
The latest commit should address the feedback so far.  I don't feel like 
the top hits aggregation is necessary at this point but it could be in the 
future.  I would prefer to wait and keep it simple until we have a need for it.


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/740
  
So, one way to do this is to allow users to refer to stellar fields:
```
my_complex_field := { 'blah' : 7 }
%define nicks_new_global_field=_complex_field
```

Another option to support this is to just insist that `%define` is a string 
and you have a `replace_global` that takes a field as an argument and the field 
contains the new global config (in JSON form) and replaces the global config:
```
new_global_config := SHELL_EDIT()
%replace_global new_global_config
```



---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/740
  
you could also make `%define nicks_new_global_field := my_complex_field`  
That makes somewhat more sense since stellar assignment is `:=` and property 
assignment is `=`


---

[GitHub] metron pull request #744: METRON-1169: dependency checker has not been runni...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/744

METRON-1169: dependency checker has not been running in travis

## Contributor Comments
Name says it all


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-1169

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #744


commit d197143561fb89752bc9da3eaa3f697e16a3cea0
Author: cstella 
Date:   2017-09-09T00:21:19Z

METRON-1169: dependency checker has not been running in travis




---

[GitHub] metron pull request #741: METRON-1153 HDFS HdfsWriter never recovers from ex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/741


---

[GitHub] metron pull request #741: METRON-1153 HDFS HdfsWriter never recovers from ex...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/741#discussion_r137913278
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
 ---
@@ -64,19 +64,34 @@ public SourceHandler(List 
rotationActions
 this.rotationPolicy = rotationPolicy;
 this.syncPolicy = syncPolicy;
 this.fileNameFormat = fileNameFormat;
+this.cleanupCallback = cleanupCallback;
 initialize();
   }
 
 
   protected void handle(JSONObject message, String sensor, 
WriterConfiguration config, SyncPolicyCreator syncPolicyCreator) throws 
IOException {
 byte[] bytes = (message.toJSONString() + "\n").getBytes();
 synchronized (this.writeLock) {
-  out.write(bytes);
+  try {
+out.write(bytes);
+  } catch (IOException writeException) {
+LOG.warn("IOException while writing output", writeException);
+// If the stream is closed, attempt to rotate the file and try 
again, hoping it's transient
+if (writeException.getMessage().contains("Stream Closed")) {
+  LOG.warn("Output Stream was closed. Attempting to rotate file 
and continue");
+  rotateOutputFile();
+  // If this write fails, the exception will be allowed to bubble 
up.
+  out.write(bytes);
--- End diff --

If there is a non-recoverable error, you are in the same position as before.
I was thinking of having a 'failed' flag ( similar to what hadoop has in 
it's screams where they track closed explicitly )


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137913436
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
 ---
@@ -321,6 +321,46 @@ public Object apply(List args) {
 }
   }
 
+  @Stellar( name="SUBSTRING"
+  , description = "Returns a substring of a string"
+  , params = {
+"input - The string to take the substring of",
+"start - The starting position (0-based and inclusive)",
+"end? - The ending position (0-based and exclusive)"
+ }
+  , returns = "The substring of the input"
+  )
+  public static class Substring extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings == null || strings.size() < 2 ) {
+throw new IllegalArgumentException("[SUBSTRING] required 2 
arguments: the input and the start position (inclusive)");
--- End diff --

Yeah, I was following the leader with `CHOMP`, but I agree, I will correct 
this in a follow-on JIRA with the rest of your issues covered.


---

[GitHub] metron pull request #741: METRON-1153 HDFS HdfsWriter never recovers from ex...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/741#discussion_r137913406
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
 ---
@@ -64,19 +64,34 @@ public SourceHandler(List 
rotationActions
 this.rotationPolicy = rotationPolicy;
 this.syncPolicy = syncPolicy;
 this.fileNameFormat = fileNameFormat;
+this.cleanupCallback = cleanupCallback;
 initialize();
   }
 
 
   protected void handle(JSONObject message, String sensor, 
WriterConfiguration config, SyncPolicyCreator syncPolicyCreator) throws 
IOException {
 byte[] bytes = (message.toJSONString() + "\n").getBytes();
 synchronized (this.writeLock) {
-  out.write(bytes);
+  try {
+out.write(bytes);
+  } catch (IOException writeException) {
+LOG.warn("IOException while writing output", writeException);
+// If the stream is closed, attempt to rotate the file and try 
again, hoping it's transient
+if (writeException.getMessage().contains("Stream Closed")) {
--- End diff --

Using string compare on the message seems kind of thin.  And there may be 
other exceptions that come up that we want to handle.

Part of what I hoped would come from the discuss thread was a discussion 
about what was recoverable and what was not recoverable, and a better way to 
handle both cases.



---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137913443
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
 ---
@@ -321,6 +321,46 @@ public Object apply(List args) {
 }
   }
 
+  @Stellar( name="SUBSTRING"
+  , description = "Returns a substring of a string"
+  , params = {
+"input - The string to take the substring of",
+"start - The starting position (0-based and inclusive)",
+"end? - The ending position (0-based and exclusive)"
+ }
+  , returns = "The substring of the input"
+  )
+  public static class Substring extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings == null || strings.size() < 2 ) {
+throw new IllegalArgumentException("[SUBSTRING] required 2 
arguments: the input and the start position (inclusive)");
+  }
+  String var = strings.get(0) == null?null: (String) strings.get(0);
+  Integer start = strings.get(1) == null?null:(Integer)strings.get(1);
--- End diff --

Agreed


---

[GitHub] metron issue #742: METRON-1168: Add SUBSTRING method to stellar

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/742
  
I am going to create a follow-on PR to this to address your issues.


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137913498
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/StringFunctionsTest.java
 ---
@@ -408,6 +408,23 @@ public void testAppendIfMissing() throws Exception {
 
   }
 
+  @Test
+  public void testSubstring() throws Exception {
+Map variables = ImmutableMap.of("s", "apache metron");
--- End diff --

Agreed, we should expect an exception in that case.


---

[GitHub] metron pull request #741: METRON-1153 HDFS HdfsWriter never recovers from ex...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/741#discussion_r137913296
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
 ---
@@ -64,19 +64,34 @@ public SourceHandler(List 
rotationActions
 this.rotationPolicy = rotationPolicy;
 this.syncPolicy = syncPolicy;
 this.fileNameFormat = fileNameFormat;
+this.cleanupCallback = cleanupCallback;
 initialize();
   }
 
 
   protected void handle(JSONObject message, String sensor, 
WriterConfiguration config, SyncPolicyCreator syncPolicyCreator) throws 
IOException {
 byte[] bytes = (message.toJSONString() + "\n").getBytes();
 synchronized (this.writeLock) {
-  out.write(bytes);
+  try {
+out.write(bytes);
+  } catch (IOException writeException) {
+LOG.warn("IOException while writing output", writeException);
+// If the stream is closed, attempt to rotate the file and try 
again, hoping it's transient
+if (writeException.getMessage().contains("Stream Closed")) {
+  LOG.warn("Output Stream was closed. Attempting to rotate file 
and continue");
+  rotateOutputFile();
+  // If this write fails, the exception will be allowed to bubble 
up.
+  out.write(bytes);
--- End diff --

If there is a non-recoverable error, you are in the same position as before.
I was thinking of having a 'failed' flag ( similar to what hadoop has in 
it's screams where they track closed explicitly )


---

[GitHub] metron issue #741: METRON-1153 HDFS HdfsWriter never recovers from exception...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/741
  
@justinleet My question with regards to the necessity for a refactor came 
from wanting to handle the exception in the HDFSWriter, where the exceptions 
where being caught at the time.

The fact that the tuples are 'unpaired' from their messages, and that we 
handle them all together seems to me to be problematic if you want to take per 
message/sourcehandle action.

I think your approach removes the immediate problem with that, although 
having both the hdfs writer and the source handler itself take actions and 
split the 'ownership' doesn't feel quite right to me.

@cestella It would be good to know why this is happening, but in truth, any 
persistent network connection in a long lived 'forever' type application needs 
to guard against these kinds of errors.

For reliability reasons, and where we want to get to, we require a more 
clear documentation of the failure and recovery states of the writers, esp. the 
hdfs as we are batching.  We also need to understand all the ways the stream 
can fail, to the extent that that is possible.

I am not sure that limiting it to this one case is enough,  there will 
still be many possible ways to end up with a very unclear situation, that the 
writer is failing continuously and but the source handler is not removed from 
'service'.  Users, as we have seen on the list will be left to track through 
system to work their way to this problem.  

Failure to store is a critical failure of the system.  Esp. in systems 
where there are data retention rules or SLA's on data loss.  Thus we need in 
addition to the handling of this, an alerting strategy.

While this fix ( questions pending ) is an improvement of the symptom, it 
does not address the higher level issue or severity of this problem.


---

[GitHub] metron issue #741: METRON-1153 HDFS HdfsWriter never recovers from exception...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/741
  
I absolutely agree, this does not address the higher level issue or 
severity.  This is merely a stopgap to fix something that we are seeing in the 
field and on the list.  I made a JIRA to capture this: 
https://issues.apache.org/jira/browse/METRON-1170  


---

[GitHub] metron issue #742: METRON-1168: Add SUBSTRING method to stellar

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/742
  
sounds good to me.  Thanks for the contribution!  I'm jealous I didn't 
think of it first.


---

[GitHub] metron pull request #745: METRON-1171: Better validation for the SUBSTRING s...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/745

METRON-1171: Better validation for the SUBSTRING stellar function

## Contributor Comments
As per the comments on #742 we need better structural validation on the 
SUBSTRING function.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-1171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #745


commit dd7129fcb769da6d049f06ef198c8834f5f78883
Author: cstella 
Date:   2017-09-09T01:53:21Z

METRON-1171: Better validation for the SUBSTRING stellar function




---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/740
  
I don't know if you saw my question, and it is hidden now so I'll restate:

What about functions that check for the GLOBAL_ flag, and may make 
assumptions that do not hold with a non-z partial configuration?




---

Re: Unclear recent commit

[Otto Fowler]

Yes, that was me.

I was committing a pr, and got an error on this while building and testing.
it was introduced by an earlier pr.  I sent an email to the list at the
time:

http://mail-archives.apache.org/mod_mbox/metron-dev/201708.mbox/%3ccao2evt6q-dcgjs3_dcgjanxauyfx-s7zgn+mfcmpkhuak9t...@mail.gmail.com%3e

Basically, while preparing a commit, verify lic. failed because of a
previously committed dependency change ( which went in without verify being
run I can only assume ).
I fixed the issue in my commit repo and pushed that commit with the pr
commit after re-testing etc.



On September 8, 2017 at 20:52:20, zeo...@gmail.com (zeo...@gmail.com) wrote:

I was looking through some of the recent commits and I noticed this[1],
anybody know what the back story is there?

1:
https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8a58265148a

Jon
-- 

Jon

[GitHub] metron issue #745: METRON-1171: Better validation for the SUBSTRING stellar ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/745
  
+1 LGTM - Thanks!


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/742


---

Re: [VOTE] Metron Release Candidate 0.4.1-RC3

[Casey Stella]

Yeah, I fixed the dependencies_with_url problem in METRON-1169.  Let's pull
that one in and the rat check one and cut a new RC.  I'm -1

On Fri, Sep 8, 2017 at 7:38 PM, Matt Foley  wrote:

> Couple pieces of info that may affect your vote:
>
> 1. This does not include today’s commits of
> METRON-1162 Apply Live Messages to the Profile Debugger
> (nickwallen)
> METRON-1077 Use column meta data end point in alerts ui
> (iraghumitra)
>
> 2. This does not pass RAT check (apache-rat-plugin) when pulled from
> tarball, altho it does when cloned from git.  I don’t know why it doesn’t
> happen with a git clone.  See https://github.com/apache/metron/pull/743
> and decide whether it needs to be in this release.
>
> 3. This does not pass build_utils/verify_licenses.sh (at least when pulled
> from tarball), the result is:
> Traceback (most recent call last):
>   File "build_utils/verify_license.py", line 44, in 
> raise ValueError("Unable to find these components: \n  " + "\n
> ".join(components_not_found) + "\nin the acceptable list of components: " +
> sys.argv[1])
> ValueError: Unable to find these components:
>   ch.hsr:geohash:jar:1.3.0:compile
>   org.locationtech.spatial4j:spatial4j:jar:0.6:compile
> in the acceptable list of components: ./dependencies_with_url.csv
>
> I suspect GeoHash requires an update in the “dependencies_with_url.csv”
> file.
> Thanks,
> --Matt
>
> On 9/8/17, 4:21 PM, "Matt Foley"  wrote:
>
> Colleagues,
> This is a call to vote on releasing Apache Metron 0.4.1.
> The release candidate is available at https://dist.apache.org/repos/
> dist/dev/metron/0.4.1-RC3/
>
> The candidate is RC3 because along the way I committed tags for two
> previous versions that were later invalidated before they were offered for
> vote.  I’ve modified the Release Process to avoid wasted tags in the future.
>
> Full list of changes in this release:
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/CHANGES
>
> The github tag to be voted upon is:
> apache-metron-0.4.1-rc3
>
> The source archive being voted upon can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/
> apache-metron-0.4.1-rc3.tar.gz
>
> The site-book is at:
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/
> site-book/index.html
>
> Other release files, signatures and digests can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/
>
> The release artifacts are signed with the following key:
> 4169 AA27 ECB3 1663 in https://dist.apache.org/repos/
> dist/dev/metron/0.4.1-RC3/KEYS
>
> Please vote on releasing this package as Apache Metron 0.4.1
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted
> here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This vote will be open until 5pm PDT Wednesday 13 Sep, due to the
> weekend.
>
>
>
>
>
>
>

[GitHub] metron pull request #743: METRON-1163 RAT failures for metron-interface/metr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/743


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137912849
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
 ---
@@ -321,6 +321,46 @@ public Object apply(List args) {
 }
   }
 
+  @Stellar( name="SUBSTRING"
+  , description = "Returns a substring of a string"
+  , params = {
+"input - The string to take the substring of",
+"start - The starting position (0-based and inclusive)",
+"end? - The ending position (0-based and exclusive)"
+ }
+  , returns = "The substring of the input"
+  )
+  public static class Substring extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings == null || strings.size() < 2 ) {
+throw new IllegalArgumentException("[SUBSTRING] required 2 
arguments: the input and the start position (inclusive)");
--- End diff --

The [] is not consistent with other functions #nit


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137913052
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
 ---
@@ -321,6 +321,46 @@ public Object apply(List args) {
 }
   }
 
+  @Stellar( name="SUBSTRING"
+  , description = "Returns a substring of a string"
+  , params = {
+"input - The string to take the substring of",
+"start - The starting position (0-based and inclusive)",
+"end? - The ending position (0-based and exclusive)"
+ }
+  , returns = "The substring of the input"
+  )
+  public static class Substring extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings == null || strings.size() < 2 ) {
+throw new IllegalArgumentException("[SUBSTRING] required 2 
arguments: the input and the start position (inclusive)");
+  }
+  String var = strings.get(0) == null?null: (String) strings.get(0);
+  Integer start = strings.get(1) == null?null:(Integer)strings.get(1);
--- End diff --

Type validation?


---

[GitHub] metron pull request #742: METRON-1168: Add SUBSTRING method to stellar

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/742#discussion_r137912941
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
 ---
@@ -321,6 +321,46 @@ public Object apply(List args) {
 }
   }
 
+  @Stellar( name="SUBSTRING"
+  , description = "Returns a substring of a string"
+  , params = {
+"input - The string to take the substring of",
+"start - The starting position (0-based and inclusive)",
+"end? - The ending position (0-based and exclusive)"
+ }
+  , returns = "The substring of the input"
+  )
+  public static class Substring extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+  if(strings == null || strings.size() < 2 ) {
+throw new IllegalArgumentException("[SUBSTRING] required 2 
arguments: the input and the start position (inclusive)");
+  }
+  String var = strings.get(0) == null?null: (String) strings.get(0);
+  Integer start = strings.get(1) == null?null:(Integer)strings.get(1);
+  Integer end = null;
+  if(strings.size() > 2) {
+ end = strings.get(2) == null ? null : (Integer) strings.get(2);
+  }
+  if(var == null || start == null) {
--- End diff --

We are again looking at the behavior with regards to invalid arguments, 
exception vs default return   We need some clarification document.  It 
seems to come up every addition to Stellar


---

Unclear recent commit

[zeo...@gmail.com]

I was looking through some of the recent commits and I noticed this[1],
anybody know what the back story is there?

1:
https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8a58265148a

Jon
-- 

Jon

[GitHub] metron issue #739: METRON-1165 [FEATURE-BRANCH] Add ability for BundleSystem...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/739
  
+1 by inspection, this looks very interesting


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/740
  
Can't we turn everything passed in into json and use the json patch stuff 
to "build out' the configuration?  Why not have the globals as a 
map all the time?


---

[GitHub] metron issue #740: METRON-1167 Define Session Specific Global Configuration ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/740
  
Even if they don't now, I believe that is the intent of the feature


---

Re: Unclear recent commit

[Casey Stella]

My guess is that what happened was otto did a pull of METRON-1061 into his
master, but it wasn't *quite* in sync with master in apache and there was a
conflict in the dependencies_with_url.csv, so that commit was made locally
to fix the conflict.  both commits, the squashed and the merge conflict
commit were pushed to apache master.

Otto, is that correct or am I just babbling on a friday night late? ;)

On Fri, Sep 8, 2017 at 8:52 PM, zeo...@gmail.com  wrote:

> I was looking through some of the recent commits and I noticed this[1],
> anybody know what the back story is there?
>
> 1:
> https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8
> a58265148a
>
> Jon
> --
>
> Jon
>

Re: Unclear recent commit

[Casey Stella]

I don't have an issue with this; it's good to have the explanation.  I also
found that we weren't running the dependency analyzer as part of travis
(or, we thought we were, but we weren't) until METRON-1169.  Thanks otto
for the explanation and jon for catching it.

On Fri, Sep 8, 2017 at 10:09 PM, Otto Fowler 
wrote:

> Yes, that was me.
>
> I was committing a pr, and got an error on this while building and testing.
> it was introduced by an earlier pr.  I sent an email to the list at the
> time:
>
> http://mail-archives.apache.org/mod_mbox/metron-dev/
> 201708.mbox/%3cCAO2EVT6q-Dcgjs3_DCGjaNXaUyfX-S7ZGn+
> mfcmpkhuak9t...@mail.gmail.com%3e
>
> Basically, while preparing a commit, verify lic. failed because of a
> previously committed dependency change ( which went in without verify being
> run I can only assume ).
> I fixed the issue in my commit repo and pushed that commit with the pr
> commit after re-testing etc.
>
>
>
> On September 8, 2017 at 20:52:20, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> I was looking through some of the recent commits and I noticed this[1],
> anybody know what the back story is there?
>
> 1:
> https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8
> a58265148a
>
> Jon
> --
>
> Jon
>

Re: [VOTE] Metron Release Candidate 0.4.1-RC3

[Casey Stella]

Just a FYI, I think all the relevant PRs to fix the rat check and
dependency analyzer are committed now and we can cut another RC whenever
we're ready.

On Fri, Sep 8, 2017 at 8:24 PM, Casey Stella  wrote:

> Yeah, I fixed the dependencies_with_url problem in METRON-1169.  Let's
> pull that one in and the rat check one and cut a new RC.  I'm -1
>
> On Fri, Sep 8, 2017 at 7:38 PM, Matt Foley  wrote:
>
>> Couple pieces of info that may affect your vote:
>>
>> 1. This does not include today’s commits of
>> METRON-1162 Apply Live Messages to the Profile Debugger
>> (nickwallen)
>> METRON-1077 Use column meta data end point in alerts ui
>> (iraghumitra)
>>
>> 2. This does not pass RAT check (apache-rat-plugin) when pulled from
>> tarball, altho it does when cloned from git.  I don’t know why it doesn’t
>> happen with a git clone.  See https://github.com/apache/metron/pull/743
>> and decide whether it needs to be in this release.
>>
>> 3. This does not pass build_utils/verify_licenses.sh (at least when
>> pulled from tarball), the result is:
>> Traceback (most recent call last):
>>   File "build_utils/verify_license.py", line 44, in 
>> raise ValueError("Unable to find these components: \n  " + "\n
>> ".join(components_not_found) + "\nin the acceptable list of components: " +
>> sys.argv[1])
>> ValueError: Unable to find these components:
>>   ch.hsr:geohash:jar:1.3.0:compile
>>   org.locationtech.spatial4j:spatial4j:jar:0.6:compile
>> in the acceptable list of components: ./dependencies_with_url.csv
>>
>> I suspect GeoHash requires an update in the “dependencies_with_url.csv”
>> file.
>> Thanks,
>> --Matt
>>
>> On 9/8/17, 4:21 PM, "Matt Foley"  wrote:
>>
>> Colleagues,
>> This is a call to vote on releasing Apache Metron 0.4.1.
>> The release candidate is available at https://dist.apache.org/repos/
>> dist/dev/metron/0.4.1-RC3/
>>
>> The candidate is RC3 because along the way I committed tags for two
>> previous versions that were later invalidated before they were offered for
>> vote.  I’ve modified the Release Process to avoid wasted tags in the future.
>>
>> Full list of changes in this release:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/CHANGES
>>
>> The github tag to be voted upon is:
>> apache-metron-0.4.1-rc3
>>
>> The source archive being voted upon can be found here:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/apac
>> he-metron-0.4.1-rc3.tar.gz
>>
>> The site-book is at:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/site
>> -book/index.html
>>
>> Other release files, signatures and digests can be found here:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/
>>
>> The release artifacts are signed with the following key:
>> 4169 AA27 ECB3 1663 in https://dist.apache.org/repos/
>> dist/dev/metron/0.4.1-RC3/KEYS
>>
>> Please vote on releasing this package as Apache Metron 0.4.1
>>
>> When voting, please list the actions taken to verify the release.
>>
>> Recommended build validation and verification instructions are posted
>> here:
>> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>>
>> This vote will be open until 5pm PDT Wednesday 13 Sep, due to the
>> weekend.
>>
>>
>>
>>
>>
>>
>>
>

Re: Unclear recent commit

[Otto Fowler]

without re-writing my other reply:
I use the prepare-commit into a clean repo every time, so I believe that
this issue
*was* in the apache repo.


On September 8, 2017 at 22:08:02, Casey Stella (ceste...@gmail.com) wrote:

My guess is that what happened was otto did a pull of METRON-1061 into his
master, but it wasn't *quite* in sync with master in apache and there was a
conflict in the dependencies_with_url.csv, so that commit was made locally
to fix the conflict. both commits, the squashed and the merge conflict
commit were pushed to apache master.

Otto, is that correct or am I just babbling on a friday night late? ;)

On Fri, Sep 8, 2017 at 8:52 PM, zeo...@gmail.com  wrote:

> I was looking through some of the recent commits and I noticed this[1],
> anybody know what the back story is there?
>
> 1:
> https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8
> a58265148a
>
> Jon
> --
>
> Jon
>

Re: Unclear recent commit

[Casey Stella]

That's very weird.

On Fri, Sep 8, 2017 at 10:12 PM, Otto Fowler 
wrote:

> without re-writing my other reply:
> I use the prepare-commit into a clean repo every time, so I believe that
> this issue
> *was* in the apache repo.
>
>
> On September 8, 2017 at 22:08:02, Casey Stella (ceste...@gmail.com) wrote:
>
> My guess is that what happened was otto did a pull of METRON-1061 into his
> master, but it wasn't *quite* in sync with master in apache and there was
> a
> conflict in the dependencies_with_url.csv, so that commit was made locally
> to fix the conflict. both commits, the squashed and the merge conflict
> commit were pushed to apache master.
>
> Otto, is that correct or am I just babbling on a friday night late? ;)
>
> On Fri, Sep 8, 2017 at 8:52 PM, zeo...@gmail.com 
> wrote:
>
> > I was looking through some of the recent commits and I noticed this[1],
> > anybody know what the back story is there?
> >
> > 1:
> > https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8
> > a58265148a
> >
> > Jon
> > --
> >
> > Jon
> >
>
>

Re: Unclear recent commit

[Otto Fowler]

Is what I did against policy?  What should I have done?
I think the committer’s guide is archived btw.


On September 8, 2017 at 22:12:27, Casey Stella (ceste...@gmail.com) wrote:

I don't have an issue with this; it's good to have the explanation. I also
found that we weren't running the dependency analyzer as part of travis
(or, we thought we were, but we weren't) until METRON-1169. Thanks otto
for the explanation and jon for catching it.

On Fri, Sep 8, 2017 at 10:09 PM, Otto Fowler 
wrote:

> Yes, that was me.
>
> I was committing a pr, and got an error on this while building and
testing.
> it was introduced by an earlier pr. I sent an email to the list at the
> time:
>
> http://mail-archives.apache.org/mod_mbox/metron-dev/
> 201708.mbox/%3cCAO2EVT6q-Dcgjs3_DCGjaNXaUyfX-S7ZGn+
> mfcmpkhuak9t...@mail.gmail.com%3e
>
> Basically, while preparing a commit, verify lic. failed because of a
> previously committed dependency change ( which went in without verify
being
> run I can only assume ).
> I fixed the issue in my commit repo and pushed that commit with the pr
> commit after re-testing etc.
>
>
>
> On September 8, 2017 at 20:52:20, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> I was looking through some of the recent commits and I noticed this[1],
> anybody know what the back story is there?
>
> 1:
> https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8
> a58265148a
>
> Jon
> --
>
> Jon
>

[GitHub] metron pull request #743: METRON-1163 RAT failures for metron-interface/metr...

[mattf-horton]

GitHub user mattf-horton opened a pull request:

https://github.com/apache/metron/pull/743

METRON-1163 RAT failures for metron-interface/metron-alerts

## Contributor Comments
Please see https://issues.apache.org/jira/browse/METRON-1163 for the 
problem.
I'm not entirely pleased with excluding 3rdpartylicenses.txt, as if it 
changes in future to non-empty, we could miss it.  Alternative suggestions 
welcome.

This is required for the 0.4.1 release, else RAT doesn't pass.

Reproduce: 
This problem seems to only reproduce when one unrolls a tarball rather than 
cloning from github.
Do:
```
mkdir workdir; cd workdir
wget 
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/apache-metron-0.4.1-rc3.tar.gz
tar xvzf apache-metron-0.4.1-rc3.tar.gz
cd apache-metron-0.4.1-rc3
mvn clean
mvn install -DskipTests
```
When it comes to metron-alerts module, apache-rat-plugin should error, 
complaining about:
```
Files with unapproved licenses:
.../metron-interface/metron-alerts/dist/3rdpartylicenses.txt

.../metron-interface/metron-alerts/dist/styles.a0b6b99c10d9a13dc67e.bundle.css
```
To test the patch, while in directory apache-metron-0.4.1-rc3 do:
```
curl -s -S 
https://issues.apache.org/jira/secure/attachment/12886172/METRON-1163.patch | \
patch -p 1
```
then re-run the mvn command (or just do `mvn install -DskipTests -rf 
:metron-alerts`) and it should pass.


## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [NA] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [NA] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [NA] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [NA] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [NA] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattf-horton/metron METRON-1163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/743.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #743


commit 5b10b544874c6100f7449cedd495669ac7b8c3af
Author: mattf-horton 
Date:   2017-09-08T01:27:52Z

METRON-1163 RAT failures for metron-interface/metron-alerts




---

Re: [VOTE] Metron Release Candidate 0.4.1-RC3

[Matt Foley]

Couple pieces of info that may affect your vote:

1. This does not include today’s commits of 
METRON-1162 Apply Live Messages to the Profile Debugger (nickwallen)
METRON-1077 Use column meta data end point in alerts ui (iraghumitra)

2. This does not pass RAT check (apache-rat-plugin) when pulled from tarball, 
altho it does when cloned from git.  I don’t know why it doesn’t happen with a 
git clone.  See https://github.com/apache/metron/pull/743 and decide whether it 
needs to be in this release.

3. This does not pass build_utils/verify_licenses.sh (at least when pulled from 
tarball), the result is:
Traceback (most recent call last):
  File "build_utils/verify_license.py", line 44, in 
raise ValueError("Unable to find these components: \n  " + "\n  
".join(components_not_found) + "\nin the acceptable list of components: " + 
sys.argv[1])
ValueError: Unable to find these components: 
  ch.hsr:geohash:jar:1.3.0:compile
  org.locationtech.spatial4j:spatial4j:jar:0.6:compile
in the acceptable list of components: ./dependencies_with_url.csv

I suspect GeoHash requires an update in the “dependencies_with_url.csv” file.
Thanks,
--Matt

On 9/8/17, 4:21 PM, "Matt Foley"  wrote:

Colleagues,
This is a call to vote on releasing Apache Metron 0.4.1.
The release candidate is available at 
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/ 

The candidate is RC3 because along the way I committed tags for two 
previous versions that were later invalidated before they were offered for 
vote.  I’ve modified the Release Process to avoid wasted tags in the future.

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/CHANGES

The github tag to be voted upon is:
apache-metron-0.4.1-rc3

The source archive being voted upon can be found here:

https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/apache-metron-0.4.1-rc3.tar.gz

The site-book is at:

https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/site-book/index.html 

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/

The release artifacts are signed with the following key:
4169 AA27 ECB3 1663 in 
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/KEYS

Please vote on releasing this package as Apache Metron 0.4.1

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted
here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open until 5pm PDT Wednesday 13 Sep, due to the weekend.

Re: [DISCUSS] Metron release 0.4.1

[Matt Foley]

METRON-1163 turns out to be invalid, due to a bad maven settings file.  It is 
no longer an issue.
Just waiting for the last couple items.

On 9/7/17, 9:26 AM, "Matt Foley"  wrote:

Okay.  Please ping when committed.
Also, any input on https://issues.apache.org/jira/browse/METRON-1163 ?

On 9/7/17, 7:39 AM, "Ryan Merriman"  wrote:

Matt,

We recently found a bug that's breaking certain features in the 
management
UI.  The fixes are still in review (
https://github.com/apache/metron/pull/729 and
https://github.com/apache/metron/pull/730) and should make it in soon.  
It
would be good to include these if possible.

Ryan

On Thu, Sep 7, 2017 at 12:23 AM, Matt Foley  wrote:

> I’ve got a blocker bug, 
https://issues.apache.org/jira/browse/METRON-1163
> , RAT failures for metron-interface/metron-alerts.  A comment in the 
jira
> suggests a way to address it, but someone familiar with the code 
should
> look at it.
>
>
>
> Raghu, would you be able to take a look?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Matt Foley  on behalf of Matt Foley <
> ma...@apache.org>
> Date: Tuesday, September 5, 2017 at 10:01 AM
> To: "dev@metron.apache.org" 
> Subject: Re: [DISCUSS] Metron release 0.4.1
>
>
>
> Great, working on it!
>
>
>
> From: Nick Allen 
> Date: Tuesday, September 5, 2017 at 8:00 AM
> To: Casey Stella , "zeo...@gmail.com" <
> zeo...@gmail.com>
> Cc: Anand Subramanian , Matt Foley <
> mfo...@hortonworks.com>, "dev@metron.apache.org" 

> Subject: Re: [DISCUSS] Metron release 0.4.1
>
>
>
> All set here.  Let's get this shipped!
>
>
>
>
>
>
>
> On Tue, Sep 5, 2017 at 10:44 AM Casey Stella  
wrote:
>
> me too
>
>
>
> On Tue, Sep 5, 2017 at 10:43 AM, zeo...@gmail.com 
> wrote:
>
> All set from my perspective.
>
>
>
> Jon
>
>
>
> On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> Hello Matt,
>
> Simon's pull request supersedes mine (METRON-1139 /
> https://github.com/apache/metron/pull/722) and is already merged into
> master.
>
> Thanks,
> Anand
>
>
>
> On 9/1/17, 12:41 AM, "Matt Foley"  wrote:
>
> >Please mark them 0.4.1, as that’s what the community says we want to 
call
> the upcoming release, and everything that’s there when I throw the 
switch
> will be included.
> >
> >Jon and Anand, will they be in by end/day Friday?
> >Thanks,
> >--Matt
> >
> >On 8/31/17, 7:45 AM, "Nick Allen"  wrote:
> >
> >Matt, et al - For JIRAs that are going into master, should we be
> marking
> >these as "Next + 1" or "0.4.1" ?
> >
> >On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com 

> wrote:
> >
> >> Can I advocate to get METRON-1129 in the RC, and throw in a 
second
> vote for
> >> METRON-1134?  Both in an attempt to better support of 
prod/offline
> use.
> >> Happy to provide testing cycles for the former.
> >>
> >> Jon
> >>
> >> On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
> >> asubraman...@hortonworks.com> wrote:
> >>
> >> > Hi Matt,
> >> >
> >> > This defect needs to be included as a follow-on to 
METRON-1122:
> >> > * METRON-1141 (https://github.com/apache/metron/pull/723)
> >> >
> >> >
> >> > Thanks,
> >> > Anand
> >> >
> >> >
> >> >
> >> > On 8/30/17, 8:57 PM, "Michael Miklavcic" <
> michael.miklav...@gmail.com>
> >> > wrote:
> >> >
> >> > >I have some work around fixing how we handle config with 
Ambari
> that I'd
> >> > >like to see go in. No PR yet, but coming soon. I expect to 
have
> this by
> >> > the
> > 

[GitHub] metron issue #741: METRON-1153 HDFS HdfsWriter never recovers from exception...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/741
  
Adjusted to only catch the "Stream Closed" exception, not the general 
IOException.  If it's stream closed, attempt the rotate and write, otherwise 
rethrow


---

[GitHub] metron pull request #739: METRON-1165 [FEATURE-BRANCH] Add ability for Bundl...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/739


---

Re: [VOTE] Metron Release Candidate 0.4.1-RC3

[Matt Foley]

Thank you, Casey.  RC3 is WITHDRAWN.  I’ll post RC4 shortly, and start a new 
VOTE thread.
--Matt

On 9/8/17, 7:14 PM, "Casey Stella"  wrote:

Just a FYI, I think all the relevant PRs to fix the rat check and
dependency analyzer are committed now and we can cut another RC whenever
we're ready.

On Fri, Sep 8, 2017 at 8:24 PM, Casey Stella  wrote:

> Yeah, I fixed the dependencies_with_url problem in METRON-1169.  Let's
> pull that one in and the rat check one and cut a new RC.  I'm -1
>
> On Fri, Sep 8, 2017 at 7:38 PM, Matt Foley  wrote:
>
>> Couple pieces of info that may affect your vote:
>>
>> 1. This does not include today’s commits of
>> METRON-1162 Apply Live Messages to the Profile Debugger
>> (nickwallen)
>> METRON-1077 Use column meta data end point in alerts ui
>> (iraghumitra)
>>
>> 2. This does not pass RAT check (apache-rat-plugin) when pulled from
>> tarball, altho it does when cloned from git.  I don’t know why it doesn’t
>> happen with a git clone.  See https://github.com/apache/metron/pull/743
>> and decide whether it needs to be in this release.
>>
>> 3. This does not pass build_utils/verify_licenses.sh (at least when
>> pulled from tarball), the result is:
>> Traceback (most recent call last):
>>   File "build_utils/verify_license.py", line 44, in 
>> raise ValueError("Unable to find these components: \n  " + "\n
>> ".join(components_not_found) + "\nin the acceptable list of components: 
" +
>> sys.argv[1])
>> ValueError: Unable to find these components:
>>   ch.hsr:geohash:jar:1.3.0:compile
>>   org.locationtech.spatial4j:spatial4j:jar:0.6:compile
>> in the acceptable list of components: ./dependencies_with_url.csv
>>
>> I suspect GeoHash requires an update in the “dependencies_with_url.csv”
>> file.
>> Thanks,
>> --Matt
>>
>> On 9/8/17, 4:21 PM, "Matt Foley"  wrote:
>>
>> Colleagues,
>> This is a call to vote on releasing Apache Metron 0.4.1.
>> The release candidate is available at https://dist.apache.org/repos/
>> dist/dev/metron/0.4.1-RC3/
>>
>> The candidate is RC3 because along the way I committed tags for two
>> previous versions that were later invalidated before they were offered 
for
>> vote.  I’ve modified the Release Process to avoid wasted tags in the 
future.
>>
>> Full list of changes in this release:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/CHANGES
>>
>> The github tag to be voted upon is:
>> apache-metron-0.4.1-rc3
>>
>> The source archive being voted upon can be found here:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/apac
>> he-metron-0.4.1-rc3.tar.gz
>>
>> The site-book is at:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/site
>> -book/index.html
>>
>> Other release files, signatures and digests can be found here:
>> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC3/
>>
>> The release artifacts are signed with the following key:
>> 4169 AA27 ECB3 1663 in https://dist.apache.org/repos/
>> dist/dev/metron/0.4.1-RC3/KEYS
>>
>> Please vote on releasing this package as Apache Metron 0.4.1
>>
>> When voting, please list the actions taken to verify the release.
>>
>> Recommended build validation and verification instructions are posted
>> here:
>> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>>
>> This vote will be open until 5pm PDT Wednesday 13 Sep, due to the
>> weekend.
>>
>>
>>
>>
>>
>>
>>
>

[VOTE] Metron Release Candidate 0.4.1-RC4

[Matt Foley]

Colleagues,
This is a call to vote on releasing Apache Metron 0.4.1.
The release candidate is available at 
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/ 

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/CHANGES

The github tag to be voted upon is:
apache-metron-0.4.1-rc4

The source archive being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/apache-metron-0.4.1-rc4.tar.gz

The site-book is at:
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/site-book/index.html 

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/

The release artifacts are signed with the following key:
4169 AA27 ECB3 1663 in 
https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/KEYS

Please vote on releasing this package as Apache Metron 0.4.1

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open until 5pm PDT Wednesday 13 Sep, due to the weekend.
Thanks,
--Matt
(release manager)

Re: Unclear recent commit

[Casey Stella]

So, generally the goal is to commit the minimal set of commits squashed by
contributor.  For situations that are wacky and end up with a conflict, I
don't think it's a problem.  No worries, IMO.

On Fri, Sep 8, 2017 at 10:29 PM, Otto Fowler 
wrote:

> Is what I did against policy?  What should I have done?
> I think the committer’s guide is archived btw.
>
>
> On September 8, 2017 at 22:12:27, Casey Stella (ceste...@gmail.com) wrote:
>
> I don't have an issue with this; it's good to have the explanation. I also
> found that we weren't running the dependency analyzer as part of travis
> (or, we thought we were, but we weren't) until METRON-1169. Thanks otto
> for the explanation and jon for catching it.
>
> On Fri, Sep 8, 2017 at 10:09 PM, Otto Fowler 
> wrote:
>
> > Yes, that was me.
> >
> > I was committing a pr, and got an error on this while building and
> testing.
> > it was introduced by an earlier pr. I sent an email to the list at the
> > time:
> >
> > http://mail-archives.apache.org/mod_mbox/metron-dev/
> > 201708.mbox/%3cCAO2EVT6q-Dcgjs3_DCGjaNXaUyfX-S7ZGn+
> > mfcmpkhuak9t...@mail.gmail.com%3e
> >
> > Basically, while preparing a commit, verify lic. failed because of a
> > previously committed dependency change ( which went in without verify
> being
> > run I can only assume ).
> > I fixed the issue in my commit repo and pushed that commit with the pr
> > commit after re-testing etc.
> >
> >
> >
> > On September 8, 2017 at 20:52:20, zeo...@gmail.com (zeo...@gmail.com)
> > wrote:
> >
> > I was looking through some of the recent commits and I noticed this[1],
> > anybody know what the back story is there?
> >
> > 1:
> > https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8
> > a58265148a
> >
> > Jon
> > --
> >
> > Jon
> >
>
>