[DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron

[Michael Miklavcic]

This came up earlier when discussing work around the ES upgrade:
https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E

Looks like Otto made this suggestion and Kyle is on board. I was originally
opposed to this because it did not seem worth the effort to support 2
separate MPacks. However, now that we are working on the Solr upgrade, it
seems like an appropriate solution for enabling us to make the indexing
piece pluggable. I propose that we commence with this solution.

Cheers,
Mike

[GitHub] metron pull request #579: METRON-941 native PaloAlto parser corrupts message...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/579


---

[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/579
  
I'm still +1.  This is really great, and everyone definitely appreciates 
how much work went into improving this so much.


---

[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
+1 one.  Thank you @ctramnitz.  I know how it can be to have a long review 
process.  This is really good work and I look forward to seeing what you do 
next ;)


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168817020
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HBaseConfig.java
 ---
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.hbase.HTableProvider;
+import org.apache.metron.rest.MetronRestConstants;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.service.GlobalConfigService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.Profile;
+import org.springframework.core.env.Environment;
+
+import java.io.IOException;
+
+import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
+import static 
org.apache.metron.rest.repository.UserSettingsRepository.USER_SETTINGS_HBASE_TABLE;
+
+@Configuration
+@Profile("!" + TEST_PROFILE)
--- End diff --

We don't want to use this Configuration if tests are running.  There is a 
separate TestConfig that setups up mock, inmemory components, etc.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168816712
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/SearchRequest.java
 ---
@@ -101,16 +99,16 @@ public void setSort(List sort) {
 this.sort = sort;
   }
 
-  public Optional getFields() {
-return fields == null || fields.size() == 0 ? Optional.empty() : 
Optional.of(fields);
+  public List getFields() {
+return fields;
   }
 
   public void setFields(List fields) {
 this.fields = fields;
   }
 
-  public Optional getFacetFields() {
-return facetFields == null || facetFields.size() == 0 ? 
Optional.empty() : Optional.of(facetFields);
+  public List getFacetFields() {
+return facetFields;
--- End diff --

Because null and an empty array mean 2 different things.  See point 6 here: 
 https://github.com/apache/metron/pull/853#issuecomment-356350839.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168816212
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
 ---
@@ -179,6 +203,35 @@ def status_rest_application(self, env):
 self.__params.metron_rest_port,
 self.__params.metron_user)
 
+def create_hbase_tables(self):
+Logger.info("Creating HBase Tables")
+metron_service.create_hbase_table(self.__params,
+  
self.__params.user_settings_hbase_table,
+  
self.__params.user_settings_hbase_cf)
+Logger.info("Done creating HBase Tables")
+self.set_hbase_configured()
+
+def set_hbase_acls(self):
+Logger.info("Setting HBase ACLs")
+if self.__params.security_enabled:
+kinit(self.__params.kinit_path_local,
+  self.__params.hbase_keytab_path,
+  self.__params.hbase_principal_name,
+  execute_user=self.__params.hbase_user)
+
+cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
--- End diff --

Good catch.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168809125
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/SearchRequest.java
 ---
@@ -101,16 +99,16 @@ public void setSort(List sort) {
 this.sort = sort;
   }
 
-  public Optional getFields() {
-return fields == null || fields.size() == 0 ? Optional.empty() : 
Optional.of(fields);
+  public List getFields() {
+return fields;
   }
 
   public void setFields(List fields) {
 this.fields = fields;
   }
 
-  public Optional getFacetFields() {
-return facetFields == null || facetFields.size() == 0 ? 
Optional.empty() : Optional.of(facetFields);
+  public List getFacetFields() {
+return facetFields;
--- End diff --

If we're going to change the interface, what about
```
return facetFields == null ? new ArrayList() : facetFields;
```
so it's not null.


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168810966
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HBaseConfig.java
 ---
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.metron.hbase.HTableProvider;
+import org.apache.metron.rest.MetronRestConstants;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.service.GlobalConfigService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.Profile;
+import org.springframework.core.env.Environment;
+
+import java.io.IOException;
+
+import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
+import static 
org.apache.metron.rest.repository.UserSettingsRepository.USER_SETTINGS_HBASE_TABLE;
+
+@Configuration
+@Profile("!" + TEST_PROFILE)
--- End diff --

Test profile?


---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r168802913
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
 ---
@@ -179,6 +203,35 @@ def status_rest_application(self, env):
 self.__params.metron_rest_port,
 self.__params.metron_user)
 
+def create_hbase_tables(self):
+Logger.info("Creating HBase Tables")
+metron_service.create_hbase_table(self.__params,
+  
self.__params.user_settings_hbase_table,
+  
self.__params.user_settings_hbase_cf)
+Logger.info("Done creating HBase Tables")
+self.set_hbase_configured()
+
+def set_hbase_acls(self):
+Logger.info("Setting HBase ACLs")
+if self.__params.security_enabled:
+kinit(self.__params.kinit_path_local,
+  self.__params.hbase_keytab_path,
+  self.__params.hbase_principal_name,
+  execute_user=self.__params.hbase_user)
+
+cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
--- End diff --

Do we check that the REST API has the hbase client also installed on the 
node? This would be needed for this to work for > 1 nodes.


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
I am not an expert in hbase, I cannot say how i'd implement it.
The version thing we can leave out, it will be just as good or bad as what 
we have in zookeeper right?

I think the use case is:

For any given User ( at least ) we need the ability to store currently 
known and future unknown configurations as bytes, with configuration type 
identifier lookup.  This should be done such that new configuration types can 
be added by 'putting' config bytes with a new identifier.

I think that gets us a long way.


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
I think it's a fair question @ottobackwards.  Anything that might affect 
how the HBase table is laid out should be worked out now or we're back to 
altering tables during upgrades.  Currently the row key is the user name, the 
column family is hardcoded (although configurable), and the column qualifier is 
hardcoded.  Would you add the type to the row key or store different client 
settings in different columns?

We could add versioning but that is not trivial and will increase the scope 
of this PR.  I can take it on here if we decide we want it but this PR is 
already large.  Follow on maybe?  Either way works for me.  If we decide a 
follow on is better I would make an effort to do it right away before we do a 
release.

Jackson is the standard in Metron for serialization/deserialization.  I 
think we should have a very compelling reason to introduce something new.  
Happy to discuss.




---

Re: Apache Website Required Links

[Michael Miklavcic]

That's awesome Anand, thanks for tackling this!

On Fri, Feb 16, 2018 at 7:41 AM, Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Btw, here is the output from running site-scan on my local version of the
> changes:
>
> ➜  tools git:(master) ruby site-scan.rb http://127.0.0.1:4000
> 127 http://127.0.0.1:4000 missing
> {
>   "127": {
> "display_name": "127",
> "uri": "http://127.0.0.1:4000;,
> "events": "https://www.apache.org/events/current-event;,
> "foundation": "ASF HOME",
> "license": "https://www.apache.org/licenses/LICENSE-2.0;,
> "sponsorship": "https://www.apache.org/foundation/sponsorship.html;,
> "security": "https://www.apache.org/security;,
> "trademarks": "Apache Metron and its logo are trademarks of The Apache
> Software Foundation.",
> "copyright": "Copyright © 2018, The Apache Software Foundation.",
> "image": null,
> "thanks": "https://www.apache.org/foundation/thanks.html;,
> "copyparent": true
>   }
> }
>
> Regards
> Anand
>
> On 2/16/18, 8:09 PM, "Anand Subramanian" 
> wrote:
>
> Hello All,
>
> Apache Whimsy checks for the site requisites in the main index.html
> and not inside sub-levels. I have created METRON-1457 (
> https://github.com/apache/metron/pull/938) to move the ASF links to the
> main page.
>
> Here is a screenshot of how the new metron page will look like with
> the ASF links above the page footer:
> https://imgur.com/3Y8ZLWL
>
> Please review and let me know what you think about the new look.
>
> Thanks
> Anand
>
> On 2/16/18, 2:08 AM, "Casey Stella"  wrote:
>
> Just reporting back that Anand's PR METRON-1386 (
> https://github.com/apache/metron/pull/935) has been merged into
> master and
> the asf-site branch.
> Kudos to Anand!
>
> Casey
>
> On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> > I can take a shot at this if there are no other takers.
> >
> > Regards,
> > Anand
> >
> > On 2/5/18, 8:59 PM, "Justin Leet"  wrote:
> >
> > I'd created a Jira awhile ago, but it deserves a callout to
> the
> > community.
> > Especially if someone wants to grab it, it's probably
> something pretty
> > easy
> > (and valuable!) to do.
> >
> > There's a set of required links on Apache web pages, which
> can be seen
> > at Website
> > Navigation Links Policy
> > 
> >
> > Reporting is at Site Check For Project - Metron
> > 
> >
> > This ticket is available at:
> > METRON-1386  jira/browse/METRON-1386>
> >
> >
> >
>
>
>
>
>

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[MohanDV]

Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/936
  
@JonZeolla Sorry I dint notice the METRON-1454 , I have updated the 
description for METRON-1450. 
I will mark my METRON-1458 as the duplicate of METRON-1454. 


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
I don't see a PR for METRON-1458, did you mean you opened a 
[JIRA](https://issues.apache.org/jira/projects/METRON/issues/METRON-1458)?  In 
the JIRA for 
[METRON-1450](https://issues.apache.org/jira/projects/METRON/issues/METRON-1450)
 the summary is:
```
Add documentation for splitting the indexing topology into 
random_access_indexing and batch_indexing
```

Based on that description, this would be the ticket to address the indexing 
README.  If you think it's worth splitting this into two JIRAs that's fine with 
me, but I already had 
[METRON-1454](https://issues.apache.org/jira/projects/METRON/issues/METRON-1454)
 open to remediate the main indexing README, but I closed it because the 
description of 
[METRON-1450](https://issues.apache.org/jira/projects/METRON/issues/METRON-1450)
 mirrored what I was looking to accomplish.  Let me know where you'd like to go 
from here, but regardless we should make sure that the PR title here matches 
the JIRA summary.


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[MohanDV]

Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/936
  
@JonZeolla it's out of the scope of this PR, This PR is intended to update 
the doc for rest endpoints.  I have created another PR (METRON-1458) to update 
the main README for indexing.


---

[GitHub] metron pull request #579: METRON-941 native PaloAlto parser corrupts message...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/579#discussion_r168779259
  
--- Diff: Upgrading.md ---
@@ -19,6 +19,24 @@ limitations under the License.
 This document constitutes a per-version listing of changes of
 configuration which are non-backwards compatible.
 
+## 0.4.2 to 0.4.3
+
+### [METRON-941: native PaloAlto parser corrupts message when having a 
comma in the payload](https://issues.apache.org/jira/browse/METRON-941)
+While modifying the PaloAlto log parser to support logs from newer
+PAN-OS version and to not break when a message payload contains a
--- End diff --

change/changed


---

[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

[ctramnitz]

Github user ctramnitz commented on the issue:

https://github.com/apache/metron/pull/579
  
No it's not a requirement. The parser will continue to work the same way as 
it did before if you feed it a full syslog line including header. (Which 
wouldn't produce a valid domain field, but thats exactly the same behavior as 
before)


---

[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
maybe regression isn't the right word  - 
"Until we have something generic to (pre-)parse syslog before it reaches 
the message parser I assumed the messages will be stripped off the syslog 
header for now." is this not a new requirement or behavior?


---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
Of course re-reading your comment, I see you specifically didn't account 
for these, but I think there are some basic tweaks to allow for this that don't 
presuppose far flung 20% use cases.


---

[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

[ctramnitz]

Github user ctramnitz commented on the issue:

https://github.com/apache/metron/pull/579
  
@ottobackwards Where is the regression? If a user used the parser 
previously with a full syslog header it will continue to work the same way. The 
result will be the same odd domain field ", 1" instead of "1". 
The parser hasn't changed, only the test which expects just the message now. 
And since the test was non-functional before (it was never invoked), this 
cannot be regression either.

The only thing that has changed is some field names, but since this was 
utterly broken before, i.e.
BasicPaloAltoFirewallParser.java line 84++
```
 
-  public static final String Bytes = "content_type";
-  public static final String BytesSent = "content_type";
-  public static final String BytesReceived = "content_type";
-  public static final String Packets = "content_type";
-  public static final String StartTime = "content_type";
-  public static final String ElapsedTimeInSec = "content_type";
-  public static final String Padding = "content_type";
```
I wouldn't call this a regression, it may be worth a note though.


---

Re: Apache Website Required Links

[Anand Subramanian]

Btw, here is the output from running site-scan on my local version of the 
changes:

➜  tools git:(master) ruby site-scan.rb http://127.0.0.1:4000
127 http://127.0.0.1:4000 missing
{
  "127": {
"display_name": "127",
"uri": "http://127.0.0.1:4000;,
"events": "https://www.apache.org/events/current-event;,
"foundation": "ASF HOME",
"license": "https://www.apache.org/licenses/LICENSE-2.0;,
"sponsorship": "https://www.apache.org/foundation/sponsorship.html;,
"security": "https://www.apache.org/security;,
"trademarks": "Apache Metron and its logo are trademarks of The Apache 
Software Foundation.",
"copyright": "Copyright © 2018, The Apache Software Foundation.",
"image": null,
"thanks": "https://www.apache.org/foundation/thanks.html;,
"copyparent": true
  }
}

Regards
Anand 

On 2/16/18, 8:09 PM, "Anand Subramanian"  wrote:

Hello All,

Apache Whimsy checks for the site requisites in the main index.html and not 
inside sub-levels. I have created METRON-1457 
(https://github.com/apache/metron/pull/938) to move the ASF links to the main 
page.

Here is a screenshot of how the new metron page will look like with the ASF 
links above the page footer:
https://imgur.com/3Y8ZLWL

Please review and let me know what you think about the new look.

Thanks
Anand

On 2/16/18, 2:08 AM, "Casey Stella"  wrote:

Just reporting back that Anand's PR METRON-1386 (
https://github.com/apache/metron/pull/935) has been merged into master 
and
the asf-site branch.
Kudos to Anand!

Casey

On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> I can take a shot at this if there are no other takers.
>
> Regards,
> Anand
>
> On 2/5/18, 8:59 PM, "Justin Leet"  wrote:
>
> I'd created a Jira awhile ago, but it deserves a callout to the
> community.
> Especially if someone wants to grab it, it's probably something 
pretty
> easy
> (and valuable!) to do.
>
> There's a set of required links on Apache web pages, which can be 
seen
> at Website
> Navigation Links Policy
> 
>
> Reporting is at Site Check For Project - Metron
> 
>
> This ticket is available at:
> METRON-1386 
>
>
>

Re: Apache Website Required Links

[Anand Subramanian]

Hello All,

Apache Whimsy checks for the site requisites in the main index.html and not 
inside sub-levels. I have created METRON-1457 
(https://github.com/apache/metron/pull/938) to move the ASF links to the main 
page.

Here is a screenshot of how the new metron page will look like with the ASF 
links above the page footer:
https://imgur.com/3Y8ZLWL

Please review and let me know what you think about the new look.

Thanks
Anand

On 2/16/18, 2:08 AM, "Casey Stella"  wrote:

Just reporting back that Anand's PR METRON-1386 (
https://github.com/apache/metron/pull/935) has been merged into master and
the asf-site branch.
Kudos to Anand!

Casey

On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> I can take a shot at this if there are no other takers.
>
> Regards,
> Anand
>
> On 2/5/18, 8:59 PM, "Justin Leet"  wrote:
>
> I'd created a Jira awhile ago, but it deserves a callout to the
> community.
> Especially if someone wants to grab it, it's probably something pretty
> easy
> (and valuable!) to do.
>
> There's a set of required links on Apache web pages, which can be seen
> at Website
> Navigation Links Policy
> 
>
> Reporting is at Site Check For Project - Metron
> 
>
> This ticket is available at:
> METRON-1386 
>
>
>

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
First, nice work.
Second, I have this question:

Since we will have some unknown number of rest clients, that may want to 
save 'configuration' associated with a user, and not just Alerts UI, should it 
not be factored more generically?

getUserSettingsForType()

And stored as user, type, settings byte[]  or some such?

We may also want to store the version so that it is user, type, 
settingsFmtVersion, settings.

We might also want to consider storing as Protobufs, which are more 
supporting changes.




---

[GitHub] metron pull request #938: METRON-1457: Move ASF links to main page in the Me...

[anandsubbu]

GitHub user anandsubbu opened a pull request:

https://github.com/apache/metron/pull/938

METRON-1457: Move ASF links to main page in the Metron website

## Contributor Comments
Apache Whimsy checks for the site requisites in the main index.html and not 
inside sub-levels. This PR moves the content of the ASF and links to the main 
page.

Here is a screenshot of how the new metron page will look like:
https://imgur.com/3Y8ZLWL

**Testing Steps**
Follow the below steps to test the changes

* From `metron/site` execute
```
bundle install
bundle exec jekyll serve
```
* Navigate to http://localhost:4000 to review the site

Note that these steps are an excerpt from 
https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge. Please 
follow this page to have the pre-requisites set up.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-1457

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/938.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #938


commit dda05791a3667859be6159abc814a59af506ed09
Author: Anand Subramanian 
Date:   2018-02-16T14:27:55Z

Move ASF links to the main page




---

[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/853
  
The latest commit switches the persistence for storing user settings to 
HBase rather than a RDBMS as discussed on the dev list.  Instead of fields 
being stored in RDBMS columns, the user settings object is now serialized with 
Jackson and stored in HBase as a byte[].  This required several changes 
including:

- Added code to setup HBase in the MPack, following the update table 
pattern in the indexing MPack scripts as an example
- Added default table/cf values to REST properties
- Added a Config class to the REST app to setup an HBase client
- Added a Service class to the REST app that manages user settings in HBase
- Added tests for all new classes and updated existing tests
- Updated the REST README

I also refactored some areas to make things clearer and easier to 
understand:

- Renamed AlertProfile to UserSettings and moved the endpoints to the 
UserController.  The thinking is that we may not want this to be limited to 
just the Alerts UI.
- Refactored the REST MPack scripts to more closely align with other 
components in regards to setting up Kafka, HBase, etc.
- Removed the Optional fields from a couple model classes.  This was 
causing issues with Jackson and I don't believe it benefits us enough to have 
to deal with that complexity.

There are a couple of design issues to consider when reviewing this.  The 
implementation in the REST app is specific to this use case.  I considered 
trying to make it more generic for future HBase use cases but decided to keep 
it simple for now.   Instead of trying to predict what those use cases look 
like and choose a pattern that works, I decided to leave that to whoever 
implements a new use case in the future.  I also considered making this generic 
to all of Metron but again, decided to keep it simple.  Should these settings 
be limited to just REST/UIs?  Any thoughts on this?

This has been tested in full dev and all tests are passing.  In full dev, 
navigate to the UserController and use the various endpoints to save and 
retrieve settings for the currently logged in user.  You should also be able to 
log in as the admin user and see all user settings and delete an individual 
user's settings.  I also tested evolving the user settings model by adding new 
fields and it worked without issue.  Users with existing settings just return 
null for new fields. 


---

[GitHub] metron issue #936: METRON-1450:Added documentation for random access and bat...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/936
  
I'm specifically referring to 
[this](https://github.com/MohanDV/metron/blob/4cb99e7186c408159cb41b0e5e8761fdcbfe74d2/metron-platform/metron-indexing/README.md#indexing-architecture)
 which is no longer accurate.


---

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/579
  
@ctramnitz Just noticed this, but could you change the name of the PR to 
match the Jira?

i.e.
> METRON-941 native PaloAlto parser corrupts message when having a comma in 
the payload



---

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
after that I'll merge



---

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
@ctramnitz one thing, since this is a regression, technically, we need to 
update the release notes / upgrade guide.
Can you add a note to the Upgrading.md about the removal of the Syslog 
-ness?


---

Re: Apache Website Required Links

[Justin Leet]

Whimsy has run the site checks since then, and unfortunately we're still
showing that the links are missing.  Like I said, I think it's because the
links aren't actually on the homepage itself.

On Thu, Feb 15, 2018 at 5:02 PM, Justin Leet  wrote:

> Hey, I took a look at this, and am unsure of the implementation.
> Specifically, the links are in a subpage, not on the homepage itself.  I'm
> unsure if that's allowable or not, but I'm pretty sure whimsy won't pick it
> up unless they do something more clever than the script I glanced at.
>
> I checked out whimsy and ran tools/site-scan.rb and got this result (which
> shows nulls for the links):
>
>> {16:52}~/Documents/workspace/whimsy/tools:master ✓ ➭ ruby site-scan.rb
>>> http://metron.apache.org/
>>
>> Metron http://metron.apache.org/ recent
>>
>> {
>>
>>   "Metron": {
>>
>> "display_name": "Metron",
>>
>> "uri": "http://metron.apache.org/;,
>>
>> "events": null,
>>
>> "foundation": null,
>>
>> "license": "http://www.apache.org/licenses/LICENSE-2.0;,
>>
>> "sponsorship": null,
>>
>> "security": null,
>>
>> "trademarks": "Apache Metron and its logo are trademarks of the The
>>> Apache Software Foundation.",
>>
>> "copyright": "Copyright © 2018, The Apache Software Foundation.",
>>
>> "image": null,
>>
>> "copyparent": true
>>
>>   }
>>
>> }
>>
>>
> If you hit the subpage itself, things look better
>
>> {16:53}~/Documents/workspace/whimsy/tools:master ✓ ➭ ruby site-scan.rb
>>> http://metron.apache.org/asf
>>
>> Metron http://metron.apache.org/asf/ missing
>>
>> {
>>
>>   "Metron": {
>>
>> "display_name": "Metron",
>>
>> "uri": "http://metron.apache.org/asf/;,
>>
>> "events": "https://www.apache.org/events/current-event;,
>>
>> "foundation": "APACHE",
>>
>> "license": "http://www.apache.org/licenses/LICENSE-2.0;,
>>
>> "sponsorship": "https://www.apache.org/foundation/sponsorship.html;,
>>
>> "security": "https://www.apache.org/security/;,
>>
>> "trademarks": "Apache Metron and its logo are trademarks of the The
>>> Apache Software Foundation.",
>>
>> "copyright": "Copyright © 2018, The Apache Software Foundation.",
>>
>> "image": null,
>>
>> "thanks": "https://www.apache.org/foundation/thanks.html;,
>>
>> "copyparent": true
>>
>>   }
>>
>> }
>>
>>
>
> On Thu, Feb 15, 2018 at 3:38 PM, Casey Stella  wrote:
>
>> Just reporting back that Anand's PR METRON-1386 (
>> https://github.com/apache/metron/pull/935) has been merged into master
>> and
>> the asf-site branch.
>> Kudos to Anand!
>>
>> Casey
>>
>> On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
>> asubraman...@hortonworks.com> wrote:
>>
>> > I can take a shot at this if there are no other takers.
>> >
>> > Regards,
>> > Anand
>> >
>> > On 2/5/18, 8:59 PM, "Justin Leet"  wrote:
>> >
>> > I'd created a Jira awhile ago, but it deserves a callout to the
>> > community.
>> > Especially if someone wants to grab it, it's probably something
>> pretty
>> > easy
>> > (and valuable!) to do.
>> >
>> > There's a set of required links on Apache web pages, which can be
>> seen
>> > at Website
>> > Navigation Links Policy
>> > 
>> >
>> > Reporting is at Site Check For Project - Metron
>> > 
>> >
>> > This ticket is available at:
>> > METRON-1386 
>> >
>> >
>> >
>>
>
>

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/579
  
I'm +1, this is definitely a major improvement and I agree, getting it 
merged would be great. @simonellistonball Any comment as @ottobackwards asked, 
or are we good to pull this in?


---

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/579
  
I don't believe I have any further things to add. I'm +1 and keen to see 
this get in.


---

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

[ctramnitz]

Github user ctramnitz commented on the issue:

https://github.com/apache/metron/pull/579
  
Any further comments? The old parser was plain broken. So even if there is 
room for improvement here, I would really like to see this merged.

Thanks!


---

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

[iraghumitra]

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/857
  
Got busy with too many things happening around me, will yield this a 
conclusion next week.


---