[GitHub] metron pull request #928: METRON-1444: Add Ubuntu Repositories for Elasticse...

[mmiklavc]

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/928

METRON-1444: Add Ubuntu Repositories for Elasticsearch to the Mpack

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1444

As of Ambari 2.6.0, the Ubuntu repoinfo.xml translation into an apt-get 
sources file has been fixed. See 
https://github.com/apache/ambari/commit/f8b29df9685b443d4a5c06c6e1725e4428c95b49#diff-6f26c26ed59462200d018c5e1e71e773

This PR updates Ambari from 2.5.2 to 2.6.1.0 in order to get the fix for 
Ubuntu repoinfo.xml processing.

**Test**

Spin up the Ubuntu full dev env. ES and Kibana should be installed without 
issue. Spot check that you see data populating the dashboard when you open 
Kibana.

Also check that centos full dev still works as expected with the Ambari 
upgrade.

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron ubuntu-deb-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/928.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #928


commit 59df0fa919fc220cf81a829de30354886f91537f
Author: Michael Miklavcic 
Date:   2018-02-07T07:23:43Z

Upgrade Ambari and update Elasticsearch, Kibana, and Curator repoinfo.xml

commit 76ad7dbbb1e897929ceb13f9cca0571b7d0b4ad7
Author: Michael Miklavcic 
Date:   2018-02-07T07:28:59Z

Roll back METRON-1443 now that we have a proper fix




---

Re: [DISCUSS] Time to remove github updates from dev?

[Andre]

All,

Turns out the process is simpler:

A PMC member must create the lists using the self-management potal:


selfserve.apache.org


Once this is done someone can update the INFRA-15988 ticket and the folks
will execute the changes.



On Wed, Jan 31, 2018 at 12:15 AM, Otto Fowler 
wrote:

> We could also just skip ‘b’ and go directly to ‘c’ like apache-commons and
> have
> commits@ issues@.
>
>
>
>
> On January 30, 2018 at 08:03:37, Andre (andre-li...@fucs.org) wrote:
>
> James,
>
> Give nobody opposed, I would suggest one of the PMCs contact the INFRA to
> get this actioned.
>
> They would need to assist with:
>
> 1. Creation of the new "issues" list
> 2. redirect both GitHub and JIRA integrations to the new list
>
> Cheers
>
> On Sat, Jan 27, 2018 at 9:40 AM, James Sirota  wrote:
>
> > Should we file an infra ticket on this?
> >
> > 19.01.2018, 13:56, "zeo...@gmail.com" :
> > > I would give that +1 as well.
> > >
> > > Jon
> > >
> > > On Fri, Jan 19, 2018 at 3:32 PM Casey Stella 
> wrote:
> > >
> > >> I could get behind that.
> > >>
> > >> On Fri, Jan 19, 2018 at 3:31 PM, Andre  wrote:
> > >>
> > >> > Folks,
> > >> >
> > >> > May I suggest Metron follows the NiFi mailing list strategy (we got
> > >> > inspired by another project but I don't recall the name) and remove
> > the
> > >> > github comments from the dev list?
> > >> >
> > >> > Within NiFi we have both the dev and the issues lists. dev is for
> > humans,
> > >> > issues is for JIRA and github commits.[1]
> > >> >
> > >> > This allows the list thread list to be cleaner and is particularly
> > >> helpful
> > >> > for those reading the list from a list aggregation service.
> > >> >
> > >> > Cheers
> > >> >
> > >> >
> > >> > [1] https://lists.apache.org/list.html?iss...@nifi.apache.org
> > >> >
> > >
> > > --
> > >
> > > Jon
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
> >
>
>

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/622


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/922
  
@simonellistonball Yes, we should.  I added the relevant context and 
grouping for each of the schemas that we ship by default.  Bro has more context 
as there were more comments in the ES schema.  I brought over whatever comments 
were relevant and grouped appropriately and as best I could in the others.

Let me know if I missed anything.



---

[GitHub] metron pull request #926: METRON-1446: Fix openjdk issue with Ubuntu

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/926


---

[GitHub] metron pull request #927: METRON-1447 Heap Size Not Set Correctly by MPack f...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/927

METRON-1447 Heap Size Not Set Correctly by MPack for ES 5.x

The preferred way in which the heap size and other JVM options are set 
changed between ES 2.x and ES 5.x.  The project upgraded to ES 5.x as part of 
#840 , but the way the heap size is set by the Mpack was not changed.

This resulted in the heap size for Elasticsearch to be set incorrectly.  
This also allows Elasticsearch to use up to 2G of heap when launched in the 
development environments, which is too much for a constrained single VM.

## Changes

The user can set the heap size by populating the "heap_size" field under 
"Advanced elastic-jvm-options" in Ambari.  

Elasticsearch also exposes a large number of other settings in this file.  
The entire content of the file was exposed in Ambari to allow users to also 
alter any other JVM options as needed.

![screen shot 2018-02-06 at 11 27 34 
am](https://user-images.githubusercontent.com/2475409/35870877-c3d310ce-0b30-11e8-9b07-e77ae3b7074c.png)

## Testing

1. Launch a development environment; either Ubuntu or CentOS.  
* Ensure that telemetry reaches the Alerts UI.
* Run the Metron Service Check
* Run the Elasticsearch Service Check

1. Login to the node and ensure that only a single `-Xms` and `-Xmx` option 
was passed to the JVM when launching Elasticsearch.  Ensure these are both set 
to the default heap size of 512mb.

```
root@node1:/etc/elasticsearch# ps -ef | grep Elastic
root  1084 31038  0 16:09 pts/400:00:00 grep --color=auto 
Elastic
elastic+ 30048 1 23 16:08 ?00:00:17 
/usr/jdk64/jdk1.8.0_112/bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true 
-Dfile.encoding=UTF-8 -Djna.nosys=true 
-Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true 
-Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 
-Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true 
-Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError 
-Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/* 
org.elasticsearch.bootstrap.Elasticsearch -d -p 
/var/run/elasticsearch/elasticsearch.pid 
-Edefault.path.logs=/var/log/elasticsearch 
-Edefault.path.data=/var/lib/elasticsearch/ 
-Edefault.path.conf=/etc/elasticsearch/
```

1. Alter the JVM options template, save the settings, restart 
Elasticsearch, and ensure that the changes are reflected in the 
`/etc/elasticsearch/jvm.options` file.

## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1447

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/927.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #927


commit 4224bfaa44ac0ad1512e7b4e13f77769c8708f32
Author: Nick Allen 
Date:   2018-02-06T16:14:15Z

METRON-1447 Heap Size Not Set Correctly by MPack for ES 5.x




---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/922
  
Should we tidy up the ordering of the schema files for better legibility 
(uniquekey next to the field, dynamic catch alls in a consistent location, some 
semantic ordering of the key elements)?


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/922
  
Looks good to me.  +1


---

[GitHub] metron issue #926: METRON-1446: Fix openjdk issue with Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/926
  
+1 LGTM.  I have not been able to replicate this problem, which I am really 
confused about.  Although this seems harmless enough of a change.


---

[GitHub] metron issue #926: METRON-1446: Fix openjdk issue with Ubuntu

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/926
  
2.0.2


---

[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/922
  
Ugh, sorry, missed the guid field.  *now* it's ready to test with guid 
field set up.


---

[GitHub] metron issue #926: METRON-1446: Fix openjdk issue with Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/926
  
Was this tested against Vagrant 2.0.2 or 1.8.1 ?


---