[jira] [Commented] (METRON-437) The Profile Definition's 'inputTopic' field is Extraneous

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511619#comment-15511619
 ] 

ASF GitHub Bot commented on METRON-437:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/264
  
+1 by inspection


> The Profile Definition's 'inputTopic' field is Extraneous
> -
>
> Key: METRON-437
> URL: https://issues.apache.org/jira/browse/METRON-437
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The 'inputTopic' field that is part of the Profiler definition is not used.  
> The input topic is defined by the topology configuration contained within 
> profiler.properties.  The 'inputTopic' field should be removed from the 
> Profiler definition as it is misleading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-438) Back the Stellar REPL with a readline implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511507#comment-15511507
 ] 

ASF GitHub Bot commented on METRON-438:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/265


> Back the Stellar REPL with a readline implementation
> 
>
> Key: METRON-438
> URL: https://issues.apache.org/jira/browse/METRON-438
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>   Original Estimate: 2.5h
>  Remaining Estimate: 2.5h
>
> The REPL as it currently stands is great, but it could be closer in-line with 
> the more popular REPL's by being backed by a readline implementation.
> Specifically, we should:
> * add variable assignment
> * add the ability to do reverse searches in the history
> * add the ability to use the arrow keys to recover previous inputs
> * add the ability to adjust input via ~/.inputrc
> * add autocomplete for function names and variable names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-426) Stellar does not support scientific notation as a literal

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510777#comment-15510777
 ] 

ASF GitHub Bot commented on METRON-426:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/257


> Stellar does not support scientific notation as a literal
> -
>
> Key: METRON-426
> URL: https://issues.apache.org/jira/browse/METRON-426
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Stellar does not support scientific notation for expressing literal doubles.  
> It should.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-426) Stellar does not support scientific notation as a literal

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510562#comment-15510562
 ] 

ASF GitHub Bot commented on METRON-426:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/257
  
@justinleet I think everything you mentioned has been addressed.  Are you 
good with this PR?


> Stellar does not support scientific notation as a literal
> -
>
> Key: METRON-426
> URL: https://issues.apache.org/jira/browse/METRON-426
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Stellar does not support scientific notation for expressing literal doubles.  
> It should.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-448) Upgrading via Ansible deployment does not add topology.classpath

[Michael Miklavcic (JIRA)]

Michael Miklavcic created METRON-448:


 Summary: Upgrading via Ansible deployment does not add 
topology.classpath
 Key: METRON-448
 URL: https://issues.apache.org/jira/browse/METRON-448
 Project: Metron
  Issue Type: Bug
Reporter: Michael Miklavcic


When using Ansible to deploy the latest Metron bits to an existing 
installation, storm-site is not being updated with the new 0.2.1BETA parameter 
topology.classpath. Topologies are unable to find the client configs as a 
result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-434) JSON Parser

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510332#comment-15510332
 ] 

ASF GitHub Bot commented on METRON-434:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/261
  
Casey, would you feel about adding in recursion, such that nested maps will 
be unfolded?
I have that working.


> JSON Parser
> ---
>
> Key: METRON-434
> URL: https://issues.apache.org/jira/browse/METRON-434
> Project: Metron
>  Issue Type: New Feature
>Affects Versions: 0.2.1BETA
>Reporter: Casey Stella
>Assignee: Casey Stella
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Create a parser which will take a JSON map in.  
> Inner maps should be handled pluggably by:
> * doing nothing
> * unfolding maps (e.g. `{ "foo" : { "bar" : 7 } }` is transformed into 
> `{"foo.bar" : 7 }`
> * Throwing an exception
> * Dropping the inner map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-447) Monit fails to reload when upgrading from 0.2.0BETA to master

[Jon Zeolla (JIRA)]

Jon Zeolla created METRON-447:
-

 Summary: Monit fails to reload when upgrading from 0.2.0BETA to 
master
 Key: METRON-447
 URL: https://issues.apache.org/jira/browse/METRON-447
 Project: Metron
  Issue Type: Improvement
Reporter: Jon Zeolla


As of commit 
[5ffcef8](https://github.com/apache/incubator-metron/commit/5ffcef8d4a9ec0788dca094f54b07d99c44c6bef),
 `/etc/monit.d/enrichment-elasticsearch.monit` was renamed to 
`/etc/monit.d/indexing-elasticsearch.monit`, however the old file isn't removed 
via ansible, which causes the below error during an upgrade:

`Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service name 
conflict, enrichment already defined 
'/usr/local/monit/status_enrichment_topology.sh'`

Not sure of other downstream impacts, but the remedy I suggest is to modify 
`/etc/monit.conf:23` to be more specific (`include /etc/monit.d/*.monit` and 
rename `/etc/monit.d/logging`?) and rename 
`/etc/monit.d/enrichment-elasticsearch.monit` to append a backup string as a 
part of the run.  

Alternatively, we could just add a `state: absent` to 
`incubator-metron/metron-deployment/roles/monit/tasks/monit-definitions.yml` to 
delete the old file, if it exists.  

Looking for feedback on the approach, but happy to fix this myself and submit a 
PR.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-438) Back the Stellar REPL with a readline implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510290#comment-15510290
 ] 

ASF GitHub Bot commented on METRON-438:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/265
  
@cestella Great.  Remember auto-complete for doc strings would need to 
continue auto-completing with a trailing space.


> Back the Stellar REPL with a readline implementation
> 
>
> Key: METRON-438
> URL: https://issues.apache.org/jira/browse/METRON-438
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>   Original Estimate: 2.5h
>  Remaining Estimate: 2.5h
>
> The REPL as it currently stands is great, but it could be closer in-line with 
> the more popular REPL's by being backed by a readline implementation.
> Specifically, we should:
> * add variable assignment
> * add the ability to do reverse searches in the history
> * add the ability to use the arrow keys to recover previous inputs
> * add the ability to adjust input via ~/.inputrc
> * add autocomplete for function names and variable names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-438) Back the Stellar REPL with a readline implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510306#comment-15510306
 ] 

ASF GitHub Bot commented on METRON-438:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/265
  
+1 Works great.  I'm a go as-is, pending CI build.  Anything else you can 
do with auto-complete earns a gold star.


> Back the Stellar REPL with a readline implementation
> 
>
> Key: METRON-438
> URL: https://issues.apache.org/jira/browse/METRON-438
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>   Original Estimate: 2.5h
>  Remaining Estimate: 2.5h
>
> The REPL as it currently stands is great, but it could be closer in-line with 
> the more popular REPL's by being backed by a readline implementation.
> Specifically, we should:
> * add variable assignment
> * add the ability to do reverse searches in the history
> * add the ability to use the arrow keys to recover previous inputs
> * add the ability to adjust input via ~/.inputrc
> * add autocomplete for function names and variable names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-438) Back the Stellar REPL with a readline implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510264#comment-15510264
 ] 

ASF GitHub Bot commented on METRON-438:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/265
  
@nickwallen I think I can do something about that, yep.  Let me noodle on 
just the right way to do it.


> Back the Stellar REPL with a readline implementation
> 
>
> Key: METRON-438
> URL: https://issues.apache.org/jira/browse/METRON-438
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>   Original Estimate: 2.5h
>  Remaining Estimate: 2.5h
>
> The REPL as it currently stands is great, but it could be closer in-line with 
> the more popular REPL's by being backed by a readline implementation.
> Specifically, we should:
> * add variable assignment
> * add the ability to do reverse searches in the history
> * add the ability to use the arrow keys to recover previous inputs
> * add the ability to adjust input via ~/.inputrc
> * add autocomplete for function names and variable names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-438) Back the Stellar REPL with a readline implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510250#comment-15510250
 ] 

ASF GitHub Bot commented on METRON-438:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/265
  
Love it.  

One other thing I was noticing.  Not a deal breaker, but want to get your 
feedback on it.  If I auto-complete a function, it adds a space after the 
function name.  If I type "WEEK_OF_YE" then tab, it auto-completes to 
"WEEK_OF_YEAR " with a trailing space.  

If I then immediately hit ENTER, Stellar executes it, interprets it as a 
variable name, sees no value for the variable and does nothing.  Would it be 
possible to auto-complete a function with a trailing '(' so that I am forced to 
provide arguments?  Or maybe there is a better way to handle this situation.


> Back the Stellar REPL with a readline implementation
> 
>
> Key: METRON-438
> URL: https://issues.apache.org/jira/browse/METRON-438
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>   Original Estimate: 2.5h
>  Remaining Estimate: 2.5h
>
> The REPL as it currently stands is great, but it could be closer in-line with 
> the more popular REPL's by being backed by a readline implementation.
> Specifically, we should:
> * add variable assignment
> * add the ability to do reverse searches in the history
> * add the ability to use the arrow keys to recover previous inputs
> * add the ability to adjust input via ~/.inputrc
> * add autocomplete for function names and variable names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-444) Empty file gets created when fixed/stellar pcap query fails

[Michael Miklavcic (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510203#comment-15510203
 ] 

Michael Miklavcic commented on METRON-444:
--

Can you check if https://issues.apache.org/jira/browse/METRON-257 is in the 
code you're testing? Changes made for pagination may address this.

> Empty file gets created when fixed/stellar pcap query fails
> ---
>
> Key: METRON-444
> URL: https://issues.apache.org/jira/browse/METRON-444
> Project: Metron
>  Issue Type: Bug
>Reporter: Neha Sinha
>
> Empty file gets dumped/created even when the stellar/fixed query fails with 
> exceptions and errors.
> The expected behaviour is that the file should be generated only when the 
> query passes without any error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-421) Make Stellar Profiler Client API Accessible in Parser and Enrichment Topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510113#comment-15510113
 ] 

ASF GitHub Bot commented on METRON-421:
---

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/268

METRON-421 Make Stellar Profiler Client API Accessible in Parser and 
Enrichment Topologies

[METRON-421](https://issues.apache.org/jira/browse/METRON-421)

These changes were required to make the Stellar Profiler Client API 
accessible from the Parser and Enrichment topologies.

* Pushing the metron-profiler-client archive to the Metron host to make the 
library accessible to the Stellar Shell.
* Added the metron-profiler-client as a dependency to the Parser and 
Enrichment topologies so that the Uber jar contains the Profiler Client API.
* Fixed an issue with potentially unsafe String to Bytes conversion that 
may cause issues on different platforms.
* Added a README for the metron-profiler-client to describe its usage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-421

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/268.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #268


commit 1ef0bf6670b3054eb0561061dc3f05fc56fa3123
Author: Nick Allen 
Date:   2016-09-19T21:23:44Z

METRON-421 Added profiler-client dependency to Enrichment and Parser 
topology so that Profile API can be used

commit 1a902c839007a8ddfcc26bf608a91c5f0d921cea
Author: Nick Allen 
Date:   2016-09-19T21:25:28Z

METRON-421 Need to push profiler-client archive to the deployed host

commit 11416ce40fad361fd8db52635f0952fd52fe25f1
Author: Nick Allen 
Date:   2016-09-20T21:47:34Z

METRON-421 Row key builder using unsafe string to bytes conversion

commit 8c46cd9be14c324f03f1b08f464ef727a52d66cb
Author: Nick Allen 
Date:   2016-09-21T14:22:54Z

METRON-421 Added README for the profiler client




> Make Stellar Profiler Client API Accessible in Parser and Enrichment 
> Topologies
> ---
>
> Key: METRON-421
> URL: https://issues.apache.org/jira/browse/METRON-421
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> A user should be able to use the Stellar Profiler Client API from the Parser 
> and Enrichment topologies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-445) Fix typos in metron-deployment roles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510089#comment-15510089
 ] 

ASF GitHub Bot commented on METRON-445:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/267
  
+1 from me


> Fix typos in metron-deployment roles
> 
>
> Key: METRON-445
> URL: https://issues.apache.org/jira/browse/METRON-445
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Priority: Trivial
>  Labels: documentation, easyfix
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The following two files currently say "Get Default mysql passowrd" in name 
> fields.  
> metron-deployment/roles/metron_streaming/tasks/topologies.yml
> metron-deployment/roles/mysql_client/tasks/main.yml
> Need to s/passowrd/password/.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-445) Fix typos in metron-deployment roles

[Jon Zeolla (JIRA)]

Jon Zeolla created METRON-445:
-

 Summary: Fix typos in metron-deployment roles
 Key: METRON-445
 URL: https://issues.apache.org/jira/browse/METRON-445
 Project: Metron
  Issue Type: Improvement
Reporter: Jon Zeolla
Priority: Trivial


The following two files currently say "Get Default mysql passowrd" in name 
fields.  

metron-deployment/roles/metron_streaming/tasks/topologies.yml
metron-deployment/roles/mysql_client/tasks/main.yml

Need to s/passowrd/password/.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-427) Create Ambari Management Pack for Metron Installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509868#comment-15509868
 ] 

ASF GitHub Bot commented on METRON-427:
---

GitHub user justinleet opened a pull request:

https://github.com/apache/incubator-metron/pull/266

METRON-427 Create Ambari Management Pack for Metron Installation

This is an initial cut at an MVP for the Ambari Management Pack.  Most of 
the ground work was completed in other Jiras (detailed in METRON-427 
description).  This takes that work, adds the enrichment portion similiar to 
indexing, elasticsearch, etc., and bundles it in an mpack.

This includes end to end from creating appropriate Kafka topics, preloading 
a MySQL database with GeoIP data for enrichment, and making it available to an 
Elasticsearch and Kibana instance. ES is optional, and can be redirected to an 
outside instance.

The directory structuring changed to reflect the mpack MVP, and accordingly 
a lot of files got moved/renamed.  For most of the ground work, there are 
various fixes and tweaks for consistency to each other and to fix bugs found 
during testing.

Detailed instructions are found in the metron-deployment README.md in the 
'Ambari Management Pack' section.  Essentially these steps are:
- Build the RPMs per the previously existing instructions (make sure to run 
the main build first).
- Build the mpack using the metron-deployment POM. From metron-deployment 
just run `mvn clean package -DskipTests`.  This will produce an mpack tar.gz, 
which is what will be used in the installation to the Ambari server.
- Place the RPMs in /localrepo on the nodes.  We don't have remote RPMs 
yet, so this is a necessary evil that will go away.
- Place the mpack file on the ambari-server node.
- Install the mpack with `ambari-server` install-mpack 
--mpack= --verbose`
- Update Storm's configuration in Ambari for topology.classpath to 
/etc/hbase/conf:/etc/hadoop/conf.  Ideally this will be automated after this 
MVP.
- Run through the normal Ambari install process.  There are some caveats 
about this (colocation of Metron services isn't enforced, and Storm + Kafka + 
HBase client need to be available).  More details are in the README, but just 
make sure Storm + Kafka + HBase client are available and that you aren't 
installing to a node with MySQL already installed.  The MySQL issue is most 
likely on a single node install
- Use the action to install Elasticsearch templates where you can start / 
stop services.

This was tested on an AWS cluster, and on a set of Docker instances where a 
lot of components (e.g. MapReduce) disabled to avoid memory issues.

Steps used on Docker were (the server and agent images already have most of 
the unnecessary components disabled.  Make sure to change jleet to your user 
home:
`docker-machine rm default
docker-machine create --driver virtualbox --virtualbox-disk-size "51200" 
--virtualbox-memory "8192" default

docker-machine start default
eval $(docker-machine env default)
git clone https://github.com/sequenceiq/docker-ambari
cd docker-ambari
. ambari-functions
export AMBARI_SERVER_IMAGE=dlyle65535/ambari-server:2.4.0.1-jdk8
export AMBARI_AGENT_IMAGE=dlyle65535/ambari-agent:2.4.0.1-jdk8

export DOCKER_OPTS="-v /Users/jleet:/home/host -v 
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch:/localrepo
 -v 
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/:/mpack"

(mac only) sudo route add -net 172.17.0.0/16 $(docker-machine ip default)

amb-start-cluster 5
amb-shell
blueprint add --url 
https://raw.githubusercontent.com/dlyle65535/ambari-rest-client/docker-metron-test/src/main/resources/blueprints/docker-metron
cluster build --blueprint docker-metron
cluster assign --host amb1.service.consul --hostGroup master_1
cluster assign --host amb2.service.consul --hostGroup master_2
cluster assign --host amb3.service.consul --hostGroup master_3
cluster assign --host amb4.service.consul --hostGroup master_4
cluster create



docker exec -it amb-server bash
ambari-server install-mpack 
--mpack=/mpack/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
ambari-server restart
http://amb-server.service.consul:8080 - install Metron service on 
amb4.service.consul
es_url: can be invalid for test purpose otherwise, use authentic 
   One can docker it - docker run -d -p 9200:9200 -p 9300:9300 
elasticsearch -Des.cluster.name=metron
Storm UI Server: amb1.service.consul:8744
`




Limitations of the MVP (also detailed in README)
- MySQL install should be optional (and allow for using an