[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975219#comment-15975219 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 We should ask @dlyle65535 whether he completely obsoleted this one. In the meantime, METRON-634 PR#532 is definitely needed. > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.3.1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974587#comment-15974587 ] ASF GitHub Bot commented on METRON-608: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/408 @mattf-horton Still needed? > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.3.1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839424#comment-15839424 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 @dlyle65535 , please see https://github.com/apache/incubator-metron/pull/425 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838721#comment-15838721 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 @dlyle65535 , I'll be happy to share my wip. It will take me a few hours to brush it off, then I'll post it as a PR referring to [METRON-609](https://issues.apache.org/jira/browse/METRON-609). > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837895#comment-15837895 ] ASF GitHub Bot commented on METRON-608: --- Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/408 @mattf-horton - I need an MPack that works for 1-N nodes in order to complete [METRON-671](https://issues.apache.org/jira/browse/METRON-671). I suspect I'm working through stuff you've already fixed. Would it make sense to push out a PR with the unified branch and let me test it? Then I could use finish METRON-671 on top of your changes. > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837360#comment-15837360 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 @JonZeolla , I removed the obsolete comments about MySQL installation in Mpack. Thanks for the suggestion. > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837347#comment-15837347 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 Hi @dlyle65535 , from a design perspective, I totally agree that we should have a single MPack that works on any size cluster, and I hope to replace both the 4(+)-node and single-node versions in the future with a more general solution. I even have a prototype of such. My reasons for proceeding with the split-off version at this time were entirely pragmatic, due to time pressure: - I really needed a single-node that works with Ambari, for testing other PRs. - I spent several days trying to make Elasticsearch work on a single node installation, while retaining the generality of the multi-node case, but couldn't make it work. This chewed up a lot of time. - I put together a unified Mpack that used two different template files for Elasticsearch, depending whether it was for a single-node or multi-node install. That worked just fine, on the surface, but was clearly going to require in-depth testing of 1, 2, 3, and 4 node cases. I do not currently have time to thoroughly test all the special cases. - Doing it this way I only had to test the single-node case, which was fairly fast. - Having made one that works, I'd like to make it available to others. - Now, if it is not accepted into the code, the high rate of change in metron-deployment will soon obsolete it. That's pretty much it. It's a throw-away, but not until we do the work of actually producing the generalized solution and testing it thoroughly. Which I evaluate as lower priority than a number of other more urgent needs. Thanks. > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830027#comment-15830027 ] ASF GitHub Bot commented on METRON-608: --- Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/408 This is great stuff. Can you talk a bit about why you split it off instead of just updated the existing MPack to support 1+ nodes? I'd prefer not to maintain two if we don't have to. > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830008#comment-15830008 ] ASF GitHub Bot commented on METRON-608: --- Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/408 Is "MySQL should not be installed on a node already running a MySQL instance" still accurate wrt Ambari MPack? I thought this was fixed. I see it noted as the 2nd bullet point [here](https://github.com/mattf-horton/incubator-metron/blob/b5b3a348527c03bc778a5fc5dd0d94ec3a00470b/metron-deployment/README.md#current-limitations). Would we be able to add that tweak in to this PR since it's so minor, and it's [semi-]related to the changes made here? I believe the actual fix was a part of [METRON-410](https://issues.apache.org/jira/browse/METRON-410) (#317) > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798985#comment-15798985 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 Oops, now the 2017 fix (METRON-647) is committed. Kicking Travis again. > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798986#comment-15798986 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton closed the pull request at: https://github.com/apache/incubator-metron/pull/408 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798987#comment-15798987 ] ASF GitHub Bot commented on METRON-608: --- GitHub user mattf-horton reopened a pull request: https://github.com/apache/incubator-metron/pull/408 METRON-608 Mpack to install a single-node test cluster This "metron-mpack-singlenode" is very similar to "metron-mpack", and can be usefully compared with it for code review, altho it also includes many bug fixes proposed in METRON-634. As it says in the Jira, this is a short-term fix by providing a completely separate Mpack just for the single-node scenario, until we can work the bugs out of an mpack that works on all sizes of cluster. This was tested on a Centos7 VM with 16GB of RAM, and all prerequisites installed, including Python 2.7.11. The usual install process was followed, with Ambari 2.4.2 and HDP stack 2.5.3. After successful startup, a few thousand 'bro' packets were injected, and were seen to correctly propagate through parser, enricher, and indexer topologies, and finally into elasticsearch. An outstanding issue is that the configuration validator in the mpack doesn't work. It fails with "unknown error" and must be ignored. The multi-node mpack already had this problem, and I have run out of time to try to fix it here. Consequently, my enhancement of trying to automatically flag the user if less than five Storm Supervisor Slots are allocated (commit 9ad8706), is untested. I would like to open a new Jira for this issue, since it was pre-existing before these additions. Thanks, --Matt You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf-horton/incubator-metron METRON-608 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 05571b501222b91457bd2f1b2ccc096af015f4cb Author: mattf-horton Date: 2016-12-08T05:31:40Z METRON-608 Mpack to install a single-node test cluster commit f2245900cd94389bc1c004c60f1606a49ca6ea8c Author: mattf-horton Date: 2016-12-10T01:01:41Z Add quicklinks to Elasticsearch Service page in Ambari, for health and indexes. commit 542a6f01376a656b539040992e9fa727efd8a047 Author: mattf-horton Date: 2016-12-10T01:02:40Z METRON-608 fix several bugs, especially in METRON use of 'es_url' commit fb646fdda345107b0debd8395743630cc5e0 Author: mattf-horton Date: 2016-12-10T09:05:43Z update version to 0.3.0 commit 8e30344e0b50fdff0cf8a1a66e7f41868a384171 Author: mattf-horton Date: 2016-12-10T09:09:30Z Merge/rebase updates from 0.3.0 commit 9ad8706e5209ca9506710ca6498ac387e667ddf0 Author: mattf-horton Date: 2016-12-13T02:03:07Z METRON-608 add service_advisor validation for number of Storm slots, enhance README, and a couple bug fixes commit d4c97e6de4838265b72139727a5fd1082388117b Author: mattf-horton Date: 2016-12-13T05:48:22Z tweak pom.xml file commit 95d83702ef844bc94c2ce54bccaebb7b7cb0513a Author: mattf-horton Date: 2016-12-20T00:54:07Z METRON-608 multiple small bug fixes commit b5b3a348527c03bc778a5fc5dd0d94ec3a00470b Author: mattf-horton Date: 2016-12-20T01:39:37Z rebase to Dec 19, 2016 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798756#comment-15798756 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton closed the pull request at: https://github.com/apache/incubator-metron/pull/408 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798757#comment-15798757 ] ASF GitHub Bot commented on METRON-608: --- GitHub user mattf-horton reopened a pull request: https://github.com/apache/incubator-metron/pull/408 METRON-608 Mpack to install a single-node test cluster This "metron-mpack-singlenode" is very similar to "metron-mpack", and can be usefully compared with it for code review, altho it also includes many bug fixes proposed in METRON-634. As it says in the Jira, this is a short-term fix by providing a completely separate Mpack just for the single-node scenario, until we can work the bugs out of an mpack that works on all sizes of cluster. This was tested on a Centos7 VM with 16GB of RAM, and all prerequisites installed, including Python 2.7.11. The usual install process was followed, with Ambari 2.4.2 and HDP stack 2.5.3. After successful startup, a few thousand 'bro' packets were injected, and were seen to correctly propagate through parser, enricher, and indexer topologies, and finally into elasticsearch. An outstanding issue is that the configuration validator in the mpack doesn't work. It fails with "unknown error" and must be ignored. The multi-node mpack already had this problem, and I have run out of time to try to fix it here. Consequently, my enhancement of trying to automatically flag the user if less than five Storm Supervisor Slots are allocated (commit 9ad8706), is untested. I would like to open a new Jira for this issue, since it was pre-existing before these additions. Thanks, --Matt You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf-horton/incubator-metron METRON-608 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 05571b501222b91457bd2f1b2ccc096af015f4cb Author: mattf-horton Date: 2016-12-08T05:31:40Z METRON-608 Mpack to install a single-node test cluster commit f2245900cd94389bc1c004c60f1606a49ca6ea8c Author: mattf-horton Date: 2016-12-10T01:01:41Z Add quicklinks to Elasticsearch Service page in Ambari, for health and indexes. commit 542a6f01376a656b539040992e9fa727efd8a047 Author: mattf-horton Date: 2016-12-10T01:02:40Z METRON-608 fix several bugs, especially in METRON use of 'es_url' commit fb646fdda345107b0debd8395743630cc5e0 Author: mattf-horton Date: 2016-12-10T09:05:43Z update version to 0.3.0 commit 8e30344e0b50fdff0cf8a1a66e7f41868a384171 Author: mattf-horton Date: 2016-12-10T09:09:30Z Merge/rebase updates from 0.3.0 commit 9ad8706e5209ca9506710ca6498ac387e667ddf0 Author: mattf-horton Date: 2016-12-13T02:03:07Z METRON-608 add service_advisor validation for number of Storm slots, enhance README, and a couple bug fixes commit d4c97e6de4838265b72139727a5fd1082388117b Author: mattf-horton Date: 2016-12-13T05:48:22Z tweak pom.xml file commit 95d83702ef844bc94c2ce54bccaebb7b7cb0513a Author: mattf-horton Date: 2016-12-20T00:54:07Z METRON-608 multiple small bug fixes commit b5b3a348527c03bc778a5fc5dd0d94ec3a00470b Author: mattf-horton Date: 2016-12-20T01:39:37Z rebase to Dec 19, 2016 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798753#comment-15798753 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 kicking travis > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796243#comment-15796243 ] ASF GitHub Bot commented on METRON-608: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/408 Please note the Travis failures are not relevant. All nine are due to https://issues.apache.org/jira/browse/METRON-648 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794446#comment-15794446 ] ASF GitHub Bot commented on METRON-608: --- GitHub user mattf-horton opened a pull request: https://github.com/apache/incubator-metron/pull/408 METRON-608 Mpack to install a single-node test cluster This "metron-mpack-singlenode" is very similar to "metron-mpack", and can be usefully compared with it for code review, altho it also includes many bug fixes proposed in METRON-634. As it says in the Jira, this is a short-term fix by providing a completely separate Mpack just for the single-node scenario, until we can work the bugs out of an mpack that works on all sizes of cluster. This was tested on a Centos7 VM with 16GB of RAM, and all prerequisites installed, including Python 2.7.11. The usual install process was followed, with Ambari 2.4.2 and HDP stack 2.5.3. After successful startup, a few thousand 'bro' packets were injected, and were seen to correctly propagate through parser, enricher, and indexer topologies, and finally into elasticsearch. An outstanding issue is that the configuration validator in the mpack doesn't work. It fails with "unknown error" and must be ignored. The multi-node mpack already had this problem, and I have run out of time to try to fix it here. Consequently, my enhancement of trying to automatically flag the user if less than five Storm Supervisor Slots are allocated (commit 9ad8706), is untested. I would like to open a new Jira for this issue, since it was pre-existing before these additions. Thanks, --Matt You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf-horton/incubator-metron METRON-608 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 05571b501222b91457bd2f1b2ccc096af015f4cb Author: mattf-horton Date: 2016-12-08T05:31:40Z METRON-608 Mpack to install a single-node test cluster commit f2245900cd94389bc1c004c60f1606a49ca6ea8c Author: mattf-horton Date: 2016-12-10T01:01:41Z Add quicklinks to Elasticsearch Service page in Ambari, for health and indexes. commit 542a6f01376a656b539040992e9fa727efd8a047 Author: mattf-horton Date: 2016-12-10T01:02:40Z METRON-608 fix several bugs, especially in METRON use of 'es_url' commit fb646fdda345107b0debd8395743630cc5e0 Author: mattf-horton Date: 2016-12-10T09:05:43Z update version to 0.3.0 commit 8e30344e0b50fdff0cf8a1a66e7f41868a384171 Author: mattf-horton Date: 2016-12-10T09:09:30Z Merge/rebase updates from 0.3.0 commit 9ad8706e5209ca9506710ca6498ac387e667ddf0 Author: mattf-horton Date: 2016-12-13T02:03:07Z METRON-608 add service_advisor validation for number of Storm slots, enhance README, and a couple bug fixes commit d4c97e6de4838265b72139727a5fd1082388117b Author: mattf-horton Date: 2016-12-13T05:48:22Z tweak pom.xml file commit 95d83702ef844bc94c2ce54bccaebb7b7cb0513a Author: mattf-horton Date: 2016-12-20T00:54:07Z METRON-608 multiple small bug fixes commit b5b3a348527c03bc778a5fc5dd0d94ec3a00470b Author: mattf-horton Date: 2016-12-20T01:39:37Z rebase to Dec 19, 2016 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: Next + 1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
