[jira] [Commented] (METRON-634) Mpack bug fixes and improvements (not related to singlenode install)
[
https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974945#comment-15974945
]
ASF GitHub Bot commented on METRON-634:
---
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/532
+1, ran it up in full-dev and on EC2 (Centos7) and worked like a champ.
Thanks!
> Mpack bug fixes and improvements (not related to singlenode install)
>
>
> Key: METRON-634
> URL: https://issues.apache.org/jira/browse/METRON-634
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
> Environment: Centos7
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). This jira provides the work items for merging those changes into
> the mainline Mpack. Almost all items relate to Elasticsearch.
> Some changes impact the Ambari database, and should therefore be done with an
> Mpack version bump and database upgrade script. These are in the second
> block below.
> h2. Not impacting Ambari database:
> * ES pid_dir specification and usage:
> ** Currently pid_dir is multiply specified in elastic-env.xml and params.py.
> The config parameter should not be over-ridden in params.py.
> ** PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It needs
> to be added to the template in elastic-sysconfig, as it must be provided to
> ES at launch-time (else the default directory will be used).
> ** pid_file is specified in params.py, but is not used anywhere. (The ES
> internal launcher synthesizes it from PID_DIR, and this is appropriate.)
> ** status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated
> in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the
> ES launch, unless /bin/java is defined (which it isn't necessarily).
> * Also in the /etc/sysconfig/elasticsearch template in elastic-sysconfig.xml,
> the value of ES_JAVA_OPTS incorrectly spans 3 lines. The lines must be
> terminated with backslashes to effectively become a single line. The current
> inclusion of newlines in the long string value is acceptable (although
> unusual) in shellscript, but not in a systemd EnvironmentFile.
> /etc/sysconfig/elasticsearch must function as both.
> * Also in ES_JAVA_OPTS, the two instances of {{log_dir}} needs to be followed
> by a slash '/'
> * Recommend adding QuickLinks for Elasticsearch. This is a low-impact change
> that is very helpful to validate ES health.
> ** ES health status
> ** ES indexes list
> * In elastic.py, when directories are being pre-created and permissions set,
> the directory $CONF_DIR/scripts should also be pre-created. I intermittently
> hit permissions issues with this directory being created later by root, and
> not properly assigned to elastic_user.
> * In several places in elastic.py, "params.elastic_user" is incorrectly used
> when "params.user_group" should be used.
> * Undefined "format()" method is used in elastic.py, unnecessarily in
> File(format("/etc/sysconfig/elasticsearch")...
> * Undefined "format()" method is similarly used several times unnecessarily
> in elastic_master.py and elastic_slave.py
> * The comments and descriptions in elastic-site.xml have multiple suggested
> improvements.
> * The comment description in
> common-services/METRON/0.3.0/configuration/metron-env.xml for "es_hosts"
> should clarify that this should be a list of ES Master hosts, not any ES
> hosts.
> h2. Affects Ambari database, needs db upgrade script:
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name (to distinguish ES cluster from Stack
> cluster)
>
[jira] [Commented] (METRON-634) Mpack bug fixes and improvements (not related to singlenode install)
[
https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971573#comment-15971573
]
ASF GitHub Bot commented on METRON-634:
---
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/532
This is great, thanks for cleaning a lot of this up. When I get a chance,
I look forward to spinning this up.
> Mpack bug fixes and improvements (not related to singlenode install)
>
>
> Key: METRON-634
> URL: https://issues.apache.org/jira/browse/METRON-634
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
> Environment: Centos7
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). This jira provides the work items for merging those changes into
> the mainline Mpack. Almost all items relate to Elasticsearch.
> Some changes impact the Ambari database, and should therefore be done with an
> Mpack version bump and database upgrade script. These are in the second
> block below.
> h2. Not impacting Ambari database:
> * ES pid_dir specification and usage:
> ** Currently pid_dir is multiply specified in elastic-env.xml and params.py.
> The config parameter should not be over-ridden in params.py.
> ** PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It needs
> to be added to the template in elastic-sysconfig, as it must be provided to
> ES at launch-time (else the default directory will be used).
> ** pid_file is specified in params.py, but is not used anywhere. (The ES
> internal launcher synthesizes it from PID_DIR, and this is appropriate.)
> ** status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated
> in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the
> ES launch, unless /bin/java is defined (which it isn't necessarily).
> * Also in the /etc/sysconfig/elasticsearch template in elastic-sysconfig.xml,
> the value of ES_JAVA_OPTS incorrectly spans 3 lines. The lines must be
> terminated with backslashes to effectively become a single line. The current
> inclusion of newlines in the long string value is acceptable (although
> unusual) in shellscript, but not in a systemd EnvironmentFile.
> /etc/sysconfig/elasticsearch must function as both.
> * Also in ES_JAVA_OPTS, the two instances of {{log_dir}} needs to be followed
> by a slash '/'
> * Recommend adding QuickLinks for Elasticsearch. This is a low-impact change
> that is very helpful to validate ES health.
> ** ES health status
> ** ES indexes list
> * In elastic.py, when directories are being pre-created and permissions set,
> the directory $CONF_DIR/scripts should also be pre-created. I intermittently
> hit permissions issues with this directory being created later by root, and
> not properly assigned to elastic_user.
> * In several places in elastic.py, "params.elastic_user" is incorrectly used
> when "params.user_group" should be used.
> * Undefined "format()" method is used in elastic.py, unnecessarily in
> File(format("/etc/sysconfig/elasticsearch")...
> * Undefined "format()" method is similarly used several times unnecessarily
> in elastic_master.py and elastic_slave.py
> * The comments and descriptions in elastic-site.xml have multiple suggested
> improvements.
> * The comment description in
> common-services/METRON/0.3.0/configuration/metron-env.xml for "es_hosts"
> should clarify that this should be a list of ES Master hosts, not any ES
> hosts.
> h2. Affects Ambari database, needs db upgrade script:
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name (to distinguish ES cluster from Stack
> cluster)
>
[jira] [Commented] (METRON-634) Mpack bug fixes and improvements (not related to singlenode install)
[ https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971501#comment-15971501 ] ASF GitHub Bot commented on METRON-634: --- GitHub user mattf-horton opened a pull request: https://github.com/apache/incubator-metron/pull/532 METRON-634 Mpack bug fixes and improvements, not impacting Ambari database These changes fix the following significant bugs: * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml * Fixes necessary for Centos7: PID_DIR and JAVA_HOME added for systemd launch * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic.py * fix permissions on scripts directory * fix elastic_group usage (missed one in previous patch) * remove incorrect and spurious use of undefined format() method * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py * remove incorrect and spurious uses of undefined format() method * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml * Fix default value of "network_host" (ie network binding names), which breaks in some envs Also the following minor issue corrections and description/comment improvements are made: * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py * fix incorrect override of pid_dir and pid_file, which are correctly defined in elastic-env.xml and ELASTICSEARCH status_params.py * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/metainfo.xml * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/quicklinks/quicklinks.json * Add quicklinks to Elastic page in Ambari, for ES health status page and indexes list page; very useful for debugging the install. * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml * metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml * Non-substantive changes to descriptions or comments, meant to improve usability. * Also an ordering change and add newline at end of file. These fixes are only the first half of METRON-634. The second half cause changes to the Ambari database, and therefore require an update script. These will be done in a separate PR, when I have time to do it. The changes here are stand-alone improvements. ## Contributor Comments These issues were worked out over several weeks of working with Mpack on Centos7, and documented in METRON-634. @dlyle65535 adopted some into the Ansible work, but most remain to be done. They have been tested manually, and through a full run of integration test. If you are running Centos7, you will only notice the bug fixes as smoother startup of Elasticsearch. The description improvements make it easier for manual users of Mpack Ambari install. ## Pull Request Checklist In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - The issues were not noticed by most users anyway. Recommend that developers familiar with the guts of Mpack review the changes by inspection. - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [NA] Have you written or updated unit tests and
[jira] [Commented] (METRON-634) Mpack bug fixes and improvements (not related to singlenode install)
[
https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840778#comment-15840778
]
ASF GitHub Bot commented on METRON-634:
---
Github user mattf-horton commented on the issue:
https://github.com/apache/incubator-metron/pull/425
@dlyle65535 , I see that changes necessary to single-node, in slave.py and
elastic_slave.py, were in my METRON-634 commit rather than the METRON-609
commit. Sorry for the oversight.
> Mpack bug fixes and improvements (not related to singlenode install)
>
>
> Key: METRON-634
> URL: https://issues.apache.org/jira/browse/METRON-634
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
> Environment: Centos7
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). This jira provides the work items for merging those changes into
> the mainline Mpack. Almost all items relate to Elasticsearch.
> Some changes impact the Ambari database, and should therefore be done with an
> Mpack version bump and database upgrade script. These are in the second
> block below.
> h2. Not impacting Ambari database:
> * ES pid_dir specification and usage:
> ** Currently pid_dir is multiply specified in elastic-env.xml and params.py.
> The config parameter should not be over-ridden in params.py.
> ** PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It needs
> to be added to the template in elastic-sysconfig, as it must be provided to
> ES at launch-time (else the default directory will be used).
> ** pid_file is specified in params.py, but is not used anywhere. (The ES
> internal launcher synthesizes it from PID_DIR, and this is appropriate.)
> ** status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated
> in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the
> ES launch, unless /bin/java is defined (which it isn't necessarily).
> * Also in the /etc/sysconfig/elasticsearch template in elastic-sysconfig.xml,
> the value of ES_JAVA_OPTS incorrectly spans 3 lines. The lines must be
> terminated with backslashes to effectively become a single line. The current
> inclusion of newlines in the long string value is acceptable (although
> unusual) in shellscript, but not in a systemd EnvironmentFile.
> /etc/sysconfig/elasticsearch must function as both.
> * Also in ES_JAVA_OPTS, the two instances of {{log_dir}} needs to be followed
> by a slash '/'
> * Recommend adding QuickLinks for Elasticsearch. This is a low-impact change
> that is very helpful to validate ES health.
> ** ES health status
> ** ES indexes list
> * In elastic.py, when directories are being pre-created and permissions set,
> the file $CONF_DIR/scripts should also be pre-created. I intermittently hit
> permissions issues with this directory being created later by root, and not
> properly assigned to elastic_user.
> * In several places in elastic.py, "params.elastic_user" is incorrectly used
> when "params.user_group" should be used.
> * Undefined "format()" method is used in elastic.py, unnecessarily in
> File(format("/etc/sysconfig/elasticsearch")...
> * Undefined "format()" method is similarly used several times unnecessarily
> in elastic_master.py
> * The comments and descriptions in elastic-site.xml have multiple suggested
> improvements.
> * The comment description in
> common-services/METRON/0.3.0/configuration/metron-env.xml for "es_hosts"
> should clarify that this should be a list of ES Master hosts, not any ES
> hosts.
> h2. Affects Ambari database, needs db upgrade script:
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name
[jira] [Commented] (METRON-634) Mpack bug fixes and improvements (not related to singlenode install)
[
https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840598#comment-15840598
]
Matt Foley commented on METRON-634:
---
Since most of these issues have been included in
[PR#425|https://github.com/apache/incubator-metron/pull/425] for METRON-609,
I'm going to go ahead and change this to be a sub-task of METRON-609. Most of
these changes (for METRON-634), especially those "not affecting the Ambari
database" are bug fixes, not enhancements, without which some piece or other of
the ES installation doesn't work correctly in today's Mpack, at least with
Centos7.
> Mpack bug fixes and improvements (not related to singlenode install)
>
>
> Key: METRON-634
> URL: https://issues.apache.org/jira/browse/METRON-634
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
> Environment: Centos7
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). This jira provides the work items for merging those changes into
> the mainline Mpack. Almost all items relate to Elasticsearch.
> Some changes impact the Ambari database, and should therefore be done with an
> Mpack version bump and database upgrade script. These are in the second
> block below.
> h2. Not impacting Ambari database:
> * ES pid_dir specification and usage:
> ** Currently pid_dir is multiply specified in elastic-env.xml and params.py.
> The config parameter should not be over-ridden in params.py.
> ** PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It needs
> to be added to the template in elastic-sysconfig, as it must be provided to
> ES at launch-time (else the default directory will be used).
> ** pid_file is specified in params.py, but is not used anywhere. (The ES
> internal launcher synthesizes it from PID_DIR, and this is appropriate.)
> ** status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated
> in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the
> ES launch, unless /bin/java is defined (which it isn't necessarily).
> * Also in the /etc/sysconfig/elasticsearch template in elastic-sysconfig.xml,
> the value of ES_JAVA_OPTS incorrectly spans 3 lines. The lines must be
> terminated with backslashes to effectively become a single line. The current
> inclusion of newlines in the long string value is acceptable (although
> unusual) in shellscript, but not in a systemd EnvironmentFile.
> /etc/sysconfig/elasticsearch must function as both.
> * Also in ES_JAVA_OPTS, the two instances of {{log_dir}} needs to be followed
> by a slash '/'
> * Recommend adding QuickLinks for Elasticsearch. This is a low-impact change
> that is very helpful to validate ES health.
> ** ES health status
> ** ES indexes list
> * In elastic.py, when directories are being pre-created and permissions set,
> the file $CONF_DIR/scripts should also be pre-created. I intermittently hit
> permissions issues with this directory being created later by root, and not
> properly assigned to elastic_user.
> * In several places in elastic.py, "params.elastic_user" is incorrectly used
> when "params.user_group" should be used.
> * Undefined "format()" method is used in elastic.py, unnecessarily in
> File(format("/etc/sysconfig/elasticsearch")...
> * Undefined "format()" method is similarly used several times unnecessarily
> in elastic_master.py
> * The comments and descriptions in elastic-site.xml have multiple suggested
> improvements.
> * The comment description in
> common-services/METRON/0.3.0/configuration/metron-env.xml for "es_hosts"
> should clarify that this should be a list of ES Master hosts, not any ES
> hosts.
> h2. Affects Ambari database, needs db upgrade script:
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
>
