[
https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated METRON-634:
------------------------------
Issue Type: Sub-task (was: Bug)
Parent: METRON-609
> Mpack bug fixes and improvements (not related to singlenode install)
> --------------------------------------------------------------------
>
> Key: METRON-634
> URL: https://issues.apache.org/jira/browse/METRON-634
> Project: Metron
> Issue Type: Sub-task
> Affects Versions: 0.3.0
> Environment: Centos7
> Reporter: Matt Foley
> Assignee: Matt Foley
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). This jira provides the work items for merging those changes into
> the mainline Mpack. Almost all items relate to Elasticsearch.
> Some changes impact the Ambari database, and should therefore be done with an
> Mpack version bump and database upgrade script. These are in the second
> block below.
> h2. Not impacting Ambari database:
> * ES pid_dir specification and usage:
> ** Currently pid_dir is multiply specified in elastic-env.xml and params.py.
> The config parameter should not be over-ridden in params.py.
> ** PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It needs
> to be added to the template in elastic-sysconfig, as it must be provided to
> ES at launch-time (else the default directory will be used).
> ** pid_file is specified in params.py, but is not used anywhere. (The ES
> internal launcher synthesizes it from PID_DIR, and this is appropriate.)
> ** status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated
> in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the
> ES launch, unless /bin/java is defined (which it isn't necessarily).
> * Also in the /etc/sysconfig/elasticsearch template in elastic-sysconfig.xml,
> the value of ES_JAVA_OPTS incorrectly spans 3 lines. The lines must be
> terminated with backslashes to effectively become a single line. The current
> inclusion of newlines in the long string value is acceptable (although
> unusual) in shellscript, but not in a systemd EnvironmentFile.
> /etc/sysconfig/elasticsearch must function as both.
> * Also in ES_JAVA_OPTS, the two instances of {{log_dir}} needs to be followed
> by a slash '/'
> * Recommend adding QuickLinks for Elasticsearch. This is a low-impact change
> that is very helpful to validate ES health.
> ** ES health status
> ** ES indexes list
> * In elastic.py, when directories are being pre-created and permissions set,
> the file $CONF_DIR/scripts should also be pre-created. I intermittently hit
> permissions issues with this directory being created later by root, and not
> properly assigned to elastic_user.
> * In several places in elastic.py, "params.elastic_user" is incorrectly used
> when "params.user_group" should be used.
> * Undefined "format()" method is used in elastic.py, unnecessarily in
> File(format("/etc/sysconfig/elasticsearch")...
> * Undefined "format()" method is similarly used several times unnecessarily
> in elastic_master.py
> * The comments and descriptions in elastic-site.xml have multiple suggested
> improvements.
> * The comment description in
> common-services/METRON/0.3.0/configuration/metron-env.xml for "es_hosts"
> should clarify that this should be a list of ES Master hosts, not any ES
> hosts.
> h2. Affects Ambari database, needs db upgrade script:
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name (to distinguish ES cluster from Stack
> cluster)
> zen_discovery_ping_unicast_hosts -> es_cluster_hosts
> network_host -> network_bindings (these are in fact interface names, not
> host names)
> {code}
> * There are at least two places in elasticsearch.master.yaml.j2
> (zen_discovery_ping_unicast_hosts and network_host) where needed square
> brackets are either missing or included in the configuration string. To be
> consistent with other usages, and less prone to human error, the square
> brackets should not be in the configuration string but rather should be
> provided in the template text.
> * "data_dir" apparently should be eliminated (from elastic-sysconfig) in
> preference for "path_data" (in elastic-site.xml). The latter value ends up
> overriding the former anyway, but the existence of the former is confusing
> and unnecessary.
> * All four configuration parameters in elastic-env.xml should be moved to
> elastic-site.xml, because they are all reasonable to set in a "site.xml" file
> and do end up in the .yml file that ES uses instead of "site.xml", and do NOT
> end up in environment variables. The only parameters that end up in env vars
> are set in elastic-sysconfig, and the ES launch process in fact ignores the
> elastic-env.sh file that is templated in elastic-env.xml (which consists only
> of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and
> hence entirely remove elastic-env.xml, or we could choose to keep the small
> elastic-env.sh file and its template, just to remind people that it is
> necessary to have JAVA_HOME defined.
> * In METRON/0.3.0/configuration/metron-env.xml and
> METRON/0.3.0/package/scripts/params/params_linux.py, the value
> "metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is
> appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus
> it can be removed from metron-env.xml and set to
> "\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also
> eliminates a really unacceptable use of "double format".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
