[ https://issues.apache.org/jira/browse/METRON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974945#comment-15974945 ]
ASF GitHub Bot commented on METRON-634: --------------------------------------- Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/532 +1, ran it up in full-dev and on EC2 (Centos7) and worked like a champ. Thanks! > Mpack bug fixes and improvements (not related to singlenode install) > -------------------------------------------------------------------- > > Key: METRON-634 > URL: https://issues.apache.org/jira/browse/METRON-634 > Project: Metron > Issue Type: Sub-task > Affects Versions: 0.3.0 > Environment: Centos7 > Reporter: Matt Foley > Assignee: Matt Foley > > Multiple bug fixes and recommended improvements were found in the course of > implementing METRON-608 that are unrelated to METRON-609 (singlenode > install). This jira provides the work items for merging those changes into > the mainline Mpack. Almost all items relate to Elasticsearch. > Some changes impact the Ambari database, and should therefore be done with an > Mpack version bump and database upgrade script. These are in the second > block below. > h2. Not impacting Ambari database: > * ES pid_dir specification and usage: > ** Currently pid_dir is multiply specified in elastic-env.xml and params.py. > The config parameter should not be over-ridden in params.py. > ** PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It needs > to be added to the template in elastic-sysconfig, as it must be provided to > ES at launch-time (else the default directory will be used). > ** pid_file is specified in params.py, but is not used anywhere. (The ES > internal launcher synthesizes it from PID_DIR, and this is appropriate.) > ** status_params.py, which redundantly defines pid_dir as a python variable, > is unnecessary and unused by the ES portion of the Mpack. It can be removed. > * JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated > in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the > ES launch, unless /bin/java is defined (which it isn't necessarily). > * Also in the /etc/sysconfig/elasticsearch template in elastic-sysconfig.xml, > the value of ES_JAVA_OPTS incorrectly spans 3 lines. The lines must be > terminated with backslashes to effectively become a single line. The current > inclusion of newlines in the long string value is acceptable (although > unusual) in shellscript, but not in a systemd EnvironmentFile. > /etc/sysconfig/elasticsearch must function as both. > * Also in ES_JAVA_OPTS, the two instances of {{log_dir}} needs to be followed > by a slash '/' > * Recommend adding QuickLinks for Elasticsearch. This is a low-impact change > that is very helpful to validate ES health. > ** ES health status > ** ES indexes list > * In elastic.py, when directories are being pre-created and permissions set, > the directory $CONF_DIR/scripts should also be pre-created. I intermittently > hit permissions issues with this directory being created later by root, and > not properly assigned to elastic_user. > * In several places in elastic.py, "params.elastic_user" is incorrectly used > when "params.user_group" should be used. > * Undefined "format()" method is used in elastic.py, unnecessarily in > File(format("/etc/sysconfig/elasticsearch")... > * Undefined "format()" method is similarly used several times unnecessarily > in elastic_master.py and elastic_slave.py > * The comments and descriptions in elastic-site.xml have multiple suggested > improvements. > * The comment description in > common-services/METRON/0.3.0/configuration/metron-env.xml for "es_hosts" > should clarify that this should be a list of ES Master hosts, not any ES > hosts. > h2. Affects Ambari database, needs db upgrade script: > * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than > elastic-env.xml, as it is a parameter that must be provided to ES at > launch-time, but is not something there's any reason for the admin to change > in usual circumstances. > * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in > elastic-sysconfig.xml. While it too is a parameter that must be provided to > ES at launch-time, it is typically left to the installing admin where to put > the config files. > * The configuration parameter names in elastic-site.xml should be improved in > several instances to make the semantics more obvious to the human reader (who > may not be real familiar with Elasticsearch configuration). Mouse-over > documentation will continue to provide the ES config parameter equivalents. > In particular, suggest: > {code} > cluster_name -> es_cluster_name (to distinguish ES cluster from Stack > cluster) > zen_discovery_ping_unicast_hosts -> es_cluster_hosts > network_host -> network_bindings (these are in fact interface names, not > host names) > {code} > * There are at least two places in elasticsearch.master.yaml.j2 > (zen_discovery_ping_unicast_hosts and network_host) where needed square > brackets are either missing or included in the configuration string. To be > consistent with other usages, and less prone to human error, the square > brackets should not be in the configuration string but rather should be > provided in the template text. > * "data_dir" apparently should be eliminated (from elastic-sysconfig) in > preference for "path_data" (in elastic-site.xml). The latter value ends up > overriding the former anyway, but the existence of the former is confusing > and unnecessary. > * All four configuration parameters in elastic-env.xml should be moved to > elastic-site.xml, because they are all reasonable to set in a "site.xml" file > and do end up in the .yml file that ES uses instead of "site.xml", and do NOT > end up in environment variables. The only parameters that end up in env vars > are set in elastic-sysconfig, and the ES launch process in fact ignores the > elastic-env.sh file that is templated in elastic-env.xml (which consists only > of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and > hence entirely remove elastic-env.xml, or we could choose to keep the small > elastic-env.sh file and its template, just to remind people that it is > necessary to have JAVA_HOME defined. > * In METRON/0.3.0/configuration/metron-env.xml and > METRON/0.3.0/package/scripts/params/params_linux.py, the value > "metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is > appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus > it can be removed from metron-env.xml and set to > "\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also > eliminates a really unacceptable use of "double format". -- This message was sent by Atlassian JIRA (v6.3.15#6346)