Re: Elasticsearch - 5.x
On 3/21/19 4:10 PM, Neale Ferguson wrote: Thanks anyway. I'll take a deeper look into what's happening inside the container when it's started. I think I should add an infra node rather than tie up the master as the logging stuff looks a little heavy on resources. Yes. We use a minimum memory of 16GB for Elasticsearch 5 in OpenShift 3.11 - up from 8GB with 3.10/ES 2.x I've had a couple of instances where when logging is brought up the api controller etc start timing out or liveness probes timeout. I can't really identify what causes the cascading of pod failures. I think creating a small ceph cluster might be useful as well rather than using the unsupported NFS. CEPH block is known to work with Elasticsearch 5. Neale ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
Thanks anyway. I'll take a deeper look into what's happening inside the container when it's started. I think I should add an infra node rather than tie up the master as the logging stuff looks a little heavy on resources. I've had a couple of instances where when logging is brought up the api controller etc start timing out or liveness probes timeout. I can't really identify what causes the cascading of pod failures. I think creating a small ceph cluster might be useful as well rather than using the unsupported NFS. Neale ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
On 3/21/19 2:31 PM, Neale Ferguson wrote: Are you asking how to erase everything and start over? - No, just how to get information out of the elasticsearch container to tell me why it's failing. Can you share your inventory files with the logging parameters (be sure to redact any sensitive information)? - The configuration was an all-in-one when it was first created. I added a compute node shortly after then added the logging. I don't see anything obviously wrong. I guess I'd have to view the detailed ansible log (e.g. ansible-playbook -vvv) to see what went wrong. [nfs] okcd-master.sinenomine.net # Create an OSEv3 group that contains the masters, nodes, and etcd groups [OSEv3:children] masters nodes etcd nfs new_nodes # Set variables common for all OSEv3 hosts [OSEv3:vars] openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider',}] openshift_master_htpasswd_users={} os_firewall_use_firewalld=True openshift_logging_install_logging=true openshift_logging_storage_kind=nfs openshift_logging_storage_access_modes=['ReadWriteOnce'] openshift_logging_storage_nfs_directory=/exports openshift_logging_storage_nfs_options='*(rw,root_squash)' openshift_logging_storage_volume_name=logging openshift_logging_storage_volume_size=200Mi openshift_logging_storage_labels={'storage': 'logging'} openshift_logging_kibana_hostname=logging.origin.z openshift_logging_es_nodeselector={'node-role.kubernetes.io/infra': 'true'} ansible_ssh_user=root openshift_deployment_type=origin oreg_url=docker.io/clefos/origin-${component}:${version} openshift_examples_modify_imagestreams=true openshift_web_console_prefix=docker.io/clefos/ openshift_disable_check=disk_availability,docker_storage,memory_availability openshift_hosted_router_selector='node-role.kubernetes.io/infra=true' openshift_hosted_manage_registry=true openshift_enable_unsupported_configurations=True openshift_hosted_registry_storage_kind=nfs openshift_hosted_registry_storage_nfs_directory=/exports openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)' openshift_hosted_registry_storage_volume_name=registry openshift_hosted_registry_storage_volume_size=5Gi openshift_cockpit_deployer_image=docker.io/clefos/cockpit-kubernetes:latest openshift_console_install=False # host group for masters [masters] okcd-master.sinenomine.net # host group for etcd [etcd] master.example.com # host group for nodes, includes region info [nodes] master.example.com openshift_node_group_name='node-config-all-in-one' node.example.com openshift_node_group_name='node-config-compute' # Adding new node to the cluster [new_nodes] ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
Are you asking how to erase everything and start over? - No, just how to get information out of the elasticsearch container to tell me why it's failing. Can you share your inventory files with the logging parameters (be sure to redact any sensitive information)? - The configuration was an all-in-one when it was first created. I added a compute node shortly after then added the logging. [nfs] okcd-master.sinenomine.net # Create an OSEv3 group that contains the masters, nodes, and etcd groups [OSEv3:children] masters nodes etcd nfs new_nodes # Set variables common for all OSEv3 hosts [OSEv3:vars] openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider',}] openshift_master_htpasswd_users={} os_firewall_use_firewalld=True openshift_logging_install_logging=true openshift_logging_storage_kind=nfs openshift_logging_storage_access_modes=['ReadWriteOnce'] openshift_logging_storage_nfs_directory=/exports openshift_logging_storage_nfs_options='*(rw,root_squash)' openshift_logging_storage_volume_name=logging openshift_logging_storage_volume_size=200Mi openshift_logging_storage_labels={'storage': 'logging'} openshift_logging_kibana_hostname=logging.origin.z openshift_logging_es_nodeselector={'node-role.kubernetes.io/infra': 'true'} ansible_ssh_user=root openshift_deployment_type=origin oreg_url=docker.io/clefos/origin-${component}:${version} openshift_examples_modify_imagestreams=true openshift_web_console_prefix=docker.io/clefos/ openshift_disable_check=disk_availability,docker_storage,memory_availability openshift_hosted_router_selector='node-role.kubernetes.io/infra=true' openshift_hosted_manage_registry=true openshift_enable_unsupported_configurations=True openshift_hosted_registry_storage_kind=nfs openshift_hosted_registry_storage_nfs_directory=/exports openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)' openshift_hosted_registry_storage_volume_name=registry openshift_hosted_registry_storage_volume_size=5Gi openshift_cockpit_deployer_image=docker.io/clefos/cockpit-kubernetes:latest openshift_console_install=False # host group for masters [masters] okcd-master.sinenomine.net # host group for etcd [etcd] master.example.com # host group for nodes, includes region info [nodes] master.example.com openshift_node_group_name='node-config-all-in-one' node.example.com openshift_node_group_name='node-config-compute' # Adding new node to the cluster [new_nodes] ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
On 3/21/19 1:56 PM, Neale Ferguson wrote: > - So this leads to the question of how to debug the elasticsearch container not coming up. The console log information in the 1st email came from me running it manually so I could add the -v option to the elasticsearch command. Setting the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I need to try and add the -v option to the run.sh script. This simply should not be happening. Did you use openshift-ansible to upgrade from 3.10 to 3.11, or did you deploy logging from scratch on 3.11? - Initially, I created the 3.11 cluster from scratch but without logging. I then added the logging statements to the hosts file and then ran the logging playbook. Are you asking how to erase everything and start over? Can you share your inventory files with the logging parameters (be sure to redact any sensitive information)? ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
> - So this leads to the question of how to debug the elasticsearch container not coming up. The console log information in the 1st email came from me running it manually so I could add the -v option to the elasticsearch command. Setting the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I need to try and add the -v option to the run.sh script. This simply should not be happening. Did you use openshift-ansible to upgrade from 3.10 to 3.11, or did you deploy logging from scratch on 3.11? - Initially, I created the 3.11 cluster from scratch but without logging. I then added the logging statements to the hosts file and then ran the logging playbook. ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
On 3/21/19 1:41 PM, Neale Ferguson wrote: elasticsearch 5 is not tech preview in 3.11 - it is fully supported - and 2.x is gone - Understood. Did that change with the move from 3.10 to 3.11? Yes I must've been scanning another host which was running 3.10 to spot those preview vars. - So this leads to the question of how to debug the elasticsearch container not coming up. The console log information in the 1st email came from me running it manually so I could add the -v option to the elasticsearch command. Setting the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I need to try and add the -v option to the run.sh script. This simply should not be happening. Did you use openshift-ansible to upgrade from 3.10 to 3.11, or did you deploy logging from scratch on 3.11? ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
elasticsearch 5 is not tech preview in 3.11 - it is fully supported - and 2.x is gone - Understood. Did that change with the move from 3.10 to 3.11? I must've been scanning another host which was running 3.10 to spot those preview vars. - So this leads to the question of how to debug the elasticsearch container not coming up. The console log information in the 1st email came from me running it manually so I could add the -v option to the elasticsearch command. Setting the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I need to try and add the -v option to the run.sh script. ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
On 3/21/19 1:33 PM, Neale Ferguson wrote: On 3/21/19, 15:33, "Neale Ferguson" wrote: Right. The image is now https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there are similar changes for origin-logging-curator5, origin-logging-kibana5, origin-logging-fluentd What version of openshift-ansible did you use to deploy logging? 3.11. Strangely, I didn't set the preview option in the hosts file but the only images available for me (the ones that I built) are the "5" series and these are the ones that were started. Elasticsearch 5 is _not_ tech preview in 3.11 - it is fully supported - 2.x is gone ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
On 3/21/19, 15:33, "Neale Ferguson" wrote: Right. The image is now https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there are similar changes for origin-logging-curator5, origin-logging-kibana5, origin-logging-fluentd What version of openshift-ansible did you use to deploy logging? 3.11. Strangely, I didn't set the preview option in the hosts file but the only images available for me (the ones that I built) are the "5" series and these are the ones that were started. ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Re: Elasticsearch - 5.x
On 3/21/19 1:23 PM, Neale Ferguson wrote: If I want to use the techpreview elasticsearch5 images are there any other changes I need to make as I am seeing this on startup: Found index level settings on node level configuration. Since elasticsearch 5.x index level settings can NOT be set on the nodes configuration like the elasticsearch.yaml, in system properties or command line arguments.In order to upgrade all indices the settings must be updated via the /${index}/_settings API. Unless all settings are dynamic all indices must be closed in order to apply the upgradeIndices created in the future should use index templates to set default values. Please ensure all required values are updated on all indices by executing: curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{ "index.number_of_replicas" : "0", "index.number_of_shards" : "1", "index.translog.flush_threshold_period" : "5m", "index.translog.flush_threshold_size" : "256mb", "index.unassigned.node_left.delayed_timeout" : "2m" }' * [2019-03-21T18:22:36,493][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [logging-es-data-master-1jbe2qib] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: node settings must not contain any index level settings The Dockerfile is building 5.6.10 but the 3.11.0 image from docker.io/openshift appears to be based on 2.4.4 (which was changed to 5.x in June 2018). Right. The image is now https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there are similar changes for origin-logging-curator5, origin-logging-kibana5, origin-logging-fluentd What version of openshift-ansible did you use to deploy logging? Neale ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
Elasticsearch - 5.x
If I want to use the techpreview elasticsearch5 images are there any other changes I need to make as I am seeing this on startup: Found index level settings on node level configuration. Since elasticsearch 5.x index level settings can NOT be set on the nodes configuration like the elasticsearch.yaml, in system properties or command line arguments.In order to upgrade all indices the settings must be updated via the /${index}/_settings API. Unless all settings are dynamic all indices must be closed in order to apply the upgradeIndices created in the future should use index templates to set default values. Please ensure all required values are updated on all indices by executing: curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{ "index.number_of_replicas" : "0", "index.number_of_shards" : "1", "index.translog.flush_threshold_period" : "5m", "index.translog.flush_threshold_size" : "256mb", "index.unassigned.node_left.delayed_timeout" : "2m" }' * [2019-03-21T18:22:36,493][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [logging-es-data-master-1jbe2qib] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: node settings must not contain any index level settings The Dockerfile is building 5.6.10 but the 3.11.0 image from docker.io/openshift appears to be based on 2.4.4 (which was changed to 5.x in June 2018). Neale ___ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev