Re: Elasticsearch - 5.x

2019-03-21 Thread Rich Megginson

On 3/21/19 4:10 PM, Neale Ferguson wrote:

Thanks anyway. I'll take a deeper look into what's happening inside the 
container when it's started.

I think I should add an infra node rather than tie up the master as the logging 
stuff looks a little heavy on resources.



Yes.  We use a minimum memory of 16GB for Elasticsearch 5 in OpenShift 3.11 - 
up from 8GB with 3.10/ES 2.x



I've had a couple of instances where when logging is brought up the api 
controller etc start timing out or liveness probes timeout. I can't really 
identify what causes the cascading of pod failures. I think creating a small 
ceph cluster might be useful as well rather than using the unsupported NFS.



CEPH block is known to work with Elasticsearch 5.




Neale
  



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Rich Megginson

On 3/21/19 2:31 PM, Neale Ferguson wrote:

 Are you asking how to erase everything and start over?
- No, just how to get information out of the elasticsearch container to tell me 
why it's failing.
 
 Can you share your inventory files with the logging parameters (be sure to redact any sensitive information)?


- The configuration was an all-in-one when it was first created. I added a 
compute node shortly after then added the logging.


I don't see anything obviously wrong.  I guess I'd have to view the detailed 
ansible log (e.g. ansible-playbook -vvv) to see what went wrong.



[nfs]
okcd-master.sinenomine.net

# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd
nfs
new_nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 
'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider',}]
openshift_master_htpasswd_users={}
os_firewall_use_firewalld=True
openshift_logging_install_logging=true
openshift_logging_storage_kind=nfs
openshift_logging_storage_access_modes=['ReadWriteOnce']
openshift_logging_storage_nfs_directory=/exports
openshift_logging_storage_nfs_options='*(rw,root_squash)'
openshift_logging_storage_volume_name=logging
openshift_logging_storage_volume_size=200Mi
openshift_logging_storage_labels={'storage': 'logging'}
openshift_logging_kibana_hostname=logging.origin.z
openshift_logging_es_nodeselector={'node-role.kubernetes.io/infra': 'true'}
ansible_ssh_user=root
openshift_deployment_type=origin
oreg_url=docker.io/clefos/origin-${component}:${version}
openshift_examples_modify_imagestreams=true
openshift_web_console_prefix=docker.io/clefos/
openshift_disable_check=disk_availability,docker_storage,memory_availability
openshift_hosted_router_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_manage_registry=true
openshift_enable_unsupported_configurations=True
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=5Gi
openshift_cockpit_deployer_image=docker.io/clefos/cockpit-kubernetes:latest
openshift_console_install=False

# host group for masters
[masters]
okcd-master.sinenomine.net

# host group for etcd
[etcd]
master.example.com

# host group for nodes, includes region info
[nodes]
master.example.com openshift_node_group_name='node-config-all-in-one'
node.example.com openshift_node_group_name='node-config-compute'

# Adding new node to the cluster
[new_nodes]


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Neale Ferguson
Are you asking how to erase everything and start over?
- No, just how to get information out of the elasticsearch container to tell me 
why it's failing. 

Can you share your inventory files with the logging parameters (be sure to 
redact any sensitive information)?

- The configuration was an all-in-one when it was first created. I added a 
compute node shortly after then added the logging.

[nfs]
okcd-master.sinenomine.net

# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd
nfs
new_nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 
'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider',}]
openshift_master_htpasswd_users={}
os_firewall_use_firewalld=True 
openshift_logging_install_logging=true
openshift_logging_storage_kind=nfs
openshift_logging_storage_access_modes=['ReadWriteOnce']
openshift_logging_storage_nfs_directory=/exports
openshift_logging_storage_nfs_options='*(rw,root_squash)'
openshift_logging_storage_volume_name=logging
openshift_logging_storage_volume_size=200Mi
openshift_logging_storage_labels={'storage': 'logging'}
openshift_logging_kibana_hostname=logging.origin.z
openshift_logging_es_nodeselector={'node-role.kubernetes.io/infra': 'true'} 
ansible_ssh_user=root
openshift_deployment_type=origin
oreg_url=docker.io/clefos/origin-${component}:${version}
openshift_examples_modify_imagestreams=true
openshift_web_console_prefix=docker.io/clefos/
openshift_disable_check=disk_availability,docker_storage,memory_availability
openshift_hosted_router_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_manage_registry=true
openshift_enable_unsupported_configurations=True
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=5Gi
openshift_cockpit_deployer_image=docker.io/clefos/cockpit-kubernetes:latest
openshift_console_install=False

# host group for masters
[masters]
okcd-master.sinenomine.net

# host group for etcd
[etcd]
master.example.com

# host group for nodes, includes region info
[nodes]
master.example.com openshift_node_group_name='node-config-all-in-one'
node.example.com openshift_node_group_name='node-config-compute'

# Adding new node to the cluster
[new_nodes]


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Rich Megginson

On 3/21/19 1:56 PM, Neale Ferguson wrote:

 > - So this leads to the question of how to debug the elasticsearch 
container not coming up. The console log information in the 1st email came from me 
running it manually so I could add the -v option to the elasticsearch command.  
Setting the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess 
I need to try and add the -v option to the run.sh script.
 
 This simply should not be happening.  Did you use openshift-ansible to upgrade from 3.10 to 3.11, or did you deploy logging from scratch on 3.11?

- Initially, I created the 3.11 cluster from scratch  but without logging. I 
then added the logging statements to the hosts file and then ran the logging 
playbook.
  



Are you asking how to erase everything and start over?

Can you share your inventory files with the logging parameters (be sure to 
redact any sensitive information)?




___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Neale Ferguson
> - So this leads to the question of how to debug the elasticsearch 
container not coming up. The console log information in the 1st email came from 
me running it manually so I could add the -v option to the elasticsearch 
command.  Setting the DEBUG and LOGLEVEL environment variables wasn't 
illuminating. I guess I need to try and add the -v option to the run.sh script.

This simply should not be happening.  Did you use openshift-ansible to 
upgrade from 3.10 to 3.11, or did you deploy logging from scratch on 3.11?
- Initially, I created the 3.11 cluster from scratch  but without logging. I 
then added the logging statements to the hosts file and then ran the logging 
playbook.
 


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Rich Megginson

On 3/21/19 1:41 PM, Neale Ferguson wrote:

 elasticsearch 5 is not tech preview in 3.11 - it is fully supported - and 
2.x is gone
- Understood. Did that change with the move from 3.10 to 3.11?



Yes



I must've been scanning another host which was running 3.10 to spot those 
preview vars.
- So this leads to the question of how to debug the elasticsearch container not 
coming up. The console log information in the 1st email came from me running it 
manually so I could add the -v option to the elasticsearch command.  Setting 
the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I 
need to try and add the -v option to the run.sh script.



This simply should not be happening.  Did you use openshift-ansible to upgrade 
from 3.10 to 3.11, or did you deploy logging from scratch on 3.11?


 
 



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Neale Ferguson
elasticsearch 5 is not tech preview in 3.11 - it is fully supported - and 
2.x is gone
- Understood. Did that change with the move from 3.10 to 3.11? I must've been 
scanning another host which was running 3.10 to spot those preview vars. 
- So this leads to the question of how to debug the elasticsearch container not 
coming up. The console log information in the 1st email came from me running it 
manually so I could add the -v option to the elasticsearch command.  Setting 
the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I 
need to try and add the -v option to the run.sh script.




___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Rich Megginson

On 3/21/19 1:33 PM, Neale Ferguson wrote:


On 3/21/19, 15:33, "Neale Ferguson"  wrote:

 Right. The image is now 
https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there 
are similar changes for origin-logging-curator5, origin-logging-kibana5,
 origin-logging-fluentd
 
 What version of openshift-ansible did you use to deploy logging?

 3.11. Strangely, I didn't set the preview option in the hosts file but the only 
images available for me (the ones that I built) are the "5" series and these 
are the ones that were started.



Elasticsearch 5 is _not_ tech preview in 3.11 - it is fully supported - 2.x is 
gone


  
 
 



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Neale Ferguson


On 3/21/19, 15:33, "Neale Ferguson"  wrote:

Right. The image is now 
https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there 
are similar changes for origin-logging-curator5, origin-logging-kibana5, 
origin-logging-fluentd

What version of openshift-ansible did you use to deploy logging?
3.11. Strangely, I didn't set the preview option in the hosts file but the 
only images available for me (the ones that I built) are the "5" series and 
these are the ones that were started.
 




___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: Elasticsearch - 5.x

2019-03-21 Thread Rich Megginson

On 3/21/19 1:23 PM, Neale Ferguson wrote:


If I want to use the techpreview elasticsearch5 images are there any other 
changes I need to make as I am seeing this on startup:

Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes

configuration like the elasticsearch.yaml, in system properties or command line

arguments.In order to upgrade all indices the settings must be updated via the

/${index}/_settings API. Unless all settings are dynamic all indices must be 
closed

in order to apply the upgradeIndices created in the future should use index 
templates

to set default values.

Please ensure all required values are updated on all indices by executing:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{

"index.number_of_replicas" : "0",

"index.number_of_shards" : "1",

"index.translog.flush_threshold_period" : "5m",

"index.translog.flush_threshold_size" : "256mb",

"index.unassigned.node_left.delayed_timeout" : "2m"

}'

*

[2019-03-21T18:22:36,493][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] 
[logging-es-data-master-1jbe2qib] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: 
java.lang.IllegalArgumentException: node settings must not contain any index 
level settings

The Dockerfile is building 5.6.10 but the 3.11.0 image from docker.io/openshift 
appears to be based on 2.4.4 (which was changed to 5.x in June 2018).



Right. The image is now https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there are similar changes for origin-logging-curator5, origin-logging-kibana5, 
origin-logging-fluentd


What version of openshift-ansible did you use to deploy logging?



Neale


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev