I am trying to set up aggregate logging (via
https://docs.openshift.org/latest/install_config/aggregate_logging.html) on
my Openshift-Origin cluster. The cluster was installed with the 'advanced'
Ansible playbook and is us
ing Origin v1.1.1 with kubernetes v1.1.0-origin-1107-g4c8e6f4. The one
modification I've made to the aggregate logging set up from the docs is to
use an NFS-based persistent volume and a corresponding
persistentVolumeClaim. The NFS server's export and the PV and PVC are set
up as outlined in the
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_nfs.html
.
When the `logging-es-XXXX` pod gets created it immediately goes into a
crash cycle with these errors in the log:
[2016-02-08 23:53:33,745][INFO ][plugins ] [Dmitri
Bukharin] loaded [searchguard, openshift-elasticsearch-plugin,
cloud-kubernetes], sites []
{1.5.2}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to created node environment]
AccessDeniedException[/elasticsearch/persistent/logging-es]
Clearly, it is having trouble with permissions of existence of the mounted
persistentVolume. I can't `exec` in to the real pod to verify anything
since it crashes too quickly, but I did create a duplicate pod that runs a
`sleep` instead of the default `run.sh` `CMD`. In this pod I can verify
that the PV is properly mounted and running the `run.sh` (
https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/Dockerfile#L32)
by hand fails in the same way. If I modify this one-off pod object
definition again to remove the `runAsUser: 1000040000` from the
`securityContext` for the container, it seems to work (or at least not fail
outright).
[master]# oc exec -ti logging-es-068wj8on-2-12xle-rcw bash
bash-4.2$ id
uid=1000 gid=0(root)
bash-4.2$ ls -l /elasticsearch/persistent/
total 0
bash-4.2$ ls -ld /elasticsearch/persistent/
drwxr-xr-x 2 nobody nobody 4096 Feb 8 18:29 /elasticsearch/persistent/
bash-4.2$ /opt/app-root/src/run.sh
[2016-02-09 00:03:13,831][INFO ][node ] [Captain
Omen] version[1.5.2], pid[22], build[62ff986/2015-04-27T09:21:06Z]
[2016-02-09 00:03:13,832][INFO ][node ] [Captain
Omen] initializing ...
[2016-02-09 00:03:14,882][INFO ][plugins ] [Captain
Omen] loaded [searchguard, openshift-elasticsearch-plugin,
cloud-kubernetes], sites []
[2016-02-09 00:03:20,026][INFO ][node ] [Captain
Omen] initialized
[2016-02-09 00:03:20,027][INFO ][node ] [Captain
Omen] starting ...
[2016-02-09 00:03:20,265][INFO ][transport ] [Captain
Omen] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
172.50.2.53:9300]}
[2016-02-09 00:03:20,303][INFO ][discovery ] [Captain
Omen] logging-es/Xpb-Cet6RaGdQdOgiAcM0w
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/share/elasticsearch/plugins/cloud-kubernetes/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/share/elasticsearch/plugins/openshift-elasticsearch-plugin/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2016-02-09 00:03:23,516][INFO ][cluster.service ] [Captain
Omen] new_master [Captain
Omen][Xpb-Cet6RaGdQdOgiAcM0w][logging-es-068wj8on-2-12xle-rcw][inet[/172.50.2.53:9300]],
reason: zen-disco-join (elected_as_master)
[2016-02-09 00:03:23,562][INFO ][http ] [Captain
Omen] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
172.50.2.53:9200]}
[2016-02-09 00:03:23,563][INFO ][node ] [Captain
Omen] started
I've run through the docs several times and rebuilt the project environment
each time, but get the same result. Does anyone have an idea of where I
might look next to figure out why using an NFS-based PV/PVC doesn't seem to
work with the default Origin aggregated logging setup?
Thanks.
--
Robert Wehner
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
