I saw on https://github.com/openshift/origin/issues/8358:
$ oc debug pod/logging-fluentd-80xzt -- cat /proc/self/attr/current
Debugging with pod/debug-logging-fluentd-80xzt, original command: <image
entrypoint>
Waiting for pod to start ...
system_u:system_r:svirt_lxc_net_t:s0:c216,c576
Removing debug pod ...
Yup. The problem was what I thought: it's being run under the
svirt_lsc_net_t SELinux type, which doesn't have access to var_log_t.
If you don't want to disable SELinux, you'll need to follow the
instructions for creating a new SELinux type that I posted above.
So I understand what's wrong but I don't see why the workaround (changing the
service account permissions from anyuid to privileged) isn't working for me + I
don't want to create a new selinuxtype.
From: [email protected]
To: [email protected]
CC: [email protected]
Subject: RE: Aggregating container logs using Kibana
Date: Wed, 13 Apr 2016 09:30:48 +0200
Fixed the issue with nodeselectormismatching:
So now I have 3 fluentd pods on my 2 normal nodes and my infranode:
But still the same permission issue:
NAME READY STATUS RESTARTS AGE
logging-curator-1-j7mz0 1/1 Running 0 17m
logging-deployer-39qcz 0/1 Completed 0 47m
logging-es-605u5g7g-1-36owl 1/1 Running 0 17m
logging-fluentd-4uqx1 1/1 Running 0 46m
logging-fluentd-dez5r 1/1 Running 0 2m
logging-fluentd-m50nj 1/1 Running 0 46m
logging-kibana-1-wfog2 2/2 Running 0 16m
From: [email protected]
To: [email protected]
CC: [email protected]
Subject: RE: Aggregating container logs using Kibana
Date: Wed, 13 Apr 2016 09:21:47 +0200
Hi Eric,
Thanks for your reply and the follow up of this issue.
I've created a new origin 1.1.6 cluster (2 days ago) but still have the same
issue:
My environment is one master (with node) non schedulable, 2 'normal' nodes and
one infra node.
I still got the permission denied (The documentation is up to date so I even
don't had to perform the workaround manually).
- system:serviceaccount:logging:aggregated-logging-fluentd is in scc privileged
by default.
The logging-deployer-template creates services and 2 pods of fluentd (on the
normal nodes).
The pods appear after performing this command:
oc label nodes --all logging-infra-fluentd=trueSo my nodes got that label. also
the unschedulable node on my master. So that's normal that it failed but why it
fails on my infra-node I don't know. (I defined in my master-config that
projects are by default on the other 2 nodes, maybe that's why but I don't know
it's relevant for my issue).
I also don't really understand why 'oc process logging-support-tempalte | oc
create -f -' is only be cited at the troubleshooting part.
Still the error: [error]: unexpected error error_class=Errno::EACCES
error=#<Errno::EACCES: Permission denied -
/var/log/es-containers.log.pos>
oc get is
NAME DOCKER REPO TAGS
UPDATED
logging-auth-proxy docker.io/openshift/origin-logging-auth-proxy
latest,v0.0.1 4 minutes ago
logging-curator docker.io/openshift/origin-logging-curator
latest 4 minutes ago
logging-elasticsearch docker.io/openshift/origin-logging-elasticsearch
latest 4 minutes ago
logging-fluentd docker.io/openshift/origin-logging-fluentd
latest 4 minutes ago
logging-kibana docker.io/openshift/origin-logging-kibana
latest 4 minutes ago
oc get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
logging-es 172.30.68.xx <none> 9200/TCP 33m
logging-es-cluster None <none> 9300/TCP 33m
logging-es-ops 172.30.18.xx <none> 9200/TCP 33m
logging-es-ops-cluster None <none> 9300/TCP 33m
logging-kibana 172.30.216.xx <none> 443/TCP 33m
logging-kibana-ops 172.30.186.xx <none> 443/TCP 33m
oc get pods
NAME READY STATUS RESTARTS AGE
logging-curator-1-j7mz0 1/1 Running 0 4m
logging-deployer-39qcz 0/1 Completed 0 34m
logging-es-605u5g7g-1-36owl 1/1 Running 0 4m
logging-fluentd-4uqx1 1/1 Running 0 33m
logging-fluentd-ex34j 0/1 NodeSelectorMismatching 0 33m
logging-fluentd-injz7 0/1 NodeSelectorMismatching 0 33m
logging-fluentd-m50nj 1/1 Running 0 33m
logging-kibana-1-wfog2 2/2 Running 0 4m
oc get daemonset
NAME DESIRED CURRENT NODE-SELECTOR AGE
logging-fluentd 4 4 logging-infra-fluentd=true 34m
oc get dc
NAME REVISION REPLICAS TRIGGERED BY
logging-curator 1 1 config,image(logging-curator:latest)
logging-es-605u5g7g 1 1
config,image(logging-elasticsearch:latest)
logging-kibana 1 1
config,image(logging-auth-proxy:latest),image(logging-kibana:latest)
oc get routes
[centos@ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops)
NAME HOST/PORT PATH SERVICE
TERMINATION LABELS
kibana kibana.test.xxx.eu logging-kibana passthrough
component=support,logging-infra=support,provider=openshift
kibana-ops kibana-ops.example.com logging-kibana-ops
passthrough component=support,logging-infra=support,provider=openshift
oc get oauthclient
NAME SECRET
WWW-CHALLENGE REDIRECT URIS
kibana-proxy
j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX FALSE
https://kibana.xxx.eu,https://kibana-ops.example.com
openshift-browser-client 71724303-b823-4435-8568-bcafxxxx4
FALSE
https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display
openshift-challenging-client ac7c9942-9a55-4e1e-8e5f-9fxxxxx
TRUE
https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit
openshift-web-console 6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx
FALSE
https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000
From: [email protected]
Date: Tue, 12 Apr 2016 17:27:06 -0500
Subject: Re: Aggregating container logs using Kibana
To: [email protected]
CC: [email protected]; [email protected]
On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo
<[email protected]> wrote:
This are all the steps I'm performing:
oc new-project logging
$ oc secrets new logging-deployer nothing=/dev/null
$ oc process logging-deployer-account-template -n openshift \
| oc create -f -
$ oc policy add-role-to-user edit --serviceaccount logging-deployer
$ oc policy add-role-to-user daemonset-admin --serviceaccount logging-deployer
$ oadm policy add-cluster-role-to-user oauth-editor \
system:serviceaccount:logging:logging-deployer
$ oadm policy add-scc-to-user \
privileged system:serviceaccount:logging:aggregated-logging-fluentd
$ oadm policy add-cluster-role-to-user cluster-reader \
system:serviceaccount:logging:aggregated-logging-fluentd
Than I execute the deployer template:
$ oc process logging-deployer-template -n openshift \
-v
KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443
\
| oc create -f -
This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on master
machine) and some empty services (the logs of the pods are telling me the
permission error)
When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's all
fine.
$ oc label nodes --all logging-infra-fluentd=true
I've edited /master/master-config.yaml + restart
$ oc scale dc/logging-kibana --replicas=2
$ oc delete oauthclient/kibana-proxy
$ oc process logging-support-template | oc create -f -
The last step creates also some pods. It's a bit weird for me that this step is
only mentioned for troubleshooting or is
it an issue that I don't have those pods after executing the deployer-template?
The template 'logging-support-template' creates your ImageStreams (along with
your routes and oauthclient) so it shouldn't be creating your pods. There may
have been a delay in scheduling your pods initially or the image stream tags
could have been in the processes of being fetched.
What does the following output?oc get is, svc, pods, daemonset, dc, routes,
oauthclient -n logging
And do you still see the same permission denied errors in the Fluentd logs?
From: [email protected]
To: [email protected]
CC: [email protected]
Subject: RE: Aggregating container logs using Kibana
Date: Tue, 5 Apr 2016 18:00:02 +0200
I still have the same issue:
I've deleted it from scc hostmount-anyuid and added it on scc privileged.
I've deleted all fluentd pods but still the same issue. Even after recreating
the project.
From: [email protected]
Date: Tue, 5 Apr 2016 10:29:04 -0400
Subject: Re: Aggregating container logs using Kibana
To: [email protected]
CC: [email protected]
On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <[email protected]> wrote:
2016-04-05 10:55:13 +0000 [error]: unexpected error error_class=Errno::EACCES
error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos>
This looks like
https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps
fluentd from reading any logs on the node.
You should be able to resolve this by adding the fluentd service account to the
privileged SCC, then having fluentd restart everywhere.
oadm policy add-scc-to-user privileged
system:serviceaccount:logging:aggregated-logging-fluentd
Oh; probably need to also remove them from the hostmount-anyuid SCC.
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
