Just a heads up, the latest deployer image on Dockerhub has an updated fluentd template that already contains the change for Fluentd to run in the privileged security context.
On Wed, Apr 13, 2016 at 11:24 AM, Eric Wolinetz <[email protected]> wrote: > > > On Wed, Apr 13, 2016 at 3:16 AM, Lorenz Vanthillo < > [email protected]> wrote: > >> I saw on https://github.com/openshift/origin/issues/8358: >> >> >> $ oc debug pod/logging-fluentd-80xzt -- cat /proc/self/attr/current >> Debugging with pod/debug-logging-fluentd-80xzt, original command: <image >> entrypoint> >> Waiting for pod to start ... >> system_u:system_r:svirt_lxc_net_t:s0:c216,c576 >> >> Removing debug pod ... >> >> >> Yup. The problem was what I thought: it's being run under the >> svirt_lsc_net_t SELinux type, which doesn't have access to var_log_t. If >> you don't want to disable SELinux, you'll need to follow the instructions >> for creating a new SELinux type that I posted above. >> >> So I understand what's wrong but I don't see why the workaround (changing >> the service account permissions from anyuid to privileged) isn't working >> for me + I don't want to create a new selinuxtype. >> > > Sorry about that, we had missed a step. You'll need to delete your > daemonset, edit your logging-fluentd-template to add a property to your > container spec and recreate your daemonset to let it properly run as > privileged to escape the SELinux enforcing. > > $ oc delete daemonset logging-fluentd > > $ oc edit template/logging-fluentd-template > > > # Please edit the object below. Lines beginning with a '#' will be ignored, > # and an empty file will abort the edit. If an error occurs while saving > this file will be > # reopened with the relevant failures. > # > apiVersion: v1 > kind: Template > labels: > component: fluentd > . . . > objects: > - apiVersion: extensions/v1beta1 > kind: DaemonSet > . . . > spec: > selector: > matchLabels: > component: fluentd > provider: openshift > template: > metadata: > labels: > component: fluentd > provider: openshift > name: fluentd-elasticsearch > spec: > containers: > . . . > name: fluentd-elasticsearch > > # insert below here > securityContext: > privileged: true > # insert above here > > resources: > limits: > cpu: 100m > . . . > > $ oc process logging-fluentd-template | oc create -f - > > >> ------------------------------ >> From: [email protected] >> To: [email protected] >> CC: [email protected] >> Subject: RE: Aggregating container logs using Kibana >> Date: Wed, 13 Apr 2016 09:30:48 +0200 >> >> >> Fixed the issue with nodeselectormismatching: >> So now I have 3 fluentd pods on my 2 normal nodes and my infranode: >> But still the same permission issue: >> NAME READY STATUS RESTARTS AGE >> logging-curator-1-j7mz0 1/1 Running 0 17m >> logging-deployer-39qcz 0/1 Completed 0 47m >> logging-es-605u5g7g-1-36owl 1/1 Running 0 17m >> logging-fluentd-4uqx1 1/1 Running 0 46m >> logging-fluentd-dez5r 1/1 Running 0 2m >> logging-fluentd-m50nj 1/1 Running 0 46m >> logging-kibana-1-wfog2 2/2 Running 0 16m >> >> ------------------------------ >> From: [email protected] >> To: [email protected] >> CC: [email protected] >> Subject: RE: Aggregating container logs using Kibana >> Date: Wed, 13 Apr 2016 09:21:47 +0200 >> >> Hi Eric, >> >> Thanks for your reply and the follow up of this issue. >> I've created a new origin 1.1.6 cluster (2 days ago) but still have the >> same issue: >> My environment is one master (with node) non schedulable, 2 'normal' >> nodes and one infra node. >> I still got the permission denied (The documentation is up to date so I >> even don't had to perform the workaround manually). >> - system:serviceaccount:logging:aggregated-logging-fluentd is in scc >> privileged by default. >> >> The logging-deployer-template creates services and 2 pods of fluentd (on >> the normal nodes). >> The pods appear after performing this command: >> >> oc label nodes --all logging-infra-fluentd=true >> >> So my nodes got that label. also the unschedulable node on my master. So >> that's normal that it failed but why it fails on my infra-node I don't >> know. (I defined in my master-config that projects are by default on the >> other 2 nodes, maybe that's why but I don't know it's relevant for my >> issue). >> I also don't really understand why 'oc process logging-support-tempalte | >> oc create -f -' is only be cited at the troubleshooting part. >> Still the error: [error]: unexpected error error_class=Errno::EACCES >> error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos> >> >> oc get is >> NAME DOCKER >> REPO TAGS UPDATED >> logging-auth-proxy docker.io/openshift/origin-logging-auth-proxy >> latest,v0.0.1 4 minutes ago >> logging-curator docker.io/openshift/origin-logging-curator >> latest 4 minutes ago >> logging-elasticsearch docker.io/openshift/origin-logging-elasticsearch >> latest 4 minutes ago >> logging-fluentd docker.io/openshift/origin-logging-fluentd >> latest 4 minutes ago >> logging-kibana docker.io/openshift/origin-logging-kibana >> latest 4 minutes ago >> >> oc get svc >> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE >> logging-es 172.30.68.xx <none> 9200/TCP 33m >> logging-es-cluster None <none> 9300/TCP 33m >> logging-es-ops 172.30.18.xx <none> 9200/TCP 33m >> logging-es-ops-cluster None <none> 9300/TCP 33m >> logging-kibana 172.30.216.xx <none> 443/TCP 33m >> logging-kibana-ops 172.30.186.xx <none> 443/TCP 33m >> >> oc get pods >> NAME READY STATUS >> RESTARTS AGE >> logging-curator-1-j7mz0 1/1 Running >> 0 4m >> logging-deployer-39qcz 0/1 Completed >> 0 34m >> logging-es-605u5g7g-1-36owl 1/1 Running >> 0 4m >> logging-fluentd-4uqx1 1/1 Running >> 0 33m >> logging-fluentd-ex34j 0/1 NodeSelectorMismatching >> 0 33m >> logging-fluentd-injz7 0/1 NodeSelectorMismatching >> 0 33m >> logging-fluentd-m50nj 1/1 Running >> 0 33m >> logging-kibana-1-wfog2 2/2 Running >> 0 4m >> >> oc get daemonset >> NAME DESIRED CURRENT NODE-SELECTOR AGE >> logging-fluentd 4 4 logging-infra-fluentd=true 34m >> >> oc get dc >> NAME REVISION REPLICAS TRIGGERED BY >> logging-curator 1 1 >> config,image(logging-curator:latest) >> logging-es-605u5g7g 1 1 >> config,image(logging-elasticsearch:latest) >> logging-kibana 1 1 >> config,image(logging-auth-proxy:latest),image(logging-kibana:latest) >> >> oc get routes >> [centos@ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops) >> NAME HOST/PORT PATH SERVICE >> TERMINATION LABELS >> kibana kibana.test.xxx.eu logging-kibana >> passthrough component=support,logging-infra=support,provider=openshift >> kibana-ops kibana-ops.example.com logging-kibana-ops >> passthrough component=support,logging-infra=support,provider=openshift >> >> oc get oauthclient >> NAME >> SECRET >> WWW-CHALLENGE REDIRECT URIS >> kibana-proxy >> j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX FALSE >> https://kibana.xxx.eu,https://kibana-ops.example.com >> openshift-browser-client >> 71724303-b823-4435-8568-bcafxxxx4 >> FALSE >> https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display >> openshift-challenging-client ac7c9942-9a55-4e1e-8e5f-9fxxxxx >> TRUE >> https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit >> openshift-web-console 6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx >> FALSE >> https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000 >> >> >> >> >> >> >> >> ------------------------------ >> From: [email protected] >> Date: Tue, 12 Apr 2016 17:27:06 -0500 >> Subject: Re: Aggregating container logs using Kibana >> To: [email protected] >> CC: [email protected]; [email protected] >> >> >> >> On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo < >> [email protected]> wrote: >> >> This are all the steps I'm performing: >> >> oc new-project logging >> >> $ oc secrets new logging-deployer nothing=/dev/null >> >> $ oc process logging-deployer-account-template -n openshift \ >> | oc create -f - >> >> $ oc policy add-role-to-user edit --serviceaccount logging-deployer >> $ oc policy add-role-to-user daemonset-admin --serviceaccount >> logging-deployer >> $ oadm policy add-cluster-role-to-user oauth-editor \ >> system:serviceaccount:logging:logging-deployer >> >> $ oadm policy add-scc-to-user \ >> privileged system:serviceaccount:logging:aggregated-logging-fluentd >> >> $ oadm policy add-cluster-role-to-user cluster-reader \ >> system:serviceaccount:logging:aggregated-logging-fluentd >> >> Than I execute the deployer template: >> >> $ oc process logging-deployer-template -n openshift \ >> -v >> KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443 >> \ >> | oc create -f - >> >> This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on >> master machine) and some empty services (the logs of the pods are telling me >> the permission error) >> When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's >> all fine. >> >> $ oc label nodes --all logging-infra-fluentd=true >> >> I've edited */master/master-config.yaml* + restart >> $ oc scale dc/logging-kibana --replicas=2 >> >> >> $ oc delete oauthclient/kibana-proxy >> $ oc process logging-support-template | oc create -f - >> >> The last step creates also some pods. It's a bit weird for me that this step >> is only mentioned for troubleshooting or is >> it an issue that I don't have those pods after executing the >> deployer-template? >> >> The template 'logging-support-template' creates your ImageStreams (along >> with your routes and oauthclient) so it shouldn't be creating your pods. >> There may have been a delay in scheduling your pods initially or the image >> stream tags could have been in the processes of being fetched. >> >> What does the following output? >> oc get is, svc, pods, daemonset, dc, routes, oauthclient -n logging >> >> And do you still see the same permission denied errors in the Fluentd >> logs? >> >> >> >> ------------------------------ >> From: [email protected] >> To: [email protected] >> CC: [email protected] >> Subject: RE: Aggregating container logs using Kibana >> Date: Tue, 5 Apr 2016 18:00:02 +0200 >> >> >> I still have the same issue: >> >> I've deleted it from scc hostmount-anyuid and added it on scc privileged. >> I've deleted all fluentd pods but still the same issue. Even after >> recreating the project. >> >> ------------------------------ >> From: [email protected] >> Date: Tue, 5 Apr 2016 10:29:04 -0400 >> Subject: Re: Aggregating container logs using Kibana >> To: [email protected] >> CC: [email protected] >> >> >> >> On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <[email protected]> wrote: >> >> >> 2016-04-05 10:55:13 +0000 [error]: unexpected error >> error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - >> /var/log/es-containers.log.pos> >> >> >> This looks like >> https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps >> fluentd from reading any logs on the node. >> >> You should be able to resolve this by adding the fluentd service account >> to the privileged SCC, then having fluentd restart everywhere. >> >> oadm policy add-scc-to-user privileged >> system:serviceaccount:logging:aggregated-logging-fluentd >> >> >> Oh; probably need to also remove them from the hostmount-anyuid SCC. >> >> >> _______________________________________________ >> users mailing list >> [email protected] >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >>
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
