Yes, I tried it with the newest images and just following the documentation and it's working fine! Thanks
From: [email protected] Date: Thu, 14 Apr 2016 15:03:40 -0500 Subject: Re: Aggregating container logs using Kibana To: [email protected] CC: [email protected] Just a heads up, the latest deployer image on Dockerhub has an updated fluentd template that already contains the change for Fluentd to run in the privileged security context. On Wed, Apr 13, 2016 at 11:24 AM, Eric Wolinetz <[email protected]> wrote: On Wed, Apr 13, 2016 at 3:16 AM, Lorenz Vanthillo <[email protected]> wrote: I saw on https://github.com/openshift/origin/issues/8358: $ oc debug pod/logging-fluentd-80xzt -- cat /proc/self/attr/current Debugging with pod/debug-logging-fluentd-80xzt, original command: <image entrypoint> Waiting for pod to start ... system_u:system_r:svirt_lxc_net_t:s0:c216,c576 Removing debug pod ... Yup. The problem was what I thought: it's being run under the svirt_lsc_net_t SELinux type, which doesn't have access to var_log_t. If you don't want to disable SELinux, you'll need to follow the instructions for creating a new SELinux type that I posted above. So I understand what's wrong but I don't see why the workaround (changing the service account permissions from anyuid to privileged) isn't working for me + I don't want to create a new selinuxtype. Sorry about that, we had missed a step. You'll need to delete your daemonset, edit your logging-fluentd-template to add a property to your container spec and recreate your daemonset to let it properly run as privileged to escape the SELinux enforcing. $ oc delete daemonset logging-fluentd $ oc edit template/logging-fluentd-template # Please edit the object below. Lines beginning with a '#' will be ignored,# and an empty file will abort the edit. If an error occurs while saving this file will be# reopened with the relevant failures.#apiVersion: v1kind: Templatelabels: component: fluentd. . .objects:- apiVersion: extensions/v1beta1 kind: DaemonSet. . .spec: selector: matchLabels: component: fluentd provider: openshift template: metadata: labels: component: fluentd provider: openshift name: fluentd-elasticsearch spec: containers:. . . name: fluentd-elasticsearch # insert below here securityContext: privileged: true # insert above here resources: limits: cpu: 100m. . . $ oc process logging-fluentd-template | oc create -f - From: [email protected] To: [email protected] CC: [email protected] Subject: RE: Aggregating container logs using Kibana Date: Wed, 13 Apr 2016 09:30:48 +0200 Fixed the issue with nodeselectormismatching: So now I have 3 fluentd pods on my 2 normal nodes and my infranode: But still the same permission issue: NAME READY STATUS RESTARTS AGE logging-curator-1-j7mz0 1/1 Running 0 17m logging-deployer-39qcz 0/1 Completed 0 47m logging-es-605u5g7g-1-36owl 1/1 Running 0 17m logging-fluentd-4uqx1 1/1 Running 0 46m logging-fluentd-dez5r 1/1 Running 0 2m logging-fluentd-m50nj 1/1 Running 0 46m logging-kibana-1-wfog2 2/2 Running 0 16m From: [email protected] To: [email protected] CC: [email protected] Subject: RE: Aggregating container logs using Kibana Date: Wed, 13 Apr 2016 09:21:47 +0200 Hi Eric, Thanks for your reply and the follow up of this issue. I've created a new origin 1.1.6 cluster (2 days ago) but still have the same issue: My environment is one master (with node) non schedulable, 2 'normal' nodes and one infra node. I still got the permission denied (The documentation is up to date so I even don't had to perform the workaround manually). - system:serviceaccount:logging:aggregated-logging-fluentd is in scc privileged by default. The logging-deployer-template creates services and 2 pods of fluentd (on the normal nodes). The pods appear after performing this command: oc label nodes --all logging-infra-fluentd=trueSo my nodes got that label. also the unschedulable node on my master. So that's normal that it failed but why it fails on my infra-node I don't know. (I defined in my master-config that projects are by default on the other 2 nodes, maybe that's why but I don't know it's relevant for my issue). I also don't really understand why 'oc process logging-support-tempalte | oc create -f -' is only be cited at the troubleshooting part. Still the error: [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos> oc get is NAME DOCKER REPO TAGS UPDATED logging-auth-proxy docker.io/openshift/origin-logging-auth-proxy latest,v0.0.1 4 minutes ago logging-curator docker.io/openshift/origin-logging-curator latest 4 minutes ago logging-elasticsearch docker.io/openshift/origin-logging-elasticsearch latest 4 minutes ago logging-fluentd docker.io/openshift/origin-logging-fluentd latest 4 minutes ago logging-kibana docker.io/openshift/origin-logging-kibana latest 4 minutes ago oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es 172.30.68.xx <none> 9200/TCP 33m logging-es-cluster None <none> 9300/TCP 33m logging-es-ops 172.30.18.xx <none> 9200/TCP 33m logging-es-ops-cluster None <none> 9300/TCP 33m logging-kibana 172.30.216.xx <none> 443/TCP 33m logging-kibana-ops 172.30.186.xx <none> 443/TCP 33m oc get pods NAME READY STATUS RESTARTS AGE logging-curator-1-j7mz0 1/1 Running 0 4m logging-deployer-39qcz 0/1 Completed 0 34m logging-es-605u5g7g-1-36owl 1/1 Running 0 4m logging-fluentd-4uqx1 1/1 Running 0 33m logging-fluentd-ex34j 0/1 NodeSelectorMismatching 0 33m logging-fluentd-injz7 0/1 NodeSelectorMismatching 0 33m logging-fluentd-m50nj 1/1 Running 0 33m logging-kibana-1-wfog2 2/2 Running 0 4m oc get daemonset NAME DESIRED CURRENT NODE-SELECTOR AGE logging-fluentd 4 4 logging-infra-fluentd=true 34m oc get dc NAME REVISION REPLICAS TRIGGERED BY logging-curator 1 1 config,image(logging-curator:latest) logging-es-605u5g7g 1 1 config,image(logging-elasticsearch:latest) logging-kibana 1 1 config,image(logging-auth-proxy:latest),image(logging-kibana:latest) oc get routes [centos@ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops) NAME HOST/PORT PATH SERVICE TERMINATION LABELS kibana kibana.test.xxx.eu logging-kibana passthrough component=support,logging-infra=support,provider=openshift kibana-ops kibana-ops.example.com logging-kibana-ops passthrough component=support,logging-infra=support,provider=openshift oc get oauthclient NAME SECRET WWW-CHALLENGE REDIRECT URIS kibana-proxy j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX FALSE https://kibana.xxx.eu,https://kibana-ops.example.com openshift-browser-client 71724303-b823-4435-8568-bcafxxxx4 FALSE https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display openshift-challenging-client ac7c9942-9a55-4e1e-8e5f-9fxxxxx TRUE https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit openshift-web-console 6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx FALSE https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000 From: [email protected] Date: Tue, 12 Apr 2016 17:27:06 -0500 Subject: Re: Aggregating container logs using Kibana To: [email protected] CC: [email protected]; [email protected] On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo <[email protected]> wrote: This are all the steps I'm performing: oc new-project logging $ oc secrets new logging-deployer nothing=/dev/null $ oc process logging-deployer-account-template -n openshift \ | oc create -f - $ oc policy add-role-to-user edit --serviceaccount logging-deployer $ oc policy add-role-to-user daemonset-admin --serviceaccount logging-deployer $ oadm policy add-cluster-role-to-user oauth-editor \ system:serviceaccount:logging:logging-deployer $ oadm policy add-scc-to-user \ privileged system:serviceaccount:logging:aggregated-logging-fluentd $ oadm policy add-cluster-role-to-user cluster-reader \ system:serviceaccount:logging:aggregated-logging-fluentd Than I execute the deployer template: $ oc process logging-deployer-template -n openshift \ -v KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443 \ | oc create -f - This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on master machine) and some empty services (the logs of the pods are telling me the permission error) When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's all fine. $ oc label nodes --all logging-infra-fluentd=true I've edited /master/master-config.yaml + restart $ oc scale dc/logging-kibana --replicas=2 $ oc delete oauthclient/kibana-proxy $ oc process logging-support-template | oc create -f - The last step creates also some pods. It's a bit weird for me that this step is only mentioned for troubleshooting or is it an issue that I don't have those pods after executing the deployer-template? The template 'logging-support-template' creates your ImageStreams (along with your routes and oauthclient) so it shouldn't be creating your pods. There may have been a delay in scheduling your pods initially or the image stream tags could have been in the processes of being fetched. What does the following output?oc get is, svc, pods, daemonset, dc, routes, oauthclient -n logging And do you still see the same permission denied errors in the Fluentd logs? From: [email protected] To: [email protected] CC: [email protected] Subject: RE: Aggregating container logs using Kibana Date: Tue, 5 Apr 2016 18:00:02 +0200 I still have the same issue: I've deleted it from scc hostmount-anyuid and added it on scc privileged. I've deleted all fluentd pods but still the same issue. Even after recreating the project. From: [email protected] Date: Tue, 5 Apr 2016 10:29:04 -0400 Subject: Re: Aggregating container logs using Kibana To: [email protected] CC: [email protected] On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <[email protected]> wrote: 2016-04-05 10:55:13 +0000 [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos> This looks like https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps fluentd from reading any logs on the node. You should be able to resolve this by adding the fluentd service account to the privileged SCC, then having fluentd restart everywhere. oadm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd Oh; probably need to also remove them from the hostmount-anyuid SCC. _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
