Hello Lorenz, how did you fix the issue with NodeSelectorsMismatching? Having the same error on a 1.1.4 cluster.
Regards v Am 2016-04-13 um 09:30 schrieb Lorenz Vanthillo:
Fixed the issue with nodeselectormismatching: So now I have 3 fluentd pods on my 2 normal nodes and my infranode: But still the same permission issue: NAME READY STATUS RESTARTS AGE logging-curator-1-j7mz0 1/1 Running 0 17m logging-deployer-39qcz 0/1 Completed 0 47m logging-es-605u5g7g-1-36owl 1/1 Running 0 17m logging-fluentd-4uqx1 1/1 Running 0 46m logging-fluentd-dez5r 1/1 Running 0 2m logging-fluentd-m50nj 1/1 Running 0 46m logging-kibana-1-wfog2 2/2 Running 0 16m ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From: [email protected] To: [email protected] CC: [email protected] Subject: RE: Aggregating container logs using Kibana Date: Wed, 13 Apr 2016 09:21:47 +0200 Hi Eric, Thanks for your reply and the follow up of this issue. I've created a new origin 1.1.6 cluster (2 days ago) but still have the same issue: My environment is one master (with node) non schedulable, 2 'normal' nodes and one infra node. I still got the permission denied (The documentation is up to date so I even don't had to perform the workaround manually). - system:serviceaccount:logging:aggregated-logging-fluentd is in scc privileged by default. The logging-deployer-template creates services and 2 pods of fluentd (on the normal nodes). The pods appear after performing this command: oc label nodes --all logging-infra-fluentd=true So my nodes got that label. also the unschedulable node on my master. So that's normal that it failed but why it fails on my infra-node I don't know. (I defined in my master-config that projects are by default on the other 2 nodes, maybe that's why but I don't know it's relevant for my issue). I also don't really understand why 'oc process logging-support-tempalte | oc create -f -' is only be cited at the troubleshooting part. Still the error: [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos> oc get is NAME DOCKER REPO TAGS UPDATED logging-auth-proxy docker.io/openshift/origin-logging-auth-proxy latest,v0.0.1 4 minutes ago logging-curator docker.io/openshift/origin-logging-curator latest 4 minutes ago logging-elasticsearch docker.io/openshift/origin-logging-elasticsearch latest 4 minutes ago logging-fluentd docker.io/openshift/origin-logging-fluentd latest 4 minutes ago logging-kibana docker.io/openshift/origin-logging-kibana latest 4 minutes ago oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es 172.30.68.xx <none> 9200/TCP 33m logging-es-cluster None <none> 9300/TCP 33m logging-es-ops 172.30.18.xx <none> 9200/TCP 33m logging-es-ops-cluster None <none> 9300/TCP 33m logging-kibana 172.30.216.xx <none> 443/TCP 33m logging-kibana-ops 172.30.186.xx <none> 443/TCP 33m oc get pods NAME READY STATUS RESTARTS AGE logging-curator-1-j7mz0 1/1 Running 0 4m logging-deployer-39qcz 0/1 Completed 0 34m logging-es-605u5g7g-1-36owl 1/1 Running 0 4m logging-fluentd-4uqx1 1/1 Running 0 33m logging-fluentd-ex34j 0/1 NodeSelectorMismatching 0 33m logging-fluentd-injz7 0/1 NodeSelectorMismatching 0 33m logging-fluentd-m50nj 1/1 Running 0 33m logging-kibana-1-wfog2 2/2 Running 0 4m oc get daemonset NAME DESIRED CURRENT NODE-SELECTOR AGE logging-fluentd 4 4 logging-infra-fluentd=true 34m oc get dc NAME REVISION REPLICAS TRIGGERED BY logging-curator 1 1 config,image(logging-curator:latest) logging-es-605u5g7g 1 1 config,image(logging-elasticsearch:latest) logging-kibana 1 1 config,image(logging-auth-proxy:latest),image(logging-kibana:latest) oc get routes [centos@ip-172-29-20-200 ~]$ oc get routes (don't use kibana-ops) NAME HOST/PORT PATH SERVICE TERMINATION LABELS kibana kibana.test.xxx.eu logging-kibana passthrough component=support,logging-infra=support,provider=openshift kibana-ops kibana-ops.example.com logging-kibana-ops passthrough component=support,logging-infra=support,provider=openshift oc get oauthclient NAME SECRET WWW-CHALLENGE REDIRECT URIS kibana-proxy j8AUaLABCLaAOSw5Iun2DeRqeDbZtRWzXBzT7NXoxZlWs1m49PXXXXXX FALSE https://kibana.xxx.eu,https://kibana-ops.example.com openshift-browser-client 71724303-b823-4435-8568-bcafxxxx4 FALSE https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/display openshift-challenging-client ac7c9942-9a55-4e1e-8e5f-9fxxxxx TRUE https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/oauth/token/implicit openshift-web-console 6a7e9ff6-0c1b-4888-9d17-5e16xxxxxx FALSE https://ec2-xx-xx-xx-xx.xx-xx-1.compute.amazonaws.com:8443/console/,http://localhost:9000,https://localhost:9000 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From: [email protected] Date: Tue, 12 Apr 2016 17:27:06 -0500 Subject: Re: Aggregating container logs using Kibana To: [email protected] CC: [email protected]; [email protected] On Tue, Apr 5, 2016 at 11:50 AM, Lorenz Vanthillo <[email protected] <mailto:[email protected]>> wrote: This are all the steps I'm performing: oc new-project logging $ oc secrets new logging-deployer nothing=/dev/null $ oc process logging-deployer-account-template -n openshift \ | oc create -f - $ oc policy add-role-to-user edit --serviceaccount logging-deployer $ oc policy add-role-to-user daemonset-admin --serviceaccount logging-deployer $ oadm policy add-cluster-role-to-user oauth-editor \ system:serviceaccount:logging:logging-deployer $ oadm policy add-scc-to-user \ privileged system:serviceaccount:logging:aggregated-logging-fluentd $ oadm policy add-cluster-role-to-user cluster-reader \ system:serviceaccount:logging:aggregated-logging-fluentd Than I execute the deployer template: $ oc process logging-deployer-template -n openshift \ -v KIBANA_HOSTNAME=kibana.example.com <http://kibana.example.com>,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443 \ | oc create -f - This creates 3 logging-fluentd pods (I have 3 nodes, 1 unschedulable on master machine) and some empty services (the logs of the pods are telling me the permission error) When I check oc edit scc privileged and oc edit scc hostmount-anyuid it's all fine. $ oc label nodes --all logging-infra-fluentd=true I've edited *//master/master-config.yaml/* + restart $ oc scale dc/logging-kibana --replicas=2 $ oc delete oauthclient/kibana-proxy $ oc process logging-support-template | oc create -f - The last step creates also some pods. It's a bit weird for me that this step is only mentioned for troubleshooting or is it an issue that I don't have those pods after executing the deployer-template? The template 'logging-support-template' creates your ImageStreams (along with your routes and oauthclient) so it shouldn't be creating your pods. There may have been a delay in scheduling your pods initially or the image stream tags could have been in the processes of being fetched. What does the following output? oc get is, svc, pods, daemonset, dc, routes, oauthclient -n logging And do you still see the same permission denied errors in the Fluentd logs? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From: [email protected] <mailto:[email protected]> To: [email protected] <mailto:[email protected]> CC: [email protected] <mailto:[email protected]> Subject: RE: Aggregating container logs using Kibana Date: Tue, 5 Apr 2016 18:00:02 +0200 I still have the same issue: I've deleted it from scc hostmount-anyuid and added it on scc privileged. I've deleted all fluentd pods but still the same issue. Even after recreating the project. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From: [email protected] <mailto:[email protected]> Date: Tue, 5 Apr 2016 10:29:04 -0400 Subject: Re: Aggregating container logs using Kibana To: [email protected] <mailto:[email protected]> CC: [email protected] <mailto:[email protected]> On Tue, Apr 5, 2016 at 10:26 AM, Luke Meyer <[email protected] <mailto:[email protected]>> wrote: 2016-04-05 10:55:13 +0000 [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos> This looks like https://github.com/openshift/origin-aggregated-logging/issues/89 - keeps fluentd from reading any logs on the node. You should be able to resolve this by adding the fluentd service account to the privileged SCC, then having fluentd restart everywhere. |oadm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd| Oh; probably need to also remove them from the hostmount-anyuid SCC. _______________________________________________ users mailing list [email protected] <mailto:[email protected]> http://lists.openshift.redhat.com/openshiftmm/listinfo/users _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
