Is there more info on this? I'm having this problem one OCP 3.7 right now too. If a node is rebooted, it comes back up but is missing the logging-infra-fluentd=true label.
On Thu, Dec 21, 2017 at 10:15 AM, Clayton Coleman <[email protected]> wrote: > There was an open bug on this previously - I’m having trouble finding it > at the moment. The node may be racing with the cloud controller and then > not updating the labels. One workaround is to simply add an “oc label > node/$(hostname) ...” command to the origin-node services as a prestart > command. > > On Dec 21, 2017, at 9:13 AM, Ernst, Chad <[email protected]> wrote: > > > > Running Origin 3.6 on AWS, we’ve found that if our EC2 instances go down > for any length of time and come back up (as opposed to the EC2 instance > getting terminated) the nodes are automatically deleted from openshift then > re-registered after the ec2 instance is restarted. The activity is logged > in /var/log/messages > > > > Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 > 21:59:30.297638 26242 nodecontroller.go:761] Deleting node (no longer > present in cloud provider): ip-172-21-20-30.ec2.internal > > Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 > 21:59:30.297662 26242 controller_utils.go:273] Recording Deleting Node > ip-172-21-20-30.ec2.internal because it's not present according to cloud > provider event message for node ip-172-21-20-30.ec2.internal > > Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 > 21:59:30.297895 26242 event.go:217] Event(v1.ObjectReference{Kind:"Node", > Namespace:"", Name:"ip-172-21-20-30.ec2.internal", > UID:"36c8dca4-e5c9-11e7-b2ce-0e69b80c212e", APIVersion:"", > ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeletingNode' > Node ip-172-21-20-30.ec2.internal event: Deleting Node > ip-172-21-20-30.ec2.internal because it's not present according to cloud > provider > > > > > > Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 > 23:10:06.303567 26242 nodecontroller.go:616] NodeController observed a > new Node: "ip-172-21-22-30.ec2.internal" > > Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 > 23:10:06.303597 26242 controller_utils.go:273] Recording Registered Node > ip-172-21-22-30.ec2.internal in NodeController event message for node > ip-172-21-22-30.ec2.internal > > Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 > 23:10:06.303899 26242 event.go:217] Event(v1.ObjectReference{Kind:"Node", > Namespace:"", Name:"ip-172-21-22-30.ec2.internal", > UID:"e850129f-e5da-11e7-ac5e-027542a418ee", APIVersion:"", > ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' > Node ip-172-21-22-30.ec2.internal event: Registered Node > ip-172-21-22-30.ec2.internal in NodeController > > > > The issue we are running into is that when the nodes come back they don’t > have all of our labels on them. They don’t get labelled to run the fluentd > pods “logging-infra-fluentd=true” and my masters aren’t set for “Scheduling > Disabled”. > > > > Can anybody point me to the any doc regarding the automatic registration > of the node from the cloudprovider or knows how to adjust the behavior when > a node is re-registered so they can be tagged properly. > > > > Thanks > > > > Chad > > ######################################################################## > The information contained in this message, and any attachments thereto, > is intended solely for the use of the addressee(s) and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination, copying, or other use of the transmitted information is > prohibited. If you received this in error, please contact the sender > and delete the material from any computer. UNIGROUP.COM > ######################################################################## > > _______________________________________________ > users mailing list > [email protected] > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > > _______________________________________________ > users mailing list > [email protected] > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > -- Mark McKinstry Senior Consultant , RHCA Red Hat Consulting (West) [email protected] M: 510-646-1280
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
