There was an open bug on this previously - I’m having trouble finding it at the moment. The node may be racing with the cloud controller and then not updating the labels. One workaround is to simply add an “oc label node/$(hostname) ...” command to the origin-node services as a prestart command.
On Dec 21, 2017, at 9:13 AM, Ernst, Chad <[email protected]> wrote: Running Origin 3.6 on AWS, we’ve found that if our EC2 instances go down for any length of time and come back up (as opposed to the EC2 instance getting terminated) the nodes are automatically deleted from openshift then re-registered after the ec2 instance is restarted. The activity is logged in /var/log/messages Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 21:59:30.297638 26242 nodecontroller.go:761] Deleting node (no longer present in cloud provider): ip-172-21-20-30.ec2.internal Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 21:59:30.297662 26242 controller_utils.go:273] Recording Deleting Node ip-172-21-20-30.ec2.internal because it's not present according to cloud provider event message for node ip-172-21-20-30.ec2.internal Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 21:59:30.297895 26242 event.go:217] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-172-21-20-30.ec2.internal", UID:"36c8dca4-e5c9-11e7-b2ce-0e69b80c212e", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeletingNode' Node ip-172-21-20-30.ec2.internal event: Deleting Node ip-172-21-20-30.ec2.internal because it's not present according to cloud provider Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 23:10:06.303567 26242 nodecontroller.go:616] NodeController observed a new Node: "ip-172-21-22-30.ec2.internal" Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 23:10:06.303597 26242 controller_utils.go:273] Recording Registered Node ip-172-21-22-30.ec2.internal in NodeController event message for node ip-172-21-22-30.ec2.internal Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 23:10:06.303899 26242 event.go:217] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-172-21-22-30.ec2.internal", UID:"e850129f-e5da-11e7-ac5e-027542a418ee", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ip-172-21-22-30.ec2.internal event: Registered Node ip-172-21-22-30.ec2.internal in NodeController The issue we are running into is that when the nodes come back they don’t have all of our labels on them. They don’t get labelled to run the fluentd pods “logging-infra-fluentd=true” and my masters aren’t set for “Scheduling Disabled”. Can anybody point me to the any doc regarding the automatic registration of the node from the cloudprovider or knows how to adjust the behavior when a node is re-registered so they can be tagged properly. Thanks Chad ######################################################################## The information contained in this message, and any attachments thereto, is intended solely for the use of the addressee(s) and may contain confidential and/or privileged material. Any review, retransmission, dissemination, copying, or other use of the transmitted information is prohibited. If you received this in error, please contact the sender and delete the material from any computer. UNIGROUP.COM ######################################################################## _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
