Re: Looking for documentation on cloud provider delete node and register node

[Clayton Coleman]

There was an open bug on this previously - I’m having trouble finding it at
the moment.  The node may be racing with the cloud controller and then not
updating the labels.  One workaround is to simply add an “oc label
node/$(hostname) ...” command to the origin-node services as a prestart
command.

On Dec 21, 2017, at 9:13 AM, Ernst, Chad  wrote:



Running Origin 3.6 on AWS, we’ve found that if our EC2 instances go down
for any length of time and come back up (as opposed to the EC2 instance
getting terminated) the nodes are automatically deleted from openshift then
re-registered after the ec2 instance is restarted.  The activity is logged
in /var/log/messages



Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220
21:59:30.297638   26242 nodecontroller.go:761] Deleting node (no longer
present in cloud provider): ip-172-21-20-30.ec2.internal

Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220
21:59:30.297662   26242 controller_utils.go:273] Recording Deleting Node
ip-172-21-20-30.ec2.internal because it's not present according to cloud
provider event message for node ip-172-21-20-30.ec2.internal

Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220
21:59:30.297895   26242 event.go:217] Event(v1.ObjectReference{Kind:"Node",
Namespace:"", Name:"ip-172-21-20-30.ec2.internal",
UID:"36c8dca4-e5c9-11e7-b2ce-0e69b80c212e", APIVersion:"",
ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DeletingNode'
Node ip-172-21-20-30.ec2.internal event: Deleting Node
ip-172-21-20-30.ec2.internal because it's not present according to cloud
provider





Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220
23:10:06.303567   26242 nodecontroller.go:616] NodeController observed a
new Node: "ip-172-21-22-30.ec2.internal"

Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220
23:10:06.303597   26242 controller_utils.go:273] Recording Registered Node
ip-172-21-22-30.ec2.internal in NodeController event message for node
ip-172-21-22-30.ec2.internal

Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220
23:10:06.303899   26242 event.go:217] Event(v1.ObjectReference{Kind:"Node",
Namespace:"", Name:"ip-172-21-22-30.ec2.internal",
UID:"e850129f-e5da-11e7-ac5e-027542a418ee", APIVersion:"",
ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode'
Node ip-172-21-22-30.ec2.internal event: Registered Node
ip-172-21-22-30.ec2.internal in NodeController



The issue we are running into is that when the nodes come back they don’t
have all of our labels on them.  They don’t get labelled to run the fluentd
pods “logging-infra-fluentd=true” and my masters aren’t set for “Scheduling
Disabled”.



Can anybody point me to the any doc regarding the automatic registration of
the node from the cloudprovider or knows how to adjust the behavior when a
node is re-registered so they can be tagged properly.



Thanks



Chad


The information contained in this message, and any attachments thereto,
is intended solely for the use of the addressee(s) and may contain
confidential and/or privileged material. Any review, retransmission,
dissemination, copying, or other use of the transmitted information is
prohibited. If you received this in error, please contact the sender
and delete the material from any computer. UNIGROUP.COM


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Looking for documentation on cloud provider delete node and register node

[Ernst, Chad]

Running Origin 3.6 on AWS, we’ve found that if our EC2 instances go down for 
any length of time and come back up (as opposed to the EC2 instance getting 
terminated) the nodes are automatically deleted from openshift then 
re-registered after the ec2 instance is restarted.  The activity is logged in 
/var/log/messages

Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 
21:59:30.297638   26242 nodecontroller.go:761] Deleting node (no longer present 
in cloud provider): ip-172-21-20-30.ec2.internal
Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 
21:59:30.297662   26242 controller_utils.go:273] Recording Deleting Node 
ip-172-21-20-30.ec2.internal because it's not present according to cloud 
provider event message for node ip-172-21-20-30.ec2.internal
Dec 20 21:59:30 ip-172-21-21-30 origin-master-controllers: I1220 
21:59:30.297895   26242 event.go:217] Event(v1.ObjectReference{Kind:"Node", 
Namespace:"", Name:"ip-172-21-20-30.ec2.internal", 
UID:"36c8dca4-e5c9-11e7-b2ce-0e69b80c212e", APIVersion:"", ResourceVersion:"", 
FieldPath:""}): type: 'Normal' reason: 'DeletingNode' Node 
ip-172-21-20-30.ec2.internal event: Deleting Node ip-172-21-20-30.ec2.internal 
because it's not present according to cloud provider


Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 
23:10:06.303567   26242 nodecontroller.go:616] NodeController observed a new 
Node: "ip-172-21-22-30.ec2.internal"
Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 
23:10:06.303597   26242 controller_utils.go:273] Recording Registered Node 
ip-172-21-22-30.ec2.internal in NodeController event message for node 
ip-172-21-22-30.ec2.internal
Dec 20 23:10:06 ip-172-21-21-30 origin-master-controllers: I1220 
23:10:06.303899   26242 event.go:217] Event(v1.ObjectReference{Kind:"Node", 
Namespace:"", Name:"ip-172-21-22-30.ec2.internal", 
UID:"e850129f-e5da-11e7-ac5e-027542a418ee", APIVersion:"", ResourceVersion:"", 
FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node 
ip-172-21-22-30.ec2.internal event: Registered Node 
ip-172-21-22-30.ec2.internal in NodeController

The issue we are running into is that when the nodes come back they don’t have 
all of our labels on them.  They don’t get labelled to run the fluentd pods 
“logging-infra-fluentd=true” and my masters aren’t set for “Scheduling 
Disabled”.

Can anybody point me to the any doc regarding the automatic registration of the 
node from the cloudprovider or knows how to adjust the behavior when a node is 
re-registered so they can be tagged properly.

Thanks

Chad


The information contained in this message, and any attachments thereto,
is intended solely for the use of the addressee(s) and may contain
confidential and/or privileged material. Any review, retransmission,
dissemination, copying, or other use of the transmitted information is
prohibited. If you received this in error, please contact the sender
and delete the material from any computer. UNIGROUP.COM


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users