Zack,
I've tried to reproduce this multiple ways, but was unable to do so
(yet), but I have a theory about what happened (which is difficult to
intentionally replicate).
Reviewing the stack trace, I noticed that the exception is occurring
after the actual deletion of the node, in a subsequent call to
cleanUpIncidentalResourcesOfDeadNodes(), which apparently attempts to
delete orphaned groups. I don't understand enough of the concept of a
"group" to know with certainty what's going on here, and my reading of
Rackspace documentation only implies "security groups" which I don't
have; I do, however, utilize their "RackConnect" system to connect my
cloud nodes to a managed server and it is possible that setup is
involved somehow. After my nodes are created, they are subsequently
switched over (by Rackspace using their scripts) to RackConnect.
I don't know what resources were being cleaned up, but it may be a
possibility that the following factors contributed:
- I had previously created a server named "Cloud-Server-35" which failed
to fully create (this happens about 1% of the time) and never got
switched over to the RackConnect group.
- I had issued the "delete" server command on the failed server, but the
node may have still been hanging around in the system; or at least, not
fully deleted by the time the second server got created.
- I had created another server named "Cloud-Server-35" to replace the
failed one. This one operated normally and was switched over to
RackConnect.
- When I went to delete the second server by IP, jclouds found leftover
"resources" associated with the original server, and attempted to clean
them up.
- Somehow in this cleanup, the code/API detected duplicate names/groups,
and attempted to rename one of the groups, encountering the exception
because the group name was mixed case.
Dan
On 8/11/14, 6:45 AM, Zack Shoylev wrote:
Hi Daniel,
Can this issue be reproduced consistently? Does the number of servers matter?
From your stack traces it seems this code is called
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/predicates/validators/DnsNameValidator.java#L56
so it could be some kind of weird interaction.
________________________________________
From: Daniel Widdis [[email protected]]
Sent: Saturday, August 09, 2014 1:57 PM
To:[email protected]
Subject: IllegalArgumentException when deleting node on Rackspace
This issue may be related, or may be completely different than Sunil
Shah's recent thread on 2-letter DNS names when creating GCE nodes, but
ultimately boils down to the same exception at the same line of code. I
debated whether to reply to that thread or start a new one, and elected
the latter. I will take a look at JIRA where the issues may possibly be
combined.
My specific symptoms/situation:
- jclouds 1.7.3
- I had manually created (through website API) several Rackspace cloud
servers, with the names Cloud-Server-30, Cloud-Server-31, etc. These
were the default names provided on the website and I chose (by virtue of
laziness) to leave them as they were.
- My code deletes these servers when it is done using them. I do this
by passing the private IP address of the server to this code:
nodes = computeService
.destroyNodesMatching(new Predicate<NodeMetadata>() {
@Override
public boolean apply(final NodeMetadata input) {
return input.getPrivateAddresses().contains(ipAddr);
}
});
- Note that my code never uses the server name, nor is even aware of
it. I simply want to tell Rackspace, "delete the server with this IP
address."
- This code worked fine for 34 out of 35 servers, 15 of which had
Cloud-Server-## names. However, on the very last server (named
Cloud-Server-35), passing the String ipAddr = "10.208.232.65" , I got
this exception:
java.lang.IllegalArgumentException: Object 'Cloud-Server' doesn't
match dns naming constraints. Reason: Should be only lowercase.
- Full stack trace here:http://pastebin.com/PKuMqkqz (ServerDeletor:110
is the destroyNodesMatching() call listed above.)
