RE: IllegalArgumentException when deleting node on Rackspace
Hi Daniel,
Can this issue be reproduced consistently? Does the number of servers matter?
From your stack traces it seems this code is called
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/predicates/validators/DnsNameValidator.java#L56
so it could be some kind of weird interaction.
From: Daniel Widdis [wid...@gmail.com]
Sent: Saturday, August 09, 2014 1:57 PM
To: user@jclouds.apache.org
Subject: IllegalArgumentException when deleting node on Rackspace
This issue may be related, or may be completely different than Sunil
Shah's recent thread on 2-letter DNS names when creating GCE nodes, but
ultimately boils down to the same exception at the same line of code. I
debated whether to reply to that thread or start a new one, and elected
the latter. I will take a look at JIRA where the issues may possibly be
combined.
My specific symptoms/situation:
- jclouds 1.7.3
- I had manually created (through website API) several Rackspace cloud
servers, with the names Cloud-Server-30, Cloud-Server-31, etc. These
were the default names provided on the website and I chose (by virtue of
laziness) to leave them as they were.
- My code deletes these servers when it is done using them. I do this
by passing the private IP address of the server to this code:
nodes = computeService
.destroyNodesMatching(new PredicateNodeMetadata() {
@Override
public boolean apply(final NodeMetadata input) {
return input.getPrivateAddresses().contains(ipAddr);
}
});
- Note that my code never uses the server name, nor is even aware of
it. I simply want to tell Rackspace, delete the server with this IP
address.
- This code worked fine for 34 out of 35 servers, 15 of which had
Cloud-Server-## names. However, on the very last server (named
Cloud-Server-35), passing the String ipAddr = 10.208.232.65 , I got
this exception:
java.lang.IllegalArgumentException: Object 'Cloud-Server' doesn't
match dns naming constraints. Reason: Should be only lowercase.
- Full stack trace here: http://pastebin.com/PKuMqkqz (ServerDeletor:110
is the destroyNodesMatching() call listed above.)
Re: IllegalArgumentException when deleting node on Rackspace
Zack,
I've tried to reproduce this multiple ways, but was unable to do so
(yet), but I have a theory about what happened (which is difficult to
intentionally replicate).
Reviewing the stack trace, I noticed that the exception is occurring
after the actual deletion of the node, in a subsequent call to
cleanUpIncidentalResourcesOfDeadNodes(), which apparently attempts to
delete orphaned groups. I don't understand enough of the concept of a
group to know with certainty what's going on here, and my reading of
Rackspace documentation only implies security groups which I don't
have; I do, however, utilize their RackConnect system to connect my
cloud nodes to a managed server and it is possible that setup is
involved somehow. After my nodes are created, they are subsequently
switched over (by Rackspace using their scripts) to RackConnect.
I don't know what resources were being cleaned up, but it may be a
possibility that the following factors contributed:
- I had previously created a server named Cloud-Server-35 which failed
to fully create (this happens about 1% of the time) and never got
switched over to the RackConnect group.
- I had issued the delete server command on the failed server, but the
node may have still been hanging around in the system; or at least, not
fully deleted by the time the second server got created.
- I had created another server named Cloud-Server-35 to replace the
failed one. This one operated normally and was switched over to
RackConnect.
- When I went to delete the second server by IP, jclouds found leftover
resources associated with the original server, and attempted to clean
them up.
- Somehow in this cleanup, the code/API detected duplicate names/groups,
and attempted to rename one of the groups, encountering the exception
because the group name was mixed case.
Dan
On 8/11/14, 6:45 AM, Zack Shoylev wrote:
Hi Daniel,
Can this issue be reproduced consistently? Does the number of servers matter?
From your stack traces it seems this code is called
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/predicates/validators/DnsNameValidator.java#L56
so it could be some kind of weird interaction.
From: Daniel Widdis [wid...@gmail.com]
Sent: Saturday, August 09, 2014 1:57 PM
To:user@jclouds.apache.org
Subject: IllegalArgumentException when deleting node on Rackspace
This issue may be related, or may be completely different than Sunil
Shah's recent thread on 2-letter DNS names when creating GCE nodes, but
ultimately boils down to the same exception at the same line of code. I
debated whether to reply to that thread or start a new one, and elected
the latter. I will take a look at JIRA where the issues may possibly be
combined.
My specific symptoms/situation:
- jclouds 1.7.3
- I had manually created (through website API) several Rackspace cloud
servers, with the names Cloud-Server-30, Cloud-Server-31, etc. These
were the default names provided on the website and I chose (by virtue of
laziness) to leave them as they were.
- My code deletes these servers when it is done using them. I do this
by passing the private IP address of the server to this code:
nodes = computeService
.destroyNodesMatching(new PredicateNodeMetadata() {
@Override
public boolean apply(final NodeMetadata input) {
return input.getPrivateAddresses().contains(ipAddr);
}
});
- Note that my code never uses the server name, nor is even aware of
it. I simply want to tell Rackspace, delete the server with this IP
address.
- This code worked fine for 34 out of 35 servers, 15 of which had
Cloud-Server-## names. However, on the very last server (named
Cloud-Server-35), passing the String ipAddr = 10.208.232.65 , I got
this exception:
java.lang.IllegalArgumentException: Object 'Cloud-Server' doesn't
match dns naming constraints. Reason: Should be only lowercase.
- Full stack trace here:http://pastebin.com/PKuMqkqz (ServerDeletor:110
is the destroyNodesMatching() call listed above.)
How does jclouds storage my files?
Hi, I am new with jclouds. What I am doing is: Opening a disk with java and formatting it with jclouds. So, instead of write Files.write() I use jclouds. When I format I can see about 63 blocks of 2MB. The blocks next to it, say the 30 neighbors, are the indexes of my files copied to the disk. It is so true that if I save 100 files for example, I can keep these blocks and move the another blocks to my cloud and I can still open the disk and see all files inside. Only see, of course. If I tried to open I need to move back the exactly blocks of that file I want to open. My question: When I copied 50GB of movies I still can see all movies on my disk. When I open the movie I need to restore all blocks. It is what my app does. But I got a claim saying the index of the movie are not integrate. I still can see the size of the movie is 800MB, but when I open it says only 20 minutes of movie. So I would like to restore all the blocks of that movie, not only what my app is asking to read. Did I make myself clear? Thanks in advanced, Felipe.
Re: How does jclouds storage my files?
Hi Filipe I am new with jclouds. What I am doing is: Opening a disk with java and formatting it with jclouds. So, instead of write Files.write() I use jclouds. Which provider or API are you using? And which underlying system (a file system, I suspect?) are you talking to? Which commands are you using to store the blobs, and which to read them? What is the error? Could you create a small test case that we could run to reproduce the behaviour you're seeing? Regards ap
How to configure Jclouds S3 API to put the container name *after* the hostname?
I'm trying to use jclouds' S3 API to connect to an internally-hosted Ceph server, which exposes an S3-compliant API. Use other (ruby and perl) clients, I've noticed that my requests work fine when I put the container/bucket name *after* the hostname, like this: https://opower.internal/*mybucket/*?acl However, jclouds is putting the container name *in the hostname* (S3 calls this a virtual bucket), like so: https://*mybucket.*opower.internal/?acl And that's failing with an AccessDenied error that I haven't figured out yet, but am still working on. In the mean time, is there a way to configure jclouds to put the container name *after* the host name, in the request? *Steve Kingsland* Senior Software Engineer * Opower * http://www.opower.com/ * We’re hiring! See jobs here http://www.opower.com/careers *
Re: How to configure Jclouds S3 API to put the container name *after* the hostname?
On Mon, Aug 11, 2014 at 08:19:56PM -0400, Steve Kingsland wrote: I'm trying to use jclouds' S3 API to connect to an internally-hosted Ceph server, which exposes an S3-compliant API. Use other (ruby and perl) clients, I've noticed that my requests work fine when I put the container/bucket name *after* the hostname, like this: https://opower.internal/*mybucket/*?acl However, jclouds is putting the container name *in the hostname* (S3 calls this a virtual bucket), like so: https://*mybucket.*opower.internal/?acl And that's failing with an AccessDenied error that I haven't figured out yet, but am still working on. In the mean time, is there a way to configure jclouds to put the container name *after* the host name, in the request? Which version of jclouds do you use? jclouds 1.7.0 should have resolved this issue: https://issues.apache.org/jira/browse/JCLOUDS-305 If you use 1.7.0 or newer, can you share the failing operations with wire logs? Also make sure you use the generic s3 provider and not the aws-s3 specific provider. -- Andrew Gaul http://gaul.org/
Re: How does jclouds storage my files?
On Mon, Aug 11, 2014 at 05:08:16PM -0300, felipe gutierrez wrote: I am new with jclouds. What I am doing is: Opening a disk with java and formatting it with jclouds. So, instead of write Files.write() I use jclouds. When I format I can see about 63 blocks of 2MB. The blocks next to it, say the 30 neighbors, are the indexes of my files copied to the disk. It is so true that if I save 100 files for example, I can keep these blocks and move the another blocks to my cloud and I can still open the disk and see all files inside. Only see, of course. If I tried to open I need to move back the exactly blocks of that file I want to open. My question: When I copied 50GB of movies I still can see all movies on my disk. When I open the movie I need to restore all blocks. It is what my app does. But I got a claim saying the index of the movie are not integrate. I still can see the size of the movie is 800MB, but when I open it says only 20 minutes of movie. So I would like to restore all the blocks of that movie, not only what my app is asking to read. Can you explain how you open a disk and what it means to format it with jclouds? Also what does an index mean? The jclouds filesystem provider does not provide any indexing; the underlying operating system provides this. -- Andrew Gaul http://gaul.org/
RE: IllegalArgumentException when deleting node on Rackspace
Hi Daniel,
I am still working on reproducing this, but have not been able to yet.
My (generic) advice for now would be to update to the latest jclouds (1.8.0)
and let us know if this happens again.
I will let you know if we make some progress reproducing this.
Also, if I remember correctly, the group is usually based on how the server is
named.
-Zack
From: Daniel Widdis [wid...@gmail.com]
Sent: Monday, August 11, 2014 1:40 PM
To: user@jclouds.apache.org
Subject: Re: IllegalArgumentException when deleting node on Rackspace
Zack,
I've tried to reproduce this multiple ways, but was unable to do so
(yet), but I have a theory about what happened (which is difficult to
intentionally replicate).
Reviewing the stack trace, I noticed that the exception is occurring
after the actual deletion of the node, in a subsequent call to
cleanUpIncidentalResourcesOfDeadNodes(), which apparently attempts to
delete orphaned groups. I don't understand enough of the concept of a
group to know with certainty what's going on here, and my reading of
Rackspace documentation only implies security groups which I don't
have; I do, however, utilize their RackConnect system to connect my
cloud nodes to a managed server and it is possible that setup is
involved somehow. After my nodes are created, they are subsequently
switched over (by Rackspace using their scripts) to RackConnect.
I don't know what resources were being cleaned up, but it may be a
possibility that the following factors contributed:
- I had previously created a server named Cloud-Server-35 which failed
to fully create (this happens about 1% of the time) and never got
switched over to the RackConnect group.
- I had issued the delete server command on the failed server, but the
node may have still been hanging around in the system; or at least, not
fully deleted by the time the second server got created.
- I had created another server named Cloud-Server-35 to replace the
failed one. This one operated normally and was switched over to
RackConnect.
- When I went to delete the second server by IP, jclouds found leftover
resources associated with the original server, and attempted to clean
them up.
- Somehow in this cleanup, the code/API detected duplicate names/groups,
and attempted to rename one of the groups, encountering the exception
because the group name was mixed case.
Dan
On 8/11/14, 6:45 AM, Zack Shoylev wrote:
Hi Daniel,
Can this issue be reproduced consistently? Does the number of servers matter?
From your stack traces it seems this code is called
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/predicates/validators/DnsNameValidator.java#L56
so it could be some kind of weird interaction.
From: Daniel Widdis [wid...@gmail.com]
Sent: Saturday, August 09, 2014 1:57 PM
To:user@jclouds.apache.org
Subject: IllegalArgumentException when deleting node on Rackspace
This issue may be related, or may be completely different than Sunil
Shah's recent thread on 2-letter DNS names when creating GCE nodes, but
ultimately boils down to the same exception at the same line of code. I
debated whether to reply to that thread or start a new one, and elected
the latter. I will take a look at JIRA where the issues may possibly be
combined.
My specific symptoms/situation:
- jclouds 1.7.3
- I had manually created (through website API) several Rackspace cloud
servers, with the names Cloud-Server-30, Cloud-Server-31, etc. These
were the default names provided on the website and I chose (by virtue of
laziness) to leave them as they were.
- My code deletes these servers when it is done using them. I do this
by passing the private IP address of the server to this code:
nodes = computeService
.destroyNodesMatching(new PredicateNodeMetadata() {
@Override
public boolean apply(final NodeMetadata input) {
return input.getPrivateAddresses().contains(ipAddr);
}
});
- Note that my code never uses the server name, nor is even aware of
it. I simply want to tell Rackspace, delete the server with this IP
address.
- This code worked fine for 34 out of 35 servers, 15 of which had
Cloud-Server-## names. However, on the very last server (named
Cloud-Server-35), passing the String ipAddr = 10.208.232.65 , I got
this exception:
java.lang.IllegalArgumentException: Object 'Cloud-Server' doesn't
match dns naming constraints. Reason: Should be only lowercase.
- Full stack trace here:http://pastebin.com/PKuMqkqz (ServerDeletor:110
is the destroyNodesMatching() call listed above.)
