Re: [jclouds] JCLOUDS-137: Retry on HTTP 500 AtmosError 1040 (#285)
The turnaround time on this is INCREDIBLE!! :+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/285#issuecomment-34685027
Re: [jclouds] JCLOUDS-137: Retry on HTTP 500 AtmosError 1040 (#285)
+ private int retryCountLimit = 5;
+ @Resource
+ protected Logger logger = Logger.NULL;
+
+ public boolean shouldRetryRequest(HttpCommand command, HttpResponse
response) {
+ if (command.getFailureCount() retryCountLimit) {
+ return false;
+ }
+ if (response.getStatusCode() == 500) {
+ byte[] content =
HttpUtils.closeClientButKeepContentStream(response);
+ // Content can be null in the case of HEAD requests
+ if (content != null) {
+try {
+ AtmosError error = utils.parseAtmosErrorFromContent(command,
response,
+new String(content));
+ if (error.getCode() == 1040) { // The server is busy. Please
try again.
I haven't seen these failures myself. I'm fine with the current change if
you're not comfortable with retrying on all returned error codes except 1020.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/285/files#r9637835
Re: [jclouds] Delete objects in a container efficiently. (#214)
I tried varying the number of threads from 10 to 50 and deleting a container with at least 5K blobs. Things worked fine. I'll rebase this patch on top of the current ToT and send out. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/214#issuecomment-36594782
Re: [jclouds] Delete objects in a container efficiently. (#214)
I am trying to reproduce this locally. I have run 50 iterations of this test so far, but haven't been able to reproduce the problem. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/214#issuecomment-36681788
Re: [jclouds] Retry on S3 HTTP 504 Gateway Timeout status codes (#317)
Looks good to me. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/317#issuecomment-37567286
Re: [jclouds] Delete objects in a container efficiently. (#214)
+ // If a future to delete a blob/directory actually got created
above,
+ // keep a reference of that in the outstandingFutures list. This is
+ // useful in case of a timeout exception. All outstanding futures
can
+ // then be cancelled.
+ if (blobDelFuture != null) {
+final AtomicReferenceListenableFutureVoid
atomicBlobDelFuture =
+ new AtomicReferenceListenableFutureVoid(blobDelFuture);
+outstandingFutures.add(atomicBlobDelFuture);
+
+// Add a callback to release the semaphore. This is required for
+// other threads waiting to acquire a semaphore above to make
+// progress.
+Futures.addCallback(blobDelFuture, new FutureCallbackObject() {
+ @Override
+ public void onSuccess(final Object o) {
+ outstandingFutures.remove(atomicBlobDelFuture);
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/214/files#r10872324
Re: [jclouds] Delete objects in a container efficiently. (#214)
import com.google.common.util.concurrent.ListenableFuture; import com.google.common.util.concurrent.ListeningExecutorService; import com.google.inject.Inject; /** * Deletes all keys in the container - * + * * @author Adrian Cole Done --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/214/files#r10872338
Re: [jclouds] Delete objects in a container efficiently. (#214)
+ * a timeout. Also, when the reference is removed from this list and
when
+ * the executorService removes the reference that it has maintained,
the
+ * future will be marked for GC since there should be no other
references
+ * to it. This is important because this code can generate an unbounded
+ * number of futures.
+ */
+ final ListAtomicReferenceListenableFutureVoid outstandingFutures
= Collections
+.synchronizedList(new
ArrayListAtomicReferenceListenableFutureVoid());
+ while (retries 0) {
+ deleteFailure.set(false);
+ executeOneIteration(containerName, listOptions.clone(), semaphore,
+ outstandingFutures, deleteFailure);
+ // Wait for all futures to complete by waiting to acquire all
+ // semaphores.
+ try {
+if (!semaphore.tryAcquire(numOutStandingRequests, maxTime,
Added a TODO for this.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/214/files#r10872424
Re: [jclouds] Delete objects in a container efficiently. (#214)
+ listing = blobStore.list(containerName, options);
+ } catch (ContainerNotFoundException ce) {
+ return listing;
+ }
+
+ // recurse on subdirectories
+ if (options.isRecursive()) {
+ for (StorageMetadata md : listing) {
+String fullPath = parentIsFolder(options, md) ? options.getDir()
+ + / + md.getName() : md.getName();
+switch (md.getType()) {
+case BLOB:
+ break;
+case FOLDER:
+case RELATIVE_PATH:
+ if (options.isRecursive()
Removed.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/214/files#r10872428
Re: [jclouds] Delete objects in a container efficiently. (#214)
+ String marker = listing.getNextMarker();
+ if (marker != null) {
+logger.debug(%s with marker %s, message, marker);
+options = options.afterMarker(marker);
+listing = getListing(containerName, options, semaphore,
+ outstandingFutures, deleteFailure);
+ } else {
+break;
+ }
+ }
+ }
+
+ public void execute(final String containerName,
+ ListContainerOptions listOptions) {
+ final AtomicBoolean deleteFailure = new AtomicBoolean();
+ final int numOutStandingRequests = 1024;
Done.
There is a configurable property called jclouds.max-parallel-deletes that can
be used for this.
You're point about whether the # of semaphores should be equal to the # of
threads in the executorService is valid. The default value of
jclouds.max-parallel-threads is equal to the # of user threads.
Given that these two are equal, I thought about whether we need the extra
config option. Can we just use the value of PROPERTY_USER_THREADS instead.
However, I'm not sure if the executorService object injected into
DeleteAllKeysInList is also being used somewhere else. If it is used
somewhere, I guess, it would be good to have an option where the use can limit
the number of parallel deletes being issued when deleting a container thereby
limiting the number of threads being used from the executorService.
Let me know if this isn't necessary.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/214/files#r10872420
Re: [jclouds] Delete objects in a container efficiently. (#214)
Looks like a network glitch on cloudbees. code ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Could not clone git://github.com/jclouds/jclouds.git ... stderr: fatal: unable to connect to github.com: github.com[0: 192.30.252.129]: errno=Connection timed out /code Can we start another build? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/214#issuecomment-38413431
Re: [jclouds] Azureblobs live-test Bug Fix (#336)
@@ -87,7 +87,9 @@ public void testListContainers() throws Exception {
public void testCreateContainer() throws Exception {
boolean created = false;
while (!created) {
- privateContainer = prefix + new SecureRandom().nextInt();
+
+ privateContainer = (CONTAINER_PREFIX + new
SecureRandom().nextInt());
Aah.. I missed the .toLowerCase() in the declaration of CONTAINER_PREFIX.
That explains it.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/336/files#r11126006
Re: [jclouds] Azureblobs live-test Bug Fix (#336)
@@ -87,7 +87,9 @@ public void testListContainers() throws Exception {
public void testCreateContainer() throws Exception {
boolean created = false;
while (!created) {
- privateContainer = prefix + new SecureRandom().nextInt();
+
+ privateContainer = (CONTAINER_PREFIX + new
SecureRandom().nextInt());
You don't need the parenthesis.
(CONTAINER_PREFIX + new SecureRandom().nextInt());
could simply be
CONTAINER_PREFIX + new SecureRandom().nextInt();
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/336/files#r11126993
Re: [jclouds] Add deprecation warnings and provide links to new Swift APIs (#237)
Do we know what's missing in the new implementation (in jclouds-labs-openstack) that is present in the old implementation? In other words, what's the effort required to have feature parity between the new implementation and the old one? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/237#issuecomment-39243860
Re: [jclouds] Azureblobs live-test Bug Fix (#336)
Congratulations on your first commit to jclouds @hsbhathiya. Looking forward to more :-) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/336#issuecomment-39277522
Re: [jclouds] Mark Swift blobstores as eventually consistent (#343)
This commit looks good to me. +1. I haven't seen too many uses of the getConsistencyModel API. I agree though, that it seems useful. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/343#issuecomment-40336557
[jclouds] Create a separate function to delete directories. (#349)
You can merge this Pull Request by running: git pull https://github.com/maginatics/jclouds deleteallkeys-cleanup Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/349 -- Commit Summary -- * Create a separate function to delete directories. -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/strategy/internal/DeleteAllKeysInList.java (45) -- Patch Links -- https://github.com/jclouds/jclouds/pull/349.patch https://github.com/jclouds/jclouds/pull/349.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/349
Re: [jclouds] Remove deprecation from getConsistencyModel (#348)
Looks good to me too! --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/348#issuecomment-40762944
Re: [jclouds] Allow Swift to set content type (#338)
Apart from what @andrewgaul said, we should have a unit test for this. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/338#issuecomment-40763386
Re: [jclouds] Create a separate function to delete directories. (#349)
Your're right. This is a purely refactoring commit. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/349#issuecomment-40847919
Re: [jclouds] Create a separate function to delete directories. (#349)
@@ -177,6 +177,26 @@ private String getMessage(final String containerName, return listing; } + private ListenableFutureVoid deleteDirectory( Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/349/files#r11788434
Re: [jclouds] First attempt at fixing 538 by adding dummy override methods for TestNG (#351)
Trying to fix this without knowing the root-cause does seems like a bad idea to me. We may just hit this again in some other environment and/or in some other circumstances and then this change simply lands up being dead code. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/351#issuecomment-40953703
Re: [jclouds] First attempt at fixing 538 by adding dummy override methods for TestNG (#351)
I see. The change itself looks fine to me. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/351#issuecomment-41104636
Re: [jclouds] First attempt at fixing 538 by adding dummy override methods for TestNG (#351)
I definitely want to fix these test. I'll see if I can run the tests later tonight. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/351#issuecomment-41106560
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
Even if we separate it, a signedGet will require a blob to present in the container, right? This will require putting a blob and then trying to read it using a signed url. Instead this commit uses a single test to do a signed put first followed by a signed get. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94#issuecomment-42326554
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
+ final ByteSource input = ByteSource.wrap(new byte[1]); + final HttpClient client = ctx.utils().http(); + + // test signed put + String blobName = test- + UUID.randomUUID(); + Blob blob2 = region.blobBuilder(blobName).forSigning() + .payload(input).contentLength(input.size()) + .contentMD5(input.hash(Hashing.md5()).asBytes()) + .contentType(MediaType.OCTET_STREAM.toString()).build(); + HttpRequest putRequest; + BlobRequestSigner signer = ctx.signerInRegion(regionId); + putRequest = signer.signPutBlob(containerName, blob2, 600); + + HttpResponse response = client.invoke(putRequest); + assertTrue(response.getStatusCode() == 200 + || response.getStatusCode() == 201); Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12565178
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
+ + // test signed put + String blobName = test- + UUID.randomUUID(); + Blob blob2 = region.blobBuilder(blobName).forSigning() + .payload(input).contentLength(input.size()) + .contentMD5(input.hash(Hashing.md5()).asBytes()) + .contentType(MediaType.OCTET_STREAM.toString()).build(); + HttpRequest putRequest; + BlobRequestSigner signer = ctx.signerInRegion(regionId); + putRequest = signer.signPutBlob(containerName, blob2, 600); + + HttpResponse response = client.invoke(putRequest); + assertTrue(response.getStatusCode() == 200 + || response.getStatusCode() == 201); + + // test signed get HttpRequest request = ctx.signerInRegion(regionId).signGetBlob(containerName, blobName); Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12565180
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
HttpRequest request = ctx.signerInRegion(regionId).signGetBlob(containerName, blobName); assertNotNull(request, regionId= + regionId + , container= + containerName + , blob= + blobName); + response = client.invoke(request); + assertEquals(response.getStatusCode(), 200); Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12565187
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
+ + // test signed put + String blobName = test- + UUID.randomUUID(); + Blob blob2 = region.blobBuilder(blobName).forSigning() + .payload(input).contentLength(input.size()) + .contentMD5(input.hash(Hashing.md5()).asBytes()) + .contentType(MediaType.OCTET_STREAM.toString()).build(); + HttpRequest putRequest; + BlobRequestSigner signer = ctx.signerInRegion(regionId); + putRequest = signer.signPutBlob(containerName, blob2, 600); + + HttpResponse response = client.invoke(putRequest); + assertTrue(response.getStatusCode() == 200 + || response.getStatusCode() == 201); + + // test signed get HttpRequest request = ctx.signerInRegion(regionId).signGetBlob(containerName, blobName); assertNotNull(request, regionId= + regionId + , container= + containerName + , blob= + blobName); Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12565185
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
@@ -83,13 +94,29 @@ public void trySign() throws InterruptedException,
ExecutionException {
continue;
}
String containerName = Iterables.getLast(containers).getName();
- PageSet? extends StorageMetadata blobs =
region.list(containerName);
- if (blobs.isEmpty()) {
-continue;
- }
- String blobName = Iterables.getLast(blobs).getName();
+
+ final ByteSource input = ByteSource.wrap(new byte[1]);
+ final HttpClient client = ctx.utils().http();
+
+ // test signed put
+ String blobName = test- + UUID.randomUUID();
+ Blob blob2 = region.blobBuilder(blobName).forSigning()
+ .payload(input).contentLength(input.size())
Done. Added the payload to the put request. And added a check for comparing the
blob obtained by getBlob with the input.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12565208
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
- String blobName = Iterables.getLast(blobs).getName(); + + final ByteSource input = ByteSource.wrap(new byte[1]); + final HttpClient client = ctx.utils().http(); + + // test signed put + String blobName = test- + UUID.randomUUID(); + Blob blob2 = region.blobBuilder(blobName).forSigning() + .payload(input).contentLength(input.size()) + .contentMD5(input.hash(Hashing.md5()).asBytes()) + .contentType(MediaType.OCTET_STREAM.toString()).build(); + HttpRequest putRequest; + BlobRequestSigner signer = ctx.signerInRegion(regionId); + putRequest = signer.signPutBlob(containerName, blob2, 600); + + HttpResponse response = client.invoke(putRequest); Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12565175
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
@@ -83,13 +94,29 @@ public void trySign() throws InterruptedException,
ExecutionException {
continue;
}
String containerName = Iterables.getLast(containers).getName();
- PageSet? extends StorageMetadata blobs =
region.list(containerName);
- if (blobs.isEmpty()) {
-continue;
- }
- String blobName = Iterables.getLast(blobs).getName();
+
+ final ByteSource input = ByteSource.wrap(new byte[1]);
+ final HttpClient client = ctx.utils().http();
+
+ // test signed put
+ String blobName = test- + UUID.randomUUID();
+ Blob blob2 = region.blobBuilder(blobName).forSigning()
+ .payload(input).contentLength(input.size())
Yeah... I found it the hard way that the payload does not get added. I think
the behavior here is different from the jclouds/apis/swift. There the payload
gets added when we call signPutBlob(). I will make that change and update the
review request.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r12441518
Re: [jclouds] Fixes an issue where the number of retries was always set to 1. (#368)
The change itself looks good. Can we have a unit test for this? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/368#issuecomment-43235344
Re: [jclouds] Fixes an issue where the number of retries was always set to 1. (#368)
Duh... nevermind then! :-) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/368#issuecomment-43424762
Re: [jclouds] Enforce correct MD5 when provided by a client (#382)
Looks good to me. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/382#issuecomment-44369485
Re: [jclouds] Allow Guava MediaType for Content-Type (#383)
@@ -303,13 +303,15 @@ Options can also be specified for extension modules blob-builder (if content-length ;; Special case, arg is prim. (.contentLength blob-builder content-length) blob-builder) + blob-builder (if content-type How does one set the content type of a blob using the Guava MediaType after this change? I only see a method that returns the media type. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/383/files#r13117334
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)
Hey Chris, Thanks for adding support for the 13.5 API. Could you list some points (maybe in the commit message itself) about what the actual changes are? Will make it easier to understand the code changes. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/394#issuecomment-45181404
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
+
+ // test signed get
+ try {
+HttpRequest getRequest = signer.signGetBlob(containerName,
+ blobName);
+assertNotNull(getRequest, regionId= + regionId + , container=
+ + containerName + , blob= + blobName);
+response = client.invoke(getRequest);
+if (response.getStatusCode() != 200) {
+ fail(Signed GET expected to return 200 but returned
+ + response.getStatusCode());
+}
+Payload payload = response.getPayload();
+byte[] output = ByteStreams.toByteArray(payload);
+if (!Arrays.equals(input.read(), output)) {
+ fail(Data with signed GET not identical to what was put);
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r13478685
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
+ .addHeader(HttpHeaders.CONTENT_TYPE,
+metadata.getContentType());
+putRequestBuilder.addHeader(HttpHeaders.CONTENT_LENGTH,
+ String.valueOf(input.size()));
+putRequestBuilder.payload(input);
+putRequest = putRequestBuilder.build();
+assertNotNull(putRequest, regionId= + regionId + , container=
+ + containerName + , blob= + blobName);
+response = client.invoke(putRequest);
+if (response.getStatusCode() != 200
+ response.getStatusCode() != 201) {
+ fail(Signed PUT expected to return 200 or 201 but returned
+ + response.getStatusCode());
+}
+ } catch (Exception e) {
+fail(Failed signed put test);
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r13478682
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
+HttpRequest getRequest = signer.signGetBlob(containerName,
+ blobName);
+assertNotNull(getRequest, regionId= + regionId + , container=
+ + containerName + , blob= + blobName);
+response = client.invoke(getRequest);
+if (response.getStatusCode() != 200) {
+ fail(Signed GET expected to return 200 but returned
+ + response.getStatusCode());
+}
+Payload payload = response.getPayload();
+byte[] output = ByteStreams.toByteArray(payload);
+if (!Arrays.equals(input.read(), output)) {
+ fail(Data with signed GET not identical to what was put);
+}
+ } catch (Exception e) {
+fail(Failed signed GET test);
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs-openstack/pull/94/files#r13478689
Re: [jclouds] Remove LocalAsyncBlobStore. (#220)
Is there anything else that needs to be done here? There is one feature that I am developing in jclouds that is made easier because of this change. Let me know if there's anything I can do to help make progress on this. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/220#issuecomment-45361526
[jclouds] Port fixes for JCLOUDS-178 to SwiftAuth and Keystone v1.1 (JCLOUDS-589) (#399)
Reauthenticate on HTTP 401. Orig commit: 578a77d6313ce0945f8d29e82103e09787622c58 Closes #589. You can merge this Pull Request by running: git pull https://github.com/maginatics/jclouds fix_401_unauthorized_non_keystone Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/399 -- Commit Summary -- * Port fixes for JCLOUDS-178 to SwiftAuth and Keystone v1.1 (JCLOUDS-589) -- File Changes -- M common/openstack/src/main/java/org/jclouds/openstack/handlers/RetryOnRenew.java (51) M common/openstack/src/main/java/org/jclouds/openstack/keystone/v1_1/handlers/RetryOnRenew.java (45) M common/openstack/src/test/java/org/jclouds/openstack/handlers/RetryOnRenewTest.java (53) M common/openstack/src/test/java/org/jclouds/openstack/keystone/v1_1/handlers/RetryOnRenewTest.java (37) -- Patch Links -- https://github.com/jclouds/jclouds/pull/399.patch https://github.com/jclouds/jclouds/pull/399.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/399
Re: [jclouds] Port fixes for JCLOUDS-178 to SwiftAuth and Keystone v1.1 (JCLOUDS-589) (#399)
These are related to the change I made. I'll take a look and update the review request. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/399#issuecomment-45554781
Re: [jclouds] Port fixes for JCLOUDS-178 to SwiftAuth and Keystone v1.1 (JCLOUDS-589) (#399)
@Inject(optional = true) @Named(Constants.PROPERTY_MAX_RETRIES) - private int retryCountLimit = 5; + static final int NUM_RETRIES = 5; @andrewgaul Done. @zack-shoylev There is already a test for this. test401ShouldRetry4Times(). --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/399/files#r13880093
Re: [jclouds] Port fixes for JCLOUDS-178 to SwiftAuth and Keystone v1.1 (JCLOUDS-589) (#399)
@@ -60,6 +67,16 @@ protected RetryOnRenew(LoadingCacheCredentials, Auth authenticationResponseCac this.backoffHandler = backoffHandler; } + /* +* The reason retries need to be tracked is that it is possible that a token +* * can be expired at any time. The reason we track by request is that only Done. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/399/files#r13880127
Re: [jclouds] JCLOUDS-622: remove calls to InputSupplier methods (#436)
I like the BlahAndClose methods. Makes code reading easier and using these methods is less error prone. I'm still a little fuzzy about about MultiPartForm gets used in the context of other blobstore operations. But the change itself seems to do what is says. Looks good! --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/436#issuecomment-48978037
Re: [jclouds] Add deleteContainerIfEmpty to BlobStore (#451)
deleteContainerIfEmpty seems useful. However, I'm not sure about deprecating deleteContainer itself. I think of deleteContainerIfEmpty as rm and deleteContainer as rm -rf. Both are useful. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/451#issuecomment-49801813
Re: [jclouds] Add deleteContainerIfEmpty to BlobStore (#451)
@@ -272,10 +274,10 @@ public void deleteContainerWithContents() throws
InterruptedException {
String containerName = getContainerName();
try {
addBlobToContainer(containerName, test);
- view.getBlobStore().deleteContainer(containerName);
- assertNotExists(containerName);
+
assertFalse(view.getBlobStore().deleteContainerIfEmpty(containerName));
+ assertTrue(view.getBlobStore().containerExists(containerName));
Maybe add a test for deleteContainerIfEmpty again to make sure that a
ContainerNotFoundException results in a return true.
assertTrue(view.getBlobStore().deleteContainerIfEmpty(containerName));
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/451/files#r15257037
Re: [jclouds] Add deleteContainerIfEmpty to BlobStore (#451)
This change itself looks good to me. Let's have a separate discussion about whether to deprecate deleteContainer. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/451#issuecomment-49951167
[jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)
S3 compatible blobStores sometimes return date in the format: quot;2014-07-23T20:53:17+quot; instead of the more common quot;2014-07-23T18:09:39.944Zquot;. This caused jclouds to barf with an IllegalArgumentException. This commit tries to parse both the formats for S3. The exception is thrown if both fail. Added unit tests for the same. You can merge this Pull Request by running: git pull https://github.com/maginatics/jclouds parse-date-seconds-format Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/456 -- Commit Summary -- * Try iso8601SecondsDateParse if iso8601DateParse fails. -- File Changes -- M apis/s3/src/main/java/org/jclouds/s3/xml/CopyObjectHandler.java (12) M apis/s3/src/main/java/org/jclouds/s3/xml/ListAllMyBucketsHandler.java (12) M apis/s3/src/main/java/org/jclouds/s3/xml/ListBucketHandler.java (12) M apis/s3/src/test/java/org/jclouds/s3/xml/CopyObjectHandlerTest.java (19) M apis/s3/src/test/java/org/jclouds/s3/xml/ListBucketHandlerTest.java (20) -- Patch Links -- https://github.com/jclouds/jclouds/pull/456.patch https://github.com/jclouds/jclouds/pull/456.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/456
Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)
@@ -51,7 +51,17 @@ public void endElement(String uri, String name, String
qName) {
if (qName.equals(ETag)) {
this.currentETag = currentOrNull(currentText);
} else if (qName.equals(LastModified)) {
- this.currentLastModified =
dateParser.iso8601DateParse(currentOrNull(currentText));
+ try {
+this.currentLastModified = dateParser
+
.iso8601DateParseWithOptionalTZ(currentOrNull(currentText));
Done.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15437217
Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)
Thanks for pushing this to master @andrewgaul. I believe the jclouds 1.8.0 release next week will be based off of master so this fix will make it there. Do we need to backport this fix to the 1.7.x branch for any further 1.7.x releases? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/456#issuecomment-50284492
Re: [jclouds-site] Update stale BlobStore docs (#68)
-### Multipart upload - -Providers may implement multipart upload for large or very files. -Here's an example of `multipart upload` using aws-s3 provider, which [allow uploading files large as 5TB.](http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?qfacts.html) +Providers may implement multipart upload for large or very files. Here's an example of multipart for large or very files -- ??? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/68/files#r15486910
Re: [jclouds] Remove LocalAsyncBlobStore. (#220)
Closed #220. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/220#event-147810874
Re: [jclouds] Add BlobStoreBench to jclouds. (#239)
Closed #239. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/239#event-147810983
Re: [jclouds-site] Release Notes for jclouds 1.8.0 (#114)
+* TODO
+
+## a id=issues/aKnown Issues
+
+* TODO
+
+## a id=reminder/aReminder
+
+The jclouds Maven group ID for versions since [1.6.1-incubating](../1.6.1)
is `org.apache.jclouds` rather than `org.jclouds`, so a pom.xml dependency
would now look like:
+
+{% highlight xml %}
+dependencies
+ dependency
+groupIdorg.apache.jclouds/groupId
+artifactIdjclouds-all/artifactId
+version1.7.3/version
This should be 1.8.0, right?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-site/pull/114/files#r15724722
Re: [jclouds-site] Release Notes for jclouds 1.8.0 (#114)
+1. [Highlights](#highlights) +1. [API Changes](#api) +1. [Known Issues](#issues) +1. [Reminder](#reminder) +1. [Credits](#credits) +1. [Test Results](#test) + +## a id=intro/aIntroduction + +You can read the official announcement at TODO ([Apache jclouds 1.7.3 released](http://jclouds.markmail.org/message/6ormro7calbt5tjn)). You can read the details of the specific JIRA issues addressed in this release at the [JIRA Release Notes](https://issues.apache.org/jira/issues/?jql=project%20%3D%20JCLOUDS%20AND%20fixVersion%20%3D%201.8.0). + +To get jclouds, please see the [jclouds installation guide](/start/install/). + +## a id=highlights/aHighlights + +* **This is the last version of jclouds to support Java 6** Based on bug 645: Starting with jclouds-1.8.0, it is mandatory to set the region for the HPCloud Object Storage provider. Previously the region and/or the apiVersion was used for choosing the correct endpoint. With jclouds-1.8.0, simply setting the region is enough. The value of the apiVersion is ignored when choosing the endpoint. Failure to set the region can result in an arbitrary region and therefore an arbitrary endpoint getting selected which may be different from previous jclouds releases. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/114/files#r15724889
Re: [jclouds] Use HEAD /name to check bucket existence with S3 (#475)
.addHeader(Date, CONSTANT_DATE) - .addHeader(Authorization, AWS identity:p32RsBr2inawMBeCkkiA228BT2w=) Why was the change in identity needed? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/475/files#r16065111
Re: [jclouds] Use HEAD /name to check bucket existence with S3 (#475)
.addHeader(Date, CONSTANT_DATE) - .addHeader(Authorization, AWS identity:p32RsBr2inawMBeCkkiA228BT2w=) I see. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/475/files#r16069034
Re: [jclouds] Use HEAD /name to check bucket existence with S3 (#475)
Change looks good to me. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/475#issuecomment-51816730
Re: [jclouds] JCLOUDS-654: Add object size to StorageMetadata (#465)
Sorry I missed this earlier. I will take a look at this later today. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/465#issuecomment-51953773
Re: [jclouds] JCLOUDS-654: Add object size to StorageMetadata (#465)
currentName = currentText.toString().trim();
- if (currentName.equals())
-currentName = null;
+ } else if (qName.equals(Value)) {
+ if (currentName.equals(size)) {
The way you have it, size is in the correct case as per
[this](https://community.emc.com/servlet/JiveServlet/previewBody/3481-102-6-16527/Atmos%20Online%20Programmer's%20Guide%201.2.4A.pdf;jsessionid=360D5B1A4D136E51874AB885545A7FA4.node0).
However, wouldn't it be better if this was a case insensitive comparison?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/465/files#r16152290
Re: [jclouds] JCLOUDS-654: Add object size to StorageMetadata (#465)
Generally, looks good. I'll take a look again when other concerns are addressed. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/465#issuecomment-5199
Re: [jclouds-labs-openstack] Add live test for signPutBlob. (#94)
Not sure what to make of the failures above. The checkstyle violations are not in my commit. Apparently they are in the neutron-api that I have not touched. Any ideas? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/94#issuecomment-53463063
[jclouds/jclouds] JCLOUDS-1161: Make AWSS3BlobRequestSignerV4 the default signer. (#1008)
Fix unit tests accordingly. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1008 -- Commit Summary -- * JCLOUDS-1161: Make AWSS3BlobRequestSignerV4 the default signer. -- File Changes -- M providers/aws-s3/src/main/java/org/jclouds/aws/s3/blobstore/config/AWSS3BlobStoreContextModule.java (4) M providers/aws-s3/src/test/java/org/jclouds/aws/s3/blobstore/AWSS3BlobSignerExpectTest.java (21) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1008.patch https://github.com/jclouds/jclouds/pull/1008.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1008
Re: [jclouds/jclouds] JCLOUDS-1161: Make AWSS3BlobRequestSignerV4 the default signer. (#1008)
Do the live tests run successfully on current master? The same 4 tests above failed for me on current master. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1008#issuecomment-246837745
Re: [jclouds/jclouds] JCLOUDS-1161: Make AWSS3BlobRequestSignerV4 the default signer. (#1008)
Added a new live test, specifically run against eu-central-1 region. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1008#issuecomment-246511656
Re: [jclouds/jclouds] JCLOUDS-1161: Make AWSS3BlobRequestSignerV4 the default signer. (#1008)
Thanks for fixing the tests on master @andrewgaul. I am able to run the tests successfully on master and also able to reproduce the test failures with my change. I'll fix them and upload a new commit. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1008#issuecomment-248364824
[jira] [Commented] (JCLOUDS-311) Allow hpcloud-objectstorage to see regions with mixed api versions
[
https://issues.apache.org/jira/browse/JCLOUDS-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13886840#comment-13886840
]
Shri Javadekar commented on JCLOUDS-311:
FWIW, versionId, versionInfo and versionList were never supposed to be
returned by Keystone. The documentation had a bug [1] that is now being fixed.
[1] https://bugs.launchpad.net/keystone/+bug/1273831
Allow hpcloud-objectstorage to see regions with mixed api versions
--
Key: JCLOUDS-311
URL: https://issues.apache.org/jira/browse/JCLOUDS-311
Project: jclouds
Issue Type: Improvement
Components: jclouds-core
Affects Versions: 1.6.2
Reporter: Adrian Cole
Assignee: Adrian Cole
Fix For: 1.7.0, 1.6.3
API versions for hpcloud objectstorage region-a.geo-1 and region-b.geo-1
differ, 1.0 and 1 respectively. This prevents multiple regions from being
visible without creating contexts for each version.
Let's configure hpcloud to ignore api versions until they can correct this
mistake.
See below for the server-side misconfiguration:
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main] {[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]name: Object
Storage,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]type:
object-store,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]endpoints:
[[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main] {[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]tenantId:
10448598368512,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
publicURL:
https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/10448598368512,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]region:
region-a.geo-1,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
versionId: 1.0,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
versionInfo: https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1.0\/,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
versionList: https:\/\/region-a.geo-1.objects.hpcloudsvc.com[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main] },[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main] {[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]tenantId:
10448598368512,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
publicURL:
https:\/\/region-b.geo-1.objects.hpcloudsvc.com:443\/v1\/10448598368512,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]region:
region-b.geo-1,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
versionId: 1,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
versionInfo:
https:\/\/region-b.geo-1.objects.hpcloudsvc.com:443\/v1\/,[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main]
versionList: https:\/\/region-b.geo-1.objects.hpcloudsvc.com:443[\n]
2013-09-30 14:16:18,397 DEBUG [jclouds.wire] [main] }[\n]
2013-09-30 14:16:18,398 DEBUG [jclouds.wire] [main]][\n]
2013-09-30 14:16:18,398 DEBUG [jclouds.wire] [main] },[\n]
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (JCLOUDS-510) Head-of-line blocking problem with DeleteAllKeysInList
[ https://issues.apache.org/jira/browse/JCLOUDS-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shri Javadekar updated JCLOUDS-510: --- Description: The current implementation of DeleteAllKeysInList suffers from the head-of-line blocking problem. It gets a PageSet of blobs from the container, creates futures for deleting them and waits for the futures to complete before getting the next PageSet. This issue was originally reported by Andrew Gaul [~gaul] [1] -Shri [1] https://github.com/jclouds/legacy-jclouds/issues/1087 was: The current implementation of DeleteAllKeysInList suffers from the head-of-line blocking problem. It gets a PageSet of blobs from the container, creates futures for deleting them and waits for the futures to complete before getting the next PageSet. This issue was originally reported by [~gaul] [1] -Shri [1] https://github.com/jclouds/legacy-jclouds/issues/1087 Head-of-line blocking problem with DeleteAllKeysInList -- Key: JCLOUDS-510 URL: https://issues.apache.org/jira/browse/JCLOUDS-510 Project: jclouds Issue Type: Improvement Components: jclouds-blobstore Reporter: Shri Javadekar Fix For: 1.7.2 The current implementation of DeleteAllKeysInList suffers from the head-of-line blocking problem. It gets a PageSet of blobs from the container, creates futures for deleting them and waits for the futures to complete before getting the next PageSet. This issue was originally reported by Andrew Gaul [~gaul] [1] -Shri [1] https://github.com/jclouds/legacy-jclouds/issues/1087 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-510) Head-of-line blocking problem with DeleteAllKeysInList
[ https://issues.apache.org/jira/browse/JCLOUDS-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962009#comment-13962009 ] Shri Javadekar commented on JCLOUDS-510: This didn't make it into 1.7.2 because of certain time constraints. It's already in master and therefore should make it into 1.8.0. Head-of-line blocking problem with DeleteAllKeysInList -- Key: JCLOUDS-510 URL: https://issues.apache.org/jira/browse/JCLOUDS-510 Project: jclouds Issue Type: Improvement Components: jclouds-blobstore Affects Versions: 1.8.0, 1.7.1 Reporter: Shri Javadekar Assignee: Shri Javadekar Fix For: 1.8.0 The current implementation of DeleteAllKeysInList suffers from the head-of-line blocking problem. It gets a PageSet of blobs from the container, creates futures for deleting them and waits for the futures to complete before getting the next PageSet. This issue was originally reported by Andrew Gaul [~gaul] [1] -Shri [1] https://github.com/jclouds/legacy-jclouds/issues/1087 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (JCLOUDS-510) Head-of-line blocking problem with DeleteAllKeysInList
[ https://issues.apache.org/jira/browse/JCLOUDS-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shri Javadekar updated JCLOUDS-510: --- Fix Version/s: (was: 1.7.2) Head-of-line blocking problem with DeleteAllKeysInList -- Key: JCLOUDS-510 URL: https://issues.apache.org/jira/browse/JCLOUDS-510 Project: jclouds Issue Type: Improvement Components: jclouds-blobstore Affects Versions: 1.8.0, 1.7.1 Reporter: Shri Javadekar Assignee: Shri Javadekar Fix For: 1.8.0 The current implementation of DeleteAllKeysInList suffers from the head-of-line blocking problem. It gets a PageSet of blobs from the container, creates futures for deleting them and waits for the futures to complete before getting the next PageSet. This issue was originally reported by Andrew Gaul [~gaul] [1] -Shri [1] https://github.com/jclouds/legacy-jclouds/issues/1087 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-589) Port fix for JCLOUDS-178 to non-keystone-v2.0 auth implementations
Shri Javadekar created JCLOUDS-589: -- Summary: Port fix for JCLOUDS-178 to non-keystone-v2.0 auth implementations Key: JCLOUDS-589 URL: https://issues.apache.org/jira/browse/JCLOUDS-589 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Fix For: 1.8.0 As part of fixing JCLOUDS-178 [1], jclouds added a mechanism for re-authenticating when it got back a 401 Unauthorized response. However, this fix only went into the Keystone v2.0 implementation. The problem still exists in the Keystone v1.1 and non-keystone auth implementations. This bug should track the porting of that fix to the keystone v1.1 and non-keystone auth implementations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-590) Port fix for JCLOUDS-178 to non-keystone-v2.0 auth implementations
Shri Javadekar created JCLOUDS-590: -- Summary: Port fix for JCLOUDS-178 to non-keystone-v2.0 auth implementations Key: JCLOUDS-590 URL: https://issues.apache.org/jira/browse/JCLOUDS-590 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Fix For: 1.8.0 As part of fixing JCLOUDS-178 [1], jclouds added a mechanism for re-authenticating when it got back a 401 Unauthorized response. However, this fix only went into the Keystone v2.0 implementation. The problem still exists in the Keystone v1.1 and non-keystone auth implementations. This bug should track the porting of that fix to the keystone v1.1 and non-keystone auth implementations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-598) Atmos ParseSystemMetadataFromHeaders parses blob hash incorrectly
[ https://issues.apache.org/jira/browse/JCLOUDS-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030052#comment-14030052 ] Shri Javadekar commented on JCLOUDS-598: Thanks for filing this Andrew. I'll take a look. Atmos ParseSystemMetadataFromHeaders parses blob hash incorrectly - Key: JCLOUDS-598 URL: https://issues.apache.org/jira/browse/JCLOUDS-598 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Andrew Gaul Assignee: Shri Javadekar Priority: Minor ParseSystemMetadataFromHeaders incorrectly looks for Content-MD5 instead of x-emc-wschecksum. Note that the value has has several formats and we should only parse those which have a MD5/0/ prefix. Fixing this will allow removal of the overridden and disabled checkMD5 methods in AtmosContainerIntegrationLiveTest, AtmosIntegrationLiveTest, and AtmosLiveTest. See b2016703adbc254347f0093171238b5f36e3fce4 for more context. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-628) PROPERTY_MAX_CONNECTIONS_PER_CONTEXT not honored for https connections
Shri Javadekar created JCLOUDS-628: -- Summary: PROPERTY_MAX_CONNECTIONS_PER_CONTEXT not honored for https connections Key: JCLOUDS-628 URL: https://issues.apache.org/jira/browse/JCLOUDS-628 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Fix For: 1.8.0, 1.7.4 The property PROPERTY_MAX_CONNECTIONS_PER_CONTEXT in jclouds governs how many connections are open to an endpoint per blobstore context. This along with PROPERTY_MAX_CONNECTION_REUSE makes sure that we don't run out of ports or fds on the underlying OS. I ran a simple test for verifying the PROPERTY_MAX_CONNECTIONS_PER_CONTEXT option. I created a container with ~10K blobs. I then used jclouds-cli to clear the container. I explicitly set jclouds.max-connections-per-context to 20. When the endpoint was http, things worked just fine. $ netstat -a | grep blah | grep ESTABL | wc -l 20 $ netstat -a | grep blah | grep ESTABL | wc -l 21 $ netstat -a | grep blah | grep ESTABL | wc -l 20 However, when the endpoint was https, this config option did not seem to be honored. The endpoint was https://api.atmosonline.com for the numbers below, but I also tried with a local swift endpoint which was https. $ netstat -a | grep atmos | grep ESTABL | wc -l 516 $ netstat -a | grep atmos | grep ESTABL | wc -l 602 I've already run into a situation where we exhausted all the available ports and operations such as container-clear or container-delete failed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-628) PROPERTY_MAX_CONNECTIONS_PER_CONTEXT not honored for https connections
[ https://issues.apache.org/jira/browse/JCLOUDS-628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064033#comment-14064033 ] Shri Javadekar commented on JCLOUDS-628: The JavaUrlHttpCommandExecutorService internally uses the java.net.HTTPUrlConnection and javax.net.ssl.HttpsURLConnection objects. JavaUrlHttpCommandExecutorService correctly sets the http.maxConnections system variable when it initializes. But looks like this does not get honored? Unfortunately, there is no way to find out whether a new HTTPUrlConnection object is internally using the same port as any other HTTPUrlConnection object. So the only way I could test is to create several connections to some https endpoint and separately use netstat -a to find out the open connections. I created a simple unit test [1] in BaseHttpCommandExecutorServiceIntegrationTest that created thousands of such http connections to a MockWebServer. It did show that the number of opened connections far exceeds PROPERTY_MAX_CONNECTIONS_PER_CONTEXT. However, I still don't know the root cause of this problem. One problem I found that looked suspicious was that in BaseHttpCommandExecutoreService.invoke(), the nativeRequest gets set to null in the try block. As a result when the finally block calls cleanup(), nativeRequest is null and therefore does not get disconnected. I fixed that and retried my test. But that does not seem to fix the problem. Also, when the InputStream from the URLConnection gets closed, IOExceptions if any are swallowed. I changed that too such that they don't get swallowed. However, this didn't provide any new information. I didn't find any documentation about this being a know problem with javax.net.ssl.HttpsURLConnection. [1] https://gist.github.com/shrinandj/be84c912eb3a0a6024b1 PROPERTY_MAX_CONNECTIONS_PER_CONTEXT not honored for https connections -- Key: JCLOUDS-628 URL: https://issues.apache.org/jira/browse/JCLOUDS-628 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Assignee: Shri Javadekar Fix For: 1.8.0, 1.7.4 The property PROPERTY_MAX_CONNECTIONS_PER_CONTEXT in jclouds governs how many connections are open to an endpoint per blobstore context. This along with PROPERTY_MAX_CONNECTION_REUSE makes sure that we don't run out of ports or fds on the underlying OS. I ran a simple test for verifying the PROPERTY_MAX_CONNECTIONS_PER_CONTEXT option. I created a container with ~10K blobs. I then used jclouds-cli to clear the container. I explicitly set jclouds.max-connections-per-context to 20. When the endpoint was http, things worked just fine. $ netstat -a | grep blah | grep ESTABL | wc -l 20 $ netstat -a | grep blah | grep ESTABL | wc -l 21 $ netstat -a | grep blah | grep ESTABL | wc -l 20 However, when the endpoint was https, this config option did not seem to be honored. The endpoint was https://api.atmosonline.com for the numbers below, but I also tried with a local swift endpoint which was https. $ netstat -a | grep atmos | grep ESTABL | wc -l 516 $ netstat -a | grep atmos | grep ESTABL | wc -l 602 I've already run into a situation where we exhausted all the available ports and operations such as container-clear or container-delete failed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (JCLOUDS-628) PROPERTY_MAX_CONNECTIONS_PER_CONTEXT not honored for https connections
[ https://issues.apache.org/jira/browse/JCLOUDS-628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064033#comment-14064033 ] Shri Javadekar edited comment on JCLOUDS-628 at 7/21/14 5:56 PM: - The JavaUrlHttpCommandExecutorService internally uses the java.net.HTTPUrlConnection and javax.net.ssl.HttpsURLConnection objects. JavaUrlHttpCommandExecutorService correctly sets the http.maxConnections system variable when it initializes. But looks like this does not get honored? Unfortunately, there is no way to find out whether a new HTTPUrlConnection object is internally using the same port as any other HTTPUrlConnection object. So the only way I could test is to create several connections to some https endpoint and separately use netstat -a to find out the open connections. I created a simple unit test [1] in BaseHttpCommandExecutorServiceIntegrationTest that created thousands of such http connections to a MockWebServer. It did show that the number of opened connections far exceeds PROPERTY_MAX_CONNECTIONS_PER_CONTEXT. However, I still don't know the root cause of this problem. One problem I found that looked suspicious was that in BaseHttpCommandExecutoreService.invoke(), the nativeRequest gets set to null in the try block. As a result when the finally block calls cleanup(), nativeRequest is null and therefore does not get disconnected. I fixed that and retried my test. But that does not seem to fix the problem. Also, when the InputStream from the URLConnection gets closed, IOExceptions if any are swallowed. I changed that too such that they don't get swallowed. However, this didn't provide any new information. I didn't find any documentation about this being a known problem with javax.net.ssl.HttpsURLConnection. [1] https://gist.github.com/shrinandj/be84c912eb3a0a6024b1 was (Author: shrinand): The JavaUrlHttpCommandExecutorService internally uses the java.net.HTTPUrlConnection and javax.net.ssl.HttpsURLConnection objects. JavaUrlHttpCommandExecutorService correctly sets the http.maxConnections system variable when it initializes. But looks like this does not get honored? Unfortunately, there is no way to find out whether a new HTTPUrlConnection object is internally using the same port as any other HTTPUrlConnection object. So the only way I could test is to create several connections to some https endpoint and separately use netstat -a to find out the open connections. I created a simple unit test [1] in BaseHttpCommandExecutorServiceIntegrationTest that created thousands of such http connections to a MockWebServer. It did show that the number of opened connections far exceeds PROPERTY_MAX_CONNECTIONS_PER_CONTEXT. However, I still don't know the root cause of this problem. One problem I found that looked suspicious was that in BaseHttpCommandExecutoreService.invoke(), the nativeRequest gets set to null in the try block. As a result when the finally block calls cleanup(), nativeRequest is null and therefore does not get disconnected. I fixed that and retried my test. But that does not seem to fix the problem. Also, when the InputStream from the URLConnection gets closed, IOExceptions if any are swallowed. I changed that too such that they don't get swallowed. However, this didn't provide any new information. I didn't find any documentation about this being a know problem with javax.net.ssl.HttpsURLConnection. [1] https://gist.github.com/shrinandj/be84c912eb3a0a6024b1 PROPERTY_MAX_CONNECTIONS_PER_CONTEXT not honored for https connections -- Key: JCLOUDS-628 URL: https://issues.apache.org/jira/browse/JCLOUDS-628 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Assignee: Shri Javadekar Fix For: 1.8.0 The property PROPERTY_MAX_CONNECTIONS_PER_CONTEXT in jclouds governs how many connections are open to an endpoint per blobstore context. This along with PROPERTY_MAX_CONNECTION_REUSE makes sure that we don't run out of ports or fds on the underlying OS. I ran a simple test for verifying the PROPERTY_MAX_CONNECTIONS_PER_CONTEXT option. I created a container with ~10K blobs. I then used jclouds-cli to clear the container. I explicitly set jclouds.max-connections-per-context to 20. When the endpoint was http, things worked just fine. $ netstat -a | grep blah | grep ESTABL | wc -l 20 $ netstat -a | grep blah | grep ESTABL | wc -l 21 $ netstat -a | grep blah | grep ESTABL | wc -l 20 However, when the endpoint was https, this config option did not seem to be honored. The endpoint was https://api.atmosonline.com for the numbers below, but I also tried with a local swift endpoint
[jira] [Commented] (JCLOUDS-635) Incorrect error codes when writing a blob to a non existent container for swift based object stores.
[ https://issues.apache.org/jira/browse/JCLOUDS-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072121#comment-14072121 ] Shri Javadekar commented on JCLOUDS-635: Pull request out for review: https://github.com/jclouds/jclouds-karaf/pull/50 Incorrect error codes when writing a blob to a non existent container for swift based object stores. Key: JCLOUDS-635 URL: https://issues.apache.org/jira/browse/JCLOUDS-635 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.2 Environment: Ubuntu 12.04 Reporter: Deepak Bobbarjung Assignee: Shri Javadekar Fix For: 1.8.0 Writing a blob to non existent container in an AWS objectstore fails with a Container not found error and a status code of 2. However writing a blob to a non existent container in a swift based objectstore (we tried both generic swift and rackspace) results in a IO error: Server rejected operation and a status code of 5. pre user@ubuntu-host:/opt/some/directory$ /opt/some/directory/jclouds-cli/bin/jclouds blobstore write --provider=swift-keystone --identity=redacted --credential=redacted --endpoint=http://swift-endpoint non-existent-container some-key /tmp/some-file shell: JAVA_HOME not set; results may vary log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /var/log/jclouds.log (Permission denied) ... .. IO error: Server rejected operation user@ubuntu-host:/opt/some/directory$ echo $? 5 user@ubuntu-host:/opt/some/direcoty$ /opt/some/directory/jclouds-cli/bin/jclouds blobstore write --provider=aws-s3 --identity=redacted --credential=redacted non-existent-container somekey /tmp/somefile shell: JAVA_HOME not set; results may vary log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /var/log/jclouds.log (Permission denied) at java.io.FileOutputStream.open(Native Method) at org.jclouds.cli.runner.Main.main(Main.java:111) Container not found: non-existent-container.s3.amazonaws.com not found: The specified bucket does not exist user@ubuntu-host:/opt/some/directory$ echo $? 2 /pre This makes it hard to programmatically catch the container not found error and create containers only if the container does not exist without hacky workarounds. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-641) rackspace-cloudfiles-us integration tests fail
[
https://issues.apache.org/jira/browse/JCLOUDS-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076525#comment-14076525
]
Shri Javadekar commented on JCLOUDS-641:
We've running nightly tests using the RegionScopedBlobStoreContext for a few
days now. Haven't run into problems involving the new openstack-swift provider.
Thanks for checking in though.
rackspace-cloudfiles-us integration tests fail
--
Key: JCLOUDS-641
URL: https://issues.apache.org/jira/browse/JCLOUDS-641
Project: jclouds
Issue Type: Bug
Components: jclouds-labs-openstack
Affects Versions: 1.8.0
Reporter: Andrew Gaul
Assignee: Jeremy Daggett
Labels: swift
{noformat}
$ mvn integration-test -pl :rackspace-cloudfiles-us -Plive
-Dtest.rackspace-cloudfiles-us.identity=${JCLOUDS_IDENTITY}
-Dtest.rackspace-cloudfiles-us.credential=${JCLOUDS_CREDENTIAL}
...
Failed tests:
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.testGetRange:373
» NullPointer
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.deleteObject:440-BaseBlobIntegrationTest.assertContainerEmptyDeleting:456
deleting qu?stion, we still have 1 blobs left in container gaul-blobstore0,
using encoding UTF-8 expected [0] but found [1]
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.testMetadata:605-BaseBlobIntegrationTest.validateMetadata:621
» NullPointer
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.deleteObject:440-BaseBlobIntegrationTest.assertContainerEmptyDeleting:456
deleting path/foo, we still have 1 blobs left in container gaul-blobstore2,
using encoding UTF-8 expected [0] but found [1]
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.testGetIfModifiedSince:213-BaseBlobIntegrationTest.addObjectAndValidateContent:398-BaseBlobStoreIntegrationTest.validateContent:293
null
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.testGetIfNoneMatch:348-BaseBlobStoreIntegrationTest.validateContent:293
null
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.testPutObjectStream:516
» HttpResponse
CloudFilesUSContainerIntegrationLiveTestBaseContainerIntegrationTest.deleteContainerIfEmptyWithContents:301
» IllegalState
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.deleteObject:438-BaseBlobStoreIntegrationTest.addBlobToContainer:269-BaseBlobStoreIntegrationTest.addBlobToContainer:281
» HttpResponse
CloudFilesUSBlobIntegrationLiveTestBaseBlobIntegrationTest.deleteObject:440-BaseBlobIntegrationTest.assertContainerEmptyDeleting:456
deleting p|pe, we still have 3 blobs left in container gaul-blobstore5,
using encoding UTF-8 expected [0] but found [3]
CloudFilesUSBlobSignerLiveTestBaseBlobSignerLiveTest.testSignGetUrlWithTime:92
» Authorization
CloudFilesUSBlobSignerLiveTestBaseBlobSignerLiveTest.testSignPutUrlWithTime:140
» Authorization
CloudFilesUSContainerIntegrationLiveTestBaseContainerIntegrationTest.testDirectory:206
[]
CloudFilesUSContainerIntegrationLiveTestBaseContainerIntegrationTest.testListContainerPrefix:241
expected [10] but found [0]
CloudFilesUSContainerIntegrationLiveTestBaseContainerIntegrationTest.testListContainerMarker:119-BaseContainerIntegrationTest.addAlphabetUnderRoot:340-BaseContainerIntegrationTest.assertContainerSize:345-BaseBlobStoreIntegrationTest.assertConsistencyAware:248-BaseBlobStoreIntegrationTest.assertConsistencyAware:235
expected [26] but found [19]
CloudFilesUSContainerIntegrationLiveTestBaseContainerIntegrationTest.testListContainerMaxResults:252-BaseContainerIntegrationTest.addAlphabetUnderRoot:340-BaseContainerIntegrationTest.assertContainerSize:345-BaseBlobStoreIntegrationTest.assertConsistencyAware:248-BaseBlobStoreIntegrationTest.assertConsistencyAware:235
expected [26] but found [0]
CloudFilesUSContainerIntegrationLiveTestSwiftContainerIntegrationLiveTest.testListRootUsesDelimiter:50
expected [16] but found [13] expected [true] but found [false]
Tests run: 55, Failures: 17, Errors: 0, Skipped: 6
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
Shri Javadekar created JCLOUDS-645: -- Summary: Incorrect endpoint url chosen for default region of HPCloud Object Storage Key: JCLOUDS-645 URL: https://issues.apache.org/jira/browse/JCLOUDS-645 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Fix For: 1.8.0 JClouds seems to be choosing the wrong endpoint from the service catalog returned by HPCloud Object Storage. From what I see there are two endpoints returned in the Object Storage part of the service catalog. D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] I do not have any region configured. With 1.7.3, the first url was getting used and my tests were succeeding. However, with 1.8.0 the second url get's chosen and my tests failed with a 404 Not Found error. See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081528#comment-14081528
]
Shri Javadekar commented on JCLOUDS-645:
HPCloudObjectStorageEndpointModule does this:
code class=java
private static SupplierURI getUriSupplier(String serviceType, String
apiVersion, RegionIdToURISupplier.Factory factory, String region) {
SupplierMapString, SupplierURI endpointsSupplier =
factory.createForApiTypeAndVersion(serviceType, apiVersion);
if (region.isEmpty()) {
return getLastValueInMap(endpointsSupplier);
- Shouldn't this get the first value in the map?
} else {
return getValueInMapOrNull(endpointsSupplier, region);
}
}
/code
Incorrect endpoint url chosen for default region of HPCloud Object Storage
--
Key: JCLOUDS-645
URL: https://issues.apache.org/jira/browse/JCLOUDS-645
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.8.0
Reporter: Shri Javadekar
Assignee: Chris Custine
Fix For: 1.8.0
JClouds seems to be choosing the wrong endpoint from the service catalog
returned by HPCloud Object Storage.
From what I see there are two endpoints returned in the Object Storage part
of the service catalog.
D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::]
publicURL:
https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n]
D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::]
publicURL:
https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n]
I do not have any region configured.
With 1.7.3, the first url was getting used and my tests were succeeding.
However, with 1.8.0 the second url get's chosen and my tests failed with a
404 Not Found error.
See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081528#comment-14081528
]
Shri Javadekar edited comment on JCLOUDS-645 at 7/31/14 9:42 PM:
-
HPCloudObjectStorageEndpointModule does this:
private static SupplierURI getUriSupplier(String serviceType, String
apiVersion, RegionIdToURISupplier.Factory factory, String region) {
SupplierMapString, SupplierURI endpointsSupplier =
factory.createForApiTypeAndVersion(serviceType, apiVersion);
if (region.isEmpty()) {
return getLastValueInMap(endpointsSupplier);
- Shouldn't this get the first value in the map?
} else {
return getValueInMapOrNull(endpointsSupplier, region);
}
}
was (Author: shrinand):
HPCloudObjectStorageEndpointModule does this:
code class=java
private static SupplierURI getUriSupplier(String serviceType, String
apiVersion, RegionIdToURISupplier.Factory factory, String region) {
SupplierMapString, SupplierURI endpointsSupplier =
factory.createForApiTypeAndVersion(serviceType, apiVersion);
if (region.isEmpty()) {
return getLastValueInMap(endpointsSupplier);
- Shouldn't this get the first value in the map?
} else {
return getValueInMapOrNull(endpointsSupplier, region);
}
}
/code
Incorrect endpoint url chosen for default region of HPCloud Object Storage
--
Key: JCLOUDS-645
URL: https://issues.apache.org/jira/browse/JCLOUDS-645
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.8.0
Reporter: Shri Javadekar
Assignee: Chris Custine
Fix For: 1.8.0
JClouds seems to be choosing the wrong endpoint from the service catalog
returned by HPCloud Object Storage.
From what I see there are two endpoints returned in the Object Storage part
of the service catalog.
D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::]
publicURL:
https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n]
D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::]
publicURL:
https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n]
I do not have any region configured.
With 1.7.3, the first url was getting used and my tests were succeeding.
However, with 1.8.0 the second url get's chosen and my tests failed with a
404 Not Found error.
See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081567#comment-14081567
]
Shri Javadekar commented on JCLOUDS-645:
Apparently not! KeystoneStorageEndpointModule does the same as above.
//Pick the matching region name (if any) otherwise just return an
arbitrary URL if no region name is set
//NOTE: The region string should never be null (it can be empty) if
this object was instantiated via guice
// as it pulls these named strings from a Properties object.
if (region.isEmpty()) {
return getLastValueInMap(endpointsSupplier);
} else {
return getValueInMapOrNull(endpointsSupplier, region);
}
Incorrect endpoint url chosen for default region of HPCloud Object Storage
--
Key: JCLOUDS-645
URL: https://issues.apache.org/jira/browse/JCLOUDS-645
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.8.0
Reporter: Shri Javadekar
Assignee: Chris Custine
Fix For: 1.8.0
JClouds seems to be choosing the wrong endpoint from the service catalog
returned by HPCloud Object Storage.
From what I see there are two endpoints returned in the Object Storage part
of the service catalog.
D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::]
publicURL:
https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n]
D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::]
publicURL:
https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n]
I do not have any region configured.
With 1.7.3, the first url was getting used and my tests were succeeding.
However, with 1.8.0 the second url get's chosen and my tests failed with a
404 Not Found error.
See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081773#comment-14081773 ] Shri Javadekar commented on JCLOUDS-645: One difference that I find between jclouds 1.7.x and jclouds 1.8.0 is that when LocationIdToURIFromAccessForTypeAndVersion.get() is invoked, in jclouds 1.7.0, the apiVersion is set to 1.0 whereas in jclouds 1.8.0, the apiVersion is null. In the service catalog returned by HPCloud the version for region-a is 1.0, region-b is 1. That explains why region-a is selected in jclouds-1.7.x. The problem now is to identify why jclouds-1.8.0 does not have the version set to 1.0. Incorrect endpoint url chosen for default region of HPCloud Object Storage -- Key: JCLOUDS-645 URL: https://issues.apache.org/jira/browse/JCLOUDS-645 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Assignee: Chris Custine Fix For: 1.8.0 JClouds seems to be choosing the wrong endpoint from the service catalog returned by HPCloud Object Storage. From what I see there are two endpoints returned in the Object Storage part of the service catalog. D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] I do not have any region configured. With 1.7.3, the first url was getting used and my tests were succeeding. However, with 1.8.0 the second url get's chosen and my tests failed with a 404 Not Found error. See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081941#comment-14081941 ] Shri Javadekar commented on JCLOUDS-645: Do you know what commit changed the default apiVersion from 1.0 to null? Also, do we know what are the other repercussions of this change? What happens if I set the region as region-a.geo-1 but set the api version to 1 (and not 1.0)? Would smiply explicitly setting the apiVersion to 1.0 be a workaround? I tried doing that but don't see any change in the behavior. region-b continues to get selected instead of region-a. The behavior of the system may not have been explicitly defined earlier and I agree downstream code ideally should not depend on this. However, changing the behavior of the system without prior notice and adequate time for action doesn't seem like the best approach to me. Incorrect endpoint url chosen for default region of HPCloud Object Storage -- Key: JCLOUDS-645 URL: https://issues.apache.org/jira/browse/JCLOUDS-645 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Assignee: Chris Custine Fix For: 1.8.0 JClouds seems to be choosing the wrong endpoint from the service catalog returned by HPCloud Object Storage. From what I see there are two endpoints returned in the Object Storage part of the service catalog. D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] I do not have any region configured. With 1.7.3, the first url was getting used and my tests were succeeding. However, with 1.8.0 the second url get's chosen and my tests failed with a 404 Not Found error. See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082626#comment-14082626 ] Shri Javadekar commented on JCLOUDS-645: I agree with the larger point about arbitrary endpoint resolution. And maybe we should make it mandatory for users to specify a region. Would be great if we give prior notice about it though. One other way to do this would be to use a specific region name as the default rather than first or last index into an array. People have accounts with cloud providers. They may not be concerned about where (what region) the data gets stored. If they were concerned about the region, they would've already set the region. For the default case, if suddenly after an upgrade they stop having access to their data because a different region was getting selected would be a bad user experience. Neways, let's have this discussion outside of this bug. To summarize what we have here: In jclouds 1.7.x, a user was required to set jclouds.region to region-b.geo-1 *and* set the apiVersion to 1 to work with region-b Ideally one wouldn't have to specify the apiVersion but for whatever reason HPCloud Object Storage has set the version to 1 instead of 1.0. A user only had to set jclouds.region to region-a.geo-1 (no need to set apiVersion) or not specify any region to work with region-a. This worked because the default apiVersion was 1.0. In jclouds 1.8.0, a user can set jclouds.region to region-b.geo-1 to work with region-b or region-a.geo-1 to work with region-a. The user never has to specify the apiVersion. Not specifying any value for jclouds.region results in region-b getting selected. This happens because the apiVersion is null and region-b is the last index in the array. If jclouds.region is null, both the above approaches suffer from a problem. jclouds 1.7.x breaks if the Keystone endpoint of HPCloud Object Storage changes their service catalog to add another region with apiVersion 1.0. jclouds 1.8.0 breaks if the Keystone endpoint in HPCloud Object Storage add another region to their service catalog. If we want to continue with what we have, I'll run some tests explicitly setting the jclouds.region and jclouds.apiVersion to make sure everything works. Incorrect endpoint url chosen for default region of HPCloud Object Storage -- Key: JCLOUDS-645 URL: https://issues.apache.org/jira/browse/JCLOUDS-645 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Assignee: Chris Custine Fix For: 1.8.0 JClouds seems to be choosing the wrong endpoint from the service catalog returned by HPCloud Object Storage. From what I see there are two endpoints returned in the Object Storage part of the service catalog. D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] I do not have any region configured. With 1.7.3, the first url was getting used and my tests were succeeding. However, with 1.8.0 the second url get's chosen and my tests failed with a 404 Not Found error. See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-645) Incorrect endpoint url chosen for default region of HPCloud Object Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082663#comment-14082663 ] Shri Javadekar commented on JCLOUDS-645: Unfortunately this is also affects jclouds-cli. We use it as part of some scripts that read/write files from blobstores and were not setting the jclouds.region environment variable. Now we'll be required to do it. And this requirement will only be for HPCloud. Incorrect endpoint url chosen for default region of HPCloud Object Storage -- Key: JCLOUDS-645 URL: https://issues.apache.org/jira/browse/JCLOUDS-645 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Assignee: Chris Custine Fix For: 1.8.0 JClouds seems to be choosing the wrong endpoint from the service catalog returned by HPCloud Object Storage. From what I see there are two endpoints returned in the Object Storage part of the service catalog. D 07-31 13:21:22.122 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-a.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] D 07-31 13:21:22.124 pool-1-thread-1 jclouds.wire:59 |::] publicURL: https:\/\/region-b.geo-1.objects.hpcloudsvc.com\/v1\/53176293441764,[\n] I do not have any region configured. With 1.7.3, the first url was getting used and my tests were succeeding. However, with 1.8.0 the second url get's chosen and my tests failed with a 404 Not Found error. See this for more details: http://pastie.org/private/yzxbnvxwidajalcucwflwa -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-648) putBlob operation does not throw a ContainerNotFoundException against Swift based blobstores
Shri Javadekar created JCLOUDS-648: -- Summary: putBlob operation does not throw a ContainerNotFoundException against Swift based blobstores Key: JCLOUDS-648 URL: https://issues.apache.org/jira/browse/JCLOUDS-648 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Fix For: 2.0.0 As per the documentation[1], the putBlob method of the BlobStore object should return throw a ContainerNotFoundException if the container does not exist. This behaves correctly against AWS. However, against Swift and Swift based blobstores such CloudFiles-US, I don't see a ContainerNotFoundException being thrown. Instead, I see the following HttpResponseException being thrown. org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_39bd836e-89b7-43d5-9b88-ddd3b0e68353/cloud-testning-5/31bbdae7404d5fb6 HTTP/1.1 at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:162) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:129) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:95) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:56) at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) at com.sun.proxy.$Proxy72.putObject(Unknown Source) at org.jclouds.openstack.swift.blobstore.SwiftBlobStore.putBlob(SwiftBlobStore.java:204) [1] http://jclouds.apache.org/reference/javadoc/1.7.x/org/jclouds/blobstore/BlobStore.html#putBlob(java.lang.String, org.jclouds.blobstore.domain.Blob) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-650) getBlob for non-existent containers does not throw a ContainerNotFoundException
Shri Javadekar created JCLOUDS-650: -- Summary: getBlob for non-existent containers does not throw a ContainerNotFoundException Key: JCLOUDS-650 URL: https://issues.apache.org/jira/browse/JCLOUDS-650 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Priority: Minor Fix For: 2.0.0 As per the documentation[1], the BlobStore object should throw a ContainerNotFoundException if the container does not exist when the getBlob API is invoked. However, I see that the call to getBlob silently succeeds when the container doesn't exist. I tested this against aws-s3, swift and cloudfiles-us. [1] http://jclouds.apache.org/reference/javadoc/1.7.x/org/jclouds/blobstore/BlobStore.html#getBlob(java.lang.String, java.lang.String) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (JCLOUDS-648) putBlob operation does not throw a ContainerNotFoundException against Swift based blobstores
[ https://issues.apache.org/jira/browse/JCLOUDS-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085592#comment-14085592 ] Shri Javadekar commented on JCLOUDS-648: This is so weird. Sometimes I get a org.jclouds.http.HttpResponseException while some other times I get org.jclouds.rest.ResourceNotFoundException: org.jclouds.http.HttpResponseException. putBlob operation does not throw a ContainerNotFoundException against Swift based blobstores Key: JCLOUDS-648 URL: https://issues.apache.org/jira/browse/JCLOUDS-648 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.3 Reporter: Shri Javadekar Fix For: 2.0.0 As per the documentation[1], the putBlob method of the BlobStore object should return throw a ContainerNotFoundException if the container does not exist. This behaves correctly against AWS. However, against Swift and Swift based blobstores such CloudFiles-US, I don't see a ContainerNotFoundException being thrown. Instead, I see the following HttpResponseException being thrown. org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_39bd836e-89b7-43d5-9b88-ddd3b0e68353/cloud-testning-5/31bbdae7404d5fb6 HTTP/1.1 at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:162) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:129) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:95) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:56) at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) at com.sun.proxy.$Proxy72.putObject(Unknown Source) at org.jclouds.openstack.swift.blobstore.SwiftBlobStore.putBlob(SwiftBlobStore.java:204) [1] http://jclouds.apache.org/reference/javadoc/1.7.x/org/jclouds/blobstore/BlobStore.html#putBlob(java.lang.String, org.jclouds.blobstore.domain.Blob) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-684) Add option to delete containers without deleting objects inside it
Shri Javadekar created JCLOUDS-684: -- Summary: Add option to delete containers without deleting objects inside it Key: JCLOUDS-684 URL: https://issues.apache.org/jira/browse/JCLOUDS-684 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Fix For: 2.0.0 Some blobstores such as Microsoft Azure[1] have the ability to delete containers even when they are not empty. These object stores simply return after marking the containers for deletion and asynchronously delete the blobs and container in which the blobs exist. jclouds should have an option by which calling deleteContainer() should simply issue the delete request to the blobstore. It should not internally try and delete all the blobs in the container. [1] http://msdn.microsoft.com/en-us/library/azure/dd179408.aspx -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (JCLOUDS-684) Add option to delete containers without deleting objects inside it
[ https://issues.apache.org/jira/browse/JCLOUDS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shri Javadekar reopened JCLOUDS-684: Great to know that jclouds already does the right thing for Azure. However, an option to delete containers without first clearing the container will also be useful for other blobstores. Cleversafe is a private object store with an S3 API. It supports deleting containers even they are not empty. Add option to delete containers without deleting objects inside it -- Key: JCLOUDS-684 URL: https://issues.apache.org/jira/browse/JCLOUDS-684 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Some blobstores such as Microsoft Azure[1] have the ability to delete containers even when they are not empty. These object stores simply return after marking the containers for deletion and asynchronously delete the blobs and container in which the blobs exist. jclouds should have an option by which calling deleteContainer() should simply issue the delete request to the blobstore. It should not internally try and delete all the blobs in the container. [1] http://msdn.microsoft.com/en-us/library/azure/dd179408.aspx -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (JCLOUDS-684) Add option to delete containers without deleting objects inside it
[ https://issues.apache.org/jira/browse/JCLOUDS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109831#comment-14109831 ] Shri Javadekar edited comment on JCLOUDS-684 at 8/25/14 10:42 PM: -- Great to know that jclouds already does the right thing for Azure. However, an option to delete containers without first clearing the container will also be useful for other blobstores. Cleversafe is a private object store with an S3 API. It supports deleting containers when they are not empty. was (Author: shrinand): Great to know that jclouds already does the right thing for Azure. However, an option to delete containers without first clearing the container will also be useful for other blobstores. Cleversafe is a private object store with an S3 API. It supports deleting containers even they are not empty. Add option to delete containers without deleting objects inside it -- Key: JCLOUDS-684 URL: https://issues.apache.org/jira/browse/JCLOUDS-684 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Some blobstores such as Microsoft Azure[1] have the ability to delete containers even when they are not empty. These object stores simply return after marking the containers for deletion and asynchronously delete the blobs and container in which the blobs exist. jclouds should have an option by which calling deleteContainer() should simply issue the delete request to the blobstore. It should not internally try and delete all the blobs in the container. [1] http://msdn.microsoft.com/en-us/library/azure/dd179408.aspx -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (JCLOUDS-712) ETAG Header with tempurls in the 'openstack-swift' provider consistently results in a 503 Service unavailable response
Shri Javadekar created JCLOUDS-712: -- Summary: ETAG Header with tempurls in the 'openstack-swift' provider consistently results in a 503 Service unavailable response Key: JCLOUDS-712 URL: https://issues.apache.org/jira/browse/JCLOUDS-712 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Fix For: 1.8.1 I am testing the new openstack-swift provider. For testing purposes, I'm using Rackspace Cloudfiles US as the Swift blobstore. I set the: provider= 'openstack-swift' region='IAD' endpoint='https://identity.api.rackspacecloud.com/v2.0' While testing PUTs with tempurls, if the ETAG header representing the MD5 checksum is absent, things work just fine. pre D 09-09 00:34:42.167 main jclouds.headers:56 |::] PUT https://storage101.iad3.clouddrive.com/v1/MossoCloudFS_2eb7963d-0f04-4cad-93e0-2227eab8bd7b/my-bucket/my-blob?temp_url_sig=1f66e0b4baa197c416ed81c44927db6c69c3c2f9temp_url_expires=1410248982 HTTP/1.1 D 09-09 00:34:42.168 main jclouds.headers:56 |::] Content-Type: application/octet-stream D 09-09 00:34:42.168 main jclouds.headers:56 |::] Content-Length: 0 D 09-09 00:34:42.619 main o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Receiving response -1558734241: HTTP/1.1 201 Created D 09-09 00:34:42.620 main jclouds.headers:56 |::] HTTP/1.1 201 Created /pre However, if the ETAG header is present, I *always* get a 503 Service Unavailable response from Swift. pre D 09-09 00:46:19.533 main jclouds.headers:56 |::] PUT https://storage101.iad3.clouddrive.com/v1/MossoCloudFS_2eb7963d-0f04-4cad-93e0-2227eab8bd7b/my-bucket/my-blob?temp_url_sig=6c03d206cee09054be9189e5b40fb30dc6cb54aetemp_url_expires=1410249679 HTTP/1.1 D 09-09 00:46:19.534 main jclouds.headers:56 |::] ETag: 93b885adfe0da089cdf634904fd59f71 D 09-09 00:46:19.534 main jclouds.headers:56 |::] Content-Type: application/octet-stream D 09-09 00:46:19.534 main jclouds.headers:56 |::] Content-Length: 0 D 09-09 00:46:19.713 main o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Receiving response -379684309: HTTP/1.1 503 Service Unavailable D 09-09 00:46:19.714 main jclouds.headers:56 |::] HTTP/1.1 503 Service Unavailable D 09-09 00:46:19.714 main jclouds.headers:56 |::] Date: Tue, 09 Sep 2014 07:46:19 GMT /pre Not sure if this happens only against Rackspace CloudFiles. Haven't tested against other Swift blobstores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-712) ETAG Header with tempurls in the 'openstack-swift' provider consistently results in a 503 Service unavailable response
[ https://issues.apache.org/jira/browse/JCLOUDS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126729#comment-14126729 ] Shri Javadekar commented on JCLOUDS-712: [~jdaggett] Any ideas? ETAG Header with tempurls in the 'openstack-swift' provider consistently results in a 503 Service unavailable response -- Key: JCLOUDS-712 URL: https://issues.apache.org/jira/browse/JCLOUDS-712 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.8.0 Reporter: Shri Javadekar Fix For: 1.8.1 I am testing the new openstack-swift provider. For testing purposes, I'm using Rackspace Cloudfiles US as the Swift blobstore. I set the: provider= 'openstack-swift' region='IAD' endpoint='https://identity.api.rackspacecloud.com/v2.0' While testing PUTs with tempurls, if the ETAG header representing the MD5 checksum is absent, things work just fine. pre D 09-09 00:34:42.167 main jclouds.headers:56 |::] PUT https://storage101.iad3.clouddrive.com/v1/MossoCloudFS_2eb7963d-0f04-4cad-93e0-2227eab8bd7b/my-bucket/my-blob?temp_url_sig=1f66e0b4baa197c416ed81c44927db6c69c3c2f9temp_url_expires=1410248982 HTTP/1.1 D 09-09 00:34:42.168 main jclouds.headers:56 |::] Content-Type: application/octet-stream D 09-09 00:34:42.168 main jclouds.headers:56 |::] Content-Length: 0 D 09-09 00:34:42.619 main o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Receiving response -1558734241: HTTP/1.1 201 Created D 09-09 00:34:42.620 main jclouds.headers:56 |::] HTTP/1.1 201 Created /pre However, if the ETAG header is present, I *always* get a 503 Service Unavailable response from Swift. pre D 09-09 00:46:19.533 main jclouds.headers:56 |::] PUT https://storage101.iad3.clouddrive.com/v1/MossoCloudFS_2eb7963d-0f04-4cad-93e0-2227eab8bd7b/my-bucket/my-blob?temp_url_sig=6c03d206cee09054be9189e5b40fb30dc6cb54aetemp_url_expires=1410249679 HTTP/1.1 D 09-09 00:46:19.534 main jclouds.headers:56 |::] ETag: 93b885adfe0da089cdf634904fd59f71 D 09-09 00:46:19.534 main jclouds.headers:56 |::] Content-Type: application/octet-stream D 09-09 00:46:19.534 main jclouds.headers:56 |::] Content-Length: 0 D 09-09 00:46:19.713 main o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Receiving response -379684309: HTTP/1.1 503 Service Unavailable D 09-09 00:46:19.714 main jclouds.headers:56 |::] HTTP/1.1 503 Service Unavailable D 09-09 00:46:19.714 main jclouds.headers:56 |::] Date: Tue, 09 Sep 2014 07:46:19 GMT /pre Not sure if this happens only against Rackspace CloudFiles. Haven't tested against other Swift blobstores. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-635) Incorrect error codes when writing a blob to a non existent container for swift based object stores.
[ https://issues.apache.org/jira/browse/JCLOUDS-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127695#comment-14127695 ] Shri Javadekar commented on JCLOUDS-635: To add to what [~gaul] said, if we explicitly strip the expect header, jclouds-cli correctly returns ENOENT (2). Incorrect error codes when writing a blob to a non existent container for swift based object stores. Key: JCLOUDS-635 URL: https://issues.apache.org/jira/browse/JCLOUDS-635 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 1.7.2 Environment: Ubuntu 12.04 Reporter: Deepak Bobbarjung Assignee: Shri Javadekar Fix For: 2.0.0 Writing a blob to non existent container in an AWS objectstore fails with a Container not found error and a status code of 2. However writing a blob to a non existent container in a swift based objectstore (we tried both generic swift and rackspace) results in a IO error: Server rejected operation and a status code of 5. pre user@ubuntu-host:/opt/some/directory$ /opt/some/directory/jclouds-cli/bin/jclouds blobstore write --provider=swift-keystone --identity=redacted --credential=redacted --endpoint=http://swift-endpoint non-existent-container some-key /tmp/some-file shell: JAVA_HOME not set; results may vary log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /var/log/jclouds.log (Permission denied) ... .. IO error: Server rejected operation user@ubuntu-host:/opt/some/directory$ echo $? 5 user@ubuntu-host:/opt/some/direcoty$ /opt/some/directory/jclouds-cli/bin/jclouds blobstore write --provider=aws-s3 --identity=redacted --credential=redacted non-existent-container somekey /tmp/somefile shell: JAVA_HOME not set; results may vary log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /var/log/jclouds.log (Permission denied) at java.io.FileOutputStream.open(Native Method) at org.jclouds.cli.runner.Main.main(Main.java:111) Container not found: non-existent-container.s3.amazonaws.com not found: The specified bucket does not exist user@ubuntu-host:/opt/some/directory$ echo $? 2 /pre This makes it hard to programmatically catch the container not found error and create containers only if the container does not exist without hacky workarounds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (JCLOUDS-727) openstack-swift provider's BlobRequestSigner.signPutBlob does not set payload and content lenght
Shri Javadekar created JCLOUDS-727:
--
Summary: openstack-swift provider's BlobRequestSigner.signPutBlob
does not set payload and content lenght
Key: JCLOUDS-727
URL: https://issues.apache.org/jira/browse/JCLOUDS-727
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.8.0
Reporter: Shri Javadekar
Fix For: 1.8.1
The BlobRequestSigner.signPutBlob method in the new openstack-swift provider
does seem to set the payload and content length. Callers have to explicitly set
these on the returned HttpRequest object.
This is different from at least the swift-keystone provider.
HttpRequest returned by the with the swift-keystone provider:
Signed put request: {method=PUT,
endpoint=http://10.50.23.43:8080/v1/AUTH_abcd/cloud-testing/erwe?temp_url_sig=6418a27f151ce00e044d798e2908bccd8ba9d3abtemp_url_expires=1411433648,
headers={Expect=[100-continue], ETag=[9c48f2a4023ba27471a8fa259a000fa4,
9c48f2a4023ba27471a8fa259a000fa4], Content-Type=[application/octet-stream]},
payload=[content=true, contentMetadata=[contentDisposition=null,
contentEncoding=null, contentLanguage=null, contentLength=null,
contentMD5=null, contentType=application/unknown, expires=null], written=false]}
HttpRequest returned by the new openstack-swift provider:
Signed put request : {method=PUT,
endpoint=https://storage101.iad3.clouddrive.com/
v1/MossoCloudFS_asd/cloud-testing/sfsdfsd?temp_url_sig=99c9676a1c
ef33ba408b6e35082a056f11a7d717temp_url_expires=1411454235, headers={}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (JCLOUDS-510) Head-of-line blocking problem with DeleteAllKeysInList
[ https://issues.apache.org/jira/browse/JCLOUDS-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shri Javadekar closed JCLOUDS-510. -- Resolution: Fixed Head-of-line blocking problem with DeleteAllKeysInList -- Key: JCLOUDS-510 URL: https://issues.apache.org/jira/browse/JCLOUDS-510 Project: jclouds Issue Type: Improvement Components: jclouds-blobstore Affects Versions: 1.7.1, 1.8.0 Reporter: Shri Javadekar Assignee: Shri Javadekar Fix For: 1.8.0 The current implementation of DeleteAllKeysInList suffers from the head-of-line blocking problem. It gets a PageSet of blobs from the container, creates futures for deleting them and waits for the futures to complete before getting the next PageSet. This issue was originally reported by Andrew Gaul [~gaul] [1] -Shri [1] https://github.com/jclouds/legacy-jclouds/issues/1087 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-874) IllegalArgumentException while setting incorrect MD5
[
https://issues.apache.org/jira/browse/JCLOUDS-874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387535#comment-14387535
]
Shri Javadekar commented on JCLOUDS-874:
How was this working in jclouds-1.8.1?
IllegalArgumentException while setting incorrect MD5
Key: JCLOUDS-874
URL: https://issues.apache.org/jira/browse/JCLOUDS-874
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.9.0
Reporter: Shri Javadekar
Assignee: Andrew Gaul
Priority: Minor
I had a negative test case that would set the incorrect md5 checksum of a
blob. After upgrade to jclouds-1.9.0, this test fails with an
IllegalArgumentException with a message that the md5 checksum should be 128
bytes long but is 144 bytes.
Here's the code for the test:
{code:title=TestIncorrectMD5.java|borderStyle=solid}
// Create a blob and set the incorrect md5 checksum
public void testIncorrectMd5() throws Exception {
{
String blobName = test;
ByteSource input = ByteSource.wrap(testdata.getBytes(
StandardCharsets.UTF_8));
String fakeHash = fakehash;
byte[] blobHash = fakeHash.getBytes(Charsets.UTF_16);
blobStore.blobBuilder(blobName)
.payload(input)
.contentLength(input.size())
.contentMD5(HashCode.fromBytes(blobHash))
.build();
}
{code}
I wonder if computing of the HashCode has changed. If changed the fakeHash
String from fakehash to fakehas, the test worked fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
