Fwd: TAC supporting Berlin Buzzwords

[Andrew Phillips]

 Original Message 
Subject: TAC supporting Berlin Buzzwords
Date: 2023-03-24 05:56
From: Gavin McDonald 
To: gmcdon...@apache.org
Reply-To: priv...@jclouds.apache.org

PMCs,

Please forward to your dev and user lists.

Hi All,

The ASF Travel Assistance Committee is supporting taking up to six (6)
people

to attend Berlin Buzzwords In June this year.

This includes Conference passes, and travel & accommodation as needed.

Please see our website at https://tac.apache.org for more information
and how to apply.

Applications close on 15th April.

Good luck to those that apply.

Gavin McDonald (VP TAC)

Re: SimpleDateFormat.rfc822DateParse exiting while holding lock

[Andrew Phillips]

Should I report this as an issue in the jclouds tracker?


That would be useful, yes, especially if you have any thoughts around 
how this could reliably be reproduced.


If you have a fix in mind, a test and pull request would definitely also 
be great.


Thanks for reporting this!

ap

Re: jclouds-labs

[Andrew Phillips]

Also +1 - thanks for kicking off this thread!

ap

Re: Missing jclouds snapshot from the apache repository

[Andrew Phillips]

But clicking on those you get a 404... Probably something wrong with
the Apache snapshots repo to be taken to ASF infra?


Is this the right thing for an INFRA ticket?

@Timur: thanks for flagging this!

ap

Re: remove Clojure support?

[Andrew Phillips]

These bindings add to jclouds maintenance and make evolving core APIs
harder.  Further they really belong outside of the jclouds repository.
I would like to remove these in 2.1.0


+1 to remove. It should be easy enough to keep them in a separate 
repository dependent on jclouds, if desired. Is there something like a 
code scan for potential usage that we could run?


Regards

ap

Re: Openstack-Swift

[Andrew Phillips]

Hi Ashkan


* Failed in command: …., org.jclouds.http.HttpResponseException:
command: POST https://identity.api.rackspacecloud.com/v2.0/tokens
HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content:
[{"badRequest":{"code":400,"message":"Invalid json request body"}}]


Could you enable wire and header logging (see [1]) so we can see what 
the exact request is that jclouds is sending? Given that your cURL 
command succeeds, we would expect some difference between that and the 
command jclouds is constructing.


Please check for any sensitive values in the log output before sharing, 
I should add.


Regards

ap

[1] http://jclouds.apache.org/reference/logging/#configure

Fwd: Jclouds with HDFS file system

[Andrew Phillips]

[forwarding to user@...]

 Original Message 
Subject: Jclouds with HDFS file system
Date: 2017-03-22 22:43
From: Chaitanya Anumalasetty 
To: andr...@apache.org

Hi Andrew,

I need one example Java code to upload/download file from HDFS file
system using Jclouds.

If you can help me that will be great.
Thanks in advance.

Re: jclouds exception

[Andrew Phillips]

so which version do i need to use ?


The version of jclouds you are pulling in was compiled against Guava 
16.0.1; from the output your provided, it appears that your code is 
trying to use Guava 18.0, which seems to be coming from spark-core.


Two options I can think of are:

1. Exclude Guava from spark-core transitive dependencies, and explicitly 
add Guava 16.0.1 as a dependency. If spark-core uses Guava 17 or 18 
features, this will not work, but it's probably at least worth a try.


2. Compile jclouds from source, changing the Guava version [1] to 18.0. 
According to the compatibility build [2], this *should* work, but I'm 
not sure how up-to-date that information is. Then depend on this 
newly-compiled jclouds version in your project.


Regards

ap

[1] https://github.com/jclouds/jclouds/blob/master/project/pom.xml#L225
[2] https://jclouds.ci.cloudbees.com/job/jclouds-compat/

Re: jclouds with trystack

[Andrew Phillips]

Hi Rupinder


I did that too. Same results.  I'll send  the output later in evg.


Could you try a cURL or similar from the box, with appropriate headers 
etc., to verify whether the TryStack endpoint is responding correctly if 
we take jclouds out of the loop?


Thanks!

ap

Fwd: Re:Is there any way to add custom headers on HTTP headers to Rackspace Cloudfiles?

[Andrew Phillips]

Er, that should have been "user@..." - sorry!

ap

 Original Message 
Subject: Re:Is there any way to add custom headers on HTTP headers to 
Rackspace Cloudfiles?

Date: 2017-02-02 15:59
From: Andrew Phillips <andr...@apache.org>
To: us...@jclouds.apache.org
Copy: jwolf...@gmail.com

[Forwarding Jeff's email to users@. @Jeff: notifications@ is just for 
automated emails.]


"In order to troubleshoot some issues downloading objects from Rackspace 
Cloudfiles, they are requesting that I add a "x-trans-id" header to my 
API requests.  Is there any way to instruct the jclouds client library 
to do this for me?  Seems like it should be straightforward to insert a 
custom HTTP header, but I can't find any way to do it."

Re: Example end-to-end walkthrough?

[Andrew Phillips]

I was completely confused by being directed to the github for the
azurecompute-arm as I thought that was an example I should be
following.



Thanks for the update! Are things working now, or is there still 
something that's not doing what it should?


Also, do you have any thoughts on what we could do to make the 
documentation clearer?


ap

Re: Example end-to-end walkthrough?

[Andrew Phillips]

But in the end, like in any Java project, the only thing you
need to use jclouds is your project is to have the jclouds .jar
files in your classpath.


Just for the avoidance of doubt: you will need the jclouds JAR files and 
the JARs for any dependencies needed by the particular jclouds JARs you 
are using.


Retrieving these dependencies is what the Maven dependency plugin's 
"copy-dependencies" goal is about:


https://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html

To Ignasi's earlier point: if you *do* happen to be looking for a 
standalone executable that could be used to interact with cloud APIs, 
the jclouds CLI is one possible option you may want to look at:


https://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html

Regards

ap

Re: Example end-to-end walkthrough?

[Andrew Phillips]

Hi Teo

The jclouds-examples repo [1] may help - I don't think it has an 
Azure-specific example at present, but the general "compute-basics" app 
should hopefully give a better idea of how the end-to-end flow might 
look like:


https://github.com/jclouds/jclouds-examples/blob/master/compute-basics/src/main/java/org/jclouds/examples/compute/basics/MainApp.java

It uses the jclouds' compute abstraction, but should also work with the 
Azure providers.


Of course, suggestions and/or PRs for ways we can improve the initial 
"how does this all fit together?" documentation are also much 
appreciated - definitely things we can do there.


Regards

ap

[1] https://github.com/jclouds/jclouds-examples/

Nice Interview with Ignasi in JAXenter about Apache jclouds

[Andrew Phillips]

Hi all

Just wanted to share a nice interview in JAXenter with Ignasi about 
Apache jclouds:


https://jaxenter.com/jclouds-project-devops-130667.html

Apart from anything else, I think it's a great example of how we can 
explain collaborative work on projects at Apache and educate the media 
and readers that, as Ignasi put it very well, "there is no 'me' in 
Apache projects."


Regards

ap

Re: Fwd: Delivery Status Notification (Failure)

[Andrew Phillips]

Hi Ken


I haven't made a break through on this yet. Any pointers will be very
useful.


Thanks for the ping - sorry to hear things are still not working. I hope 
someone on this list with more knowledge of the Openstack provider will 
be able to help; I just wanted to point out that there may be a bit of a 
delay at present as the release of jclouds 2.0 is currently in progress.


Thanks for your patience!

ap

Re: Fwd: Delivery Status Notification (Failure)

[Andrew Phillips]

Hi Ken


I want to use Jcloud swift and nova, the connection always time outs
after
successfully authenticating...


Could you provide a short code sample (with passwords etc. obfuscated) 
to help us understand how the connection is configured?


Regards

ap

Re: ApacheHC need yo support TLS1.2

[Andrew Phillips]

Hi Dileep


Due to Various security vulnerabilities, Many Servers supports/enables
 only TLS1.2 Protocols.

ApacheHCHttpCommandExecutorServiceModule Needs change to support
TLS1.2 Protocols only.


Have you considered using the OkHttp driver [1] instead? As far as I 
recall, that should allow you to support TLS 1.2 only.


Regards

ap

[1] https://github.com/apache/jclouds/tree/master/drivers/okhttp

Re: How to use jclouds with OpenStack Swift?

[Andrew Phillips]

But I got an error:


Ah, sorry to hear :-( Glad we're getting closer, though.

Could you enable header and wire logging [1], run the request again and 
please put the output (with sensitive information removed) into a Gist 
or Pastie?


That will hopefully give us a bit more insight into the exact request 
being made, and why your Openstack installation isn't happy with it.


Regards

ap

[1] https://jclouds.apache.org/reference/logging/

Re: How to use jclouds with OpenStack Swift?

[Andrew Phillips]

What is "RegionOne" ? First of all if we talk about regions in Swift
then it can't be a String. It is Integer value. That is why I can't
understand what is "RegionOne" and how to create a simple container
with SwiftApi.


Could you call "getConfiguredRegions" on your API instance to see which 
regions are available?


https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/openstack/swift/v1/SwiftApi.html#getConfiguredRegions()

Regards

ap

Re: Fwd: Re: cannot find symbol: variable SINGLE_NO_ARG_INVOCATION

[Andrew Phillips]

mvn clean package -DskipTests &> mvncleanout1 attached.


So close, this time!

[INFO] all ... SUCCESS 
[1.144s]

[INFO] Apache jclouds  FAILURE
[2.181s]


From the logs, it appears that the code you are building is failing a 
license check:


[INFO] Rat check: Summary of files. Unapproved: 2 unknown: 2 generated: 
0 approved: 1 licence.

[INFO]

...
[ERROR] Failed to execute goal 
org.apache.rat:apache-rat-plugin:0.10:check (default) on project 
jclouds: Too many files with unapproved license: 2 See RAT report in: 
/home/user/IdeaProjects/JcloudNeedHelp/target/rat.txt -> [Help 1]

[ERROR]

Could you have a look at that file to see what it's complaining about?

Regards

ap

Re: Fwd: Re: cannot find symbol: variable SINGLE_NO_ARG_INVOCATION

[Andrew Phillips]

The final compilation output is


Could you please run 'mvn clean package -DskipTests' from the root of 
the project and put the *full* output (including progress, as in the 
last attachment) in a Gist or Pastie?


Thanks!

ap

Re: jclouds 2.0.0-SNAPSHOT

[Andrew Phillips]

Hi Sathish


Can you please help us when we can expect jcloud 2.0.0-SNAPSHOT
official release.


2.0.0-SNAPSHOTs are already being published and are available from [1]. 
See also the section "Using the daily builds" from the installation 
guide [2].


For information about the progress of the 2.0 release, please follow 
this thread:


http://apache.markmail.org/thread/c56u3fgfeboimbhk

Thanks!

ap

[1] 
https://repository.apache.org/content/groups/snapshots/org/apache/jclouds/

[2] http://jclouds.apache.org/start/install/

Re: Cannot retry after server error, command has exceeded retry limit 5

[Andrew Phillips]

Hi Pratheesh

Could you enable wire and header logging (see [1]) and send us the 
output, so we can get some more details on the call that is being made?


Could you also provide a minimal, if possible, test case that 
demonstrates the problem you're seeing?


Regards

ap

[1] http://jclouds.apache.org/reference/logging/

On 2016-06-02 08:10, Pratheesh wrote:

Please let me know what is the cause of this error:

Jun 02, 2016 2:49:54 PM org.jclouds.logging.jdk.JDKLogger logError

SEVERE: Cannot retry after server error, command has exceeded retry
limit 5: [method=org.jclouds.openstack.swift.SwiftClient.public
abstract org.jclouds.openstack.swift.domain.SwiftObject
org.jclouds.openstack.swift.CommonSwiftClient.getObject(java.lang.String,java.lang.String,org.jclouds.http.options.GetOptions[])[Smart,
, [Lorg.jclouds.http.options.GetOptions;@7cb66c36], request=GET
https://sng01.objectstorage.softlayer.net/v1/AUTH_ab675443---23456ejkhgkl/Smart/
HTTP/1.1]

Regards

Pratheesh

Re: NoClassDefFoundError: Could not initialize class org.jclouds.reflect.Reflection2

[Andrew Phillips]

Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.jclouds.reflect.Reflection2


That class is in jclouds-core-1.9.1.jar, so you shouldn't really get 
that exception. Can you share the actual classpath is that is used when 
the code is run?


Regards

ap

Re: 2.0.0-SNAPSHOT Filesystemprovider permission problem

[Andrew Phillips]

and files. Maybe it's a Windows 7 specialty when not using the Windows
temp directory? Where are the ITs putting their stuff for testing? Are
there even Windows related tests for the Filesystem storage provider 
;)?


There are indeed a number of filesystem tests, although many of them are 
skipped on Windows:


https://github.com/jclouds/jclouds/tree/master/apis/filesystem/src/test/java/org/jclouds/filesystem

If you could run the tests on your setup to see if any of them fail, 
that would be helpful in terms of providing a reproducible case to work 
on.


Based on the following, I suspect PUBLIC_READ may not work reliably on 
transient blobstores such as the filesystem provider, though:


https://github.com/jclouds/jclouds/blob/master/apis/filesystem/src/test/java/org/jclouds/filesystem/integration/FilesystemContainerIntegrationTest.java#L192

To your question of testing blobstore work in general: do you have a 
testing account with the target provider that you could use? The 
semantics of local blobstores, and what is and is not supported, can be 
sufficiently different that test results are not necessarily very 
predictive of what will happen when you talk to your actual endpoint.


Regards

ap

Re: Group name not set for new nodes on EC2

[Andrew Phillips]

On 2016-03-20 10:00, cen wrote:

Agree with both statements. If there was documentation about this on
createSecurityGroup I'd knew instantly what is going on. It could even
be a generic statement that a certain method adheres to
GroupNamingConvention.

The pattern does not really match my use case because I create a single
security group on first run and then re-use it for each new VM (that's
why I hardcoded the name).


Ah, like that - in that case, I would imagine Ignasi's suggestion of 
either setting the RESOURCENAME_PREFIX property, or going a step further 
and actually overriding the naming convention "generator", might be the 
right thing to do?


Then you *know* exactly what the resources will look like, and don't 
have to save that information for reuse later.


Regards

ap

Re: Group name not set for new nodes on EC2

[Andrew Phillips]

Regarding the group name in the nodes, could you share the code you
use to build the template, including the template options?


Thanks for the additional details! More information on the resource 
prefixing Ignasi was mentioning is here, by the way:


http://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/compute/functions/GroupNamingConvention.html

Still, it seems we are short on documentation on where exactly this 
prefixing is applied. For example, the documentation for 
createSecurityGroup [1] does not mention this, even though from the code 
[2] and what we've seen in this thread a prefix *is* being added.


In order to avoid this problem, would it make sense to recommend the 
following pattern?


SecurityGroup sg = securityGroupExtension.createSecurityGroup("name", 
region);


// get the name of the actually created item!!
String sgName = sg.getName();
...
EC2TemplateOptions o = 
EC2TemplateOptions.Builder.securityGroups(sgName);

...

In other words, instead of assuming that the input parameter to 
createSecurityGroup (or similar calls) can be used as-is, always use the 
values of the resulting object?


Regards

ap

[1] 
http://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/aws/ec2/compute/extensions/AWSEC2SecurityGroupExtension.html#createSecurityGroup(java.lang.String, 
java.lang.String)
[2] 
https://github.com/jclouds/jclouds/blob/master/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/compute/extensions/AWSEC2SecurityGroupExtension.java#L73

Re: Group name not set for new nodes on EC2

[Andrew Phillips]

After changing the group name in template to "jclouds#crossbuild" it 
started working.


Ah, glad to hear! Could you share your final code and perhaps a short 
description of "this is what I was trying to do, this approach didn't 
work, this different approach did"? I suspect that may turn into pretty 
useful reference material for others trying to do the same thing in 
future ;-)


Regards

ap

Re: Using JClouds without TLSv1.0

[Andrew Phillips]

Hi Stéphane


Any idea of code to configure my http client with TLSv1.1 or TLSv1.2
instead of TLSv1.0, please ?


The following blog post doesn't provide a step-by-step solution, but 
should hopefully help:


https://jclouds.apache.org/blog/2014/10/25/poodle-and-jclouds/

Regards

ap

RE: Use JClouds to talk to non AWS cloud with S3 API

[Andrew Phillips]

Hi Julie

Thanks for the additional details! Good to see we're now trying to 
connect to the same URL in both cases (i.e. 
http://jag-itop-svr.dev.opsware.com:8080/public).


Could you attach the wire and header logs from *this* failed jclouds 
call? Now that we have the details from s3curl, we should be able to see 
what is different about the request jclouds is sending.


Regards

ap

On 2016-01-31 14:53, Yao, Julie wrote:

Debug the code
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService
protected HttpResponse invoke(HttpURLConnection connection) throws
IOException, InterruptedException {
  HttpResponse.Builder builder = HttpResponse.builder();
  InputStream in = null;
  try {
 in = connection.getInputStream();
  } catch (IOException e) {
 in = connection.getErrorStream();

detailmessage:
Server returned HTTP response code: 500 for URL:
http://jag-itop-svr.dev.opsware.com:8080/public
I checked connection.authObj: null
Is connection.authObj correct?

Here is my code:
String username = "test:tester";
String password = "testing";
Properties overrides = new Properties();
overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS, 
"false");

   overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/");

BlobStoreContext context = ContextBuilder.newBuilder(new
S3ApiMetadata())
.endpoint("http://jag-itop-svr.dev.opsware.com:8080;)
.credentials(username, password)
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext.class);

BlobStore blobStore = context.getBlobStore();
String bucket="public";
boolean exist = blobStore.containerExists(bucket);

I also have VM args:
-Djclouds.trust-all-certs=true -Djclouds.relax-hostname=true
-Djclouds.provider=s3

Thanks!

-Original Message-
From: Yao, Julie
Sent: Sunday, January 31, 2016 10:01 AM
To: Andrew Phillips; user@jclouds.apache.org
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API


Ah, OK. To confirm: are we calling a Walrus installation here?

No

I got s3curl.pl working. I have a bucket called 'public'. The rest api
to see if the bucket 'public' exist:
HEAD /public

./s3curl.pl --id 'test:tester' --key 'testing' --head --  -v  http://
jag-itop-svr.dev.opsware.com:8080/public
* About to connect() to jag-itop-svr.dev.opsware.com port 8080
*   Trying 192.168.34.176... connected
* Connected to jag-itop-svr.dev.opsware.com (192.168.34.176) port 8080

HEAD /public HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Host: jag-itop-svr.dev.opsware.com:8080
Accept: */*
Date: Sun, 31 Jan 2016 17:54:30 +
Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=


< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
x-amz-id-2: tx29a5adaa1d8345c882658-0056ae4ab0
< Content-Length: 0
Content-Length: 0
< x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
x-amz-request-id: tx29a5adaa1d8345c882658-0056ae4ab0
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
X-Trans-Id: tx29a5adaa1d8345c882658-0056ae4ab0
< Date: Sun, 31 Jan 2016 17:56:00 GMT
Date: Sun, 31 Jan 2016 17:56:00 GMT

I will download source code to debug.

Thanks!


-Original Message-
From: Andrew Phillips [mailto:andr...@apache.org]
Sent: Friday, January 29, 2016 6:35 PM
To: user@jclouds.apache.org
Cc: Yao, Julie
Subject: RE: Use JClouds to talk to non AWS cloud with S3 API


I guess
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
is not correct.
Without it, it will throw exception.


Ah, OK. To confirm: are we calling a Walrus installation here?

Also, in your boto code snippet:

 def connect(self):
 self.connection = boto.connect_s3 (
 aws_access_key_id = self.access_key_id,
 aws_secret_access_key = self.secret_access_key,
 port = self.port,
 host= self.host,
 is_secure=False,
 calling_format=boto.s3.connection.OrdinaryCallingFormat()
 )

What are the values of "self.host" and "self.port"? I'm assuming they
are "myserver" and "8080", or so? In that case, boto should also be
looking for http://myserver:8080/julie, and not
http://myserver:8080/services/Walrus/julie, which the jclouds settings
are causing us to call.

Please see if you can make a successful HEAD Bucket call using cURL or
other web request generator:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html

It will be much easier to determine the correct settings for jclouds
once we have an actual example of an HTTP request to your service that
is successful.

Regards

ap

RE: Use JClouds to talk to non AWS cloud with S3 API

[Andrew Phillips]

Hm...not much difference there:

s3curl call that succeeds:

HEAD /public HTTP/1.1
User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

Host: jag-itop-svr.dev.opsware.com:8080
Accept: */*
Date: Sun, 31 Jan 2016 17:54:30 +
Authorization: AWS test:tester:K4fukbjysjqzsArmq8EXHnc/tnE=

jclouds call that fails:

HEAD http://jag-itop-svr.dev.opsware.com:8080/public HTTP/1.1
Date: Sun, 31 Jan 2016 21:18:50 GMT
Authorization: AWS test:tester:jxcGT++8gYPHXO8Vpgt2FtIu9FI=

Can you try to reproduce the successful call using 'plain' cURL [1], 
then without each of the headers that are different ('User-Agent', 
'Host' and 'Accept')? Hopefully, that will allow us to figure out which 
of those three headers is causing the 500.


Also, have you been able to try Ranjith's suggestion [2] and use the 
ApacheHC driver?


Regards

ap

[1] https://curl.haxx.se/
[2] http://markmail.org/message/7gw6wnpbaefgp64r

Re: Problem bootstraping FreeBSD on EC2

[Andrew Phillips]

Found the solution. If I execute commands in a single line using &&,
written as a java string they work as expected.


Uff. Glad to hear you were able to get it to work in the end!

ap

RE: Use JClouds to talk to non AWS cloud with S3 API

[Andrew Phillips]

but the result is wrong.


Ah, pity :-( Caught celebrating too soon ;-) One of the things that may 
be worth trying here is to move away from the portably blobstore 
abstraction and try the API directly - that's more comparable to the 
boto calls.


...
S3Client client = ContextBuilder.newBuilder(new S3ApiMetadata())
.endpoint("http://myserver:8080/services/Walrus;)
.credentials(username, password)
.overrides(overrides)
.buildApi(S3Client.class);

client.putObject(...)

See 
https://jclouds.apache.org/reference/javadoc/1.9.x/index.html?org/jclouds/s3/S3Client.html.


If that works, then we can hopefully more easily get back to the 
blobstore abstraction to see why that is not working.


Also, please record the wire and header logs so we can see what HTTP 
calls exactly are being made [1].


Regards

ap

[1] https://jclouds.apache.org/reference/logging/

RE: Use JClouds to talk to non AWS cloud with S3 API

[Andrew Phillips]

I guess
overrides.setProperty(PROPERTY_S3_SERVICE_PATH, "/services/Walrus");
is not correct.
Without it, it will throw exception.


Ah, OK. To confirm: are we calling a Walrus installation here?

Also, in your boto code snippet:

def connect(self):
self.connection = boto.connect_s3 (
aws_access_key_id = self.access_key_id,
aws_secret_access_key = self.secret_access_key,
port = self.port,
host= self.host,
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat()
)

What are the values of "self.host" and "self.port"? I'm assuming they 
are "myserver" and "8080", or so? In that case, boto should also be 
looking for http://myserver:8080/julie, and not 
http://myserver:8080/services/Walrus/julie, which the jclouds settings 
are causing us to call.


Please see if you can make a successful HEAD Bucket call using cURL or 
other web request generator:


http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html

It will be much easier to determine the correct settings for jclouds 
once we have an actual example of an HTTP request to your service that 
is successful.


Regards

ap

RE: Use JClouds to talk to non AWS cloud with S3 API

[Andrew Phillips]

Thanks for the heads-up, Julie - glad you were able to get it to work!

ap

Re: Example - Cloudwatch Basics

[Andrew Phillips]

Hi Wesley


I'm starting studies with Apache jclouds. So, I plan to collect data
from the vcpu and memory each instance of an OpenStack cloud. The
example "CloudWatch-basics" available on GitHub can be used to
OpenStack clouds? It seems that can only be used for EC2 clouds.


The CloudWatch implementation in jclouds is an API (see [1]), so in 
principle it should be possible to point it an any endpoint the supports 
the same operations, even if that endpoint is not provided by AWS.


From what I can see (and the OpenStack experts on this list will have a 
much more qualified opinion ;-)), the only component in the OpenStack 
ecosystem that claims to be CloudWatch-compliant is Synaps [2], which is 
in the category "Related projects".


Heat seems to have a "CloudWatch-like" interface [3], but that is only a 
minimal implementation, so I'm not sure to what extent we would expect 
that to work with jclouds.


Regards

ap

[1] http://jclouds.apache.org/start/concepts/
[2] https://wiki.openstack.org/wiki/Synaps
[3] https://wiki.openstack.org/wiki/Heat/Using-CloudWatch

Re: jclouds 1.8.1 to Openstack Swift3 object store

[Andrew Phillips]

Hi Forrest


Here is the code I used to generate results file,
http://pastebin.com/x85yHTHW


Looking at the code snippet, it doesn't seem like we're actually making 
an ACL call with boto. Could you see what happens if you add that?


http://boto.cloudhackers.com/en/latest/ref/s3.html?highlight=get_acl#boto.s3.bucket.Bucket.get_acl

Regards

ap

Re: jclouds 1.8.1 to Openstack Swift3 object store

[Andrew Phillips]

Hi Forrest

Thanks for getting in touch! Could you share the part of the code where 
you're creating the jclouds context in a Gist or Pastie?


And just to clarify: you're trying to connect to an *OpenStack* 
instance, but using the S3 API?


Regards

ap

[1] http://pastie.org

Re: jclouds 1.8.1 to Openstack Swift3 object store

[Andrew Phillips]

Hi Forrest

Thanks for the update!


Where it looks like I fail is when I am doing a GET
request on this, http://:8080/?acl, more
than once and eventually getting an Internal Server Error --

2015-12-02 17:06:31.095 DEBUG [Thread-12] j.headers >> GET
http://vcloud210.storage.tucson.ibm.com:8080/418-b7ff7d008a2711e5bbb9080027001c70?acl


From your code snippets, I would guess that the ACL calls are coming 
from the putBlob call, since createContainerInLocation for the S3 API 
[1] resolves more or less to a straight putBucketInRegion call [2], 
whereas putBlob includes a call to get the bucket's ACL [3].


Having said that, the getBucketACL call [4] is a call that, according to 
these OpenStack docs [5], is "known to be supported". If that call is 
actually blowing up for some reason, that would initially seem to point 
towards a bug on the OpenStack side.


Are you able to make a successful call to get the bucket's ACL with cURL 
or similar, or do you get the same error?


Regards

ap

[1] 
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/S3Client.java
[2] 
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/blobstore/S3BlobStore.java#L433
[3] 
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/blobstore/S3BlobStore.java#L277
[4] 
https://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/s3/S3Client.html#getBucketACL(java.lang.String)

[5] https://wiki.openstack.org/wiki/Swift/APIFeatureComparison

Re: jclouds 1.8.1 to Openstack Swift3 object store

[Andrew Phillips]

Could this be related to the fact that I'm using the 's3'
builder versus the 'aws-s3'?


Intuitively, that seems unlikely: if anything, I'd expect calls made 
using the aws-s3 provider to be *more* AWS-specific than the more 
generic S3 API.


What we'd ideally like to find out is how the boto call is different 
from the call that jclouds is making. Are you able to capture some 
logging for that, and/or make successful calls using cURL?


Regards

ap

Re: can't select LOCAL disk types in softlayer

[Andrew Phillips]

Hi Subhadip

Looking at the tests (e.g. [1]), it would certainly seem as though the 
provider expects to be able to support the local disk type.


Could you enable wire and header logging [2] to see what the request 
that is actually being made to SoftLayer looks like, and post it to a 
Gist or Pastie [3] after removing any sensitive information?


Regards

ap

[1] 
https://github.com/jclouds/jclouds/blob/master/providers/softlayer/src/test/java/org/jclouds/softlayer/compute/SoftLayerTemplateBuilderLiveTest.java#L53

[2] http://jclouds.apache.org/reference/logging/
[3] http://pastie.org

PS: The mailing lists "dev-help@..." and "user-help@..." don't exist ;-) 
Please use "user@..." for questions on how to use jclouds, such as this 
one. Thanks!


On 2015-09-18 02:36, Subhadip Bagui wrote:

Hi,

Whenever I tired creating instance with softlayer it always takes 
disktype

as "SAN".

I have tried with below code, but it's always taking diskType as SAN 
only


*templateBuilder.hardwareId("cpu=2,memory=2048,disk=25,type=LOCAL");*

*options.diskType("LOCAL");*


Please suggest how to allow disk type as local also

Regards,
Subhadip

---

Re: ec2 authorization issue after upgrading to 1.9.1

[Andrew Phillips]

After upgrading to 1.9.1 it's starts happening. Any idea ?


What was the previous version you were using? Were there any other 
changes you made to your setup, e.g. the JVM type of version?


This is not something we've had anyone else report so far, so at this 
point it seems more likely to be an interplay between jclouds and other 
parts of the environment.


Regards

ap

Re: google cloud node creation request fail - trying to ssh with node ip after creation

[Andrew Phillips]

But strangely the program execution is not coming out of the
main thread called. After connecting to the node is it trying to
fetch something which never completes?


If you take a thread dump or look at it using VisualVM or so, can you 
see where it's hanging? Also, have you tried enabling wire logging [1] 
to see which requests may not be completing?


Regards

ap

[1] https://jclouds.apache.org/reference/logging/

Re: google cloud node creation request fail - trying to ssh with node ip after creation

[Andrew Phillips]

Correct. I dont want to connect to the node on creation. Will
make a separate call for bootstrap to connect and deploy
scripts. Is there any way to disable the connect to node ?


You might want to see if setting "blockUntilRunning" to false in the 
TemplateOptions [1] makes any difference. It's not the only reason 
jclouds may try to connect to the box, but setting it to false is at 
least worth a try...


Regards

ap

[1] 
https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/compute/options/TemplateOptions.Builder.html#blockUntilRunning(boolean)

Re: ec2 authorization issue after upgrading to 1.9.1

[Andrew Phillips]

[moving this to user@...]


What is the correct procedure to pass certificate for every public
clouds through jclouds? if I have to ship the application to any
remote machine ?


In terms of trusting certificates, jclouds doesn't behave any 
differently than any other Java application. The first question for me 
would be why the certificate is being rejected in the first place - the 
public cloud providers should all have certificates that are signed by 
trusted CAs, and so should be accepted by the JVM without any changes.


Are they endpoints you're connecting to using self-signed certs, or is 
there any other obvious reason (hostname mismatches, expired certs etc.) 
why they would be rejected? Or are the JVMs you're running on using a 
different set of trusted CAs?


Regards

ap

Re: google cloud node creation request fail - trying to ssh with node ip after creation

[Andrew Phillips]

My doubt is why the request is trying to do ssh while node
creation. I'm not doing any runScript on the node in my
computeService request.


Ah, thanks for clarifying - I had misunderstood your question. From what 
I recall, the SSH check is try to verify that the node is actually 
accessible and usable; experience having shown that the status as 
reported by the cloud API itself isn't necessarily a reliable indicator 
of whether the image is "ready".


I take it that in your situation you don't actually plan to access the 
node once you've created it?


Regards

ap

Re: google cloud node creation request fail - trying to ssh with node ip after creation

[Andrew Phillips]

[responding on user@]


I'm supplying actual proxy ip and port with username and password,


Ah, OK, thanks for clarifying. Does your proxy allow port 22? From the 
following message, I suspect that it might be blocking that port:


WARNING: Error checking reachability of ip:port 146.148.11.83:22

Regards

ap

Re: GCE problem 'error: group'

[Andrew Phillips]

I really appreciate it.


Hey, we're not done yet (or is it all working now?) ;-) But glad to hear 
this is helping...


Regards

ap

Re: GCE problem 'error: group'

[Andrew Phillips]

error: group(class java.lang.NullPointerException)


An NPE, eh? I see that the code seems to have changed relative to the 
current trunk - do you think you could build against 2.0.0-SNAPSHOT and 
run the example again to see if anything's changed?


I've pasted a version of the compute-basics POM with the required 
changes (basically, changing jclouds.version, changing the 
google-compute-engine group ID and adding the Apache snapshot repo) 
here:


http://pastie.org/private/zlfanz26pzc0hydsc2zwq

Thanks for sending over that additional info!

Regards

ap

Re: GCE problem 'error: group'

[Andrew Phillips]

Hi Jonathan

If you built the examples from source, could you modify the error 
handling line in [1] to also print out the type of the exception and the 
stacktrace? E.g. from:


} catch (Exception e) {
  System.err.println(error:  + e.getMessage());
  error = 1;
} finally {

to

} catch (Exception e) {
  System.err.println(error:  + e.getMessage() + ( + e.getClass() + 
));

  e.printStackTrace(System.err);
  error = 1;
} finally {

Could you then run the example again and add the output to this thread?

Thanks!

ap

[1] 
https://github.com/jclouds/jclouds-examples/blob/master/compute-basics/src/main/java/org/jclouds/examples/compute/basics/MainApp.java#L237

Re: jclouds 1.9.0 with example code returns errors

[Andrew Phillips]

I installed jclouds1.9.0 and was trying out the
JCloudsNovajava example from docs and I get an error
while compiling the code. I have copied the pom.xml from
the doc and tried to compile it.. The error is


How are you setting up this project? What libraries are on your 
classpath?


Regards

ap

Re: jClouds with https

[Andrew Phillips]

Hi Mark

How does jClouds cope with https endpoints. Do I have to set up   
anything in the client to make a https connection?


Are you getting any specific error? If so, could you share the full  
stack trace or error message?


Regards

ap

Re: AWS-EC2: Why are security groups subnets mutually exclusive?

[Andrew Phillips]

What you say makes sense and I think it is an unintentional limitation
in the current code.


Some additional context here: the securityGroups option [1]  
originally (and perhaps still - sorry, can't check the code now :-()  
refers to old style security groups, which *were* incompatible with  
VPCs for a long time.


The new style security groups are specified using the  
securityGroupIds option instead [2]. If the distinction between the  
two types no longer exists, then some code simplification should  
certainly be possible!


See also this thread: http://markmail.org/thread/gmnffrbqibfwqczr

Regards

ap

[1]  
http://jclouds-javadocs.elasticbeanstalk.com/org/jclouds/aws/ec2/compute/AWSEC2TemplateOptions.html#securityGroups(java.lang.String...)
[2]  
http://jclouds-javadocs.elasticbeanstalk.com/org/jclouds/aws/ec2/compute/AWSEC2TemplateOptions.html#securityGroupIds(java.lang.Iterable)


--
Andrew Phillips
Apache jclouds

Re: Thread leak when creating SshClient

[Andrew Phillips]

context-scoped) etc, so in general, I'd say you try to create a
context just once and reuse it when possible, and configure the thread
pool size to fit your app needs.


See also this old thread:

https://groups.google.com/forum/#!topic/jclouds/pF9ZqXPdLTk

Regards

ap

Re: jclouds-cli problems

[Andrew Phillips]

Well, I was able to update assembly/pom.xml to use a local copy
jclouds-karaf features.xml file. Indeed, maven downloaded the jar, I can
see it in the final packaging now:
./system/com/jcraft/jsch.agentproxy.usocket-jna/0.0.8/jsch.agentproxy.usocket-jna-0.0.8.jar.
However, the NoClassDefFoundError is still persisting.


Some quick investigation on my side points to a similar issue:

1. Downloaded the pre-built ZIP from  
http://search.maven.org/remotecontent?filepath=org/apache/jclouds/cli/jclouds-cli-assembly/1.9.0/jclouds-cli-assembly-1.9.0.zip

2. Unzipped and tried to run jclouds:node-list
3. Since I'm running on Windows, my error is slightly different:

8) Error injecting constructor, java.lang.NoClassDefFoundError:  
com/jcraft/jsch/

agentproxy/connector/PageantConnector
...

What I'm wondering is whether the exception being caught by the  
try/catch block that tries to find an SSH Agent isn't too restrictive  
[1].


I'll see if I can mess around a bit with the deps in my installation  
to see if that makes any difference.


Regards

ap

[1]  
https://github.com/jclouds/jclouds/blob/master/drivers/sshj/src/main/java/org/jclouds/sshj/config/SshjSshClientModule.java#L56-L58

Re: jclouds-cli problems

[Andrew Phillips]

./system/com/jcraft/jsch.agentproxy.usocket-jna/0.0.8/jsch.agentproxy.usocket-jna-0.0.8.jar.
However, the NoClassDefFoundError is still persisting.


Dang. Hm, in that case I suspect this will require some more  
investigation,  and unfortunately I don't have any cycles to look into  
this immediately :-(


To perhaps take a step back: what is your use case for the CLI at this  
point? If you're looking to issue blobstore commands, the following  
code may provide an alternative:


https://github.com/jclouds/jclouds/pull/487/files

Regards

ap

Re: ec2 security group can't be deleted

[Andrew Phillips]

Hi Liang

Is this behaviour reproducible, i.e. does it happen every time you run  
the example? If you add some wait time to the example code between the  
moment it destroys the node and tries to delete the security group, do  
you get the same result?


Regards

ap

Re: AWS Ec2 new hardware type, like t2.micro

[Andrew Phillips]

Do we know if anyone will be working one this anytime soon?


https://github.com/jclouds/jclouds/pull/605 is being reviewed right now ;-)

Thanks for the ping!

ap

Re: Problem with Proxy settings

[Andrew Phillips]

Hi Eyal

Could you enable wire and header logging [1] and then try to replicate  
this by trying the same request through the proxy that the jclouds is  
trying to make? Or is that what you tested already?


Also, are you using any of the jclouds HTTP drivers (okhttp or  
apachehc), or are you using the default java.net impelementation? You  
can see if you're using the drivers by having a look at the  
dependencies of your project.


Regards

ap

[1] http://jclouds.apache.org/reference/logging/#configure

Re: AWS Ec2 new hardware type, like t2.micro

[Andrew Phillips]

So here instanceType.T2_MICRO not supported. Only previous generation
listed here.


jclouds indeed supports only those instance types listed in the  
InstanceType [1] enum. Please raise a JIRA or open a pull request for  
any types that are missing.


Thanks for raising this!

ap

[1]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/ec2/domain/InstanceType.html

Re: AWS-EC2: Why are security groups subnets mutually exclusive?

[Andrew Phillips]

Hi Yaron


To create nodes in a subnet under Amazon VPC add the following option to
your template options. Note that VPCs and Security Groups are mutually
exclusive.

Why?


From what I understand, Security Groups here refers to the  
classic security groups that are referred to as EC2-Classic in the  
AWS docs. These state, for example:


You can't specify a security group that you created for a VPC when  
you launch an instance in EC2-Classic. and You can't specify a  
security group that you created for EC2-Classic when you launch an  
instance in a VPC. [1]


You should be able to specify the security groups *in your VPC* by  
using the securityGroupIds template builder option on  
AWSEC2TemplateOptions [2] (rather than the securityGroups option  
which, if I recall correctly, selects the old-style groups).


See also this thread [3].

Regards

ap

[1]  
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html
[2]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/aws/ec2/compute/AWSEC2TemplateOptions.Builder.html

[3] http://apache.markmail.org/thread/7ssdg7ww4c652gyj

Re: [Error] openstack-neutron is not in the list of providers or apis

[Andrew Phillips]

Is this because of incorrect Manifest file in the 1.8.1 jar that I have
reported in another thread[1]? Or what would be issue here?


I would assume that, if this were related to incorrect import package  
statements, you wouldn't be able to load the bundle in the first  
place. But that's just a guess.


Are you able to work around this by calling  
ContextBuilder.newBuilder(new NeutronApiMetadata())...build(), by any  
chance?


Regards

ap

[1] http://javadocs.jclouds.cloudbees.net/org/jclouds/ContextBuilder.html

Re: seeing jclouds.headers log lines for 401 responses

[Andrew Phillips]

Is there a way to force the headers for these 401 responses to be logged?


I'm afraid I can't help with that, but does making the request that  
jclouds makes via cURL actually cause a 401, or does it succeed when  
not going via jclouds?


Regards

ap

[ANNOUNCE] Apache jclouds 1.8.1 released

[Andrew Phillips]

The Apache jclouds team is pleased to announce the release of jclouds 1.8.1.

Apache jclouds is an open source multi-cloud toolkit for the Java  
platform that gives you the freedom to create applications that are  
portable across clouds while giving you full control to use  
cloud-specific features.


The source archives for the release are available here:
http://www.apache.org/dyn/closer.cgi/jclouds

The Maven artifacts for the release are available in Maven Central,  
under the org.apache.jclouds group ID.


The full change log is available here:
https://issues.apache.org/jira/browse/JCLOUDS/fixforversion/12327548

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://jclouds.apache.org/

The Apache jclouds Team

Re: vcloud-director support

[Andrew Phillips]

Hi Duncan

Thanks for bringing this up! Sorry for the delay in responding - 1.8.1  
has been eating up some time of late ;-)



Please chip in and maybe copy users on the dev discussion on this
particular topic.


One related topic that has recently been discussed on the dev list [1]  
is the overall size and maintenance effort related to the larger APIs.  
It would be a very useful learning experience for all, I think, if we  
could apply some of the thinking from that here by looking to create a  
minimal walking API skeleton first and seeing how far users can run  
with that.


That should hopefully not only lower the bar to entry quite a lot in  
terms of making initial reviews etc. much quicker. It would also allow  
us to hopefully get user feedback quickly on which functionality they  
are missing, so we can focus our effort on what users are actually  
asking for.


@Duncan, Rajesh, jclouds users: Does this approach sound like  
something that you think could work?


Regards

ap

[1] http://apache.markmail.org/thread/cdjk4qvvzbd5npye

Re: How to destroy an instance without node id?

[Andrew Phillips]

Forwarding your question to user@...


I am experiencing a problem like returned node id is null. But it is
creating the ec2 instance after a while.


Is this a reproducible problem, or is it happening intermittently? If  
you can reproduce this, could you enable wire and header logging [1]  
and put the data of the response that is causing the problem in a Gist  
or Pastie?


For those nodes where the ID is null, could you print out the entire  
NodeMetadata object? Are any of the values set?


Regards

ap

[1] https://jclouds.apache.org/reference/logging/

- Forwarded message from rraju1...@gmail.com -
Date: Fri, 17 Oct 2014 18:20:22 +0530
From: Rajaratnam Rajkumar rraju1...@gmail.com
Reply-To: Rajaratnam Rajkumar rraju1...@gmail.com
 Subject: Re: How to destroy an instance without node id?
  To: Andrew Phillips andr...@apache.org

Hi Andrew,

Set? extends NodeMetadata nodes = computeService
.createNodesInGroup(group, 1, template);

I guess this method call can timeout. What happens if it timeouts?

I am experiencing a problem like returned node id is null. But it is
creating the ec2 instance after a while.

Please help on this.

Re: How to destroy an instance without node id?

[Andrew Phillips]

Forwarding to user@...

Hi Rajaratnam


In some situations I am getting node id null. I want to terminate that
instance at that point.


Could you give us some more details about which APIs/providers you're  
using? It would be interesting to know where the null IDs are coming  
from...that doesn't sound right...


Thanks

ap

Quoting Rajaratnam Rajkumar rraju1...@gmail.com:


Hi,

Is there any way to destroy a node without node id?

Here is what I am trying to do.

String group = uniqueString();

 ComputeService computeService = iaasProvider
.getComputeService();
Template template = // some template

// create and start a node
Set? extends NodeMetadata nodes = computeService
.createNodesInGroup(group, 1, template);

node = nodes.iterator().next();

if (null == node.getId()) {
 // destroy the node
}

In some situations I am getting node id null. I want to terminate that
instance at that point.

Please let me know how can do it?

Thanks.





--
Andrew Phillips
Apache jclouds

Re: How to destroy an instance without node id?

[Andrew Phillips]

I am using EC2. Most of the time it works fine. But sometimes node id null.


Do you currently have a node in EC2 that exhibits this behaviour, or  
is this relatively frequent? Could you send a screenshot of what the  
EC2 console shows for a node like this?


Regards

ap

Re: Does jClouds support IPv6?

[Andrew Phillips]

Forwarding to user@...

Hi Jeffrey


Does jClouds support IPv6?


Just to clarify: do you mean does the jclouds openstack-nova API  
support servers with only IPv6 addresses?, or something related to  
IPv6 support more generally?


Could you describe what you are trying to do?

Thanks!

ap

Quoting Jeffrey Nguyen (jeffrngu) jeffr...@cisco.com:


Hi,

Does jClouds support IPv6?   I'm particularly interested in the APIs  
 for floating IP and fixed IP that already supports IPv4 for   
Openstack (and perhaps vCloud too).   I saw this JIRA   
https://issues.apache.org/jira/browse/JCLOUDS-748 so I'm suspecting   
jClouds doesn't support IPv6 yet but I thought I'd ask anyway.


Thanks,
-Jeffrey

Re: Version of jclouds 1.8.0 with jdk1.7.0_60

[Andrew Phillips]

Thanks Andrew, at the reflection jar there is another guava.jar. So I
excluded it:


Do things work now? ;-)

ap

Re: jclouds GCE provider -----END RSA PRIVATE KEY----- doesn't contain % line [-----END ]

[Andrew Phillips]

Hi Aric


Anyone here using google-compute-engine provider for jclouds there appears
to be a problem when putting the pem extracted from the .p12 into the
credential section of my jenkins config.xml


Could you first see if you can run the basic example [1] directly  
against GCE with the same key file etc.? That way, we can hopefully  
figure out whether the problem is in jclouds itself, or in the Jenkins  
plugin.


Thanks!

ap

[1]  
https://github.com/jclouds/jclouds-examples/blob/master/google/src/main/java/org/jclouds/examples/google/computeengine/CreateServer.java

Re: Is jclouds developer-friendly for Android ?

[Andrew Phillips]

So, any ideas about what could be the problem? Is jclouds compatible
with the Android platform?


On this specific point, which seems to be related to the way jclouds  
is trying to use ServiceLoader to load the appropriate implementation  
class for an API or provider:


http://stackoverflow.com/questions/5760607/using-serviceloader-on-android

Regards

ap

Re: Unable to export running jar with Jclouds code from eclipse

[Andrew Phillips]

java.util.NoSuchElementException: key [aws-ec2] not in the list of
providers or
apis: {}
at org.jclouds.ContextBuilder.newBuilder(ContextBuilder.java:174)

Seems to be a classpath issue? Any hints?


The problem here is most likely related to ServiceLoader: jclouds uses  
a META-INF/... file to determine the implementations of API/provider  
metadata classes that are available.


If you create a fat JAR, the default logic typically picks only  
*one* of these files (or none of them), rather then merging them all,  
which is what needs to be done. See [1] for some ways of achieving  
this. If you're using Maven, there's also an example in  
jclouds-examples [2].


Hope this helps!

ap

[1] http://blog.xebia.com/2011/07/20/jar-with-deps-dont-like-meta-infservices/
[2]  
https://github.com/jclouds/jclouds-examples/blob/master/compute-basics/src/main/assembly/jar-with-dependencies.xml

Re: Get all configured regions

[Andrew Phillips]

is a method in jclouds ec2 api. So can some one point out me a method or
another way for get all configured regions in cloudstack?


Are you talking about these kind of regions [1]? Could you link to the  
actual CloudStack API call that you would make to retrieve the  
information you're looking for?


Regards

ap

[1]  
https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Installation_Guide/about-regions.html

Re: Self-signed certs

[Andrew Phillips]

This sounds like an http transport concern, Yury.

Perhaps you can try a different driver, such as our okhttp one?


On this point: which driver are you currently using? The default  
(standard Java HTTP) one? As Adrian mentions, trying the OkHttp or  
Apache HC alternatives [1], [2] sounds like the most obvious first step.


Regards

ap

[1] https://github.com/jclouds/jclouds/tree/master/drivers/okhttp
[2] https://github.com/jclouds/jclouds/tree/master/drivers/apachehc

Re: Where i can get source libraries for OpenStack API's

[Andrew Phillips]

i want for version 2.0.0..


jclouds version 2.0.0 has not been released yet - the latest version is 1.8.0:

http://jclouds.apache.org/releasenotes/

ap

Re: Specifying the ip address of the iaas when creating Compute service

[Andrew Phillips]

Is there any way to stop an instance using jclouds? In ComputeService
interface there is no such a method. There is a method for destroy an
instance. But I want to an instance.


Stopping instances is not currently supported at the view level, i.e.  
in ComputeService. But for those APIs and providers that support this,  
you can usually do it using an API/provider-specific call.


See e.g. the InstanceApi in the case of EC2 [1].

Hope that helps!

ap

[1]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/ec2/features/InstanceApi.html

Re: Example of setting Blob ACL using S3 provider?

[Andrew Phillips]

although I'm not entirely sure what the difference is between the AWSS3Client
the docs are using, and the S3Client that I've used


One is the API-specific client, the other the provider-specific client  
(see [1]). I.e. S3Client should work with all S3-compatible  
providers, whereas the AWSS3Client could expose extra functionality  
that *only* Amazon offers.


Hope that helps!

ap

PS: PRs to update outdated docs much appreciated...thanks!

[1] http://jclouds.apache.org/start/concepts/

Re: Example of setting Blob ACL using S3 provider?

[Andrew Phillips]

was apparently removed in Jclouds 1.6. I think I can work around this using
contextBuilder.buildApi(S3Client.class) and the S3Object class to set the
ACL.

But then if I have to use an S3-specific API to set an ACL, why not just
use the com.amazonaws.services.s3.AmazonS3Client client directly? How is
Jclouds actually benefitting me, if I'm using it to code directly to the S3
provider?


The challenge here is that ACLs are not something that is supported by  
all the blobstores jclouds supports, so is not included in the  
BlobStore abstraction. In other words, coding with ACLs on blobs is  
implicitly making your code provider-dependent already.


In order to support provider-specific features like that, jclouds  
indeed supports access to the underlying API, as you describe (see [1]  
for more details). This is indeed not all that different from using  
the provider client directly, but if the number of provider-specific  
calls you need to make are small, using jclouds will allow you to move  
to another provider relatively easily if you can re-implement the  
provider-specific calls or remove them.


A middle-ground approach that jclouds takes for some options is the  
ability to pass provider-specific options to the abstract interface.  
E.g. something like:


Blob myBlob = ...
PutOptions options = AWSS3PutOptions.Builder.storageClass(...).otherOption...;
blobstore.putBlob(myContainer, myBlob, options); // [2]

This is *also* AWS-specific code, but not quite as tied to the  
specific underlying API. Unfortunately, withAcl is not an option  
currently supported on AWSS3PutOptions [3] (it *is* supported on  
PutObjectOptions [4], but that's the options class for the S3-specific  
call).


That should be a relatively easy fix, though - would you be interested  
in submitting a PR for that?


Hope that helps!

ap

[1] http://jclouds.apache.org/start/concepts/
[2]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/BlobStore.html#putBlob(java.lang.String, org.jclouds.blobstore.domain.Blob,  
org.jclouds.blobstore.options.PutOptions)
[3]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/aws/s3/blobstore/options/AWSS3PutOptions.html
[4]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/s3/options/PutObjectOptions.html

Re: Exception cause when connecting to cloudstack

[Andrew Phillips]

Hi Asanka

I think what Schalk was trying to point out is that's its generally  
advised never to put sensitive information such as your passwords into  
an email on a public forum such as this one. If I understand  
correctly, your CloudStack instance is running locally, so hopefully  
is not accessible to anyone else.


Still, you should consider changing your credentials and replacing  
them with things like 'dummy' in future emails.


Now on to your problem ;-) According to the API metadata [1], the  
value for identity is expected to be your API key, and the value for  
credential is expected to be your secret key.


So you should be calling .credentials(apiKey, secretKey). Could you  
try that?


Note that Everett's example is for Rackspace, which is not a CloudStack cloud.

ap

[1]  
https://github.com/jclouds/jclouds/blob/master/apis/cloudstack/src/main/java/org/jclouds/cloudstack/CloudStackApiMetadata.java#L63-L64

Re: Method initialization failed from the command prompt

[Andrew Phillips]

I have all the required jars. As I said in Eclipse it's working fine,
the problem comes from the command line.


How are you running the program from the command line?

ap

Re: Softlayer GlobalIdentifier question

[Andrew Phillips]

With the most recent code (master) I am running into other issues. (Pls
Note, I initially tried to update my maven to version 2.0.0 and that still
had the the globalIdentifier problem).


2.0.0 does not exist (yet). I take it you mean 2.0.0-SNAPSHOT [1]?

ap

[1] https:/jclouds.apache.org/start/install/ (see Using the daily builds)

Re: Guava versions

[Andrew Phillips]

Can any of the devs respond to this issue?


jclouds uses Guava pretty heavily, including some of the newer  
features that were added in more recent releases (Andrew G can  
probably provide much more technical detail here). In some instances,  
such as fixing JCLOUDS-427 [1], we actually *depend* on newer Guava  
versions.


In short, from what I know, sticking to older Guava versions is not  
really an option for new jclouds major versions going forward.


Which leaves two questions for me: support for older Guava versions by  
supporting older jclouds major releases, and looking for the  
conflicting projects/products to work with newer Guava versions.


For the former: we have historically tried to support the current  
major version and the former major version (which means 1.8.x and  
1.7.x for now). Support for the prior major version tends to be  
limited to critical fixes rather than ongoing backporting of  
everything that's making it onto master.


So you should still be able to use an older version of Guava by  
sticking to an older jclouds major version, but you are unlikely to be  
able to make use of significant new jclouds features. From what I can  
see, we simply don't have the bandwidth in the active developer  
community to handle all the backports and minor releases for older  
versions.


Which leaves the final point: is there some way to encourage the  
projects you are using that are stuck on older versions to upgrade? It  
does seem to me that they're slightly more responsible for causing  
this problem than jclouds is. I know that is easy to say and doesn't  
help resolve the issue, but if we can help by adding our voice to  
Upgrade Guava version please discussions I'm happy to pitch in.


In the case of Cassandra, by the way, I'm glad to see this discussion  
is already happening:


https://issues.apache.org/jira/browse/CASSANDRA-6639

Regards

ap

[1] https://issues.apache.org/jira/browse/JCLOUDS-427

RE: JClouds BlobStore re-authentication when auth token expires

[Andrew Phillips]

fixed delay between the 2,3,4th attempts.  Is there a way to   
configure an increasing backoff?


According to its Javadoc, [1] BackoffLimitedRetryHandler already  
provides an exponential backoff. Is that the kind of thing you're  
looking for?


Glad to hear the re-authentication is working, at least!

ap

[1]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/http/handlers/BackoffLimitedRetryHandler.html

Re: openstack keystone infra

[Andrew Phillips]

is there a plan to change it? was it forgotten?


The plan is still to de-async all APIs, including Keystone. If you  
are using the Keystone API, please use the sync interface if possible.


Do you need to use the async interfaces in your code in any specific  
situations?


ap

Re: openstack keystone infra

[Andrew Phillips]

Is there any reason it wasn't moved? Maybe the retryOnRenew or is there no
prevention to de-async it?


As far as I am aware, it's simply still work in progress. The goal  
was to be working on de-asyncing for 2.0...


ap

Re: Specifying EC2 keypair?

[Andrew Phillips]

After some more experimenting I found out how it works.


Thanks for the details, David - glad to hear it's working!

ap

Re: Error: No space left on device

[Andrew Phillips]

org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:405)


The error is in putBlob, which actually tries to write the blob to the  
target location, but may also write to the temporary directory.


Which drive on your system actually has no space left any more? And  
what is the root directory that you are passing to the filesystem  
provider?


ap

Re: Error: No space left on device

[Andrew Phillips]

How could I delete the files only the temporary directory? I also cant find
this directory at /tmp


All I can see in your stacktrace is:


java.lang.RuntimeException: java.io.IOException: No space left on device


It does not say *which* device is out of space. Could you check on  
your target system to find out which device has no space left [1]?


ap

[1] https://kb.iu.edu/d/agfe

Re: Can we retrive OpenStack Swift Object using ObjectID?

[Andrew Phillips]

Moving this to user@...


I am not sure how to get ObjectID for Swift Object. Earlier I have worked
with SNIA CDMI object storage. Where we can access an Object using its name
or unique ObjectID.


Both the portable BlobStore API [1] and the Swift-specific ObjectApi  
[2] support blob retrieval by name only, which serves as the unique  
identifier. The SwiftObject class also does not have an ObjectID  
field [3].


Looking at the Swift API documentation for Objects [4], I can also not  
see any obvious ObjectID field [5]. Could you let us know which  
attribute you have in mind here?


Regards

ap

[1]  
http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/blobstore/BlobStore.html
[2]  
http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/openstack/swift/v1/features/ObjectApi.html
[3]  
http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/openstack/swift/v1/domain/SwiftObject.html
[4]  
http://docs.openstack.org/api/openstack-object-storage/1.0/content/storage_object_services.html
[5]  
http://docs.openstack.org/api/openstack-object-storage/1.0/content/GET_getObject__v1__account___container___object__storage_object_services.html#GET_getObject__v1__account___container___object__storage_object_services-Response

Re: JClouds BlobStore re-authentication when auth token expires

[Andrew Phillips]

I've done some google searches and haven't found much information on  
 this -- is there a standard way (or automatic within jclouds) to   
perform re-authentication when the token expires?


Which provider are you talking about specifically? There a quite a few  
APIs in jclouds the try to renew authentication automatically, see  
e.g. [1].


I assume that catching an authentication exception in your calling  
code and retrying isn't an option for you?


Regards

ap

[1]  
https://github.com/jclouds/jclouds/blob/master/common/openstack/src/main/java/org/jclouds/openstack/handlers/RetryOnRenew.java

Re: SLF4JLoggingModule cannot be resolved to a type

[Andrew Phillips]

can you please explain why I am getting this error. I downloaded all
required jars.


Could you put the code you're using and the exact stacktrace you're  
getting into a Gist or Pastie [1]?


Regards

ap

[1] http://jclouds.apache.org/reference/report-a-bug/

RE: IllegalArgumentException when deleting node on Rackspace

[Andrew Phillips]

Of course, the proper fix is to make sure the DNS validator is not   
used when cleaning up key pairs...


Should an error during a cleanup process like this lead to the  
exception being caught and logged, rather than being propagated to the  
user?


ap

Re: jclouds S3 provider calls GetBucketAcl every time putBlob() is called?

[Andrew Phillips]

How would I pass in my new context module?


That would be a good question to ask on #IRC [1], I think.

Basically, as far as I recall (don't quote me on this! ;-)), you'll  
have to create your own ApiMetadata, probably by overriding  
S3ApiMetadata [2], and change the set of default modules. You can then  
pass that to newBuilder [3].


Regards

ap

[1] http://jclouds.apache.org/community/
[2]  
https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/S3ApiMetadata.java
[3]  
http://javadocs.jclouds.cloudbees.net/org/jclouds/ContextBuilder.html#ContextBuilder(org.jclouds.apis.ApiMetadata)

RE: How to get reference to NeutronApi from ComputeServiceContext

[Andrew Phillips]

Usually I would not recommend using different versions of the  
dependencies. However, I don't recall any particular reasons they  
might conflict.


Definitely would like to repeat that it's not recommended to mix  
different versions of jclouds JARs in your app. But of course, we can  
always try and see ;-)


ap

Re: jclouds S3 provider calls GetBucketAcl every time putBlob() is called?

[Andrew Phillips]

blobStore.putBlob(), the S3 provider requests the bucket's ACL first. From
looking at the code, it appears that the ACLs should be cached after the
first GET, but I'm not seeing that.


Could you point out which code (path) specifically you're looking at here?

Thanks!

ap

Re: How does jclouds storage my files?

[Andrew Phillips]

2 - I use this implementation of iSCSI http://jscsi.org/. And it is this
class that saves my blobs
http://grepcode.com/file/repo1.maven.org/maven2/org.jscsi/target/2.5.2/org/jscsi/target/storage/JCloudsStorageModule.java?av=f


As far as I can see, the questions you have are related to jSCSI,  
which is not part of jclouds. Have you tried asking on the jSCSI  
mailing list [1]?


ap

[1] https://github.com/disy/jSCSI/blob/master/pom.xml#L65-L72

Re: How does jclouds storage my files?

[Andrew Phillips]

yes. But What I am doing is editing jscsi code. So they don't know how to
answer because they don't move the blobs to other place.


Ah, OK. Still, the specific details of how a blob is stored are either  
jSCSI-specific, or (as Andrew Gaul pointed out) specific to the native  
filesystem provider in the JVM, which in turn depends on the  
underlying filesystem.


The jclouds filesystem API does not do any of this work itself: it  
simply uses the standard Java file libraries to write and write files.


So I suspect you will have to either check with jSCSI or examine the  
file handling implementation of the JVM on which you are running :-(


ap

Re: jclouds-example using up-to-date jclouds ?

[Andrew Phillips]

Latest jclouds-examples can build under 1.8.0. With jclouds 2.0, I see the
errors below:


Since jclouds 2.0.0 does not exist yet, you will not be able to use  
that version. Please use 1.8.0 instead.


ap

Re: How does jclouds storage my files?

[Andrew Phillips]

Hi Filipe


I am new with jclouds. What I am doing is: Opening a disk with java and
formatting it with jclouds. So, instead of write Files.write() I use
jclouds.


Which provider or API are you using? And which underlying system (a  
file system, I suspect?) are you talking to?


Which commands are you using to store the blobs, and which to read  
them? What is the error?


Could you create a small test case that we could run to reproduce the  
behaviour you're seeing?


Regards

ap