Noorul Islam K M <[email protected]> writes: > Ignasi <[email protected]> writes: > >> It is not intentional if there are some missing apis. We added the >> environment API time ago [1], but not all endpoints were added then, >> and are still missing. >> >> Mind opening an Improvement in JIRA to add the missing ones? >> >> Also, if you want to contribute them [2], that would be very welcome! >> :) I plan to merge JCLOUDS-246 as it might block contributions (due to >> the change of the project structure). >> > > Created https://issues.apache.org/jira/browse/JCLOUDS-256 > > But I am not able to assign it to myself. So I left a comment saying > that I am already working on this. >
Here is the initial commit on a branch. https://github.com/noorul/jclouds-chef/commit/d97e14f0a445b0d2ae731ced212664b9ff3b2896 I will wait for you to merge live-test so that I can add a live test and create a pull request. Thanks and Regards Noorul > Thanks and Regards > Noorul > >> >> Ignasi >> >> [1] https://github.com/jclouds/legacy-jclouds-chef/pull/61 >> [2] https://wiki.apache.org/jclouds/How%20to%20Contribute >> >> On 28 August 2013 11:03, Noorul Islam K M <[email protected]> wrote: >>> Ignasi <[email protected]> writes: >>> >>>> Hi Noorul, >>>> >>>> Finally, I've fixed the issue! :) Here is the commit [1] that fixes >>>> it. Basically we don't have to encode the '+' and ' ' characters in >>>> the Signature query parameter. With this fix applied, all live tests >>>> are passing for Chef 0.10, Chef 11 and Enterprise Chef. >>>> >>>> Feel free to checkout my branch [2] and validate the live tests >>>> (specially the testDownloadCookbooks). >>>> >>> >>> Thanks a lot! >>> >>> I see that some of the APIs are not exposed, for example >>> >>> http://docs.opscode.com/api_chef_server_environments_recipe.html >>> >>> Is this something that consciously avoided? >>> >>> Thanks and Regards >>> Noorul >>> >>>> >>>> Ignasi >>>> >>>> >>>> P.S. I'll submit the PR to have it merged in 1.7.0-SNAPSHOT as soon as >>>> JCLOUDS-246 [2] gets merged. >>>> >>>> >>>> [1] >>>> https://github.com/nacx/jclouds-chef/commit/67c1545aab3b742e9508f8bd3837b57de25c47e5 >>>> [2] https://github.com/nacx/jclouds-chef/tree/245-live-tests >>>> [3] https://issues.apache.org/jira/browse/JCLOUDS-246 >>>> >>>> On 27 August 2013 13:18, Noorul Islam K M <[email protected]> wrote: >>>>> Ignasi <[email protected]> writes: >>>>> >>>>>> Hi Noorul, >>>>>> >>>>>> Just a follow up. I've been able to isolate (I think) the issue and >>>>>> I've opened the corresponding JIRA [1]. I asked the chef-dev mailing >>>>>> list [2] to see if someone can give us the details of how to properly >>>>>> sign the URLs that are failing. It seems that some change may be >>>>>> required in the signing filter [3]. >>>>>> >>>>>> Let's see if we can get this fixed! >>>>>> >>>>> >>>>> Thank you! >>>>> >>>>> Thanks and Regards >>>>> Noorul >>>>> >>>>>> >>>>>> [1] https://issues.apache.org/jira/browse/JCLOUDS-254 >>>>>> [2] http://lists.opscode.com/sympa/arc/chef-dev/2013-08/msg00034.html >>>>>> [3] >>>>>> https://github.com/jclouds/jclouds-chef/blob/master/core/src/main/java/org/jclouds/chef/filters/SignedHeaderAuth.java >>>>>> >>>>>> On 26 August 2013 16:54, Ignasi <[email protected]> wrote: >>>>>>> Ok, I just added a few cookbooks more to my Hosted Chef account and >>>>>>> the download is failing the same way. >>>>>>> I still haven't figured out what can be failing in the signature (it >>>>>>> shouldn't have changed since Chef 0.10), so any help would be >>>>>>> appreciated! >>>>>>> >>>>>>> On 26 August 2013 15:55, Ignasi <[email protected]> wrote: >>>>>>>> Hi again, >>>>>>>> >>>>>>>> The problem appears in the "getResourceContents" api call. After >>>>>>>> debugging for a while I haven't found an explanation. It fails >>>>>>>> randomly with random resources, and there is no pattern for that :( >>>>>>>> Sometimes it fails to download a root file resource, sometimes not. >>>>>>>> Sometimes it fails downloading a recipe file, sometimes it just works. >>>>>>>> >>>>>>>> I executed the live test suite against a Chef 0.10.8, and against >>>>>>>> Enterprise Chef and all tests are passing, so I assume it might be an >>>>>>>> issue in Chef 11; otherwise I can not explain why the same resource >>>>>>>> sometimes is downloaded and sometimes it is not. The authentication >>>>>>>> mechanism hasn't changed, and if it works for Chef 0.10 and Enterprise >>>>>>>> Chef, it should work for Chef 11 too (or at least have some >>>>>>>> deterministic behavior). >>>>>>>> >>>>>>>> >>>>>>>> To properly reflect that in the live tests, I've separated the >>>>>>>> "testListCookbooks" in two different tests: "testListCookbooks" and >>>>>>>> "testDownloadCookbooks". If you update my branch, you should get those >>>>>>>> changes. >>>>>>>> >>>>>>>> I'll search if there is any open issue and file a new one if >>>>>>>> appropriate. >>>>>>>> >>>>>>>> Anyway, let me know if you find something that could help getting rid >>>>>>>> of this! >>>>>>>> >>>>>>>> >>>>>>>> Ignasi >>>>>>>> >>>>>>>> >>>>>>>> On 26 August 2013 14:29, Noorul Islam K M <[email protected]> wrote: >>>>>>>>> Ignasi <[email protected]> writes: >>>>>>>>> >>>>>>>>>> Thanks a lot for your feedback Noorul! >>>>>>>>>> >>>>>>>>>> You can run the 'core' live tests as follows: >>>>>>>>>> >>>>>>>>>> mvn clean install -pl core -Plive \ >>>>>>>>>> -Dtest.chef.endpoint=<you chef server url> \ >>>>>>>>>> -Dtest.chef.identity=<identity> \ >>>>>>>>>> -Dtest.chef.credential=<path to the PEM private key> >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think, I get the same as you >>>>>>>>> >>>>>>>>> >>>>>>>>> Test suite progress: tests succeeded: 41, failed: 1, skipped: 1. >>>>>>>>> Tests run: 43, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: >>>>>>>>> 187.134 sec <<< FAILURE! >>>>>>>>> >>>>>>>>> Results : >>>>>>>>> >>>>>>>>> Failed tests: testListCookbooks(org.jclouds.chef.ChefApiLiveTest): >>>>>>>>> <?xml version="1.0" >>>>>>>>> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch >>>>>>>>> </Code><Message>The request signature we calculated does not match >>>>>>>>> the signature you provided. Check your key and signing >>>>>>>>> method.</Message><S >>>>>>>>> tringToSignBytes></StringToSignBytes><RequestId>g2gCZAATYm9va3NoZWxmQDEyNy4wLjAuMWgDYgAABWFiAAef4WIAC6/x</RequestId><HostId></HostId><Signatu >>>>>>>>> reProvided>hRkpN1RzN >>>>>>>>> /pGeMmyXnDtRZUNnI=</SignatureProvided><StringToSign>GET(..) >>>>>>>>> >>>>>>>>> Tests run: 43, Failures: 1, Errors: 0, Skipped: 1 >>>>>>>>> >>>>>>>>> Thanks and Regards >>>>>>>>> Noorul >>>>>>>>> >>>>>>>>>> And if you have a hosted chef account, you can run them like this: >>>>>>>>>> >>>>>>>>>> mvn clean install -pl enterprise -Plive \ >>>>>>>>>> -Dtest.enterprisechef.org=<you organization name> \ >>>>>>>>>> -Dtest.enterprisechef.identity=<identity> \ >>>>>>>>>> -Dtest.enterprisechef.credential=<path to the PEM private key> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 26 August 2013 14:10, Noorul Islam K M <[email protected]> wrote: >>>>>>>>>>> Ignasi <[email protected]> writes: >>>>>>>>>>> >>>>>>>>>>>> Noorul, I've just uploaded to my branch [1] a change that >>>>>>>>>>>> registers a >>>>>>>>>>>> custom Map deserializer to allow repeated keys. It just keeps the >>>>>>>>>>>> last >>>>>>>>>>>> one, which seems to be the one with the right content. You can >>>>>>>>>>>> take a >>>>>>>>>>>> look at the commit [2] if you want to know the details. >>>>>>>>>>>> >>>>>>>>>>>> I've tried the api call that was failing to you, and now it works >>>>>>>>>>>> and >>>>>>>>>>>> the cookbook is properly deserialized. However, the live test to >>>>>>>>>>>> list >>>>>>>>>>>> all cookbooks is still failing due to a signature error (which is >>>>>>>>>>>> weird, because the signature is generated in a generic filter [3] >>>>>>>>>>>> that >>>>>>>>>>>> applies to all requests). >>>>>>>>>>>> >>>>>>>>>>>> I think it might be related to my Chef 11 installation. To make >>>>>>>>>>>> sure, >>>>>>>>>>>> could you update the branch and run the live tests, or try to list >>>>>>>>>>>> the >>>>>>>>>>>> cookbooks using the ChefApi and ChefServer, and share the results? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I tested these methods after pulling latest from your branch. >>>>>>>>>>> >>>>>>>>>>> getCookbook() >>>>>>>>>>> listCookbooks() >>>>>>>>>>> listEnvironmentCookbooks() >>>>>>>>>>> >>>>>>>>>>> Everything seems to be working fine. >>>>>>>>>>> >>>>>>>>>>> How do I run live tests? Is this documented somewhere? >>>>>>>>>>> >>>>>>>>>>> Thanks and Regards >>>>>>>>>>> Noorul >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ignasi >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [1] https://github.com/nacx/jclouds-chef/tree/245-live-tests >>>>>>>>>>>> [2] >>>>>>>>>>>> https://github.com/nacx/jclouds-chef/commit/59115b5a08197547cf74f6831e265fe67a3b9ab8 >>>>>>>>>>>> [3] >>>>>>>>>>>> https://github.com/nacx/jclouds-chef/blob/245-live-tests/core/src/main/java/org/jclouds/chef/filters/SignedHeaderAuth.java >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 26 August 2013 12:24, Ignasi <[email protected]> wrote: >>>>>>>>>>>>> As soon as possible. >>>>>>>>>>>>> >>>>>>>>>>>>> My branch depends on issue JCLOUDS-246 [1]. Since changes are not >>>>>>>>>>>>> backward compatible (just a massive package and artifactId >>>>>>>>>>>>> renaming), >>>>>>>>>>>>> I sent a mail last week to let users express an opinion on the >>>>>>>>>>>>> change >>>>>>>>>>>>> [2]. I plan to merge it this week, since there are no opinions >>>>>>>>>>>>> against, and as soon as it is merged, I'll merge my current >>>>>>>>>>>>> branch. >>>>>>>>>>>>> >>>>>>>>>>>>> You could expect to have it merged by the end of the week, if I >>>>>>>>>>>>> finish >>>>>>>>>>>>> the fix by then (which I hope :)). I plan to upload to my branch >>>>>>>>>>>>> the >>>>>>>>>>>>> fix this afternoon; I'll let you know when I have something >>>>>>>>>>>>> usable so >>>>>>>>>>>>> you can test and validate it. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for your help! >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Ignasi >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/JCLOUDS-246 >>>>>>>>>>>>> [2] http://markmail.org/message/5jm2ivpypewnov3h >>>>>>>>>>>>> >>>>>>>>>>>>> On 26 August 2013 12:02, Noorul Islam K M <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Ignasi <[email protected]> writes: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just tested the openssl cookbook and found that the returned >>>>>>>>>>>>>>> Metadata >>>>>>>>>>>>>>> has a duplicate key in the recipes list. I've pasted the json >>>>>>>>>>>>>>> here [1] >>>>>>>>>>>>>>> so you can have a look. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll ping Chef people to see if it is an issue in their side, >>>>>>>>>>>>>>> and try >>>>>>>>>>>>>>> to build a workaround in the meanwhile. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would like to know, usually how long it takes something to get >>>>>>>>>>>>>> into >>>>>>>>>>>>>> upstream from the branch that you are working on? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can provide any help needed to make this working but need to >>>>>>>>>>>>>> be sure >>>>>>>>>>>>>> that I can use this in production as soon as possible. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks and Regards >>>>>>>>>>>>>> Noorul >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] http://pastie.org/8270391 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 26 August 2013 08:28, Ignasi <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> Hi Noorul! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is definitely the intention to support Chef 11 as soon >>>>>>>>>>>>>>>> as possible. I >>>>>>>>>>>>>>>> have a work in progress branch [1] where I fixed all >>>>>>>>>>>>>>>> Enterprise Chef live >>>>>>>>>>>>>>>> tests and also made all live tests pass against a Chef 11 >>>>>>>>>>>>>>>> except the >>>>>>>>>>>>>>>> "listCookbooks" one. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I get the exact same error than you, and it happens when >>>>>>>>>>>>>>>> deserializing the >>>>>>>>>>>>>>>> json for the cookbook into the CookbookVersion object. It >>>>>>>>>>>>>>>> seems that the >>>>>>>>>>>>>>>> Chef Server is returning a duplicate key in the json structure >>>>>>>>>>>>>>>> (although it >>>>>>>>>>>>>>>> shouldn't because we are forcing by default a 0.10 version >>>>>>>>>>>>>>>> header to make it >>>>>>>>>>>>>>>> return the same responses than in previous versions), and the >>>>>>>>>>>>>>>> json parser >>>>>>>>>>>>>>>> fails to build the corresponding map. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm currently working on this but it's taking longer than >>>>>>>>>>>>>>>> expected, so any >>>>>>>>>>>>>>>> help would be much appreciated. If you could build my branch >>>>>>>>>>>>>>>> and enable >>>>>>>>>>>>>>>> jclouds wire logging [2], you would be able to see the json >>>>>>>>>>>>>>>> returned for >>>>>>>>>>>>>>>> your cookbook. Could you share that json? Also, any help >>>>>>>>>>>>>>>> tracking/isolating >>>>>>>>>>>>>>>> the error (or patch :)) is appreciated too! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ignasi >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] https://github.com/nacx/jclouds-chef/tree/245-live-tests >>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>> http://jclouds.incubator.apache.org/documentation/reference/jclouds-logging/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> El 26/08/2013 07:27, "Noorul Islam K M" <[email protected]> >>>>>>>>>>>>>>>> escribió: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We are planning to use jclouds-chef library to talk to >>>>>>>>>>>>>>>>> community Chef >>>>>>>>>>>>>>>>> Server 11. In my initial testings, I get the following >>>>>>>>>>>>>>>>> exception when I >>>>>>>>>>>>>>>>> try to get cookbook details using >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> api.getCookbook("openssl", "1.0.2") >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> | java.lang.IllegalArgumentException: duplicate key: openssl >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:67) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.common.collect.ImmutableMap$Builder.fromEntryList(ImmutableMap.java:249) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:235) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:313) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:278) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172) >>>>>>>>>>>>>>>>> at com.google.gson.Gson.fromJson(Gson.java:795) >>>>>>>>>>>>>>>>> at com.google.gson.Gson.fromJson(Gson.java:761) >>>>>>>>>>>>>>>>> at com.google.gson.Gson.fromJson(Gson.java:710) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.jclouds.json.internal.GsonWrapper.fromJson(GsonWrapper.java:47) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.jclouds.http.functions.ParseJson.apply(ParseJson.java:86) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> README.md says that >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It currently supports versions 0.9 and 0.10 of the standard >>>>>>>>>>>>>>>>> Chef server >>>>>>>>>>>>>>>>> apis, and an initial and very basic (still in progress) >>>>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>>> of the user and organization api of the Hosted and Private >>>>>>>>>>>>>>>>> Chef >>>>>>>>>>>>>>>>> flavours. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think that the issue I am facing is because I am using Chef >>>>>>>>>>>>>>>>> server 11. >>>>>>>>>>>>>>>>> Is there any plans to officially support Chef Server 11 in >>>>>>>>>>>>>>>>> near future? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks and Regards >>>>>>>>>>>>>>>>> Noorul
