Hey folks, For suggestions to improve the compliance testing suite, I'd highly encourage you to write to the spec list with feedback.
-Dan On Mon, Aug 25, 2008 at 2:16 AM, Chris Chabot <[EMAIL PROTECTED]> wrote: > Ps i also tried the archived old test gadget (that I'm kinda missing right > now) at > > http://opensocial-resources.googlecode.com/svn/tests/trunk/archive/compliancetests.xml > > However that seems to error out now too (java + samplecontainer & on php > shindig + partuza as well) on "request create activity" (x2), "update person > app data" and "people field return types". > > In other words, I'm having a bit of a rough time testing stuff while trying > to code up a json-rpc interface in php.. if everything is 'red', it's a good > bit harder to verify, and i simply don't have the time right now to write my > own gadget test code combined with the php code. > > I imagine containers currently working to implement shindig (and there's a > few of those i know of) might run into these issues as well > > -- Chris > > > > On Aug 25, 2008, at 11:09 AM, Kevin Brown wrote: > > On Mon, Aug 25, 2008 at 2:05 AM, Chris Chabot <[EMAIL PROTECTED]> wrote: >> >> When you load up the reference test suite in the java sample container the >>> result is: >>> 94 Passed >>> 47 Failed >>> 2 Warnings >>> 22 Unverified >>> >>> now 47 is a bit steep, especially considering that this uses java's >>> json-rpc interface, so instruction ordering shouldn't be an issue here. >>> >>> A fair bit of those are silly errors (like the empty proxy string), or >>> errors like "expected 'gadgets.Tab', got " 'gadgets.[object >>> Object],<spam>'". and some errors that make no sense to me "[PPL005.1] >>> Nonsupported Field - familyName: FAILED: (got 'Doe'), expected >>> 'undefined'" >>> (name is supported, so why complain you got a familyName?) >>> >>> However that doesn't account for all 47 errors, there's a few real ones >>> in >>> there too, and it's currently quite hard to separate the real failures >>> from >>> the ones that don't really matter. >>> >>> I'm slightly concerned that with such a volume of errors (wether they are >>> real errors or not), the tool looses it's usefulness. I mean if someone >>> checks out shindig, implements the basic services and runs the test suite >>> to >>> see if they did that correctly .... How would one not completely familiar >>> with the complete opensocial stack be able to diagnose what is or isn't >>> the >>> fault of their own code? A needle and haystack come to mind :) >>> >> >> >> I agree, which is why we should talk to the people writing the compliance >> gadget and get rid of the unnecessary stuff first, then we can look for >> real >> problems. >> >> >> >>> >>> -- Chris >>> >>> >>> On Aug 25, 2008, at 4:58 AM, Cassie wrote: >>> >>> Ahh - disclaimer for my last statement - I was only talking about the >>> >>>> opensocial related tests... I don't usually check the non-social ones >>>> :) >>>> >>>> - Cassie >>>> >>>> >>>> On Sat, Aug 23, 2008 at 4:46 PM, Chris Chabot <[EMAIL PROTECTED]> >>>> wrote: >>>> >>>> Don't suppose these are easy to fix on the javascript side? >>>>> >>>>> gadgets.io.* TestSuite: >>>>> >>>>> Description> Tests if we can get the proxy URL with given URL as proxy >>>>> [GIO101.0] gadgets.io.getProxyUrl(String) - With valid URL.: PASS: (got >>>>> 'proxy?refresh=3600&url=http%3A%2F%2Fwww%2F~user') >>>>> [GIO101.1] gadgets.io.getProxyUrl(String) - With valid URL.: PASS: (got >>>>> 'proxy?refresh=3600&url=http%3A%2F%2Fwww%2F~user') >>>>> [GIO102] [P2 ]:: gadgets.io.getProxyUrl() - With no parameters.: FAILED >>>>> [458 >>>>> ms] >>>>> Description> Tests if we can call getProxyUrl API without any parameter >>>>> and >>>>> it returns the proxy URL with empty proxy >>>>> [GIO102.0] gadgets.io.getProxyUrl() - With no parameters.: PASS: (got >>>>> 'proxy?refresh=3600&url=undefined') >>>>> [GIO102.1] gadgets.io.getProxyUrl() - With no parameters.: FAILED: (got >>>>> 'proxy?refresh=3600&url=undefined'), expected 'url=' >>>>> [GIO103] [P2 ]:: gadgets.io.getProxyUrl(null) - With null parameter.: >>>>> FAILED >>>>> [462 ms] >>>>> Description> Tests if we can call getProxyUrl API with null as >>>>> parameter >>>>> and >>>>> it returns the proxy URL with empty proxy >>>>> [GIO103.0] gadgets.io.getProxyUrl(null) - With null parameter.: PASS: >>>>> (got >>>>> 'proxy?refresh=3600&url=null') >>>>> [GIO103.1] gadgets.io.getProxyUrl(null) - With null parameter.: FAILED: >>>>> (got >>>>> 'proxy?refresh=3600&url=null'), expected 'url=' >>>>> [GIO104] [P2 ]:: gadgets.io.getProxyUrl(undefined) - With undefined >>>>> parameter.: FAILED [466 ms] >>>>> Description> Tests if we can call getProxyUrl API with undefined as >>>>> parameter and it returns the proxy URL with empty proxy >>>>> [GIO104.0] gadgets.io.getProxyUrl(undefined) - With undefined >>>>> parameter.: >>>>> PASS: (got 'proxy?refresh=3600&url=undefined') >>>>> [GIO104.1] gadgets.io.getProxyUrl(undefined) - With undefined >>>>> parameter.: >>>>> FAILED: (got 'proxy?refresh=3600&url=undefined'), expected 'url=' >>>>> >>>>> I'm just a sucker for seeing green boxes is all :) >>>>> >>>>> -- Chris >>>>> >>>>> On Aug 22, 2008, at 11:04 PM, Cassie wrote: >>>>> >>>>> I check the compliance tests regularly for the actual deployment of >>>>> >>>>>> Shindig >>>>>> that I work on at work. We are failing more now only because the tests >>>>>> are >>>>>> getting much more thorough. (The tests are also very active so >>>>>> sometimes >>>>>> they have bugs too although it is usually our code that's wrong :) >>>>>> >>>>>> I haven't found many issues with Shindig's actual js layer though - >>>>>> its >>>>>> usually been in the server layer and most often in the service >>>>>> implementations that are container specific. >>>>>> >>>>>> The non-rpc based container definitely has some issues though because >>>>>> it >>>>>> sending requests to the server in a json map format... which doesn't >>>>>> preserve order. So, some of the compliance tests would fail simply >>>>>> because >>>>>> they were fetching app data before it was updated and so forth. >>>>>> >>>>>> So... hopefully someone out there can get a patch to switch the php to >>>>>> rpc >>>>>> batching going :) >>>>>> >>>>>> - Cassie >>>>>> >>>>>> >>>>>> On Fri, Aug 22, 2008 at 12:28 PM, Dan Peterson <[EMAIL PROTECTED]> >>>>>> wrote: >>>>>> >>>>>> By the way, the docs for the compliance test suite are at: >>>>>> >>>>>>> http://code.google.com/p/opensocial-resources/wiki/ComplianceTests >>>>>>> >>>>>>> -Dan >>>>>>> On Fri, Aug 22, 2008 at 12:22 PM, Louis Ryan <[EMAIL PROTECTED]> >>>>>>> wrote: >>>>>>> >>>>>>> Im seeing some similar issues. One thing I noticed is that lookingFor >>>>>>> >>>>>>>> is >>>>>>>> now >>>>>>>> an Enum in JS but its not the Java datamodel. Im going to fix that >>>>>>>> one. >>>>>>>> >>>>>>>> On Fri, Aug 22, 2008 at 11:54 AM, Chris Chabot <[EMAIL PROTECTED]> >>>>>>>> >>>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>> When running the compliance test suite: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.7/compliance/reference/reference.xml >>>>>>> >>>>>>> >>>>>>>> I get 28 failed on my live version of partuza+php shindig ( >>>>>>>>> >>>>>>>>> >>>>>>>> www.partuza.nl is >>>>>>>> >>>>>>>> >>>>>>>>> running a checkout that is about 1.5 weeks old), while the latest >>>>>>>>> code >>>>>>>>> locally gives me 42 errors. >>>>>>>>> >>>>>>>>> To rule out that it wasn't the php code, i updated just shindig/php >>>>>>>>> on >>>>>>>>> >>>>>>>>> >>>>>>>> the >>>>>>>> >>>>>>>> >>>>>>>>> live server, and the error count didn't change, so it's probably >>>>>>>>> some >>>>>>>>> shindig//features/* changes that cause this. >>>>>>>>> >>>>>>>>> Anyone checking if our JS code is 'compliant' ? And/or working on >>>>>>>>> >>>>>>>>> >>>>>>>> fixing >>>>>>> >>>>>>> >>>>>>>> it? Seems right now it's only getting less so :) >>>>>>>>> >>>>>>>>> -- Chris >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>> >
