Chris,
Thanks for trying the gadget. It uncovered couple wrong API use. I
fixed tehm in the gadget. Rightnow this is the summary for partuza.nl
Passed Failed Warnings Unverified Total
116 18 5 23 162
It's much cleaner now :-)
PPLX01 - throws exception : line 115 : peoplesuite.js
PPL004 - gives false for
opensocial.getEnvironment().supportsField(opensocial.Environment.ObjectType.ADDRESS,
"familyName"); but it also returns value which is contradictory. Test
expects true for any field which has value.
Opensocial spec 0.7 Definition :: SPEC-DEF-TEST - Need to fix the spec
gadgets.util.* Test Suite :: UTIL003 - container returns false for
opensocial-0.7, setPrefs etc.
AppData Test Suite :: APP010 -
AppData Test Suite :: APP103
AppData Test Suite :: APP302 -
AppData Test Suite :: APP008 -
in above cases it gives Internal server error in error message but
hadError is always false.
AppData Test Suite :: APP300 - it expects error because we are making call
req.add(req.newFetchPersonAppDataRequest('VIEWER_OWNER_ABC',
[this.id + '_dataKey1']), 'invalidFetchRequest');
'VIEWER_OWNER_ABC' is invalid id.
MakeRequest Test Suite :: SMKRT001 - data.validated = false
MakeRequest Test Suite :: MKRT005 - Json parse error
MakeRequest Test Suite :: MKRT011 - json parse error
gadget.Prefs Test Suite :: PREF010 - prefs are not honored for all the
cases below
gadget.Prefs Test Suite :: PREF020
gadget.Prefs Test Suite :: PREF030
gadget.Prefs Test Suite :: PREF040
gadget.Prefs Test Suite :: PREF050
I believe that gadgets spec is very little documented and requires
some brush up. But that's definitely not in the scope of compliance
gadget.
regards,
Shreyas
- Show quoted text -
On Mon, Aug 25, 2008 at 9:53 AM, Dan Peterson <[EMAIL PROTECTED]> wrote:
> Hey folks,
>
> For suggestions to improve the compliance testing suite, I'd highly
> encourage you to write to the spec list with feedback.
>
> -Dan
> On Mon, Aug 25, 2008 at 2:16 AM, Chris Chabot <[EMAIL PROTECTED]> wrote:
>
>> Ps i also tried the archived old test gadget (that I'm kinda missing right
>> now) at
>>
>> http://opensocial-resources.googlecode.com/svn/tests/trunk/archive/compliancetests.xml
>>
>> However that seems to error out now too (java + samplecontainer & on php
>> shindig + partuza as well) on "request create activity" (x2), "update person
>> app data" and "people field return types".
>>
>> In other words, I'm having a bit of a rough time testing stuff while trying
>> to code up a json-rpc interface in php.. if everything is 'red', it's a good
>> bit harder to verify, and i simply don't have the time right now to write my
>> own gadget test code combined with the php code.
>>
>> I imagine containers currently working to implement shindig (and there's a
>> few of those i know of) might run into these issues as well
>>
>> -- Chris
>>
>>
>>
>> On Aug 25, 2008, at 11:09 AM, Kevin Brown wrote:
>>
>> On Mon, Aug 25, 2008 at 2:05 AM, Chris Chabot <[EMAIL PROTECTED]> wrote:
>>>
>>> When you load up the reference test suite in the java sample container the
>>>> result is:
>>>> 94 Passed
>>>> 47 Failed
>>>> 2 Warnings
>>>> 22 Unverified
>>>>
>>>> now 47 is a bit steep, especially considering that this uses java's
>>>> json-rpc interface, so instruction ordering shouldn't be an issue here.
>>>>
>>>> A fair bit of those are silly errors (like the empty proxy string), or
>>>> errors like "expected 'gadgets.Tab', got " 'gadgets.[object
>>>> Object],<spam>'". and some errors that make no sense to me "[PPL005.1]
>>>> Nonsupported Field - familyName: FAILED: (got 'Doe'), expected
>>>> 'undefined'"
>>>> (name is supported, so why complain you got a familyName?)
>>>>
>>>> However that doesn't account for all 47 errors, there's a few real ones
>>>> in
>>>> there too, and it's currently quite hard to separate the real failures
>>>> from
>>>> the ones that don't really matter.
>>>>
>>>> I'm slightly concerned that with such a volume of errors (wether they are
>>>> real errors or not), the tool looses it's usefulness. I mean if someone
>>>> checks out shindig, implements the basic services and runs the test suite
>>>> to
>>>> see if they did that correctly .... How would one not completely familiar
>>>> with the complete opensocial stack be able to diagnose what is or isn't
>>>> the
>>>> fault of their own code? A needle and haystack come to mind :)
>>>>
>>>
>>>
>>> I agree, which is why we should talk to the people writing the compliance
>>> gadget and get rid of the unnecessary stuff first, then we can look for
>>> real
>>> problems.
>>>
>>>
>>>
>>>>
>>>> -- Chris
>>>>
>>>>
>>>> On Aug 25, 2008, at 4:58 AM, Cassie wrote:
>>>>
>>>> Ahh - disclaimer for my last statement - I was only talking about the
>>>>
>>>>> opensocial related tests... I don't usually check the non-social ones
>>>>> :)
>>>>>
>>>>> - Cassie
>>>>>
>>>>>
>>>>> On Sat, Aug 23, 2008 at 4:46 PM, Chris Chabot <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>
>>>>> Don't suppose these are easy to fix on the javascript side?
>>>>>>
>>>>>> gadgets.io.* TestSuite:
>>>>>>
>>>>>> Description> Tests if we can get the proxy URL with given URL as proxy
>>>>>> [GIO101.0] gadgets.io.getProxyUrl(String) - With valid URL.: PASS: (got
>>>>>> 'proxy?refresh=3600&url=http%3A%2F%2Fwww%2F~user')
>>>>>> [GIO101.1] gadgets.io.getProxyUrl(String) - With valid URL.: PASS: (got
>>>>>> 'proxy?refresh=3600&url=http%3A%2F%2Fwww%2F~user')
>>>>>> [GIO102] [P2 ]:: gadgets.io.getProxyUrl() - With no parameters.: FAILED
>>>>>> [458
>>>>>> ms]
>>>>>> Description> Tests if we can call getProxyUrl API without any parameter
>>>>>> and
>>>>>> it returns the proxy URL with empty proxy
>>>>>> [GIO102.0] gadgets.io.getProxyUrl() - With no parameters.: PASS: (got
>>>>>> 'proxy?refresh=3600&url=undefined')
>>>>>> [GIO102.1] gadgets.io.getProxyUrl() - With no parameters.: FAILED: (got
>>>>>> 'proxy?refresh=3600&url=undefined'), expected 'url='
>>>>>> [GIO103] [P2 ]:: gadgets.io.getProxyUrl(null) - With null parameter.:
>>>>>> FAILED
>>>>>> [462 ms]
>>>>>> Description> Tests if we can call getProxyUrl API with null as
>>>>>> parameter
>>>>>> and
>>>>>> it returns the proxy URL with empty proxy
>>>>>> [GIO103.0] gadgets.io.getProxyUrl(null) - With null parameter.: PASS:
>>>>>> (got
>>>>>> 'proxy?refresh=3600&url=null')
>>>>>> [GIO103.1] gadgets.io.getProxyUrl(null) - With null parameter.: FAILED:
>>>>>> (got
>>>>>> 'proxy?refresh=3600&url=null'), expected 'url='
>>>>>> [GIO104] [P2 ]:: gadgets.io.getProxyUrl(undefined) - With undefined
>>>>>> parameter.: FAILED [466 ms]
>>>>>> Description> Tests if we can call getProxyUrl API with undefined as
>>>>>> parameter and it returns the proxy URL with empty proxy
>>>>>> [GIO104.0] gadgets.io.getProxyUrl(undefined) - With undefined
>>>>>> parameter.:
>>>>>> PASS: (got 'proxy?refresh=3600&url=undefined')
>>>>>> [GIO104.1] gadgets.io.getProxyUrl(undefined) - With undefined
>>>>>> parameter.:
>>>>>> FAILED: (got 'proxy?refresh=3600&url=undefined'), expected 'url='
>>>>>>
>>>>>> I'm just a sucker for seeing green boxes is all :)
>>>>>>
>>>>>> -- Chris
>>>>>>
>>>>>> On Aug 22, 2008, at 11:04 PM, Cassie wrote:
>>>>>>
>>>>>> I check the compliance tests regularly for the actual deployment of
>>>>>>
>>>>>>> Shindig
>>>>>>> that I work on at work. We are failing more now only because the tests
>>>>>>> are
>>>>>>> getting much more thorough. (The tests are also very active so
>>>>>>> sometimes
>>>>>>> they have bugs too although it is usually our code that's wrong :)
>>>>>>>
>>>>>>> I haven't found many issues with Shindig's actual js layer though -
>>>>>>> its
>>>>>>> usually been in the server layer and most often in the service
>>>>>>> implementations that are container specific.
>>>>>>>
>>>>>>> The non-rpc based container definitely has some issues though because
>>>>>>> it
>>>>>>> sending requests to the server in a json map format... which doesn't
>>>>>>> preserve order. So, some of the compliance tests would fail simply
>>>>>>> because
>>>>>>> they were fetching app data before it was updated and so forth.
>>>>>>>
>>>>>>> So... hopefully someone out there can get a patch to switch the php to
>>>>>>> rpc
>>>>>>> batching going :)
>>>>>>>
>>>>>>> - Cassie
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 22, 2008 at 12:28 PM, Dan Peterson <[EMAIL PROTECTED]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> By the way, the docs for the compliance test suite are at:
>>>>>>>
>>>>>>>> http://code.google.com/p/opensocial-resources/wiki/ComplianceTests
>>>>>>>>
>>>>>>>> -Dan
>>>>>>>> On Fri, Aug 22, 2008 at 12:22 PM, Louis Ryan <[EMAIL PROTECTED]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Im seeing some similar issues. One thing I noticed is that lookingFor
>>>>>>>>
>>>>>>>>> is
>>>>>>>>> now
>>>>>>>>> an Enum in JS but its not the Java datamodel. Im going to fix that
>>>>>>>>> one.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 22, 2008 at 11:54 AM, Chris Chabot <[EMAIL PROTECTED]>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> When running the compliance test suite:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.7/compliance/reference/reference.xml
>>>>>>>>
>>>>>>>>
>>>>>>>>> I get 28 failed on my live version of partuza+php shindig (
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> www.partuza.nl is
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> running a checkout that is about 1.5 weeks old), while the latest
>>>>>>>>>> code
>>>>>>>>>> locally gives me 42 errors.
>>>>>>>>>>
>>>>>>>>>> To rule out that it wasn't the php code, i updated just shindig/php
>>>>>>>>>> on
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> live server, and the error count didn't change, so it's probably
>>>>>>>>>> some
>>>>>>>>>> shindig//features/* changes that cause this.
>>>>>>>>>>
>>>>>>>>>> Anyone checking if our JS code is 'compliant' ? And/or working on
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> fixing
>>>>>>>>
>>>>>>>>
>>>>>>>>> it? Seems right now it's only getting less so :)
>>>>>>>>>>
>>>>>>>>>> -- Chris
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>
--
regards,
Shreyas
