On Wed, Jul 5, 2017 at 6:52 AM, Ruwan Abeykoon <[email protected]> wrote:
> Hi All, > Adjusting test *Dynamically* at runtime is wrong. Tests has to be exact > and precise. > What we should do is to add new tests specifically to target the change in > schema/schema extensions. Still each test should not be dynamic. > We will end up with many test cases, but each single test only check one > thing properly. We should be able to use common methods(not tests) to > reduce code duplication. > > Cheers, > Ruwan > > On Wed, Jul 5, 2017 at 1:25 AM, Omindu Rathnaweera <[email protected]> > wrote: > >> Since this is a 'compliance test suite' it makes sense to terminate the >> tests at the point where it deviates from the spec. However, it would be >> nice to have an option to run certain tests even if some of the said >> critical tests fail, especially the tests related to sp configs and schemas >> . The reason is that we had the same limitation when testing our product >> against the existing compliance test. The test failed since our SCIM >> implementation didn't support /ServiceProviderConfigs and /Schemas >> endpoints and from the compliance test perspective, this should be the >> correct behavior. But it prevented us from testing our other endpoints >> (/Users, /Groups). So I prefer if we can provide the option to test a >> specific endpoint at least with the default schema (or better give the >> option to upload a schema) so the user can get an idea on which parts of >> the implementation is spec compliant and which part isn't. >> >> As for the dynamic nature of the test suite, I guess we will need to >> adjust the test suite at runtime to a certain degree since we are testing >> the schema extensions. We cannot test the nature of those extended >> attributes unless we adjust the test suite according to schema. >> > In this case I don't think it's a schema extension. It's about custom schema that violates the SCIM schema. E.g. making username attribute immutable which is against SCIM schema. >> Regards, >> Omindu. >> >> On Mon, Jul 3, 2017 at 9:03 AM, Ruwan Abeykoon <[email protected]> wrote: >> >>> Hi All, >>> I would also think terminate the test is the correct move, when it is >>> not compliant >>> >>> We could add more test cases if we need to test adjusting the schema. >>> These are different test than 100% compliancy test. >>> >>> Apart from this, Each test case should be able to run irrespective of >>> the result of the other test. We need to adhere "Isolation" on test cases. >>> [1] >>> >>> [1] http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html >>> >>> Cheers, >>> Ruwan >>> >>> >>> On Sun, Jul 2, 2017 at 7:52 AM, Vindula Jayawardana < >>> [email protected]> wrote: >>> >>>> Hi Johann, >>>> >>>>> >>>>>> In SCIM 2.0 compliance test suite, there are inter dependencies >>>>>> between tests. For an example, >>>>>> >>>>>> We have identified /Schemas endpoint as a critical test which tests >>>>>> the schemas corresponding to user and group resources according to SCIM >>>>>> specification. However in a case where a SCIM service provider has >>>>>> customized the schemas according to their own requirements, this test >>>>>> will >>>>>> be failed. As the test suite uses the /Schemas endpoint to learn about >>>>>> the >>>>>> service providers schema definitions (consider the case where there is a >>>>>> user schema extension defined by the service provider), if the /Schemas >>>>>> endpoint fails, the test suite will be terminated immediately as the test >>>>>> suite cannot learn the configurations. However we can also make it not to >>>>>> terminate but to get adjusted to the service provider's configs after >>>>>> just >>>>>> failing the /Schemas endpoint test only. With that, the service provider >>>>>> will be able to run the remaining tests on the altered schemas without >>>>>> being blocked due to test dependency. But it should also be noted that, >>>>>> this approach can cause the test suite to not to adhere to the >>>>>> specification, but to adjust itself dynamically after a proper indication >>>>>> of the reason for the adjustment. >>>>>> >>>>>> Hence, as identified in the above example, there are two possible >>>>>> options in a test dependency situation. >>>>>> >>>>>> 1. Terminate >>>>>> 2. Adjust accordingly and continue the suite but fails only the >>>>>> parent test. >>>>>> >>>>>> What is the best way of handling this?. Any thoughts on this is >>>>>> highly appreciated. >>>>>> >>>>> >>>>> I would say just terminate. Why would you need to customize the core >>>>> user schema when there is the possibility of defining an extended user >>>>> schema. No one asked you to stick to any of the attributes coming from the >>>>> core schema right? >>>>> >>>> >>>> Yes, you have a good point on this. Assuming user schema can be done in >>>> the way you have specified, still there can be a problem when it comes to >>>> group schema since it does not have an extension defined. However due to >>>> the less number of attributes associated with the group schema and their >>>> nature of it, I would say, it is very likely that service providers will >>>> not be altering the group schema most of the cases. Hence putting an effort >>>> on the second option could be redundant. So I agree with the termination >>>> based on your reasoning. >>>> >>>>> >>>>> >>>>>> >>>>>> Thank you >>>>>> *Vindula Jayawardana* >>>>>> Computer Science and Engineering Dept. >>>>>> University of Moratuwa >>>>>> mobile : +713462554 >>>>>> Email : [email protected] >>>>>> >>>>>> <https://www.facebook.com/vindula.jayawardana> >>>>>> <http://lk.linkedin.com/pub/vindula-jayawardana/a7/315/53b> >>>>>> <https://plus.google.com/u/0/+VindulaJayawardana/posts> >>>>>> <https://twitter.com/vindulajay> >>>>>> >>>>>> *“Respect is how to treat everyone, not just those you want to >>>>>> impress. "* >>>>>> >>>>>> >>>>>> *-Richard Branson-* >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> >>>>> *Johann Dilantha Nallathamby* >>>>> Senior Technical Lead - WSO2 Identity Server >>>>> Governance Technologies Team >>>>> WSO2, Inc. >>>>> lean.enterprise.middleware >>>>> >>>>> Mobile - *+94777776950* >>>>> Blog - *http://nallaa.wordpress.com <http://nallaa.wordpress.com>* >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> *Ruwan Abeykoon* >>> *Associate Director/Architect**,* >>> *WSO2, Inc. http://wso2.com <https://wso2.com/signature> * >>> *lean.enterprise.middleware.* >>> >>> >> >> >> -- >> Omindu Rathnaweera >> Senior Software Engineer, WSO2 Inc. >> Mobile: +94 771 197 211 <+94%2077%20119%207211> >> > > > > -- > > *Ruwan Abeykoon* > *Associate Director/Architect**,* > *WSO2, Inc. http://wso2.com <https://wso2.com/signature> * > *lean.enterprise.middleware.* > > -- Thanks & Regards, *Johann Dilantha Nallathamby* Senior Technical Lead - WSO2 Identity Server Governance Technologies Team WSO2, Inc. lean.enterprise.middleware Mobile - *+94777776950* Blog - *http://nallaa.wordpress.com <http://nallaa.wordpress.com>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
