Since this is a 'compliance test suite' it makes sense to terminate the tests at the point where it deviates from the spec. However, it would be nice to have an option to run certain tests even if some of the said critical tests fail, especially the tests related to sp configs and schemas . The reason is that we had the same limitation when testing our product against the existing compliance test. The test failed since our SCIM implementation didn't support /ServiceProviderConfigs and /Schemas endpoints and from the compliance test perspective, this should be the correct behavior. But it prevented us from testing our other endpoints (/Users, /Groups). So I prefer if we can provide the option to test a specific endpoint at least with the default schema (or better give the option to upload a schema) so the user can get an idea on which parts of the implementation is spec compliant and which part isn't.
As for the dynamic nature of the test suite, I guess we will need to adjust the test suite at runtime to a certain degree since we are testing the schema extensions. We cannot test the nature of those extended attributes unless we adjust the test suite according to schema. Regards, Omindu. On Mon, Jul 3, 2017 at 9:03 AM, Ruwan Abeykoon <[email protected]> wrote: > Hi All, > I would also think terminate the test is the correct move, when it is not > compliant > > We could add more test cases if we need to test adjusting the schema. > These are different test than 100% compliancy test. > > Apart from this, Each test case should be able to run irrespective of the > result of the other test. We need to adhere "Isolation" on test cases. [1] > > [1] http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html > > Cheers, > Ruwan > > > On Sun, Jul 2, 2017 at 7:52 AM, Vindula Jayawardana < > [email protected]> wrote: > >> Hi Johann, >> >>> >>>> In SCIM 2.0 compliance test suite, there are inter dependencies between >>>> tests. For an example, >>>> >>>> We have identified /Schemas endpoint as a critical test which tests the >>>> schemas corresponding to user and group resources according to SCIM >>>> specification. However in a case where a SCIM service provider has >>>> customized the schemas according to their own requirements, this test will >>>> be failed. As the test suite uses the /Schemas endpoint to learn about the >>>> service providers schema definitions (consider the case where there is a >>>> user schema extension defined by the service provider), if the /Schemas >>>> endpoint fails, the test suite will be terminated immediately as the test >>>> suite cannot learn the configurations. However we can also make it not to >>>> terminate but to get adjusted to the service provider's configs after just >>>> failing the /Schemas endpoint test only. With that, the service provider >>>> will be able to run the remaining tests on the altered schemas without >>>> being blocked due to test dependency. But it should also be noted that, >>>> this approach can cause the test suite to not to adhere to the >>>> specification, but to adjust itself dynamically after a proper indication >>>> of the reason for the adjustment. >>>> >>>> Hence, as identified in the above example, there are two possible >>>> options in a test dependency situation. >>>> >>>> 1. Terminate >>>> 2. Adjust accordingly and continue the suite but fails only the parent >>>> test. >>>> >>>> What is the best way of handling this?. Any thoughts on this is highly >>>> appreciated. >>>> >>> >>> I would say just terminate. Why would you need to customize the core >>> user schema when there is the possibility of defining an extended user >>> schema. No one asked you to stick to any of the attributes coming from the >>> core schema right? >>> >> >> Yes, you have a good point on this. Assuming user schema can be done in >> the way you have specified, still there can be a problem when it comes to >> group schema since it does not have an extension defined. However due to >> the less number of attributes associated with the group schema and their >> nature of it, I would say, it is very likely that service providers will >> not be altering the group schema most of the cases. Hence putting an effort >> on the second option could be redundant. So I agree with the termination >> based on your reasoning. >> >>> >>> >>>> >>>> Thank you >>>> *Vindula Jayawardana* >>>> Computer Science and Engineering Dept. >>>> University of Moratuwa >>>> mobile : +713462554 >>>> Email : [email protected] >>>> >>>> <https://www.facebook.com/vindula.jayawardana> >>>> <http://lk.linkedin.com/pub/vindula-jayawardana/a7/315/53b> >>>> <https://plus.google.com/u/0/+VindulaJayawardana/posts> >>>> <https://twitter.com/vindulajay> >>>> >>>> *“Respect is how to treat everyone, not just those you want to impress. >>>> "* >>>> >>>> >>>> *-Richard Branson-* >>>> >>>> >>>> >>> >>> >>> -- >>> Thanks & Regards, >>> >>> *Johann Dilantha Nallathamby* >>> Senior Technical Lead - WSO2 Identity Server >>> Governance Technologies Team >>> WSO2, Inc. >>> lean.enterprise.middleware >>> >>> Mobile - *+94777776950* >>> Blog - *http://nallaa.wordpress.com <http://nallaa.wordpress.com>* >>> >> >> > > > -- > > *Ruwan Abeykoon* > *Associate Director/Architect**,* > *WSO2, Inc. http://wso2.com <https://wso2.com/signature> * > *lean.enterprise.middleware.* > > -- Omindu Rathnaweera Senior Software Engineer, WSO2 Inc. Mobile: +94 771 197 211
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
