Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Johann Nallathamby]

On Wed, Jul 5, 2017 at 6:52 AM, Ruwan Abeykoon  wrote:

> Hi All,
> Adjusting test *Dynamically* at runtime is wrong. Tests has to be exact
> and precise.
> What we should do is to add new tests specifically to target the change in
> schema/schema extensions. Still each test should not be dynamic.
> We will end up with many test cases, but each single test only check one
> thing properly. We should be able to use common methods(not tests) to
> reduce code duplication.
>
> Cheers,
> Ruwan
>
> On Wed, Jul 5, 2017 at 1:25 AM, Omindu Rathnaweera 
> wrote:
>
>> Since this is a 'compliance test suite' it makes sense to terminate the
>> tests at the point where it deviates from the spec. However, it would be
>> nice to have an option to run certain tests even if some of the said
>> critical tests fail, especially the tests related to sp configs and schemas
>> . The reason is that we had the same limitation when testing our product
>> against the existing compliance test. The test failed since our SCIM
>> implementation didn't support /ServiceProviderConfigs and /Schemas
>> endpoints and from the compliance test perspective, this should be the
>> correct behavior. But it prevented us from testing our other endpoints
>> (/Users, /Groups). So I prefer if we can provide the option to test a
>> specific endpoint at least with the default schema (or better give the
>> option to upload a schema) so the user can get an idea on which parts of
>> the implementation is spec compliant and which part isn't.
>>
>> As for the dynamic nature of the test suite, I guess we will need to
>> adjust the test suite at runtime to a certain degree since we are testing
>> the schema extensions. We cannot test the nature of those extended
>> attributes unless we adjust the test suite according to schema.
>>
>
In this case I don't think it's a schema extension. It's about custom
schema that violates the SCIM schema. E.g. making username attribute
immutable which is against SCIM schema.


>> Regards,
>> Omindu.
>>
>> On Mon, Jul 3, 2017 at 9:03 AM, Ruwan Abeykoon  wrote:
>>
>>> Hi All,
>>> I would also think terminate the test is the correct move, when it is
>>> not compliant
>>>
>>> We could add more test cases if we need to test adjusting the schema.
>>> These are different test than 100% compliancy test.
>>>
>>> Apart from this, Each test case should be able to run irrespective of
>>> the result of the other test. We need to adhere "Isolation" on test cases.
>>> [1]
>>>
>>> [1] http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html
>>>
>>> Cheers,
>>> Ruwan
>>>
>>>
>>> On Sun, Jul 2, 2017 at 7:52 AM, Vindula Jayawardana <
>>> vindula...@cse.mrt.ac.lk> wrote:
>>>
 Hi Johann,

>
>> In SCIM 2.0 compliance test suite, there are inter dependencies
>> between tests. For an example,
>>
>> We have identified /Schemas endpoint as a critical test which tests
>> the schemas corresponding to user and group resources according to SCIM
>> specification. However in a case where a SCIM service provider has
>> customized the schemas according to their own requirements, this test 
>> will
>> be failed. As the test suite uses the /Schemas endpoint to learn about 
>> the
>> service providers schema definitions (consider the case where there is a
>> user schema extension defined by the service provider), if the /Schemas
>> endpoint fails, the test suite will be terminated immediately as the test
>> suite cannot learn the configurations. However we can also make it not to
>> terminate but to get adjusted to the service provider's configs after 
>> just
>> failing the /Schemas endpoint test only. With that, the service provider
>> will be able to run the remaining tests on the altered schemas without
>> being blocked due to test dependency. But it should also be noted that,
>> this approach can cause the test suite to not to adhere to the
>> specification, but to adjust itself dynamically after a proper indication
>> of the reason for the adjustment.
>>
>> Hence, as identified in the above example, there are two possible
>> options in a test dependency situation.
>>
>> 1. Terminate
>> 2. Adjust accordingly and continue the suite but fails only the
>> parent test.
>>
>> What is the best way of handling this?. Any thoughts on this is
>> highly appreciated.
>>
>
> I would say just terminate. Why would you need to customize the core
> user schema when there is the possibility of defining an extended user
> schema. No one asked you to stick to any of the attributes coming from the
> core schema right?
>

 Yes, you have a good point on this. Assuming user schema can be done in
 the way you have specified,  still there can be a problem when it comes to
 group schema since it does not have an extension defined. However due to
 the 

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Ruwan Abeykoon]

Hi All,
Adjusting test *Dynamically* at runtime is wrong. Tests has to be exact and
precise.
What we should do is to add new tests specifically to target the change in
schema/schema extensions. Still each test should not be dynamic.
We will end up with many test cases, but each single test only check one
thing properly. We should be able to use common methods(not tests) to
reduce code duplication.

Cheers,
Ruwan

On Wed, Jul 5, 2017 at 1:25 AM, Omindu Rathnaweera  wrote:

> Since this is a 'compliance test suite' it makes sense to terminate the
> tests at the point where it deviates from the spec. However, it would be
> nice to have an option to run certain tests even if some of the said
> critical tests fail, especially the tests related to sp configs and schemas
> . The reason is that we had the same limitation when testing our product
> against the existing compliance test. The test failed since our SCIM
> implementation didn't support /ServiceProviderConfigs and /Schemas
> endpoints and from the compliance test perspective, this should be the
> correct behavior. But it prevented us from testing our other endpoints
> (/Users, /Groups). So I prefer if we can provide the option to test a
> specific endpoint at least with the default schema (or better give the
> option to upload a schema) so the user can get an idea on which parts of
> the implementation is spec compliant and which part isn't.
>
> As for the dynamic nature of the test suite, I guess we will need to
> adjust the test suite at runtime to a certain degree since we are testing
> the schema extensions. We cannot test the nature of those extended
> attributes unless we adjust the test suite according to schema.
>
> Regards,
> Omindu.
>
> On Mon, Jul 3, 2017 at 9:03 AM, Ruwan Abeykoon  wrote:
>
>> Hi All,
>> I would also think terminate the test is the correct move, when it is not
>> compliant
>>
>> We could add more test cases if we need to test adjusting the schema.
>> These are different test than 100% compliancy test.
>>
>> Apart from this, Each test case should be able to run irrespective of the
>> result of the other test. We need to adhere "Isolation" on test cases. [1]
>>
>> [1] http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html
>>
>> Cheers,
>> Ruwan
>>
>>
>> On Sun, Jul 2, 2017 at 7:52 AM, Vindula Jayawardana <
>> vindula...@cse.mrt.ac.lk> wrote:
>>
>>> Hi Johann,
>>>

> In SCIM 2.0 compliance test suite, there are inter dependencies
> between tests. For an example,
>
> We have identified /Schemas endpoint as a critical test which tests
> the schemas corresponding to user and group resources according to SCIM
> specification. However in a case where a SCIM service provider has
> customized the schemas according to their own requirements, this test will
> be failed. As the test suite uses the /Schemas endpoint to learn about the
> service providers schema definitions (consider the case where there is a
> user schema extension defined by the service provider), if the /Schemas
> endpoint fails, the test suite will be terminated immediately as the test
> suite cannot learn the configurations. However we can also make it not to
> terminate but to get adjusted to the service provider's configs after just
> failing the /Schemas endpoint test only. With that, the service provider
> will be able to run the remaining tests on the altered schemas without
> being blocked due to test dependency. But it should also be noted that,
> this approach can cause the test suite to not to adhere to the
> specification, but to adjust itself dynamically after a proper indication
> of the reason for the adjustment.
>
> Hence, as identified in the above example, there are two possible
> options in a test dependency situation.
>
> 1. Terminate
> 2. Adjust accordingly and continue the suite but fails only the parent
> test.
>
> What is the best way of handling this?. Any thoughts on this is highly
> appreciated.
>

 I would say just terminate. Why would you need to customize the core
 user schema when there is the possibility of defining an extended user
 schema. No one asked you to stick to any of the attributes coming from the
 core schema right?

>>>
>>> Yes, you have a good point on this. Assuming user schema can be done in
>>> the way you have specified,  still there can be a problem when it comes to
>>> group schema since it does not have an extension defined. However due to
>>> the less number of attributes associated with the group schema and their
>>> nature of it, I would say, it is very likely that service providers will
>>> not be altering the group schema most of the cases. Hence putting an effort
>>> on the second option could be redundant. So I agree with the termination
>>> based on your reasoning.
>>>


>
> Thank you
> *Vindula Jayawardana*

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Omindu Rathnaweera]

Since this is a 'compliance test suite' it makes sense to terminate the
tests at the point where it deviates from the spec. However, it would be
nice to have an option to run certain tests even if some of the said
critical tests fail, especially the tests related to sp configs and schemas
. The reason is that we had the same limitation when testing our product
against the existing compliance test. The test failed since our SCIM
implementation didn't support /ServiceProviderConfigs and /Schemas
endpoints and from the compliance test perspective, this should be the
correct behavior. But it prevented us from testing our other endpoints
(/Users, /Groups). So I prefer if we can provide the option to test a
specific endpoint at least with the default schema (or better give the
option to upload a schema) so the user can get an idea on which parts of
the implementation is spec compliant and which part isn't.

As for the dynamic nature of the test suite, I guess we will need to adjust
the test suite at runtime to a certain degree since we are testing the
schema extensions. We cannot test the nature of those extended attributes
unless we adjust the test suite according to schema.

Regards,
Omindu.

On Mon, Jul 3, 2017 at 9:03 AM, Ruwan Abeykoon  wrote:

> Hi All,
> I would also think terminate the test is the correct move, when it is not
> compliant
>
> We could add more test cases if we need to test adjusting the schema.
> These are different test than 100% compliancy test.
>
> Apart from this, Each test case should be able to run irrespective of the
> result of the other test. We need to adhere "Isolation" on test cases. [1]
>
> [1] http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html
>
> Cheers,
> Ruwan
>
>
> On Sun, Jul 2, 2017 at 7:52 AM, Vindula Jayawardana <
> vindula...@cse.mrt.ac.lk> wrote:
>
>> Hi Johann,
>>
>>>
 In SCIM 2.0 compliance test suite, there are inter dependencies between
 tests. For an example,

 We have identified /Schemas endpoint as a critical test which tests the
 schemas corresponding to user and group resources according to SCIM
 specification. However in a case where a SCIM service provider has
 customized the schemas according to their own requirements, this test will
 be failed. As the test suite uses the /Schemas endpoint to learn about the
 service providers schema definitions (consider the case where there is a
 user schema extension defined by the service provider), if the /Schemas
 endpoint fails, the test suite will be terminated immediately as the test
 suite cannot learn the configurations. However we can also make it not to
 terminate but to get adjusted to the service provider's configs after just
 failing the /Schemas endpoint test only. With that, the service provider
 will be able to run the remaining tests on the altered schemas without
 being blocked due to test dependency. But it should also be noted that,
 this approach can cause the test suite to not to adhere to the
 specification, but to adjust itself dynamically after a proper indication
 of the reason for the adjustment.

 Hence, as identified in the above example, there are two possible
 options in a test dependency situation.

 1. Terminate
 2. Adjust accordingly and continue the suite but fails only the parent
 test.

 What is the best way of handling this?. Any thoughts on this is highly
 appreciated.

>>>
>>> I would say just terminate. Why would you need to customize the core
>>> user schema when there is the possibility of defining an extended user
>>> schema. No one asked you to stick to any of the attributes coming from the
>>> core schema right?
>>>
>>
>> Yes, you have a good point on this. Assuming user schema can be done in
>> the way you have specified,  still there can be a problem when it comes to
>> group schema since it does not have an extension defined. However due to
>> the less number of attributes associated with the group schema and their
>> nature of it, I would say, it is very likely that service providers will
>> not be altering the group schema most of the cases. Hence putting an effort
>> on the second option could be redundant. So I agree with the termination
>> based on your reasoning.
>>
>>>
>>>

 Thank you
 *Vindula Jayawardana*
 Computer Science and Engineering Dept.
 University of Moratuwa
 mobile : +713462554
 Email : vindul...@gmail.com

 
 
 
 

 *“Respect is how to treat everyone, not just those you want to impress.
 "*


 *-Richard Branson-*



>>>
>>>
>>> --
>>> Thanks & Regards,
>>>
>>> *Johann Dilantha Nallathamby*
>>> Senior Technical Lead - WSO2 Identity 

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Ruwan Abeykoon]

Hi All,
I would also think terminate the test is the correct move, when it is not
compliant

We could add more test cases if we need to test adjusting the schema. These
are different test than 100% compliancy test.

Apart from this, Each test case should be able to run irrespective of the
result of the other test. We need to adhere "Isolation" on test cases. [1]

[1] http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html

Cheers,
Ruwan


On Sun, Jul 2, 2017 at 7:52 AM, Vindula Jayawardana <
vindula...@cse.mrt.ac.lk> wrote:

> Hi Johann,
>
>>
>>> In SCIM 2.0 compliance test suite, there are inter dependencies between
>>> tests. For an example,
>>>
>>> We have identified /Schemas endpoint as a critical test which tests the
>>> schemas corresponding to user and group resources according to SCIM
>>> specification. However in a case where a SCIM service provider has
>>> customized the schemas according to their own requirements, this test will
>>> be failed. As the test suite uses the /Schemas endpoint to learn about the
>>> service providers schema definitions (consider the case where there is a
>>> user schema extension defined by the service provider), if the /Schemas
>>> endpoint fails, the test suite will be terminated immediately as the test
>>> suite cannot learn the configurations. However we can also make it not to
>>> terminate but to get adjusted to the service provider's configs after just
>>> failing the /Schemas endpoint test only. With that, the service provider
>>> will be able to run the remaining tests on the altered schemas without
>>> being blocked due to test dependency. But it should also be noted that,
>>> this approach can cause the test suite to not to adhere to the
>>> specification, but to adjust itself dynamically after a proper indication
>>> of the reason for the adjustment.
>>>
>>> Hence, as identified in the above example, there are two possible
>>> options in a test dependency situation.
>>>
>>> 1. Terminate
>>> 2. Adjust accordingly and continue the suite but fails only the parent
>>> test.
>>>
>>> What is the best way of handling this?. Any thoughts on this is highly
>>> appreciated.
>>>
>>
>> I would say just terminate. Why would you need to customize the core user
>> schema when there is the possibility of defining an extended user schema.
>> No one asked you to stick to any of the attributes coming from the core
>> schema right?
>>
>
> Yes, you have a good point on this. Assuming user schema can be done in
> the way you have specified,  still there can be a problem when it comes to
> group schema since it does not have an extension defined. However due to
> the less number of attributes associated with the group schema and their
> nature of it, I would say, it is very likely that service providers will
> not be altering the group schema most of the cases. Hence putting an effort
> on the second option could be redundant. So I agree with the termination
> based on your reasoning.
>
>>
>>
>>>
>>> Thank you
>>> *Vindula Jayawardana*
>>> Computer Science and Engineering Dept.
>>> University of Moratuwa
>>> mobile : +713462554
>>> Email : vindul...@gmail.com
>>>
>>> 
>>> 
>>> 
>>> 
>>>
>>> *“Respect is how to treat everyone, not just those you want to impress.
>>> "*
>>>
>>>
>>> *-Richard Branson-*
>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>>
>> *Johann Dilantha Nallathamby*
>> Senior Technical Lead - WSO2 Identity Server
>> Governance Technologies Team
>> WSO2, Inc.
>> lean.enterprise.middleware
>>
>> Mobile - *+9476950*
>> Blog - *http://nallaa.wordpress.com *
>>
>
>


-- 

*Ruwan Abeykoon*
*Associate Director/Architect**,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Vindula Jayawardana]

Hi Johann,

>
>> In SCIM 2.0 compliance test suite, there are inter dependencies between
>> tests. For an example,
>>
>> We have identified /Schemas endpoint as a critical test which tests the
>> schemas corresponding to user and group resources according to SCIM
>> specification. However in a case where a SCIM service provider has
>> customized the schemas according to their own requirements, this test will
>> be failed. As the test suite uses the /Schemas endpoint to learn about the
>> service providers schema definitions (consider the case where there is a
>> user schema extension defined by the service provider), if the /Schemas
>> endpoint fails, the test suite will be terminated immediately as the test
>> suite cannot learn the configurations. However we can also make it not to
>> terminate but to get adjusted to the service provider's configs after just
>> failing the /Schemas endpoint test only. With that, the service provider
>> will be able to run the remaining tests on the altered schemas without
>> being blocked due to test dependency. But it should also be noted that,
>> this approach can cause the test suite to not to adhere to the
>> specification, but to adjust itself dynamically after a proper indication
>> of the reason for the adjustment.
>>
>> Hence, as identified in the above example, there are two possible options
>> in a test dependency situation.
>>
>> 1. Terminate
>> 2. Adjust accordingly and continue the suite but fails only the parent
>> test.
>>
>> What is the best way of handling this?. Any thoughts on this is highly
>> appreciated.
>>
>
> I would say just terminate. Why would you need to customize the core user
> schema when there is the possibility of defining an extended user schema.
> No one asked you to stick to any of the attributes coming from the core
> schema right?
>

Yes, you have a good point on this. Assuming user schema can be done in the
way you have specified,  still there can be a problem when it comes to
group schema since it does not have an extension defined. However due to
the less number of attributes associated with the group schema and their
nature of it, I would say, it is very likely that service providers will
not be altering the group schema most of the cases. Hence putting an effort
on the second option could be redundant. So I agree with the termination
based on your reasoning.

>
>
>>
>> Thank you
>> *Vindula Jayawardana*
>> Computer Science and Engineering Dept.
>> University of Moratuwa
>> mobile : +713462554
>> Email : vindul...@gmail.com
>>
>> 
>> 
>> 
>> 
>>
>> *“Respect is how to treat everyone, not just those you want to impress. "*
>>
>>
>> *-Richard Branson-*
>>
>>
>>
>
>
> --
> Thanks & Regards,
>
> *Johann Dilantha Nallathamby*
> Senior Technical Lead - WSO2 Identity Server
> Governance Technologies Team
> WSO2, Inc.
> lean.enterprise.middleware
>
> Mobile - *+9476950*
> Blog - *http://nallaa.wordpress.com *
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Vindula Jayawardana]

Hi Gayan,

>
>> In SCIM 2.0 compliance test suite, there are inter dependencies between
>> tests. For an example,
>>
>> We have identified /Schemas endpoint as a critical test which tests the
>> schemas corresponding to user and group resources according to SCIM
>> specification. However in a case where a SCIM service provider has
>> customized the schemas according to their own requirements, this test will
>> be failed. As the test suite uses the /Schemas endpoint to learn about the
>> service providers schema definitions (consider the case where there is a
>> user schema extension defined by the service provider), if the /Schemas
>> endpoint fails, the test suite will be terminated immediately as the test
>> suite cannot learn the configurations. However we can also make it not to
>> terminate but to get adjusted to the service provider's configs after just
>> failing the /Schemas endpoint test only. With that, the service provider
>> will be able to run the remaining tests on the altered schemas without
>> being blocked due to test dependency. But it should also be noted that,
>> this approach can cause the test suite to not to adhere to the
>> specification, but to adjust itself dynamically after a proper indication
>> of the reason for the adjustment.
>>
>> Hence, as identified in the above example, there are two possible options
>> in a test dependency situation.
>>
>> 1. Terminate
>> 2. Adjust accordingly and continue the suite but fails only the parent
>> test.
>>
>> What is the best way of handling this?. Any thoughts on this is highly
>> appreciated.
>>
> Thanks for bringing this question. SCIM 2.0 is an open standard for
> identity provisioning. Advantage of open standard is if two parties follow
> a common standard/specifications integration should be seamless. Idea of
> compliance test is to make sure given software product is adhere to
> particular specification hence I am +1 to terminate the test.
>

Agree with you.

What was the approach for SCIM 1.1 ?
>

 SCIM 1.1 terminates in such a situation as well.

>

>
>> Thank you
>> *Vindula Jayawardana*
>> Computer Science and Engineering Dept.
>> University of Moratuwa
>> mobile : +713462554
>> Email : vindul...@gmail.com
>>
>> 
>> 
>> 
>> 
>>
>> *“Respect is how to treat everyone, not just those you want to impress. "*
>>
>>
>> *-Richard Branson-*
>>
>>
>>
>
>
> --
> Gayan Gunawardana
> Senior Software Engineer; WSO2 Inc.; http://wso2.com/
> Email: ga...@wso2.com
> Mobile: +94 (71) 8020933
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Johann Nallathamby]

On Sat, Jul 1, 2017 at 7:24 PM, Vindula Jayawardana <
vindula...@cse.mrt.ac.lk> wrote:

> Hi,
>
> In SCIM 2.0 compliance test suite, there are inter dependencies between
> tests. For an example,
>
> We have identified /Schemas endpoint as a critical test which tests the
> schemas corresponding to user and group resources according to SCIM
> specification. However in a case where a SCIM service provider has
> customized the schemas according to their own requirements, this test will
> be failed. As the test suite uses the /Schemas endpoint to learn about the
> service providers schema definitions (consider the case where there is a
> user schema extension defined by the service provider), if the /Schemas
> endpoint fails, the test suite will be terminated immediately as the test
> suite cannot learn the configurations. However we can also make it not to
> terminate but to get adjusted to the service provider's configs after just
> failing the /Schemas endpoint test only. With that, the service provider
> will be able to run the remaining tests on the altered schemas without
> being blocked due to test dependency. But it should also be noted that,
> this approach can cause the test suite to not to adhere to the
> specification, but to adjust itself dynamically after a proper indication
> of the reason for the adjustment.
>
> Hence, as identified in the above example, there are two possible options
> in a test dependency situation.
>
> 1. Terminate
> 2. Adjust accordingly and continue the suite but fails only the parent
> test.
>
> What is the best way of handling this?. Any thoughts on this is highly
> appreciated.
>

I would say just terminate. Why would you need to customize the core user
schema when there is the possibility of defining an extended user schema.
No one asked you to stick to any of the attributes coming from the core
schema right?


>
> Thank you
> *Vindula Jayawardana*
> Computer Science and Engineering Dept.
> University of Moratuwa
> mobile : +713462554
> Email : vindul...@gmail.com
>
> 
> 
> 
> 
>
> *“Respect is how to treat everyone, not just those you want to impress. "*
>
>
> *-Richard Branson-*
>
>
>


-- 
Thanks & Regards,

*Johann Dilantha Nallathamby*
Senior Technical Lead - WSO2 Identity Server
Governance Technologies Team
WSO2, Inc.
lean.enterprise.middleware

Mobile - *+9476950*
Blog - *http://nallaa.wordpress.com *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Gayan Gunawardana]

On Sat, Jul 1, 2017 at 8:54 AM, Vindula Jayawardana <
vindula...@cse.mrt.ac.lk> wrote:

> Hi,
>
> In SCIM 2.0 compliance test suite, there are inter dependencies between
> tests. For an example,
>
> We have identified /Schemas endpoint as a critical test which tests the
> schemas corresponding to user and group resources according to SCIM
> specification. However in a case where a SCIM service provider has
> customized the schemas according to their own requirements, this test will
> be failed. As the test suite uses the /Schemas endpoint to learn about the
> service providers schema definitions (consider the case where there is a
> user schema extension defined by the service provider), if the /Schemas
> endpoint fails, the test suite will be terminated immediately as the test
> suite cannot learn the configurations. However we can also make it not to
> terminate but to get adjusted to the service provider's configs after just
> failing the /Schemas endpoint test only. With that, the service provider
> will be able to run the remaining tests on the altered schemas without
> being blocked due to test dependency. But it should also be noted that,
> this approach can cause the test suite to not to adhere to the
> specification, but to adjust itself dynamically after a proper indication
> of the reason for the adjustment.
>
> Hence, as identified in the above example, there are two possible options
> in a test dependency situation.
>
> 1. Terminate
> 2. Adjust accordingly and continue the suite but fails only the parent
> test.
>
> What is the best way of handling this?. Any thoughts on this is highly
> appreciated.
>
Thanks for bringing this question. SCIM 2.0 is an open standard for
identity provisioning. Advantage of open standard is if two parties follow
a common standard/specifications integration should be seamless. Idea of
compliance test is to make sure given software product is adhere to
particular specification hence I am +1 to terminate the test.

What was the approach for SCIM 1.1 ?

>
> Thank you
> *Vindula Jayawardana*
> Computer Science and Engineering Dept.
> University of Moratuwa
> mobile : +713462554
> Email : vindul...@gmail.com
>
> 
> 
> 
> 
>
> *“Respect is how to treat everyone, not just those you want to impress. "*
>
>
> *-Richard Branson-*
>
>
>


-- 
Gayan Gunawardana
Senior Software Engineer; WSO2 Inc.; http://wso2.com/
Email: ga...@wso2.com
Mobile: +94 (71) 8020933
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [GSoC][SCIM] SCIM 2.0 Test Dependencies

[Vindula Jayawardana]

Hi,

In SCIM 2.0 compliance test suite, there are inter dependencies between
tests. For an example,

We have identified /Schemas endpoint as a critical test which tests the
schemas corresponding to user and group resources according to SCIM
specification. However in a case where a SCIM service provider has
customized the schemas according to their own requirements, this test will
be failed. As the test suite uses the /Schemas endpoint to learn about the
service providers schema definitions (consider the case where there is a
user schema extension defined by the service provider), if the /Schemas
endpoint fails, the test suite will be terminated immediately as the test
suite cannot learn the configurations. However we can also make it not to
terminate but to get adjusted to the service provider's configs after just
failing the /Schemas endpoint test only. With that, the service provider
will be able to run the remaining tests on the altered schemas without
being blocked due to test dependency. But it should also be noted that,
this approach can cause the test suite to not to adhere to the
specification, but to adjust itself dynamically after a proper indication
of the reason for the adjustment.

Hence, as identified in the above example, there are two possible options
in a test dependency situation.

1. Terminate
2. Adjust accordingly and continue the suite but fails only the parent test.

What is the best way of handling this?. Any thoughts on this is highly
appreciated.

Thank you
*Vindula Jayawardana*
Computer Science and Engineering Dept.
University of Moratuwa
mobile : +713462554
Email : vindul...@gmail.com






*“Respect is how to treat everyone, not just those you want to impress. "*


*-Richard Branson-*
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev