Re: [Dev] REST-API: Fine Grained resources vs Coarse Grained resources

[Thamali Wijewardhana]

Thank you very much Frank. I am preparing a report including the changes
that could be done to the ML API according to the things we discussed. Once
I finished it, I will send you to be reviewed.

Thanks
Thamali

On Thu, Mar 31, 2016 at 5:12 PM, Frank Leymann  wrote:

> Yes, Thamali!
>
> Please let me know if I can be of any help. For example, we may schedule a
> call to discuss some of the APIs whenever you think that's appropriate :-)
>
>
> Best regards,
> Frank
>
> 2016-03-31 7:10 GMT+02:00 Thamali Wijewardhana :
>
>> Thank you very much  Frank for your comments.
>> I am now clear that deciding whether to use fine grained resources or
>> coarse grained resource does not depend on the size of the object. We have
>> to consider what objects change and how often they are changed when
>> defining resources. And also the number of requests does not depend on the
>> number of resources defined.  I am trying to use this knowledge and the
>> above three options.
>>
>> Best regards
>> Thamali
>>
>> On Wed, Mar 30, 2016 at 11:35 PM, Frank Leymann  wrote:
>>
>>> Hi all,
>>>
>>> first of all, I fully support Jo's comment on the importance of the
>>> intuitiveness of an API.
>>>
>>> Next, I have a few questions and comments.
>>>
>>> @Thamali:
>>>
>>>1. If you need to update more than one "fine grained resources"
>>>atomically in order to maintain consistency, you can use a Controller
>>>Resource (see section 4.4 of the Guideline document). You pass the set of
>>>fine grained resources in a single invocation of the Controller and the
>>>Controller ensures atomicity (typically by using transactions 
>>> internally).
>>>2. To reduce data exchange, REST recommends caching - for example at
>>>the client side. A resource always comes with "validators" like
>>>Last-Modified header or the ETag header. Clients store these validators 
>>> and
>>>when requesting the resource again, the request is a conditional GET
>>>passing the validators with the request to the server; a server won't
>>>return the resource if the validators did not change (see section 10.4).
>>>This will reduce data transfer but admittedly not (!) the number of
>>>requests to servers.
>>>   - If you don't expect the number of requests be orders of
>>>   magnitudes higher based on fine grained resources, I would rely on 
>>> load
>>>   balancing capabilities - unless practice proves that we run into 
>>> problems...
>>>3. What does actually change and how often?
>>>   - A client requesting a config doesn't want to retrieve the
>>>   encompassing analysis resource in case only the config changed. It add
>>>   parsing burden to the client etc.
>>>   - In your question before you wanted to avoid frequent config
>>>   retrieval, but how does including configs in the analysis resource 
>>> reduce
>>>   the numbers of requests at all? Clients now will retrieve large 
>>> analysis
>>>   results as often as they will retrieve small configs - sounds worse.
>>>   - If configs don't change often, but analysis resources do,
>>>   client's wont be happy to retrieve valid configs with each modified 
>>> analysis
>>>  - Jo's option (1) gives a solution for that problem in case
>>>  coarse-grained resources are used.
>>>   4. When designing resources, considering retrieval only is wrong.
>>>We must understand, e.g. the creation of resources. Is a config always
>>>created together with an analysis resource and vice versa? Or are 
>>> analysis
>>>resources and configs perceived as independent (but related) entities? 
>>> The
>>>latter case indicates that you have to have two separate resources, the
>>>former indicates that you should (not must!) have a single resource.
>>>   - Independent resources that are often manipulate as a whole are
>>>   covered by Composite Resources (section 4.3). I.e. APIs that allow you
>>>   manipulated groups of related resources as a whole when needed, but 
>>> still
>>>   to manipulate the individual resources because the regular GET/PUT/...
>>>   methods on these fine grained resources are available too.
>>>  - This is a generalization of Jo's option (1).
>>>   5. Retrieving pieces of a resource should be done by supporting
>>>filters (as you say yourself). E.g. retrieving the name of an algorithm 
>>> is
>>>done via the API for retrieving the algorithm an adding a corresponding
>>>filter condition as query string (see the notion of "projection" in 
>>> section
>>>10.2.).
>>>   - This is Jo's option (2).
>>>
>>> Please get back with further questions if this didn't help - I am happy
>>> to continue the discussion :-)
>>>
>>>
>>> Best regards,
>>> Frank
>>>
>>> 2016-03-29 12:47 GMT+02:00 Joseph Fonseka :
>>>
 Hi

 Having different ways to retrieve the configs of the resource 

Re: [Dev] REST-API: Fine Grained resources vs Coarse Grained resources

[Thamali Wijewardhana]

Thank you very much  Frank for your comments.
I am now clear that deciding whether to use fine grained resources or
coarse grained resource does not depend on the size of the object. We have
to consider what objects change and how often they are changed when
defining resources. And also the number of requests does not depend on the
number of resources defined.  I am trying to use this knowledge and the
above three options.

Best regards
Thamali

On Wed, Mar 30, 2016 at 11:35 PM, Frank Leymann  wrote:

> Hi all,
>
> first of all, I fully support Jo's comment on the importance of the
> intuitiveness of an API.
>
> Next, I have a few questions and comments.
>
> @Thamali:
>
>1. If you need to update more than one "fine grained resources"
>atomically in order to maintain consistency, you can use a Controller
>Resource (see section 4.4 of the Guideline document). You pass the set of
>fine grained resources in a single invocation of the Controller and the
>Controller ensures atomicity (typically by using transactions internally).
>2. To reduce data exchange, REST recommends caching - for example at
>the client side. A resource always comes with "validators" like
>Last-Modified header or the ETag header. Clients store these validators and
>when requesting the resource again, the request is a conditional GET
>passing the validators with the request to the server; a server won't
>return the resource if the validators did not change (see section 10.4).
>This will reduce data transfer but admittedly not (!) the number of
>requests to servers.
>   - If you don't expect the number of requests be orders of
>   magnitudes higher based on fine grained resources, I would rely on load
>   balancing capabilities - unless practice proves that we run into 
> problems...
>3. What does actually change and how often?
>   - A client requesting a config doesn't want to retrieve the
>   encompassing analysis resource in case only the config changed. It add
>   parsing burden to the client etc.
>   - In your question before you wanted to avoid frequent config
>   retrieval, but how does including configs in the analysis resource 
> reduce
>   the numbers of requests at all? Clients now will retrieve large analysis
>   results as often as they will retrieve small configs - sounds worse.
>   - If configs don't change often, but analysis resources do,
>   client's wont be happy to retrieve valid configs with each modified 
> analysis
>  - Jo's option (1) gives a solution for that problem in case
>  coarse-grained resources are used.
>   4. When designing resources, considering retrieval only is wrong.
>We must understand, e.g. the creation of resources. Is a config always
>created together with an analysis resource and vice versa? Or are analysis
>resources and configs perceived as independent (but related) entities? The
>latter case indicates that you have to have two separate resources, the
>former indicates that you should (not must!) have a single resource.
>   - Independent resources that are often manipulate as a whole are
>   covered by Composite Resources (section 4.3). I.e. APIs that allow you
>   manipulated groups of related resources as a whole when needed, but 
> still
>   to manipulate the individual resources because the regular GET/PUT/...
>   methods on these fine grained resources are available too.
>  - This is a generalization of Jo's option (1).
>   5. Retrieving pieces of a resource should be done by supporting
>filters (as you say yourself). E.g. retrieving the name of an algorithm is
>done via the API for retrieving the algorithm an adding a corresponding
>filter condition as query string (see the notion of "projection" in section
>10.2.).
>   - This is Jo's option (2).
>
> Please get back with further questions if this didn't help - I am happy to
> continue the discussion :-)
>
>
> Best regards,
> Frank
>
> 2016-03-29 12:47 GMT+02:00 Joseph Fonseka :
>
>> Hi
>>
>> Having different ways to retrieve the configs of the resource will affect
>> the intuitiveness of the API so IMO having a uniform way of retrieving the
>> configs will be more beneficial.
>>
>> Alternate solutions for the given problem.
>> 1. You always return the configs as part of the specific resource. Ex. /
>> analysis/{analysis_id} and give the option of turning off configs as a
>> query parameter Ex /analysis/{analysis_id}?configs=false.
>> 2. Further more you can filter configs with /analysis/{
>> analysis_id}?configs=attribute.
>> 3. If you want to do a partial update of configs you can use sub resource
>> or a controller resource. Ex. /analysis/{analysis_id}/configs,  /
>> analysis/update-config
>>
>> Regards
>> Jo
>>
>> On Tue, Mar 29, 2016 at 3:31 PM, Udara Liyanage  wrote:
>>
>>> Hi,
>>>
>>> When 

Re: [Dev] REST-API: Fine Grained resources vs Coarse Grained resources

[Thamali Wijewardhana]

Thanks all for the quick response and I will continue with those comments.

On Tue, Mar 29, 2016 at 4:17 PM, Joseph Fonseka  wrote:

> Hi
>
> Having different ways to retrieve the configs of the resource will affect
> the intuitiveness of the API so IMO having a uniform way of retrieving the
> configs will be more beneficial.
>
> Alternate solutions for the given problem.
> 1. You always return the configs as part of the specific resource. Ex. /
> analysis/{analysis_id} and give the option of turning off configs as a
> query parameter Ex /analysis/{analysis_id}?configs=false.
> 2. Further more you can filter configs with /analysis/{
> analysis_id}?configs=attribute.
> 3. If you want to do a partial update of configs you can use sub resource
> or a controller resource. Ex. /analysis/{analysis_id}/configs,  /
> analysis/update-config
>
> Regards
> Jo
>
> On Tue, Mar 29, 2016 at 3:31 PM, Udara Liyanage  wrote:
>
>> Hi,
>>
>> When designing API, we need to let the clients know the API endpoints so
>> they can invoke relevant  endpoint to fetch expected resource. Assuming
>> size of the configuration/analytic object payload is a dynamic thing, how
>> can the client know which endpoints to invoke to fetch the required
>> resource. How does the client know the size of the analytic payload. In
>> this case how does a client know which API endpoint to invoke if he want to
>> get configs. It may be GET /analysis if analysis size is low, otherwise GET
>> /analysis/configs .
>>
>> Yes, we can not go extreme in fine grain or coarse grain. IMO best
>> approach is to go in middle path, define a reasonable set of endpoints to
>> fetch resources.
>>
>> On Tue, Mar 29, 2016 at 3:08 PM, Thamali Wijewardhana 
>> wrote:
>>
>>> Hi,
>>>
>>> When creating a REST API to WSO2 machine learner, one of the important
>>> problems I faced was selection among fine grained resources and coarse
>>> grained resources. In other words, whether to define something as a
>>> separate resource or a part of a large resource.
>>>
>>> Fine grained Resources are low complex and easy to maintain. But it can
>>> make data become an inconsistent state and the server will end up receiving
>>> higher number of HTTP requests possibly impacting its ability to serve
>>> multiple API consumers.
>>>
>>> In using coarse grained resources, the data inconsistency and higher
>>> load on the server is reduced. But it may be difficult to maintain and
>>> higher JSON payload may be returned.
>>>
>>> For example, we have an API GET api/analysis/analysis_id/configs which
>>> retrieves configurations of an API. The problem is whether to use a
>>> separate resource for configs or return configs with the analysis resource.
>>> If we consider configuration as a separate resource, we have to define an
>>> API, GET api/analysis/analysis_id/configs. But if we return configs with
>>> analysis resource, then it may be only the API GET
>>> api/analysis/analysis_id  and the configuration should be added to analysis
>>> resource and returned with it.
>>>
>>> I have found an approach to solve the problem and given below is what I
>>> have understood.
>>>
>>> The decision should be taken considering the situation.
>>>
>>> Here, the decision is based on the size of the configuration object. If
>>> it has a large size, then if we return it with analysis resource, it may be
>>> a large JSON payload and time wastage because every time an analysis
>>> resource is returned configuration also have to be returned even not
>>> necessary. Therefore, if configuration is large, it is better to use a
>>> separate resource for configurations and use a separate API as
>>> api/analysis/analysis_id/configs
>>>
>>> But, when we have to access a simple property such as algorithm-name of
>>> the analysis, then it is better to return it with algorithm resource.
>>>
>>> This is the approach I have decided and highly appreciate your
>>> suggestions on this.
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> ___
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
>
> --
> *Joseph Fonseka*
> WSO2 Inc.; http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 772 512 430
> skype: jpfonseka
>
> * *
>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] REST-API: Fine Grained resources vs Coarse Grained resources

[Joseph Fonseka]

Hi

Having different ways to retrieve the configs of the resource will affect
the intuitiveness of the API so IMO having a uniform way of retrieving the
configs will be more beneficial.

Alternate solutions for the given problem.
1. You always return the configs as part of the specific resource. Ex. /
analysis/{analysis_id} and give the option of turning off configs as a
query parameter Ex /analysis/{analysis_id}?configs=false.
2. Further more you can filter configs with /analysis/{
analysis_id}?configs=attribute.
3. If you want to do a partial update of configs you can use sub resource
or a controller resource. Ex. /analysis/{analysis_id}/configs,  /
analysis/update-config

Regards
Jo

On Tue, Mar 29, 2016 at 3:31 PM, Udara Liyanage  wrote:

> Hi,
>
> When designing API, we need to let the clients know the API endpoints so
> they can invoke relevant  endpoint to fetch expected resource. Assuming
> size of the configuration/analytic object payload is a dynamic thing, how
> can the client know which endpoints to invoke to fetch the required
> resource. How does the client know the size of the analytic payload. In
> this case how does a client know which API endpoint to invoke if he want to
> get configs. It may be GET /analysis if analysis size is low, otherwise GET
> /analysis/configs .
>
> Yes, we can not go extreme in fine grain or coarse grain. IMO best
> approach is to go in middle path, define a reasonable set of endpoints to
> fetch resources.
>
> On Tue, Mar 29, 2016 at 3:08 PM, Thamali Wijewardhana 
> wrote:
>
>> Hi,
>>
>> When creating a REST API to WSO2 machine learner, one of the important
>> problems I faced was selection among fine grained resources and coarse
>> grained resources. In other words, whether to define something as a
>> separate resource or a part of a large resource.
>>
>> Fine grained Resources are low complex and easy to maintain. But it can
>> make data become an inconsistent state and the server will end up receiving
>> higher number of HTTP requests possibly impacting its ability to serve
>> multiple API consumers.
>>
>> In using coarse grained resources, the data inconsistency and higher load
>> on the server is reduced. But it may be difficult to maintain and higher
>> JSON payload may be returned.
>>
>> For example, we have an API GET api/analysis/analysis_id/configs which
>> retrieves configurations of an API. The problem is whether to use a
>> separate resource for configs or return configs with the analysis resource.
>> If we consider configuration as a separate resource, we have to define an
>> API, GET api/analysis/analysis_id/configs. But if we return configs with
>> analysis resource, then it may be only the API GET
>> api/analysis/analysis_id  and the configuration should be added to analysis
>> resource and returned with it.
>>
>> I have found an approach to solve the problem and given below is what I
>> have understood.
>>
>> The decision should be taken considering the situation.
>>
>> Here, the decision is based on the size of the configuration object. If
>> it has a large size, then if we return it with analysis resource, it may be
>> a large JSON payload and time wastage because every time an analysis
>> resource is returned configuration also have to be returned even not
>> necessary. Therefore, if configuration is large, it is better to use a
>> separate resource for configurations and use a separate API as
>> api/analysis/analysis_id/configs
>>
>> But, when we have to access a simple property such as algorithm-name of
>> the analysis, then it is better to return it with algorithm resource.
>>
>> This is the approach I have decided and highly appreciate your
>> suggestions on this.
>>
>>
>> Thanks
>>
>>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 

-- 
*Joseph Fonseka*
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

mobile: +94 772 512 430
skype: jpfonseka

* *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] REST-API: Fine Grained resources vs Coarse Grained resources

[Udara Liyanage]

Hi,

When designing API, we need to let the clients know the API endpoints so
they can invoke relevant  endpoint to fetch expected resource. Assuming
size of the configuration/analytic object payload is a dynamic thing, how
can the client know which endpoints to invoke to fetch the required
resource. How does the client know the size of the analytic payload. In
this case how does a client know which API endpoint to invoke if he want to
get configs. It may be GET /analysis if analysis size is low, otherwise GET
/analysis/configs .

Yes, we can not go extreme in fine grain or coarse grain. IMO best approach
is to go in middle path, define a reasonable set of endpoints to fetch
resources.

On Tue, Mar 29, 2016 at 3:08 PM, Thamali Wijewardhana 
wrote:

> Hi,
>
> When creating a REST API to WSO2 machine learner, one of the important
> problems I faced was selection among fine grained resources and coarse
> grained resources. In other words, whether to define something as a
> separate resource or a part of a large resource.
>
> Fine grained Resources are low complex and easy to maintain. But it can
> make data become an inconsistent state and the server will end up receiving
> higher number of HTTP requests possibly impacting its ability to serve
> multiple API consumers.
>
> In using coarse grained resources, the data inconsistency and higher load
> on the server is reduced. But it may be difficult to maintain and higher
> JSON payload may be returned.
>
> For example, we have an API GET api/analysis/analysis_id/configs which
> retrieves configurations of an API. The problem is whether to use a
> separate resource for configs or return configs with the analysis resource.
> If we consider configuration as a separate resource, we have to define an
> API, GET api/analysis/analysis_id/configs. But if we return configs with
> analysis resource, then it may be only the API GET
> api/analysis/analysis_id  and the configuration should be added to analysis
> resource and returned with it.
>
> I have found an approach to solve the problem and given below is what I
> have understood.
>
> The decision should be taken considering the situation.
>
> Here, the decision is based on the size of the configuration object. If it
> has a large size, then if we return it with analysis resource, it may be a
> large JSON payload and time wastage because every time an analysis resource
> is returned configuration also have to be returned even not necessary.
> Therefore, if configuration is large, it is better to use a separate
> resource for configurations and use a separate API as
> api/analysis/analysis_id/configs
>
> But, when we have to access a simple property such as algorithm-name of
> the analysis, then it is better to return it with algorithm resource.
>
> This is the approach I have decided and highly appreciate your suggestions
> on this.
>
>
> Thanks
>
>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev