Re: Caching / Ehcache

[Rohit Yadav]

Hi Marc-Aurele,


Thanks for starting a discussion thread. I think the GenericDaoBase uses 
ehcache to build an internal cache and the rate-limiting plugin uses ehcache, 
however I'm not sure to what extent they are used and the impact of removing 
them on a production management server. Let me evaluate this soon and get back 
on this thread (I'll be away next week on holiday).


Thank you for your contribution, it has been a pleasure working with you in the 
community. I wish you good luck in your next endeavor.


- Rohit






From: Marc-Aurèle Brothier 
Sent: Sunday, October 28, 2018 1:39:53 AM
To: dev@cloudstack.apache.org
Subject: Caching / Ehcache

Hi everyone,

(Again as the email formatting has been removed and was hard to read -
I hope it will be better this time).

While trying to lower the DB load for CloudStack I did some long
testing and here are my outcomes for the current cache mechanism in
CloudStack.

I would be interested to hear from people who try to customize the
ehcache configuration in CS. A PR (
https://github.com/apache/cloudstack/pull/2913) is also open to
desactivate (before deleting) ehcache in CS, read below to understand
why.


# Problems

The code in CS does not seem to fit any caching mechanism especially
due to the homemade DAO code. The main 3 flaws are the following:

## Entities are not expected to be shared

There is quite a lot of code with method calls passing entity IDs value
as long, which does some object fetching. Without caching, this
behavior will create distinct objects each time an entity with the same
ID is fetched. With the cache enabled, the same object will be shared
among those methods. It has been seen that it does generate some side
effects where code still expected unchanged entity attributes after
calling different methods thus generating exception/bugs.

## DAO update operations are using search queries

Some part of the code are updating entities based on a search query,
therefore the whole cache must be invalidated (see GenericDaoBase:
public int update(UpdateBuilder ub, final SearchCriteria sc, Integer
rows);).

## Entities based on views joining multiple tables

There are quite a lot of entities based on SQL views joining multiple
entities in a same object. Enabling caching on those would require a
mechanism to link and cross-remove related objects whenever one of the
sub-entity is changed.


# Final word

Based on the previously discussed points, the best approach IMHO would
be to move out of the custom DAO framework in CS and use a well known
one. It will handle caching well and the joins made by the views in the
code. It's not an easy change, but it will fix along a lot of issues
and add a proven / robust framework to an important part of the code.

The work to change the DAO layer is a huge task, I don't know how / who
will perform it.

What are the proposals for a new DAO framework ?


FYI I will stop working for Exoscale at the end of the month, so I
won't be able to tackle such challenge as I won't be working with CS
anymore. I'll try my best to continue looking at the project to give my
insights and share the experienced I have with CS.


Marc-Aurèle


rohit.ya...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 

[GitHub] falcon78921 commented on issue #17: CLOUDSTACK-10399: Added section regarding MySQL server-id

[GitBox]

falcon78921 commented on issue #17: CLOUDSTACK-10399: Added section regarding 
MySQL server-id
URL: 
https://github.com/apache/cloudstack-documentation/pull/17#issuecomment-435152140
 
 
   @rafaelweingartner Thanks Rafael! :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] falcon78921 opened a new pull request #17: CLOUDSTACK-10399: Added section regarding MySQL server-id

[GitBox]

falcon78921 opened a new pull request #17: CLOUDSTACK-10399: Added section 
regarding MySQL server-id
URL: https://github.com/apache/cloudstack-documentation/pull/17
 
 
   Fixes CLOUDSTACK-10399: Specify MySQL server-id for later versions of 
CloudStack
   
   Signed-off-by: James McClune 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wido commented on issue #16: Change systemvm template from 4.6 to 4.11

[GitBox]

wido commented on issue #16: Change systemvm template from 4.6 to 4.11
URL: 
https://github.com/apache/cloudstack-documentation/pull/16#issuecomment-435129113
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] GabrielBrascher opened a new pull request #16: Change systemvm template from 4.6 to 4.11

[GitBox]

GabrielBrascher opened a new pull request #16: Change systemvm template from 
4.6 to 4.11
URL: https://github.com/apache/cloudstack-documentation/pull/16
 
 
   Fix System VM template download path from 4.6 to 4.11.1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] DagSonsteboSB commented on issue #14: Documentation for KVM live storage migration

[GitBox]

DagSonsteboSB commented on issue #14: Documentation for KVM live storage 
migration
URL: 
https://github.com/apache/cloudstack-documentation/pull/14#issuecomment-435066148
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] houthuis opened a new pull request #15: documentation for selecting a pod, cluster or host during VM deploy

[GitBox]

houthuis opened a new pull request #15: documentation for selecting a pod, 
cluster or host during VM deploy
URL: https://github.com/apache/cloudstack-documentation/pull/15
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: API rate limiting

[Ivan Kudryavtsev]

Marc-Aurele, we are using rate-limiting. However the feature can be
implemented in Nginx, so it's not critical.

чт, 1 нояб. 2018 г., 6:53 Marc-Aurèle Brothier :

> Hi,
>
> Are people using the API rate limiting feature ?
>
> I'm working on removing the ehcache and was wondering if I need to find an
> alternate solution for that plugin or if it could be removed too.
>
> Marc-Aurèle
>

API rate limiting

[Marc-Aurèle Brothier]

Hi,

Are people using the API rate limiting feature ?

I'm working on removing the ehcache and was wondering if I need to find an 
alternate solution for that plugin or if it could be removed too.

Marc-Aurèle


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Caching / Ehcache

[Marc-Aurèle Brothier]

I wasn't in Montreal so I wasn't aware of the willing to change the JPA layer. 
That's definitively a good thing.

> On 30 Oct 2018, at 17:05, Rafael Weingärtner  
> wrote:
> 
> Hello Marc,
> 
> It is said to hear that you might not be active as you used to be. However,
> I do wish you success in your new path.
> 
> Now, regarding the ehCache. +1 to remove it. Were you in Montreal? This
> year in Montreal, one of the things we discussed is to move away from this
> ad-hoc JPA solution that we have to a standardized and proved one (e.g.
> Spring data + some commonly used JAP implementation such as Eclipse link,
> OpenJPA, and others). AS a first step towards that, we could disable and
> remove the use of ehCache.
> 
> 
> 
> P.S. sorry the late reply, but I have been busy with others topics not
> related to ACS.
> 
> On Sun, Oct 28, 2018 at 2:17 PM Daan Hoogland 
> wrote:
> 
>> Sorry to hear you are reducing your activity on CloudStack, Marc-Aurele.
>> Hope you fare well.
>> 
>> On Sat, Oct 27, 2018 at 10:10 PM Marc-Aurèle Brothier 
>> wrote:
>> 
>>> Hi everyone,
>>> 
>>> (Again as the email formatting has been removed and was hard to read -
>>> I hope it will be better this time).
>>> 
>>> While trying to lower the DB load for CloudStack I did some long
>>> testing and here are my outcomes for the current cache mechanism in
>>> CloudStack.
>>> 
>>> I would be interested to hear from people who try to customize the
>>> ehcache configuration in CS. A PR (
>>> https://github.com/apache/cloudstack/pull/2913) is also open to
>>> desactivate (before deleting) ehcache in CS, read below to understand
>>> why.
>>> 
>>> 
>>> # Problems
>>> 
>>> The code in CS does not seem to fit any caching mechanism especially
>>> due to the homemade DAO code. The main 3 flaws are the following:
>>> 
>>> ## Entities are not expected to be shared
>>> 
>>> There is quite a lot of code with method calls passing entity IDs value
>>> as long, which does some object fetching. Without caching, this
>>> behavior will create distinct objects each time an entity with the same
>>> ID is fetched. With the cache enabled, the same object will be shared
>>> among those methods. It has been seen that it does generate some side
>>> effects where code still expected unchanged entity attributes after
>>> calling different methods thus generating exception/bugs.
>>> 
>>> ## DAO update operations are using search queries
>>> 
>>> Some part of the code are updating entities based on a search query,
>>> therefore the whole cache must be invalidated (see GenericDaoBase:
>>> public int update(UpdateBuilder ub, final SearchCriteria sc, Integer
>>> rows);).
>>> 
>>> ## Entities based on views joining multiple tables
>>> 
>>> There are quite a lot of entities based on SQL views joining multiple
>>> entities in a same object. Enabling caching on those would require a
>>> mechanism to link and cross-remove related objects whenever one of the
>>> sub-entity is changed.
>>> 
>>> 
>>> # Final word
>>> 
>>> Based on the previously discussed points, the best approach IMHO would
>>> be to move out of the custom DAO framework in CS and use a well known
>>> one. It will handle caching well and the joins made by the views in the
>>> code. It's not an easy change, but it will fix along a lot of issues
>>> and add a proven / robust framework to an important part of the code.
>>> 
>>> The work to change the DAO layer is a huge task, I don't know how / who
>>> will perform it.
>>> 
>>> What are the proposals for a new DAO framework ?
>>> 
>>> 
>>> FYI I will stop working for Exoscale at the end of the month, so I
>>> won't be able to tackle such challenge as I won't be working with CS
>>> anymore. I'll try my best to continue looking at the project to give my
>>> insights and share the experienced I have with CS.
>>> 
>>> 
>>> Marc-Aurèle
>>> 
>>> 
>> 
>> --
>> Daan
>> 
> 
> 
> --
> Rafael Weingärtner



signature.asc
Description: Message signed with OpenPGP using GPGMail

CFP: Cloudstack EUG Dec 13th

[Giles Sirett]

Hi all
The next Cloudstack European User group will be held in London on December 13th

https://www.eventbrite.co.uk/e/cloudstack-european-user-group-meetup-tickets-50608731077?aff=ebdssbdestsearch

Does anybody from this community want to come and present a talk? - could be on 
use-case, ongoing projects, basically anything cloudstack related

Please ping me if you're interested


Kind regards
Giles


giles.sir...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue