IMO we should not be using caches for searches. It only makes sense to maybe cache the "all APIs" set. Even that should only be a local cache with a reasonably small time window (1 minute). The purpose of that cache also is to prevent high hits rates on the DB in abnormal situations.
My gut feeling is that we'll end up with lots of complicated SQLs if we try to filter the data set solely at the DB layer. This is because things like picking the latest version, doing permissions checks, etc might end up as complex queries. So I think we should go for a hybrid solution where we filter "as much as possible" from the DB layer (SELECT WHERE API.STATUS = 'PUBLISHED') and then process the complicated ones in memory. I don't think we'll ever come across a situation where a single tenant will have more than 2000 APIs. So loading that to memory and processing shouldn't be much of an issue. Thanks, NuwanD. On Wed, Nov 16, 2016 at 9:14 PM, Lahiru Cooray <[email protected]> wrote: > +1 for Jo's opinion on using RBMS based pagination. > > I think we are trying here to make the code unnecessarily complex by > introducing caches. As discussed this may become more complex when we do > sorting/searching operations. > Also regarding the performance, I don't see a significant performance gain > by introducing our own cache as RDBMS's normally maintain query caches. > > Only benefit I see here is avoiding the vendor specific queries. > How about addressing this concern properly by introducing a factory method > and giving the flexibility to plug a new vendor easily when needed. > > On Wed, Nov 16, 2016 at 3:23 PM, Uvindra Dias Jayasinha <[email protected]> > wrote: > >> Hi Jo, >> >> On 16 November 2016 at 14:47, Joseph Fonseka <[email protected]> wrote: >> >>> Hi Uvindra >>> >>> On Wed, Nov 16, 2016 at 1:48 PM, Uvindra Dias Jayasinha < >>> [email protected]> wrote: >>>> >>>> >>>> Searching and sorting will be done at DB level via SQL. It is only the >>>> pagination of the returned result set that will be done at Java level. >>>> >>> >>> So does this mean for different search request you will be caching >>> separate sets of data ? If that is so will it exhaust the memory and how >>> will API search perform under a load. And since the servers will be running >>> on containers available memory would be even less. >>> >> >> Thats a good point, definitely should not be caching different search >> request results.. Will discuss this with the team and get back to you. >> >>> >>> Why would this be more efficient than Java? We achieve a similar thing >>>> by using a local cache so there is no need to have an intermediate in >>>> memory DB. What was proposed makes sense when you consider that we a >>>> dealing with only a very small amount of data at DB level now. >>>> >>> >>> Considering pagination is done using DBMS, in-memory db will help to >>> work only with a one data set as well as needing to maintain one set of >>> pagination queries. >>> >> >> Hmmm I still dont understand the difference between in memory DB and in >> memory cache. Technically in both cases we are working with an in memory >> dataset. >> >>> >>> >>>> In the future if we foresee a requirement for DB level pagination we >>>> can implement that within the DAO layer, but for now this is more than >>>> enough to meet our needs. This way we can avoid introducing unwanted >>>> complexity upfront since there is no rationale for doing so. >>>> >>> >>> Agreed, but if the complexity is to support pagination for all the DBMSs >>> we can at-least focus on adding db level pagination for commonly used DBMSs >>> (ie MySQL, Oracle). >>> >> >> Its not a complexity to support for multiple DB's, we can do that but its >> an additional effort we can avoid by going down this path. >> >>> >>> Regards >>> Jo >>> >>> >>> >>> >>> >>> >>>> >>>> >>>>> Thanks & Regards >>>>> Jo >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>>>> With regards, >>>>>>> *Manu*ranga Perera. >>>>>>> >>>>>>> phone : 071 7 70 20 50 >>>>>>> mail : [email protected] >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Uvindra >>>>>> >>>>>> Mobile: 777733962 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> -- >>>>> *Joseph Fonseka* >>>>> WSO2 Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 772 512 430 >>>>> skype: jpfonseka >>>>> >>>>> * <http://lk.linkedin.com/in/rumeshbandara>* >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Uvindra >>>> >>>> Mobile: 777733962 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> -- >>> *Joseph Fonseka* >>> WSO2 Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 772 512 430 >>> skype: jpfonseka >>> >>> * <http://lk.linkedin.com/in/rumeshbandara>* >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Regards, >> Uvindra >> >> Mobile: 777733962 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Lahiru Cooray* > Software Engineer > WSO2, Inc.;http://wso2.com/ > lean.enterprise.middleware > > Mobile: +94 715 654154 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Nuwan Dias Software Architect - WSO2, Inc. http://wso2.com email : [email protected] Phone : +94 777 775 729
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
