@Rajith As per the offline discussion we had regarding the performance evaluation (using JMETER) of the two methods: - use 2 separate thread groups for evaluating the performance of 2 DB based search methods - run each thread group sequentially - run the test for a longer period under different concurrency level and record the latency and TPS
With regard to the long latencies you are noticing, we need to figure out if this is related to the database query/queries or something else. To do a quick test: simply log the execution time of the query/queries. If the execution times are high (or have spikes) then we can try to optimize the DB query. Otherwise we have to do some further investigations. Thanks Malith On Sat, Jan 14, 2017 at 10:43 PM, Nuwan Dias <[email protected]> wrote: > File system based indexers bring new challenges in the container world. It > also poses challenges in HA (multinode) and multi regional deployments. > Therefore if we're selecting a solr indexing approach, the only practical > solution is to go with a solr server based architecture. > > Even with a solr server assisted architecture, we still have complexities > when dealing with multi regional deployments. Also since API artifacts > reside in the database, if we're loading search results from an index, we > have to sync the permissions of the artifacts in the index too. Plus this > makes the deployment complex. > > Given the complexities that have to be addressed in an indexing solution, > I'd prefer to opt with the DB based search at least to start off with. > Since APIs and their permissions are both maintained in the DB, it would be > much simple if the search also works on that. Unlike in C4 this database > will only have data of 1 tenant. Hence we're not expecting the API count to > be in the thousands. Therefore I think searching through the database > should be feasible. > > Thanks, > NuwanD. > > On Sat, 14 Jan 2017 at 8:27 am, Chandana Napagoda <[email protected]> > wrote: > >> Hi Rajith, >> >> I believe indexing based approach is more suitable for any search >> solution because its design is to support fast search results. If you use >> database search, you have to use multiple DB indexes to improve search >> performance, and that will reduce the performance of insert operations. >> >> I think, REG_LOG kind of history table is not necessary for APIM product, >> since it is totally aware of the APIM artifacts(APIs, Docs, forum posts) >> and you can directly connect to the DB[1] from the text search engine and >> index the resources. As per the Solr documentation, it is capable of >> importing only the new additions(delta) for indexing. >> >> [1]. https://cwiki.apache.org/confluence/display/solr/ >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler >> >> Regards, >> Chandana >> >> >> On Fri, Jan 13, 2017 at 11:44 AM, Rajith Roshan <[email protected]> wrote: >> >> Hi all, >> >> We are currently evaluating how to perform full text search at the >> database level for C5 based API Manager. We will be evaluating this for >> different types of databases to find their implementation complexities and >> limitations. >> Other option available for us to use indexing based approach (use Solr) >> >> *Database full text search * >> *Pros* >> >> - Less complications when using container based approach >> - Clustering will require only database syncing. >> - No need to maintain and ship external search engine. >> >> *Cons* >> >> - Implementation may vary significantly based on the database type >> - There can be limitation in full text search for particular database >> types (For ex: mysql full text support only prefix search) >> - Queries will differ based on database type >> - Document search will not be available, because they are stored as >> blobs >> >> >> *Indexing based approach * >> *Pros* >> >> - Document search >> - Search will be efficient (No need to access database) >> >> *Cons* >> >> - Since indexing data is written to file system , when going for >> container based approach we would require mechanisms to file system >> mounting >> - Syncing indexers in a cluster would require something similar to >> existing C4 based registry architecture (use of REG_LOG table) >> - Maintaining (for ex: Version updates) and shipping external search >> engine. >> >> Your valuable input regrading this is highly appreciated. >> >> Thanks! >> Rajith >> >> -- >> Rajith Roshan >> Software Engineer, WSO2 Inc. >> >> >> Mobile: +94-72-642-8350 <%2B94-71-554-8430> >> >> >> >> >> >> >> >> -- >> *Chandana Napagoda* >> Associate Technical Lead >> WSO2 Inc. - http://wso2.org >> >> *Email : [email protected] <[email protected]>**Mobile : +94718169299 >> <+94%2071%20816%209299>* >> >> *Blog : http://cnapagoda.blogspot.com <http://cnapagoda.blogspot.com> >> | http://chandana.napagoda.com <http://chandana.napagoda.com>* >> >> *Linkedin : http://www.linkedin.com/in/chandananapagoda >> <http://www.linkedin.com/in/chandananapagoda>* >> >> >> >> > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Jayasinghe WSO2, Inc. (http://wso2.com) Email :[email protected] Mobile :0770704040 Blog :https://medium.com/@malith.jayasinghe <https://medium.com/@malith.jayasinghe> Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
