It is unfortuantely a little late in the game for us to switch ORM's now, otherwise I may have.
On Tue, Mar 10, 2009 at 9:15 AM, Jeremy Benoist <[email protected]>wrote: > I didn't know very well propel.But I know that Doctrine handle natively a > cache system that is easy to use > http://www.ullright.org/ullWiki/show/docid/64 > > And this cache could be use with different system (like apc, memcache, > etc..) > http://trac.doctrine-project.org/browser/trunk/lib/Doctrine/ORM/Cache > > If Propel doesn't handle cache maybe you can check how doctrine handle it > to try to do the same, like your little snippet. > > J. > > > On Tue, Mar 10, 2009 at 7:54 AM, Gareth McCumskey <[email protected]>wrote: > >> I have actually come across a rather interesting way to use the memory >> cache as specified in the book ( >> http://www.symfony-project.org/book/1_1/18-Performance#chapter_18_sub_caching_data_in_the_server >> ) >> >> The way it is described to store data in the cache needs three things; a >> unique name that can be easily determined, the value related to that name >> and how long that value stays cached. In our application, our database >> records will very very rarely see any updates. Once records are inserted >> they are only ever retrieved making them ideal for caching. But we cannot >> store every database record into memory. We do have queries that run very >> frequently, however, and each Criteria object i sunique for each query with >> the various values and so on. The problem is you cannot store an object, >> like the Criteria object that is built up before you run a Propel query with >> doSelect or similar methods. So, if you serialize the Criteria object, you >> have a string. But this is a very long string (one of the serialized >> Criteria objects I tested with was over 1000 characters long). But you can >> convert it to a hash value. Naturally there is one problem with a hash; the >> possibility that two different string would create the same hash. So create >> two hashes and concatenate them, an SHA1 hash and an MD5 hash. This can then >> be the name of the item you are storing in memory. Run the DB query and you >> have the value of the query, which can then also be stored into the cache. >> >> As a brief example for what I mean by overriding the doSelect method for a >> model class: >> >> private static function doSelect($c) >> { >> $serialized_c = serialize($c); >> $md5_serialized = md5($serialized_c); >> $sha1_serialized = sha1($serialized_c); >> >> $cache = new sfAPCCache(); >> >> if ($cache->has('doSelect'.$md5_serialized.$sha1_serialized)) >> { >> $query_value = >> $cache->get('doSelect'.$md5_serialized.$sha1_serialized); >> } >> else >> { >> $query_value = parent::doSelect($c); >> $cache->set('doSelect'.$md5_serialized.$sha1_serialized, >> $query_value, 60); >> } >> >> return $query_value; >> } >> >> Any comments would be great on this or any problems with doing this kind >> of thing that I may not have seen please feel free to let me know >> >> >> On Tue, Mar 10, 2009 at 8:06 AM, Gareth McCumskey >> <[email protected]>wrote: >> >>> Thanks for that. I have actually been looking at the function cache >>> amongst others and there is a lot we can do there as our DB records, once >>> inserted, are not likely to change. In fact if they do it means we are >>> having a problem as we store email data for a number of companies in them. >>> Therefore function caching and even memory caching records as we extract >>> them from db would probably help us a lot. It does mean more work code-wise >>> and isn't a "quick-fix", so we plan to start looking at this once we hit >>> Beta where performance will be a major requirement. >>> >>> The old system is faster simply because it follows no design pattern >>> except procedural and that is where its speed lies. There are no ORM's, >>> classes or anything like that, and SQL queries are sent straight through to >>> the database using handcoded, dynamic SQL queries as opposed to an ORM >>> generated one and the resultsets are manipulated directly in each "view". In >>> fact there are only views, there is little seperation of business logic and >>> presentation. >>> >>> The reason we need symfony for this new version is that we are going to >>> be adding more advanced features that would "complicate" the product beyond >>> what a procedural style would allow us to maintain. We are already >>> struggling to keep the older system maintained and enhanced for our >>> customers as it is. symfony, Propel and even Prototype with scriptaculous >>> help alleviate these maintenance and extensibility issues. >>> >>> >>> On Mon, Mar 9, 2009 at 9:13 PM, Richtermeister <[email protected]> wrote: >>> >>>> >>>> Hi Gareth, >>>> >>>> after reading all this I feel your time is most likely best spent in >>>> smart caching, since it sounds like the DB is not your bottleneck. >>>> What's easy to overlook when working with symfony, is that compared to >>>> straight procedural "get data -> display data" scripts, rendering >>>> templates with hydrated objects is slower, albeit more flexible. So, >>>> if your previous site was coded in a bad way, it was probably using a >>>> lot of "view specific" code, so it's hard to compete with that on pure >>>> speed considerations. The only way to mitigate that is by using all >>>> forms of caching, and yes, this may include the function cache >>>> (although I don't like it much). However, the "higher up" you can >>>> cache, i.e. a complete action, the less you have to cache on the model >>>> level. >>>> >>>> Just my 2 cents. Good luck, >>>> Daniel >>>> >>>> >>>> >>>> >>>> >>>> On Mar 9, 8:42 am, Sumedh <[email protected]> wrote: >>>> > My 2 cents...slow query log in mysql should help a lot... >>>> > >>>> > Please let us know your insights at the end of your exercise... :) >>>> > >>>> > On Mar 9, 3:41 pm, Gareth McCumskey <[email protected]> wrote: >>>> > >>>> > > I just tried using Propel 1.3 on our application and while I would >>>> love to >>>> > > continue using it (as it seemed to produce a little more efficieny) >>>> we can't >>>> > > use it for now because the servers that the app will run on are >>>> Centos 4 >>>> > > with PHP 5.1.x as its maximum version for now. The sysadmins here >>>> say that >>>> > > to force an upgrade to 5.2.x would be a hard task as to retain >>>> RedHat >>>> > > support it means they would need to upgrade to Centos 5. >>>> > >>>> > > I am currently looking at the chapter about Optimising symfony and >>>> the >>>> > > function cache seems to be something we cna consider doing in a lot >>>> of our >>>> > > model calls from the action to help speed things up, especially for >>>> model >>>> > > methods that access historical data (i.e. stuff dated in the past >>>> that >>>> > > obviously wont change on subsequent calls) but these are relatively >>>> large >>>> > > coding changes which we will probably only do during our beta >>>> development >>>> > > phase. >>>> > >>>> > > I am still looking through more advise recieved from this post and I >>>> have to >>>> > > thank everyone for their input. I honestly didn't expect this >>>> response and >>>> > > it has been fantastic and very helpful. >>>> > >>>> > > On Mon, Mar 9, 2009 at 12:49 AM, Crafty_Shadow <[email protected]> >>>> wrote: >>>> > >>>> > > > Symfony 1.1 came by default with Propel 1.2 >>>> > > > You can try upgrading to 1.3 (it isn't really a trivial task, but >>>> it >>>> > > > shouldn't be a big problem) >>>> > > > There is thorough explanation on the symfony site how to do it: >>>> > > >http://www.symfony-project.org/cookbook/1_1/en/propel_13 >>>> > > > It should fare a measurable increase in performance. Also, a site >>>> that >>>> > > > makes good use of cache should have caching for absolutely >>>> everything >>>> > > > not session-dependent. I find it hard to imagine a php app, no >>>> matter >>>> > > > how fast, that would run faster than symfony's cached output. >>>> > >>>> > > > Alvaro: >>>> > > > Is your plugin based on Propel 1.3? >>>> > > > If you believe you have made significant improvements to Propel, >>>> why >>>> > > > not suggest them for version 2.0, which is still under heavy >>>> > > > development? >>>> > >>>> > > > On Mar 8, 4:33 pm, alvaro <[email protected]> wrote: >>>> > > > > At the company I developed a symfony plugin to optimize the >>>> Propel >>>> > > > > queries and also the Propel hydrate method, improving even 5 >>>> times >>>> > > > > query speed and also memory usage. >>>> > >>>> > > > > The plugins supports joins and thanks to PHP features the plugin >>>> > > > > returns Propel objects populated with custom AS columns. >>>> > >>>> > > > > We are thinking on release it on the following weeks so stay >>>> tuned :) >>>> > >>>> > > > > Regards, >>>> > >>>> > > > > Alvaro >>>> > >>>> > > > > On Mar 8, 2009, at 10:20 PM, Gareth McCumskey wrote: >>>> > >>>> > > > > > We have put numerous caching techniques into effect, from >>>> Cache- >>>> > > > > > Expires headers to compression of static files like js and >>>> html >>>> > > > > > files. Currently we use symfony 1.1 and Propel as the ORM. We >>>> have >>>> > > > > > identified the bottleneck generally as being the application >>>> > > > > > processing after the db queries have run to extract the data. >>>> > >>>> > > > > > The entire point of my question was to get some info on >>>> general tips >>>> > > > > > and tricks we can try out to see if anything helps or if >>>> perhaps we >>>> > > > > > have missed any obvious issues that may actually be the cause >>>> of the >>>> > > > > > slow performance we are getting. As it is I have gotten quite >>>> a few >>>> > > > > > and look forward to getting into the office tomorrow to try >>>> them >>>> > > > > > out. Anymore is greatly appreciated. >>>> > >>>> > > > > > Of course I am looking through the code to see if there is >>>> anyway we >>>> > > > > > can streamline it on that end, but every little bit helps. >>>> > >>>> > > > > > Gareth >>>> > >>>> > > > > > On Sun, Mar 8, 2009 at 12:27 PM, Crafty_Shadow < >>>> [email protected]> >>>> > > > > > wrote: >>>> > >>>> > > > > > Gareth, you didn't mention what version of symfony you were >>>> using, >>>> > > > > > also what ORM (if any). >>>> > > > > > The best course of optimization will depend on those. Also, as >>>> already >>>> > > > > > mentioned, caching is your best friend. >>>> > >>>> > > > > > On Mar 8, 9:43 am, Gareth McCumskey <[email protected]> >>>> wrote: >>>> > > > > > > Well, consider a single database table that looks something >>>> like >>>> > > > > > this: >>>> > >>>> > > > > > > From_address >>>> > > > > > > to_address (possibly multiple addresses comma-seperated) >>>> > > > > > > headers >>>> > > > > > > spam_report >>>> > > > > > > subject >>>> > >>>> > > > > > > And we would have millions of those records in the database. >>>> > > > > > Repeated >>>> > > > > > > entries, especially on to_address, means the data is hugely >>>> > > > > > redundant. By >>>> > > > > > > normalising we are turning a text search across millions of >>>> > > > > > records with >>>> > > > > > > redundant repeated data into a text search over a unique >>>> list, >>>> > > > > > then an >>>> > > > > > > integer search over primary key (which of course is >>>> indexed). >>>> > >>>> > > > > > > On Sun, Mar 8, 2009 at 9:37 AM, Lawrence Krubner >>>> > > > > > <[email protected]>wrote: >>>> > >>>> > > > > > > > On Mar 8, 3:26 am, Gareth McCumskey <[email protected]> >>>> wrote: >>>> > > > > > > > > We had a speed increase because we had a lot of text >>>> searches >>>> > > > > > in the old >>>> > > > > > > > > system, all going through text fields where the same >>>> values >>>> > > > > > were repeated >>>> > > > > > > > > over and over. Its therefore a lot faster to search a >>>> much >>>> > > > > > smaller table, >>>> > > > > > > > > where the text fields are unique, and find the value >>>> once, >>>> > > > > > then use an ID >>>> > > > > > > > > comparison, being much faster to match integers than >>>> text. >>>> > >>>> > > > > > > > In sounds like you got a speed boost from doing >>>> intelligent >>>> > > > > > indexing. >>>> > > > > > > > What you are describing sounds more like indexing than >>>> > > > > > normalization, >>>> > > > > > > > at least to me. >>>> >>>> >>> >> >> >> > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -~----------~----~----~----~------~----~------~--~---
