Hi On 05.10.2010 15:13, Richard S. Hall wrote: > On 10/5/10 2:59, Felix Meschberger wrote: >> Hi, >> >> On 04.10.2010 22:26, Richard S. Hall wrote: >>> On 10/4/10 12:31, Pierre De Rop wrote: >>>> This is surprising, because I double checked (I did the same kind of >>>> tests >>>> of differents machines), and I always see that loading a populated >>>> cache is >>>> a bit longer than an empty cache. >>>> Ok, it's confusing for now, I will do further investigation and will >>>> get >>>> back to you later, if I find something. >>> Let's keep investigating this. >>> >>> In the meantime, I decided to rollback the change for the next release >>> so we can have more time to investigate it because right now I need to >>> get a bug fix out. So, we won't introduce any bundle cache structure >>> changes in 3.0.4, but maybe in 3.0.5. >> This sounds like a minor-release-number-worthy change ;-) > > I'm not sure that's necessary. It is still backward compatible, but it > isn't forward compatible.
This means: once you ran the framework with the new bundle cache and installed and/or updated bundles you cannot run the existing cache with an older version of the framework any longer. Right ? In this case, IMHO a minor version number increase might provide a hint at this situation. Regards Felix > > -> richard > >> Regards >> Felix >> >>> -> richard >>> >>>> On Mon, Oct 4, 2010 at 5:05 PM, Richard S. >>>> Hall<[email protected]>wrote: >>>> >>>>> On 10/4/10 10:52, Richard S. Hall wrote: >>>>> >>>>>> Thanks for the installer bundle, I did some tests locally on my Mac >>>>>> using >>>>>> 230 bundles from GlassFish, this is what I see (note: the "purge" >>>>>> command >>>>>> flushes disk buffers): >>>>>> >>>>>> Empty cache >>>>>> ----------------- >>>>>> [heavyweight main]$ rm -rf felix-cache; purge; date; java -jar >>>>>> bin/felix.jar >>>>>> Mon Oct 4 10:43:09 EDT 2010 >>>>>> 2010-10-04 10:43:24,541 - BundleInstaller starting and waiting for >>>>>> FrameworkEvent.STARTED event ... >>>>>> 2010-10-04 10:43:24,822 - Framework started: Installing bundles from >>>>>> ./load dir >>>>>> 2010-10-04 10:43:54,464 - Done. >>>>>> >>>>>> Populated cache >>>>>> ----------------------- >>>>>> [heavyweight main]$ purge; date; java -jar bin/felix.jar >>>>>> Mon Oct 4 10:44:49 EDT 2010 >>>>>> 2010-10-04 10:45:12,212 - BundleInstaller starting and waiting for >>>>>> FrameworkEvent.STARTED event ... >>>>>> 2010-10-04 10:45:13,340 - Framework started: Installing bundles from >>>>>> ./load dir >>>>>> 2010-10-04 10:45:13,342 - Done. >>>>>> >>>>>> You can see that the empty cache scenario takes about 45 seconds, >>>>>> whereas >>>>>> the populated cache takes about 24 seconds. So, on my machine, it >>>>>> takes a >>>>>> lot less time to re-load cached bundles...could be a function of my >>>>>> disk >>>>>> speed, I suppose, but it is what I would expect. >>>>>> >>>>>> Not sure how to proceed, other than trying to get more samples. >>>>>> >>>>> Interestingly, I tried the same test using the 3.0.3 release (the >>>>> above was >>>>> with trunk) and I got the follow results: >>>>> >>>>> Empty cache (3.0.3) >>>>> ----------------- >>>>> >>>>> rm -rf felix-cache; purge; date; java -jar bin/felix.jar >>>>> Mon Oct 4 10:58:23 EDT 2010 >>>>> 2010-10-04 10:58:43,340 - BundleInstaller starting and waiting for >>>>> FrameworkEvent.STARTED event ... >>>>> 2010-10-04 10:58:43,557 - Framework started: Installing bundles from >>>>> ./load >>>>> dir >>>>> 2010-10-04 10:59:03,373 - Done. >>>>> >>>>> Populated cache (3.0.3) >>>>> ----------------------- >>>>> >>>>> purge; date; java -jar bin/felix.jar >>>>> Mon Oct 4 10:59:50 EDT 2010 >>>>> 2010-10-04 11:00:24,738 - BundleInstaller starting and waiting for >>>>> FrameworkEvent.STARTED event ... >>>>> 2010-10-04 11:00:26,736 - Framework started: Installing bundles from >>>>> ./load >>>>> dir >>>>> 2010-10-04 11:00:26,738 - Done. >>>>> >>>>> So, the populated cache takes nearly as long as the empty cache (36 >>>>> seconds >>>>> compared to 40 seconds)...this also makes sense, since the trunk has >>>>> patches >>>>> specifically designed to improve re-loading cached bundles. >>>>> >>>>> -> richard >>>>> >>>>> >>>>> >>>>>>> On Fri, Oct 1, 2010 at 11:30 PM,<[email protected]> wrote: >>>>>>>>> On Fri, 1 Oct 2010 23:25:35 +0200, Pierre De Rop< >>>>>>>>> [email protected]> >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> On Fri, Oct 1, 2010 at 10:57 PM, Richard S. Hall >>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>> >>>>>>>>>>> One thing, was your performance comparison based on using >>>>>>>>>>> the new >>>>>>>>>>> cache >>>>>>>>>>> or >>>>>>>>>>> >>>>>>>>>>>> old? >>>>>>>>>>>> >>>>>>>>>>>> I assume you are using a newly created cache for the framework >>>>>>>>>>>> snapshot >>>>>>>>>>>> and >>>>>>>>>>>> [obviously] an old cache for the 3.0.2, correct? >>>>>>>>>>>> >>>>>>>>>>>> yes: >>>>>>>>>>>> >>>>>>>>>>> old cache with 3.0.2, and new cache with trunk. >>>>>>>>>>> but I also checked that new fwk/trunk is properly working when >>>>>>>>>>> being >>>>>>>>>>> started >>>>>>>>>>> with the old cache (from 3.0.2) >>>>>>>>>>> >>>>>>>>>>> You should also be testing an already created cache (i.e., a >>>>>>>>>>> framework >>>>>>>>>>> >>>>>>>>>>>> restart), not an empty cache. >>>>>>>>>>>> >>>>>>>>>>>> In the previous test, I compared with already created cache, >>>>>>>>>>>> not >>>>>>>>>>>> with >>>>>>>>>>>> >>>>>>>>>>> empty >>>>>>>>>> caches. >>>>>>>>>>> With empty caches, here are the results: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - old fwk 3.0.2 / fresh empty cache -> 15.8 sec. >>>>>>>>>>> (this is >>>>>>>>>>> surprising: >>>>>>>>>>> >>>>>>>>>>> I >>>>>>>>>> would expect to get a longer delay, since the cache is >>>>>>>>>> empty when >>>>>>>>>>> I >>>>>>>>>>> start >>>>>>>>>>> the fwk) >>>>>>>>>>> >>>>>>>>>>> 2010-10-01 23:10:17,403 Starting Felix 3.0.2 >>>>>>>>>>> 2010-10-01 23:10:23,885 Starting cluster node "test" >>>>>>>>>>> 2010-10-01 23:10:33,208 Cluster node "test" ready >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - new fwk / fresh empty cache ->15.2 sec >>>>>>>>>>> >>>>>>>>>>> 2010-10-01 23:15:46,2 Starting Felix 3.1.0.SNAPSHOT >>>>>>>>>>> 2010-10-01 23:15:52,417 Starting cluster node "test" >>>>>>>>>>> 2010-10-01 23:16:01,225 Cluster node "test" ready >>>>>>>>>>> >>>>>>>>>>> hmmm, I don't understand why the fwk startup time is better >>>>>>>>>>> when the >>>>>>>>>>> >>>>>>>>>>> cache >>>>>>>>>> is empty (15.2 sec) than when the cache is non-empty (21.2 sec >>>>>>>>>> from >>>>>>>>>>> previous >>>>>>>>>>> mail) >>>>>>>>>>> Ok, I will do more tests ... may be I did something wrong ? ... >>>>>>>>>>> to be >>>>>>>>>>> continued ... >>>>>>>>>>> >>>>>>>>>>> Hmm. Yeah, that seems a little odd. In that case, you should >>>>>>>>>>> just >>>>>>>>>> delete >>>>>>>>>> your cache each time to get better startup performance! ;-) >>>>>>>>>> >>>>>>>>>> Definitely let me know what you find out. I too would expect >>>>>>>>>> framework >>>>>>>>>> restarts to be faster since we don't have to copy the JAR files >>>>>>>>>> first. >>>>>>>>>> >>>>>>>>>> -> richard >>>>>>>>>> >>>>>>>>>> /pierre >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Is that what you are doing? Just want to make sure I >>>>>>>>>>> understand what >>>>>>>>>>> is >>>>>>>>>>> >>>>>>>>>>>> being measured. :-) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -> richard >>>>>>>>>>>> >>>>>>>>>>>> On 10/1/10 16:43, Pierre De Rop wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Richard, >>>>>>>>>>>> >>>>>>>>>>>>> I tried our app server with the trunk (old cache with the new >>>>>>>>>>>>> >>>>>>>>>>>>> fwk/trunk) >>>>>>>>>>> -> >>>>>>>>>>> >>>>>>>>>>>> everything works ok >>>>>>>>>>>>> (I install 195 bundles, and start 41 bundles) >>>>>>>>>>>>> >>>>>>>>>>>>> With new cache/fwk trunk -> it's also ok. >>>>>>>>>>>>> >>>>>>>>>>>>> I also compared the performance of the startup time between >>>>>>>>>>>>> felix >>>>>>>>>>>>> >>>>>>>>>>>>> 2.0.2 >>>>>>>>>>> and >>>>>>>>>>> >>>>>>>>>>>> felix trunk. >>>>>>>>>>>>> To do this, I managed to ensure that my linux IO buffers >>>>>>>>>>>>> cache are >>>>>>>>>>>>> >>>>>>>>>>>>> empty >>>>>>>>>>> before doing each test, by doing this: >>>>>>>>>>> >>>>>>>>>>>> sync >>>>>>>>>>>>> echo 1> /proc/sys/vm/drop_caches >>>>>>>>>>>>> echo 2> /proc/sys/vm/drop_caches >>>>>>>>>>>>> echo 3> /proc/sys/vm/drop_caches >>>>>>>>>>>>> >>>>>>>>>>>>> This ensures that my linux has released all buffer cache >>>>>>>>>>>>> disk, like >>>>>>>>>>>>> it >>>>>>>>>>>>> is >>>>>>>>>>>>> the case when I boot my host (cold start). >>>>>>>>>>>>> Now, for felix 3.0.2, I load my server in around *21,1 >>>>>>>>>>>>> *seconds >>>>>>>>>>>>> (it's >>>>>>>>>>>>> slow >>>>>>>>>>>>> because all disk buffers are empty): >>>>>>>>>>>>> >>>>>>>>>>>>> 2010-10-01 22:15:07,880 Starting Felix 3.0.2 >>>>>>>>>>>>> 2010-10-01 22:15:18,121 Starting cluster node "test" >>>>>>>>>>>>> 2010-10-01 22:15:28,966 Cluster node "test" ready >>>>>>>>>>>>> >>>>>>>>>>>>> and with felix/trunk, indeed, the server starts a little bit >>>>>>>>>>>>> quicker, >>>>>>>>>>>>> but >>>>>>>>>>>>> not so much: around *in 20,3* seconds (I also flushed disk >>>>>>>>>>>>> buffers >>>>>>>>>>>>> before >>>>>>>>>>>>> starting the test): >>>>>>>>>>>>> >>>>>>>>>>>>> 2010-10-01 22:17:47,505 Starting Felix 3.1.0.SNAPSHOT >>>>>>>>>>>>> 2010-10-01 22:17:56,815 Starting cluster node "test" >>>>>>>>>>>>> 2010-10-01 22:18:07,798 Cluster node "test" ready >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /pierre >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Oct 1, 2010 at 5:39 PM, Richard S. Hall< >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> p.s. I deploy snapshots of the everything, including the >>>>>>>>>>>>> convenience >>>>>>>>>>>>> >>>>>>>>>>>>> framework distribution or you can build from trunk. >>>>>>>>>>>>>> On 10/1/10 11:37, Richard S. Hall wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've made some changes to the bundle cache layout and >>>>>>>>>>>>>> handling to >>>>>>>>>>>>>>> improve >>>>>>>>>>>>>>> performance and clean up code. I've tried to maintain >>>>>>>>>>>>>>> backward >>>>>>>>>>>>>>> compatibility >>>>>>>>>>>>>>> with old caches, but I'd appreciate is someone could try it >>>>>>>>>>>>>>> out >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> caches to see if it works. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ** Back up your caches just in case! ** >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Note that the new cache format will not work with older >>>>>>>>>>>>>>> frameworks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So >>>>>>>>>>>>> you >>>>>>>>>>>> cannot use a cache created with this new framework with >>>>>>>>>>>> previous >>>>>>>>>>>>>>> frameworks; >>>>>>>>>>>>>>> however, previously created caches should mostly continue >>>>>>>>>>>>>>> to work >>>>>>>>>>>>>>> between >>>>>>>>>>>>>>> the two although there will be a loss of fidelity since >>>>>>>>>>>>>>> changes >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> state >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>> only saved to the new way or the old way depending on if >>>>>>>>>>>>>>> you are >>>>>>>>>>>>>>> running >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>> the new framework or the old one. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For the curious, I've combined five bundle cache files (id, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> location, >>>>>>>>>>>>> state, start level, last modified, and refresh count) into a >>>>>>>>>>>>> single >>>>>>>>>>>> file >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> try to avoid excessive file accesses. This appears to >>>>>>>>>>>>>>> improve >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> startup >>>>>>>>>>>>> time >>>>>>>>>>>> when you have a cache with lots of bundles, but your mileage >>>>>>>>>>>> will >>>>>>>>>>>>>>> vary >>>>>>>>>>>>> based >>>>>>>>>>>> on platform and other factors. Although, you won't necessarily >>>>>>>>>>>> see >>>>>>>>>>>>>>> any >>>>>>>>>>>>> improvements if you are using an old cache, since it will >>>>>>>>>>>>> revert to >>>>>>>>>>>> the >>>>>>>>>>>>>>> old >>>>>>>>>>>>>>> way. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -> richard >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >
