Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
Bug filed: https://bugs.eclipse.org/bugs/show_bug.cgi?id=379865 See attachment to the bug for the code and bundle to reproduce the problem. You can simply drop the bundle and the activator will run the test. Thanks, Stanley -Original Message- From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Gunnar Wagenknecht Sent: Wednesday, May 09, 2012 11:22 PM To: equinox-dev@eclipse.org Subject: Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability Stanly, Am 08.05.2012 20:18, schrieb stanley_p...@dell.com: But the exact same example runs 50X faster on the other two OSGi runtime. Any thoughts? And we observed the other runtime used more memory. Will they be using some indexing (in memory)? I find your report very valuable. I think at this time the best thing is to submit a bug and attach the sample project. The project should contain all the code that is necessary to execute the test. This allows the developers to look at the code and run the tests. Otherwise things might get lost in between mails. If Equinox is really 50x slower than two other frameworks then it looks like there is room for improvement. Thanks! -Gunnar -- Gunnar Wagenknecht gun...@wagenknecht.org http://wagenknecht.org/ ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
Bug filed: https://bugs.eclipse.org/bugs/show_bug.cgi?id=379865 From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Thomas Watson Sent: Thursday, May 10, 2012 11:50 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability I agree with Gunner. Please open a bug report and attach a test that we can reproduce with. Tom [cid:image001.gif@01CD3447.4FCD4F30]Gunnar Wagenknecht ---05/10/2012 01:23:05 AM---Stanly, From: Gunnar Wagenknecht gun...@wagenknecht.org To: equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date: 05/10/2012 01:23 AM Subject: Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability Stanly, Am 08.05.2012 20:18, schrieb stanley_p...@dell.commailto:stanley_p...@dell.com: But the exact same example runs 50X faster on the other two OSGi runtime. Any thoughts? And we observed the other runtime used more memory. Will they be using some indexing (in memory)? I find your report very valuable. I think at this time the best thing is to submit a bug and attach the sample project. The project should contain all the code that is necessary to execute the test. This allows the developers to look at the code and run the tests. Otherwise things might get lost in between mails. If Equinox is really 50x slower than two other frameworks then it looks like there is room for improvement. Thanks! -Gunnar -- Gunnar Wagenknecht gun...@wagenknecht.orgmailto:gun...@wagenknecht.org http://wagenknecht.org/ ___ equinox-dev mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev inline: image001.gifinline: image002.pnginline: image003.png___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
Stanly, Am 08.05.2012 20:18, schrieb stanley_p...@dell.com: But the exact same example runs 50X faster on the other two OSGi runtime. Any thoughts? And we observed the other runtime used more memory. Will they be using some indexing (in memory)? I find your report very valuable. I think at this time the best thing is to submit a bug and attach the sample project. The project should contain all the code that is necessary to execute the test. This allows the developers to look at the code and run the tests. Otherwise things might get lost in between mails. If Equinox is really 50x slower than two other frameworks then it looks like there is room for improvement. Thanks! -Gunnar -- Gunnar Wagenknecht gun...@wagenknecht.org http://wagenknecht.org/ ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
I agree with Gunner. Please open a bug report and attach a test that we can reproduce with. Tom From: Gunnar Wagenknecht gun...@wagenknecht.org To: equinox-dev@eclipse.org, Date: 05/10/2012 01:23 AM Subject:Re: [equinox-dev] Service Lookup by GUID very Slow - the FrameworkScalability Stanly, Am 08.05.2012 20:18, schrieb stanley_p...@dell.com: But the exact same example runs 50X faster on the other two OSGi runtime. Any thoughts? And we observed the other runtime used more memory. Will they be using some indexing (in memory)? I find your report very valuable. I think at this time the best thing is to submit a bug and attach the sample project. The project should contain all the code that is necessary to execute the test. This allows the developers to look at the code and run the tests. Otherwise things might get lost in between mails. If Equinox is really 50x slower than two other frameworks then it looks like there is room for improvement. Thanks! -Gunnar -- Gunnar Wagenknecht gun...@wagenknecht.org http://wagenknecht.org/ ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev inline: graycol.gifinline: ecblank.gif___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
The key question is about the scalability of the runtime, in terms of - Number of services, 100K, 1M, 10M, etc - Registering and unregistering them, churn over long period of time - Is it primarily memory bound? Assuming that services are not CPU heavy or thread hungry Some details of the experiment we had: The code is pretty straightforward. We only have a 5 Service Classes. We add an ID property to each service object we register and later look it up using that ID. The look up through getServiceReferences() took 4-5 seconds when we have 200K services registered. Also, can someone shed some light for the following questions: -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Service Registration: we register all the 200K service instances in a loop. They are one of 5 different server classes. BundleContext bc, DictionaryString, String props = new HashtableString, String(1); props.put(“ID”, “ID001”); bc.registerService(ServiceClass, serviceObject, createProperties(props)); Service Look up: BundleContext bc, bc.getServiceReferences(ServiceClass, (ID=ID001)”); Thank you, Stanley From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of BJ Hargrave Sent: Friday, May 04, 2012 10:19 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow Equinox also indexes by objectClass alone. So I am not sure what the discrepancy is here. Would be nice to have the test case code to analyze. Stanley, can you post a gist with the code? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliancehttp://www.osgi.org/ hargr...@us.ibm.commailto:hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From:Richard S. Hall he...@ungoverned.orgmailto:he...@ungoverned.org To:Equinox development mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date:2012/05/04 13:16 Subject:Re: [equinox-dev] Service Lookup by GUID very Slow Sent by: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org Just a side comment... On the Felix framework, it is technically possible to index services on arbitrary service properties, but we don't provide any configuration properties to do so. By default, we only index on objectClass, which I assume Equinox does as well. If all of your services have the same objectClass, then it will regress to a linear search. There is no other magic to make it faster in Felix. I would expect something similar in Equinox. If that is not the case, then yeah it sounds like there is an issue. - richard On 5/4/12 12:41 , stanley_p...@dell.commailto:stanley_p...@dell.com wrote: Tom, You are right on. I am using a simple filter. We just added a GUID property to each service. Two follow up questions: -We ran the same tests on Felix and Knopflerfish and get 100ms response time. This is about 50X. I am wondering there may be something wrong in the environment. Do you think JVM settings like Perm generation size helps? I have Xmx=2GB, Xms=1GB and XXMaxPermSize=64MB. -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Thank you, Stanley From: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Thomas Watson Sent: Friday, May 04, 2012 5:40 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow I was also not sure what you meant by GUID. After some thought I think you probably mean the service id or perhaps the service pid (service.id and service.pid properties)? And by lookup I assume you are using some kind of service filter, for example (service.id=23) with a call to BundleContext.getServiceReferences. I will say that the service registry is not optimized for this kind of lookup. You are far better keeping your own data structure that optimizes the lookup over the set of service references and indexes on the keys that you want to use to lookup service references. This can easily be done with a ServiceTrackerCustomizer. Tom [cid:image001.gif@01CD2D05.A58E8680]BJ Hargrave---05/03/2012 10:04:05 PM---What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specif From: BJ Hargrave/Austin/IBM@IBMUS To: Equinox
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
* No, services are not created lazily by default. You are creating the services yourself in your code sample, i.e., the 'serviceObject' variable. There are some frameworks such as Declarative Services that create service instances lazily, however the creation of the service (and therefore the associated cost) would not be triggered by merely calling getServiceReferences(). It would be triggered by calling getService() on one of the ServiceReference objects. * By creating a ServiceTracker, you can be notified synchronously as each service is registered. You could then very cheaply extract the ID property from each service and store it in a ConcurrentHashMap for later reference. I'm pretty sure that the poor performance of your example is due to the fact that the framework is forced to evaluate the filter _expression_ "(ID=x)" against every entry in the registry. You don't even allow it to return when it finds the first match, because you have called getServiceReferences(), plural, so it must find all of the matches not just the first one. If you called getServiceReference() then the search time would be halved, on aggregate. Regards Neil stanley_p...@dell.com 8 May 2012 18:44 The key question is about the scalability of the runtime, in terms of - Number of services, 100K, 1M, 10M, etc- Registering and unregistering them, churn over long period of time- Is it primarily memory bound? Assuming that services are not CPU heavy or thread hungry Some details of the experiment we had: The code is pretty straightforward. We only have a 5 Service Classes. We add an ID property to each service object we register and later look it up using that ID. The look up through getServiceReferences() took 4-5 seconds when we have 200K services registered. Also, can someone shed some light for the following questions: - Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? - Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Service Registration: we register all the 200K service instances in a loop. They are one of 5 different server classes. BundleContext bc, DictionaryString, String props = new HashtableString, String(1); props.put(“ID”, “ID001”); bc.registerService(ServiceClass, serviceObject, createProperties(props)); Service Look up: BundleContext bc, bc.getServiceReferences(ServiceClass, "(ID=ID001)”); Thank you,Stanley From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of BJ HargraveSent: Friday, May 04, 2012 10:19 AMTo: Equinox development mailing listSubject: Re: [equinox-dev] Service Lookup by GUID very Slow Equinox also indexes by objectClass alone. So I am not sure what the discrepancy is here. Would be nice to have the test case code to analyze. Stanley, can you post a gist with the code? -- BJ HargraveSenior Technical Staff Member, IBMOSGi Fellow and CTO of the OSGi Alliancehargr...@us.ibm.com office: +1 386 848 1781mobile: +1 386 848 3788From: "Richard S. Hall" he...@ungoverned.org To: Equinox development mailing list equinox-dev@eclipse.org, Date: 2012/05/04 13:16 Subject: Re: [equinox-dev] Service Lookup by GUID very Slow Sent by: equinox-dev-boun...@eclipse.org ___equinox-dev mailing listequinox-dev@eclipse.orghttps://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
Thank you Neil. But the exact same example runs 50X faster on the other two OSGi runtime. Any thoughts? And we observed the other runtime used more memory. Will they be using some indexing (in memory)? Thanks, Stanley From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Neil Bartlett Sent: Tuesday, May 08, 2012 11:02 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability * No, services are not created lazily by default. You are creating the services yourself in your code sample, i.e., the 'serviceObject' variable. There are some frameworks such as Declarative Services that create service instances lazily, however the creation of the service (and therefore the associated cost) would not be triggered by merely calling getServiceReferences(). It would be triggered by calling getService() on one of the ServiceReference objects. * By creating a ServiceTracker, you can be notified synchronously as each service is registered. You could then very cheaply extract the ID property from each service and store it in a ConcurrentHashMap for later reference. I'm pretty sure that the poor performance of your example is due to the fact that the framework is forced to evaluate the filter expression (ID=x) against every entry in the registry. You don't even allow it to return when it finds the first match, because you have called getServiceReferences(), plural, so it must find all of the matches not just the first one. If you called getServiceReference() then the search time would be halved, on aggregate. Regards Neil [cid:image001.jpg@01CD2D0C.3BA6B150] stanley_p...@dell.commailto:stanley_p...@dell.com 8 May 2012 18:44 The key question is about the scalability of the runtime, in terms of -Number of services, 100K, 1M, 10M, etc -Registering and unregistering them, churn over long period of time -Is it primarily memory bound? Assuming that services are not CPU heavy or thread hungry Some details of the experiment we had: The code is pretty straightforward. We only have a 5 Service Classes. We add an ID property to each service object we register and later look it up using that ID. The look up through getServiceReferences() took 4-5 seconds when we have 200K services registered. Also, can someone shed some light for the following questions: -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Service Registration: we register all the 200K service instances in a loop. They are one of 5 different server classes. BundleContext bc, DictionaryString, String props = new HashtableString, String(1); props.put(“ID”, “ID001”); bc.registerService(ServiceClass, serviceObject, createProperties(props)); Service Look up: BundleContext bc, bc.getServiceReferences(ServiceClass, (ID=ID001)”); Thank you, Stanley From: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of BJ Hargrave Sent: Friday, May 04, 2012 10:19 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow Equinox also indexes by objectClass alone. So I am not sure what the discrepancy is here. Would be nice to have the test case code to analyze. Stanley, can you post a gist with the code? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliancehttp://www.osgi.org/ hargr...@us.ibm.commailto:hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From:Richard S. Hall he...@ungoverned.orgmailto:he...@ungoverned.org To:Equinox development mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date:2012/05/04 13:16 Subject:Re: [equinox-dev] Service Lookup by GUID very Slow Sent by: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org ___ equinox-dev mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev inline: image001.jpg___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability
My guess would be something in the implementation of the filter. Sorry I have lost track of which framework implementation was slow... but nevertheless I don't know enough about the internal details of any of them. Perhaps the slow framework is recompiling the filter each time? I have created a Gist showing, in simplified form, how to use ServiceTracker to create a dynamic index for the services using a particular ID property. I would be interested in hearing whether this speeds things up, and by how much. Here's the link: https://gist.github.com/2638180 Regards Neil stanley_p...@dell.com 8 May 2012 19:18 Thank you Neil.But the exact same example runs 50X faster on the other two OSGi runtime. Any thoughts? And we observed the other runtime used more memory. Will they be using some indexing (in memory)?Thanks,StanleyFrom: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Neil BartlettSent: Tuesday, May 08, 2012 11:02 AMTo: Equinox development mailing listSubject: Re: [equinox-dev] Service Lookup by GUID very Slow - the Framework Scalability* No, services are not created lazily by default. You are creating the services yourself in your code sample, i.e., the 'serviceObject' variable.There are some frameworks such as Declarative Services that create service instances lazily, however the creation of the service (and therefore the associated cost) would not be triggered by merely calling getServiceReferences(). It would be triggered by calling getService() on one of the ServiceReference objects.* By creating a ServiceTracker, you can be notified synchronously as each service is registered. You could then very cheaply extract the ID property from each service and store it in a ConcurrentHashMap for later reference.I'm pretty sure that the poor performance of your example is due to the fact that the framework is forced to evaluate the filter _expression_ "(ID=x)" against every entry in the registry. You don't even allow it to return when it finds the first match, because you have called getServiceReferences(), plural, so it must find all of the matches not just the first one. If you called getServiceReference() then the search time would be halved, on aggregate.RegardsNeilstanley_p...@dell.com8 May 2012 18:44The key question is about the scalability of the runtime, in terms of - Number of services, 100K, 1M, 10M, etc- Registering and unregistering them, churn over long period of time- Is it primarily memory bound? Assuming that services are not CPU heavy or thread hungrySome details of the experiment we had:The code is pretty straightforward. We only have a 5 Service Classes. We add an ID property to each service object we register and later look it up using that ID. The look up through getServiceReferences() took 4-5 seconds when we have 200K services registered.Also, can someone shed some light for the following questions:- Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? - Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Service Registration: we register all the 200K service instances in a loop. They are one of 5 different server classes.BundleContext bc, DictionaryString, String props = new HashtableString, String(1); props.put(ID, ID001); bc.registerService(ServiceClass, serviceObject, createProperties(props));Service Look up: BundleContext bc, bc.getServiceReferences(ServiceClass, "(ID=ID001));Thank you,StanleyFrom: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of BJ HargraveSent: Friday, May 04, 2012 10:19 AMTo: Equinox development mailing listSubject: Re: [equinox-dev] Service Lookup by GUID very SlowEquinox also indexes by objectClass alone. So I am not sure what the discrepancy is here. Would be nice to have the test case code to analyze. Stanley, can you post a gist with the code? -- BJ HargraveSenior Technical Staff Member, IBMOSGi Fellow and CTO of the OSGi Alliancehargr...@us.ibm.com office: +1 386 848 1781mobile: +1 386 848 3788From: "Richard S. Hall" he...@ungoverned.org To: Equinox development mailing list equinox-dev@eclipse.org, Date: 2012/05/04 13:16 Subject: Re: [equinox-dev] Service Lookup by GUID very Slow Sent by:equinox-dev-boun...@eclipse.org ___equinox-dev mailing listequinox-dev@eclipse.orghttps://dev.eclipse.org/mailman/listinfo/equinox-dev___equinox-dev mailing listequinox-dev@eclipse.orghttps://dev.eclipse.org/mailman/listinfo/equinox-dev stanley_p...@dell.com 8 May 2012 18:44 The key question is about the scalability of the runtime, in terms of - Number of services, 100K,
Re: [equinox-dev] Service Lookup by GUID very Slow
The code is pretty straightforward. We only have a 5 Service Classes. We add an ID property to each service object we register and later look it up using that ID. Also, can you shed some light for these two? -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Service Registration: we register all the 200K service instances in a loop. They are one of 5 different server classes. BundleContext bc, DictionaryString, String props = new HashtableString, String(1); props.put(“ID”, “ID001”); bc.registerService(ServiceClass, serviceObject, createProperties(props)); Service Look up: BundleContext bc, bc.getServiceReferences(ServiceClass, (ID=ID001)”); Thank you, Stanley From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of BJ Hargrave Sent: Friday, May 04, 2012 10:19 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow Equinox also indexes by objectClass alone. So I am not sure what the discrepancy is here. Would be nice to have the test case code to analyze. Stanley, can you post a gist with the code? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliancehttp://www.osgi.org/ hargr...@us.ibm.commailto:hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From:Richard S. Hall he...@ungoverned.orgmailto:he...@ungoverned.org To:Equinox development mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date:2012/05/04 13:16 Subject:Re: [equinox-dev] Service Lookup by GUID very Slow Sent by: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org Just a side comment... On the Felix framework, it is technically possible to index services on arbitrary service properties, but we don't provide any configuration properties to do so. By default, we only index on objectClass, which I assume Equinox does as well. If all of your services have the same objectClass, then it will regress to a linear search. There is no other magic to make it faster in Felix. I would expect something similar in Equinox. If that is not the case, then yeah it sounds like there is an issue. - richard On 5/4/12 12:41 , stanley_p...@dell.commailto:stanley_p...@dell.com wrote: Tom, You are right on. I am using a simple filter. We just added a GUID property to each service. Two follow up questions: -We ran the same tests on Felix and Knopflerfish and get 100ms response time. This is about 50X. I am wondering there may be something wrong in the environment. Do you think JVM settings like Perm generation size helps? I have Xmx=2GB, Xms=1GB and XXMaxPermSize=64MB. -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Thank you, Stanley From: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Thomas Watson Sent: Friday, May 04, 2012 5:40 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow I was also not sure what you meant by GUID. After some thought I think you probably mean the service id or perhaps the service pid (service.id and service.pid properties)? And by lookup I assume you are using some kind of service filter, for example (service.id=23) with a call to BundleContext.getServiceReferences. I will say that the service registry is not optimized for this kind of lookup. You are far better keeping your own data structure that optimizes the lookup over the set of service references and indexes on the keys that you want to use to lookup service references. This can easily be done with a ServiceTrackerCustomizer. Tom [cid:image001.gif@01CD2C3A.DBC24640]BJ Hargrave---05/03/2012 10:04:05 PM---What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specif From: BJ Hargrave/Austin/IBM@IBMUS To: Equinox development mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date: 05/03/2012 10:04 PM Subject: Re: [equinox-dev] Service Lookup by GUID very Slow What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specifics of your lookup? Such as the code snippet? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO
Re: [equinox-dev] Service Lookup by GUID very Slow
I was also not sure what you meant by GUID. After some thought I think you probably mean the service id or perhaps the service pid (service.id and service.pid properties)? And by lookup I assume you are using some kind of service filter, for example (service.id=23) with a call to BundleContext.getServiceReferences. I will say that the service registry is not optimized for this kind of lookup. You are far better keeping your own data structure that optimizes the lookup over the set of service references and indexes on the keys that you want to use to lookup service references. This can easily be done with a ServiceTrackerCustomizer. Tom From: BJ Hargrave/Austin/IBM@IBMUS To: Equinox development mailing list equinox-dev@eclipse.org, Date: 05/03/2012 10:04 PM Subject:Re: [equinox-dev] Service Lookup by GUID very Slow What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specifics of your lookup? Such as the code snippet? -- BJ Hargrave Senior Technical Staff office: +1 386 848 Member, IBM 1781 OSGi Fellow and CTO of the mobile: +1 386 848 OSGi Alliance 3788 hargr...@us.ibm.com From:stanley_p...@dell.com To:equinox-dev@eclipse.org, Date:2012/05/03 16:54 Subject:[equinox-dev] Service Lookup by GUID very Slow Sent by:equinox-dev-boun...@eclipse.org In an experiment to have 200K of services registered, the service lookup by GUID is exceedingly slow – more the 4 seconds per lookup. There are enough RAM (8G) and heap (2G) allocated. What would be the reason of the slowness of the lookup? Any settings to start the framework to improve this? Thanks, Stanley___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev inline: graycol.gifinline: ecblank.gif___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow
Tom, You are right on. I am using a simple filter. We just added a GUID property to each service. Two follow up questions: -We ran the same tests on Felix and Knopflerfish and get 100ms response time. This is about 50X. I am wondering there may be something wrong in the environment. Do you think JVM settings like Perm generation size helps? I have Xmx=2GB, Xms=1GB and XXMaxPermSize=64MB. -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Thank you, Stanley From: equinox-dev-boun...@eclipse.org [mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Thomas Watson Sent: Friday, May 04, 2012 5:40 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow I was also not sure what you meant by GUID. After some thought I think you probably mean the service id or perhaps the service pid (service.id and service.pid properties)? And by lookup I assume you are using some kind of service filter, for example (service.id=23) with a call to BundleContext.getServiceReferences. I will say that the service registry is not optimized for this kind of lookup. You are far better keeping your own data structure that optimizes the lookup over the set of service references and indexes on the keys that you want to use to lookup service references. This can easily be done with a ServiceTrackerCustomizer. Tom [cid:image001.gif@01CD29D9.E85A0C70]BJ Hargrave---05/03/2012 10:04:05 PM---What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specif From: BJ Hargrave/Austin/IBM@IBMUS To: Equinox development mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date: 05/03/2012 10:04 PM Subject: Re: [equinox-dev] Service Lookup by GUID very Slow What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specifics of your lookup? Such as the code snippet? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliancehttp://www.osgi.org/ hargr...@us.ibm.commailto:hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From:stanley_p...@dell.commailto:stanley_p...@dell.com To:equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org, Date:2012/05/03 16:54 Subject:[equinox-dev] Service Lookup by GUID very Slow Sent by: equinox-dev-boun...@eclipse.orgmailto:equinox-dev-boun...@eclipse.org In an experiment to have 200K of services registered, the service lookup by GUID is exceedingly slow – more the 4 seconds per lookup. There are enough RAM (8G) and heap (2G) allocated. What would be the reason of the slowness of the lookup? Any settings to start the framework to improve this? Thanks, Stanley___ equinox-dev mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.orgmailto:equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev inline: image001.gifinline: image003.pnginline: image004.png___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow
Just a side comment...
On the Felix framework, it is technically possible to index services
on arbitrary service properties, but we don't provide any
configuration properties to do so. By default, we only index on
objectClass, which I assume Equinox does as well.
If all of your services have the same objectClass, then it will
regress to a linear search. There is no other magic to make it
faster in Felix. I would expect something similar in Equinox. If
that is not the case, then yeah it sounds like there is an issue.
- richard
On 5/4/12 12:41 , stanley_p...@dell.com wrote:
Tom,
You
are right on. I am using a simple filter. We just added a
GUID property to each service. Two follow up questions:
- We
ran the same tests on Felix and Knopflerfish and get 100ms
response time. This is about 50X. I am wondering there may
be something wrong in the environment. Do you think JVM
settings like Perm generation size helps? I have Xmx=2GB,
Xms=1GB and XXMaxPermSize=64MB.
- Do
you think lazy service creation may be the reason? Is lazy
creation the default? How to configure it?
- Can
you outline the steps to use ServiceTrackerCustomizer
to build index? Do you mean
trapping the registration events and build the needed
indexes?
Thank
you,
Stanley
From:
equinox-dev-boun...@eclipse.org
[mailto:equinox-dev-boun...@eclipse.org] On Behalf
Of Thomas Watson
Sent: Friday, May 04, 2012 5:40 AM
To: Equinox development mailing list
Subject: Re: [equinox-dev] Service Lookup by GUID
very Slow
I
was also not sure what you meant by GUID. After some
thought I think you probably mean the service id or perhaps
the service pid (service.id and service.pid properties)?
And
by lookup I assume you are using some kind of service
filter, for example "(service.id=23)" with a call to
BundleContext.getServiceReferences. I will say that the
service registry is not optimized for this kind of lookup.
You are far better keeping your own data structure that
optimizes the lookup over the set of service references and
indexes on the keys that you want to use to lookup service
references. This can easily be done with a
ServiceTrackerCustomizer.
Tom
BJ
Hargrave---05/03/2012 10:04:05 PM---What is service lookup
by GUID? Services don't have globally unique identifiers.
Can you provide more information on the specif
From:
BJ
Hargrave/Austin/IBM@IBMUS
To:
Equinox
development mailing list equinox-dev@eclipse.org,
Date:
05/03/2012
10:04 PM
Subject:
Re:
[equinox-dev] Service Lookup by GUID very Slow
What
is service lookup by GUID? Services don't have globally
unique identifiers. Can you provide more information on the
specifics of your lookup? Such as the code snippet?
--
BJ
Hargrave
Senior Technical Staff Member, IBM
OSGi Fellow and CTO of the OSGi
Alliance
hargr...@us.ibm.com
office: +1 386 848 1781
mobile: +1 386 848 3788
From:stanley_p...@dell.com
To:equinox-dev@eclipse.org,
Dat
Re: [equinox-dev] Service Lookup by GUID very Slow
Equinox also indexes by objectClass alone. So I am not sure what the discrepancy is here. Would be nice to have the test case code to analyze. Stanley, can you post a gist with the code? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliance hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From: Richard S. Hall he...@ungoverned.org To: Equinox development mailing list equinox-dev@eclipse.org, Date: 2012/05/04 13:16 Subject:Re: [equinox-dev] Service Lookup by GUID very Slow Sent by:equinox-dev-boun...@eclipse.org Just a side comment... On the Felix framework, it is technically possible to index services on arbitrary service properties, but we don't provide any configuration properties to do so. By default, we only index on objectClass, which I assume Equinox does as well. If all of your services have the same objectClass, then it will regress to a linear search. There is no other magic to make it faster in Felix. I would expect something similar in Equinox. If that is not the case, then yeah it sounds like there is an issue. - richard On 5/4/12 12:41 , stanley_p...@dell.com wrote: Tom, You are right on. I am using a simple filter. We just added a GUID property to each service. Two follow up questions: -We ran the same tests on Felix and Knopflerfish and get 100ms response time. This is about 50X. I am wondering there may be something wrong in the environment. Do you think JVM settings like Perm generation size helps? I have Xmx=2GB, Xms=1GB and XXMaxPermSize=64MB. -Do you think lazy service creation may be the reason? Is lazy creation the default? How to configure it? -Can you outline the steps to use ServiceTrackerCustomizer to build index? Do you mean trapping the registration events and build the needed indexes? Thank you, Stanley From: equinox-dev-boun...@eclipse.org [ mailto:equinox-dev-boun...@eclipse.org] On Behalf Of Thomas Watson Sent: Friday, May 04, 2012 5:40 AM To: Equinox development mailing list Subject: Re: [equinox-dev] Service Lookup by GUID very Slow I was also not sure what you meant by GUID. After some thought I think you probably mean the service id or perhaps the service pid (service.id and service.pid properties)? And by lookup I assume you are using some kind of service filter, for example (service.id=23) with a call to BundleContext.getServiceReferences. I will say that the service registry is not optimized for this kind of lookup. You are far better keeping your own data structure that optimizes the lookup over the set of service references and indexes on the keys that you want to use to lookup service references. This can easily be done with a ServiceTrackerCustomizer. Tom BJ Hargrave---05/03/2012 10:04:05 PM---What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specif From: BJ Hargrave/Austin/IBM@IBMUS To: Equinox development mailing list equinox-dev@eclipse.org, Date: 05/03/2012 10:04 PM Subject: Re: [equinox-dev] Service Lookup by GUID very Slow What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specifics of your lookup? Such as the code snippet? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliance hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From:stanley_p...@dell.com To:equinox-dev@eclipse.org, Date:2012/05/03 16:54 Subject:[equinox-dev] Service Lookup by GUID very Slow Sent by:equinox-dev-boun...@eclipse.org In an experiment to have 200K of services registered, the service lookup by GUID is exceedingly slow – more the 4 seconds per lookup. There are enough RAM (8G) and heap (2G) allocated. What would be the reason of the slowness of the lookup? Any settings to start the framework to improve this? Thanks, Stanley___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev image/gifimage/pngimage/pngimage/pngimage/pngimage/pngimage/pngimage/pngimage/png___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
Re: [equinox-dev] Service Lookup by GUID very Slow
What is service lookup by GUID? Services don't have globally unique identifiers. Can you provide more information on the specifics of your lookup? Such as the code snippet? -- BJ Hargrave Senior Technical Staff Member, IBM OSGi Fellow and CTO of the OSGi Alliance hargr...@us.ibm.com office: +1 386 848 1781 mobile: +1 386 848 3788 From: stanley_p...@dell.com To: equinox-dev@eclipse.org, Date: 2012/05/03 16:54 Subject:[equinox-dev] Service Lookup by GUID very Slow Sent by:equinox-dev-boun...@eclipse.org In an experiment to have 200K of services registered, the service lookup by GUID is exceedingly slow – more the 4 seconds per lookup. There are enough RAM (8G) and heap (2G) allocated. What would be the reason of the slowness of the lookup? Any settings to start the framework to improve this? Thanks, Stanley___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev ___ equinox-dev mailing list equinox-dev@eclipse.org https://dev.eclipse.org/mailman/listinfo/equinox-dev
