On 22 Nov 2010, at 02:17, Justin Edelson wrote:
> It should *not* be the case that requesting
> /system/userManager/user/<userid>.json causes listChildren() to be
> called. If it does, that's a bug (but eyeballing the code, this doesn't
> seem possible). Requesting /system/userManager/user.2.json will cause
> listChildren() to be called.
at
org.sakaiproject.nakamura.lite.authorizable.AuthorizableManagerImpl.findAuthorizable(AuthorizableManagerImpl.java:290)
at
org.sakaiproject.nakamura.api.lite.jackrabbit.SparseMapUserManager.findAuthorizables(SparseMapUserManager.java:110)
at
org.apache.jackrabbit.core.security.principal.DynamicPrincipalProvider.findUserPrincipals(DynamicPrincipalProvider.java:346)
at
org.apache.jackrabbit.core.security.principal.DynamicPrincipalProvider.findPrincipals(DynamicPrincipalProvider.java:194)
at
org.apache.jackrabbit.core.security.principal.DynamicPrincipalProvider.getPrincipals(DynamicPrincipalProvider.java:217)
at
org.apache.jackrabbit.core.security.principal.PrincipalManagerImpl.getPrincipals(PrincipalManagerImpl.java:114)
at
org.sakaiproject.nakamura.user.resource.SakaiAuthorizableResourceProvider.listChildren(SakaiAuthorizableResourceProvider.java:182)
at
org.apache.sling.jcr.resource.internal.helper.WrappedResourceProvider.listChildren(WrappedResourceProvider.java:65)
at
org.apache.sling.jcr.resource.internal.helper.ResourceProviderEntry$1.seek(ResourceProviderEntry.java:206)
at
org.apache.sling.jcr.resource.internal.helper.ResourceProviderEntry$1.<init>(ResourceProviderEntry.java:179)
at
org.apache.sling.jcr.resource.internal.helper.ResourceProviderEntry.listChildren(ResourceProviderEntry.java:145)
at
org.apache.sling.jcr.resource.internal.JcrResourceResolver.listChildren(JcrResourceResolver.java:819)
at
org.apache.sling.jcr.resource.internal.JcrResourceResolver.getChildInternal(JcrResourceResolver.java:1101)
at
org.apache.sling.jcr.resource.internal.JcrResourceResolver.resolveInternal(JcrResourceResolver.java:1036)
at
org.apache.sling.jcr.resource.internal.JcrResourceResolver.resolve(JcrResourceResolver.java:463)
at
org.apache.sling.engine.impl.request.RequestData.initResource(RequestData.java:196)
at
org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:302)
at
org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:207)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
the SakaiAuthorizableRespourceProvider.listChildren method is identical to the
standard AuthrizableResourceProvider
>
> In terms of your larger point, ResourceProvider.listChildren() returns
> an Iterator and AuthorizableResourceProvider.listChildren() return value
> simply wraps the result of getPrincipals(), so in and of itself, this
> shouldn't exhaust memory.
Agreed, and it looks like the iterator is not used, so the only expense is
executing the query which provided the lucene/jackrabbit batch size is small
and the result set is dense from an ACL point of view there should not be a
problem. However if the result set is sparse, and the first entry is retrieved
from the iterator that will cause many items to enter the shared item cache and
block all other operations that need the persistence manager.
>
> What is probably necessary is to put some guard in place so that only X
> number of items are returned and if more than X results are available,
> support some kind of paging (I don't know what X should be). This would
> be similar to SLING-1308. Actually, why isn't the code from SLING-1308
> being invoked in your case? That seems very suspicious.
I have a feeling that infinity on the parent path will invoke listChildren
without limit, which will exercise the iterator.
>
> Sorry to hear about your bad release.
One of the dangers of making a platform too flexible and releasing it to
inexperienced hands, there were some hugely complex JCR Queries written that
had no hope of scaling to more than a few users or items of data, but I also
think that our use cases are not well aligned with those of Jackrabbit and
Enterprise Content Management in general.
Ian
>
> Justin
>
>
> On 11/21/10 4:05 PM, Ian Boston wrote:
>> Hi,
>> In AuthorizableResourceProvider there is a call to
>> principalManager.getPrincipals(searchType);
>>
>> IIUC that lists all authorizables that match the search type. Eg, all users
>> or all groups or all authorizables.
>>
>> This is probably Ok with 100 users, but not so ok with 25K users and 40K
>> groups, as its called every time to do a call to
>> http://localhost:8080/system/userManager/user/<userid>.json
>>
>> Did I understand correctly or am I reading the code incorrectly ?
>>
>> I think the stack trace is something like
>> AuthorizableResourceProvider.listChildren
>> PrincipalManagerImpl.getPrincipals
>> DefaultPrincipalProvider.getPrincipals
>> DefaultPrincipalProvider.findPrincipals with a null search filter (ie all
>> the known Principals matching that type)
>>
>> Obviously AuthorizableResourceProvider.listChildren *should* list all
>> Authorizables, but since in most systems that identify the users, there
>> could be millions, this is probably not a good idea.
>>
>> AuthorizableResourceProvider should probably not list children as doing so
>> will almost certainly crash the server for anything other than a trivial
>> case.
>>
>> WDYT?
>> Ian
>>
>>
>> BTW, I must apologise for not doing much work on Sling in the last month,
>> our release did not go smoothy and I have been firefighting. (fire still
>> smouldering)
>>
>
