Hi Carsten, 2016-07-21 7:03 GMT+02:00 Carsten Ziegeler <[email protected]>:
> Hi, > > I'm not against adding such a method in general, however there are some > things to consider: > Usually you should provide a limit, unfortunately that limit can't be > passed down to the providers. I think the limit can be implemented in the loop that gets the items out of the iterator (in the code consuming the iterator). Since the down providers also provide iterators (and it seems they implement those iterators properly without loading the whole content in memory at any point), everything's in place to allow supporting a memory efficient limit mechanism. In order to have an API which allows this > I suggest a query API to be added to the resource provider. If we would > have this, we wouldn't need a new method - given that the limit is set > to a sensible value. > Do you mean that the current consumers of JobManager#findJobs would use o.a.s.s.r.p.QueryLanguageProvider#findResources instead ? This would work, already now IIUC, but it would require the current consumers of JobManager#findJobs to know about the JobManager implementation (content structure). > > But more important, if you compare Sling jobs to other messaging/queuing > systems, these other solutions have no findJobs method. It's not > possible to get the contents of the queue. I think Ian wrote some very > good analysis some time ago and create a prototype for a new messaging api. > And this means we shouldn't have added findJobs in the first place, > therefore I'm more thinking of deprecating such functionality completely. > I understand the issue with giving the ability to search for jobs in an unbounded queues. This streaming idea was an attempt to mitigate one of those issues (memory consumption). If the plan is to soon deprecate the method completely, there's no need for a streaming version. However I don't see a direct alternative to this method except using the more generic QueryLanguageProvider#findResources which would require the consumer to know about the JobManager internals. Or would we deprecate without alternative (which I think makes sense from Sling PoV) ? Regards, Timothee > Regards > Carsten > > > Hi, > > > > In our platform, we use the o.a.s.e.j.JobManager API signature: > > > > Collection<Job> findJobs(QueryType type, String topic, long limit, > > Map<String, Object>... templates); > > > > This signature and its implementation are fine except in cases where the > > number of returned jobs is very large. > > In those cases, the implementation can run the instance OOM because the > > heap is consumed by the collection of jobs to be returned. > > > > Instead of building the whole list in memory, we could instead stream the > > set of jobs to be returned, thus allowing the API consumer to avoid the > OOM. > > This streaming behaviour would only be efficient if all the calls > > implementing the signature do stream the result all the way down. > > > > Looking at the JobManager#findJobs implementation, it seems this is the > > case. > > > > So, I suggest we add a method signature in JobManager API that allows to > > return an Iterator or an Iterable. > > > > Adding a signature to the JobManager would be a backward compatible > change > > as the JobManager is a provider type. > > > > wdyt ? > > > > Regards, > > > > Timothee > > > > > > > -- > Carsten Ziegeler > Adobe Research Switzerland > [email protected] > >
