I should also mention that I'm using JDK 1.5.0_11. On Tue, Apr 8, 2008 at 11:09 PM, Norval Hope <[EMAIL PROTECTED]> wrote:
> As a side note here is an excerpt of the old thread which I mentioned with > a question from me and an answer from Alex: > > NH> I was just wanting to ask one of the ADS gurus what happens exactly at > NH> the LDAP protocol level when an extremely large result set is returned > NH> by ADS for an LDAP search(). Is a client able to start retreiving > NH> results via their NamingEnumeration before the server has completed > NH> its search? I'm hoping there is some sort of "chunking" at the > NH> protocol level so that batches of results are streamed to the client > NH> as they become available, rather then the waiting for the entire > NH> result set to be available first. Is this the case? > > Yes and no :). There is no chunking of an individual PDU sent back. > Meaning each SearchEntryResponse prepared then sent out the door. If an > entry you're returning has a 1 mb jpeg in it then this will be used to > prepare a response and return it rather than streaming out the jpeg. Old > incarnations of MINA used to allow us to do this but it was complicated. > For the search operation, we do not collect all entries before sending a > response. That would kill the server. ApacheDS will pull out and send a > result entry to you one at a time. MINA may lag a little bit and some > results may actually sit a while before going out the door. But for all > practical purposes we pull from the db and pump out results one by one. > The way it works is a bit complicated and will take some time to discuss. > I can go into it if enough people are interested and willing to document it > :) after I describe how she does this. > Alex > > On Tue, Apr 8, 2008 at 11:05 PM, Norval Hope <[EMAIL PROTECTED]> wrote: > > > Hi ApacheDS / MINA-ers, > > > > I am currently doing some load and memory testing against an extension > > made to early version of ApacheDS (after it became 1.5 but before a > > repository restructure that occurred not long after the rename from 1.0). > > What I am finding is that all search result entries are cached in memory due > > to ExecutorFilter.fireEvent() queing Events on the session buffer's event > > queue that refer to them. Only after the SearchResponseIterator queues the > > final SearchResponseDone message and it is encoded and flushed, do any of > > the search results actually get flushed out to my JNDI client (JMeter as it > > happens). As the searches concerned are attempting to return 100k results > > this behaviour leads to a huge heap requirement, and in fact imposes a > > seemingly articificial limit on the total number of results which can be > > returned by a search (I have gone to great lengths to ensure a "streaming" > > approach is used at the level of my custom partitions through the use of > > lazy custom NamingEnumerations). > > > > This seems quite bizarre to me as I can see repeated calls to flush > > encoded search results but these don't seem to have the effect of streaming > > search results incrementally to the client as they become available, which > > was my hope / expectation. The streaming behaviour of MINA / ApacheDS was > > the subject of a query I posed many months ago and the response from the dev > > list at that time was that individual search results would be streamed fine > > but that large attributes like JPEGs may pose a problem as they are not > > currently streamed. I've spent a full day trying to debug what is going on > > between the thread doing the encoding and the thread which recieved the > > original search request and then iterated through the search results, but I > > still can't see how the SearchResponseDone causes all of the results to > > finally be sent when it is eventually written and flushed (and the > > synchronize block on the queue is exited) as the same NIO actions of writing > > and flushing seem to be applied for all the previous search result messages > > too. > > > > Also it seems strange to me that the ExecutorFilter's SessionBuffer > > continues to hold references to an Event matching each search result long > > after that result was encoded and flushed out. Once the search is 100% > > completed and the final search response done message is written, then the > > ProcessEventsRunnable.processEvents() > > method is called for each of these queued events but actually does nothing > > with them. > > > > I was just wanting to check if: > > > > a) this behaviour in the processing of search results has been noticed > > before > > b) if anyone has any tips on debugging the MINA stack in this > > particular case as I've found it extremely hard to get my mind around what > > is actually going on re the individual search results which seem to have > > been flushed out to NIO but then don't get passed on to the client until the > > search is 100% finished > > > > as the MINA code concerned is very subtle and I've not tried to groc it > > before. > > > > Any help / opinions / pointers gratefully accepted. > > > > Thanks, > > Norval > > > >
