[
https://issues.apache.org/jira/browse/DIRSERVER-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588377#action_12588377
]
nrhope edited comment on DIRSERVER-1161 at 4/13/08 8:58 PM:
-----------------------------------------------------------------
Ok, these attached files represent a patch to ApacheDS 1.5.1 (release tag
http://svn.apache.org/repos/asf/directory/apacheds/releases/1.5.1) and latest
MINA 1.1 trunk (http://svn.apache.org/repos/asf/mina/branches/1.1 at 647053)
which passes the "mvn -Dintegration test" on AD and as well my own test suites
working at a higher level).
Note this patch also cover another leak I noticed in SearchHandler when a
search returns no results (refer to these lines):
// stop leak of message for each search returning no results
SessionRegistry.getSingleton().removeOutstandingRequest(
session, req.getMessageId() );
The changes are discussed in early comments / attachments this issue but are
summarized again here:
1. The major problem is that no data is streamed to the JNDI client
(and consequently the queue on the ExecutorFilter.SessionBuffer grows
in an unbounded fashion) until SearchHandler.messageRecieved()
returns. Hence it needs to return as soon as a NamingEnumeration over
the results to be returned is available, meaning a separate thread
needs to be used to drain the SearchResponseIterator asynchronously. Another
solution mentioned by Emmanuel would be to force the session to be drained by
the client for each result, but I think forcing the server and client to move
in such strict lock-step would not be the best approach scalability-wise (and
would be harder to implement too).
2. The next problem is the potential for a difference in the client
/ server speed. My patch uses a
LinkedBlockingQueue and this approach worked fine. The problems / notes are:
a. I did my patching against MINA 1.1 trunk (1.1.7-SNAPSHOT) so I
could get Trustin's feedback, but I gather you guys are aiming at
1.1.6 for your upcoming AD 1.5.2 release
b. given Trustin's feedback I'm not sure how "onboard" he is going
to be this the blocking queue approach, but it seems a simple and
clean solution to me
c. configuring the size of the blocking queue (or using a
non-blocking queue if this needs be kept as an
option for other MINA usages) is difficult as SessionBuffer is a static class in
ExecutorFilter and therefore can't be configured via ExecutorFilter's
constructor. I have used a system property as the best solution I can
think of at this stage.
was (Author: nrhope):
Ok, these attached files represent a patch to ApacheDS 1.5.1 (release tag
http://svn.apache.org/repos/asf/directory/apacheds/releases/1.5.1) and latest
MINA 1.1 trunk (http://svn.apache.org/repos/asf/mina/branches/1.1 at 647053)
which passes the "mvn -Dintegration test" on AD and as well my own test suites
working at a higher level).
The changes are discussed in early comments / attachments this issue but are
summarized again here:
1. The major problem is that no data is streamed to the JNDI client
(and consequently the queue on the ExecutorFilter.SessionBuffer grows
in an unbounded fashion) until SearchHandler.messageRecieved()
returns. Hence it needs to return as soon as a NamingEnumeration over
the results to be returned is available, meaning a separate thread
needs to be used to drain the SearchResponseIterator asynchronously. Another
solution mentioned by Emmanuel would be to force the session to be drained by
the client for each result, but I think forcing the server and client to move
in such strict lock-step would not be the best approach scalability-wise (and
would be harder to implement too).
2. The next problem is the potential for a difference in the client
/ server speed. My patch uses a
LinkedBlockingQueue and this approach worked fine. The problems / notes are:
a. I did my patching against MINA 1.1 trunk (1.1.7-SNAPSHOT) so I
could get Trustin's feedback, but I gather you guys are aiming at
1.1.6 for your upcoming AD 1.5.2 release
b. given Trustin's feedback I'm not sure how "onboard" he is going
to be this the blocking queue approach, but it seems a simple and
clean solution to me
c. configuring the size of the blocking queue (or using a
non-blocking queue if this needs be kept as an
option for other MINA usages) is difficult as SessionBuffer is a static class in
ExecutorFilter and therefore can't be configured via ExecutorFilter's
constructor. I have used a system property as the best solution I can
think of at this stage.
> search results are not streamed to the client until final done response is
> queued
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1161
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1161
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: JDK 1.5.0_11
> Reporter: Norval Hope
> Attachments: apacheds_1.5.1_streaming.patch,
> apacheds_1.5.1_streaming.patch, apacheds_1.5.1_streaming_log_output.txt,
> installers_1.5.1_streaming.patch, installers_1.5.1_streaming.patch,
> mina_1.1.2_streaming.patch, mina_1.1_trunk_streaming.patch, pom.xml,
> streaming_log_output.txt, streaming_logging.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> Search results accumulate in Events on the SessionBuffer.eventQueue within
> ExecutorFilter.fireEvent() until final done response for the search is
> written to the session and then all results for the search (possibly millions
> depending on the search and state of the directory) are written out at once.
> This is a big problem for scalability and I gather from previous
> correspondence with Alex that this behaviour is unexpected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
