Norbet Reilly a écrit :
Hi everyone,
Thanks for my inclusion to the project as a "mentored commiter" (I
didn't mind "proby" myself ;-), it's an honour to be involved with
such an exciting opensource project.
I was just wanting to ask one of the ADS gurus what happens exactly at
the LDAP protocol level when an extremely large result set is returned
by ADS for an LDAP search(). Is a client able to start retreiving
results via their NamingEnumeration before the server has completed
its search?
Yep. We don't build the full result first, and then send it to the
client. In fact, an iterator is created,
and while iterating the result, some entries are pushed back to the
client as soon as they are fetched.
I'm hoping there is some sort of "chunking" at the
protocol level so that batches of results are streamed to the client
as they become available, rather then the waiting for the entire
result set to be available first. Is this the case?
It is :) (Alex can confirm)
But there is something that can be done to improve this mechanism : Big
objects like jpeg image, for instance, are loaded in memory and sent
back to the client. This is dangerous for the server, because if you
have more than one client requesting for more than one image, then you
can exhaust the memory. Especially if you consider caching, in which
case entries are in memory, including their attributes.
Even worst, they will use twice the memory needed to store this image :
one as an Attribute, and one as a byte[] in the SearchResultEntry
encoded PDU that will be sent back to the client. This is also the case
when the cleint send this object when adding it or modifying it.
We have thought of using a streaming mechanism allowing us to avoid to
keep such objects in memory, with special accessors to get chunks of
byte[] from a file instead of loading the full object in memory, but we
never had time to build it.
Obviously, this is something we need. It goes deep into the server, but
not that much. From Ldap point of view, we just need to modify the
LockableAttributeImpl object to deal with a huge attribute, and in this
case, stream it to a temporary location on the disk (let say that if an
attribute length is longer than 1024 bytes - or char if it's a String -,
then a temp file will be used. The trikiest part will be MINA. we are
building a ByteBuffer to store the result to send, which will contain
the full block of data. What can be done is to modify the way we
cooperate with MINA to send chunks of data - 4096 bytes long? - and
continue until all the baytes have been sent.
Here is a good piece of interesting work to be done :)
If you are interested, well, we also are to have it into the server! So
just don't be afraid, this is not that complicated. We can provide
assistance and guidance if needed.
Emmanuel
Thanks
