For handling large datasets I would asume you build something like "Lazy loading" manually in your code. On client side you usually call that "paging" As a dataset is just another representation of an object I would not stick to XML in general, the data source can be almost anything.
Practically you would initialize for example an array that has the length of the complete "dataset" but only fill those nodes of the array that are really visible in the UI. For example in Adobe Flex grid work the same, you initialize the whole length but you fill the data in the nodes only when needed. So you would effectively only load 10-20 records by each RPC call. In the UI for rendering instead of initializing every view with data you would use some kind of pooling and only change the content of the views instead of re-rendering for every new data. Sebastian 2012/4/2 Justin Ellis <[email protected]> > Hello all, > > I am writing a document management application that has to deal with a > very large number of entries and related tables. Basically, I have a > database that contains a "Documents" table, which may have several > hundred thousand entries, that has a number of relationships to > various other tables, such as "Authors," which is a many-to-many > connection with "Persons," and a one-to-many connection with "Pages" > (each document may contain anywhere from one up to hundreds of pages). > > My problem is that I am trying to display all of this information in a > number of ways, including a spreadsheet-type format that would show > one line per document, along with the author and page information, but > I am having a hard time figuring out an efficient way of getting the > data into an XML format for OL. Almost none of the OL or middle layer > examples I have come across deal with nested XML elements; I have > included an example of how I am trying to format the XML at the end of > this email. > > I was using solely Struts2 for the middle layer, but was having a hard > time representing all of the relationships. I switched to a > combination of Struts2 and Hibernate, and wrote my own Struts2 result > to format the XML, but the process is prohibitively slow and memory > hungry, mostly for storing the data in the persistence layer, but also > somewhat for formatting the XML. I am looking into using paging, but > I have not found any good OL examples for that, nor have I found any > good examples in general for dealing with a large amount of > information that requires nested XML elements like the ones I describe > above. > > Can someone please help me get on the right track? Links to examples > would be great, including suggestions for better technologies or > better ways to format my data. If simple paging will solve my issue, > I would be grateful for information on how to plumb OL together with > the middle layer to accomplish this. If others have worked on > applications dealing with this kind of data volume (hundreds of > thousands of entries, with anywhere from five to a couple hundred > related entries each), I would welcome any insight or advice you might > have. > > Example XML for one document: > > <Document Id="1" Title="Example Doc Title" Date="April 2, 2012"> > <Author Id="1" FirstName="John" LastName="Smith"/> > <Author Id="2" FirstName="John" LastName="Doe"/> > <Page Id="1" ImageFile="documents/image1.tif"/> > <Page Id="2" ImageFile="documents/image2.tif"/> > <Page Id="3" ImageFile="documents/image3.tif"/> > </Document> > > > Thank you! > > -- > Justin Ellis > 646-783-9387 > Fax: 866-448-6503 > [email protected] > www.LightBulbLaw.com > -- Sebastian Wagner https://twitter.com/#!/dead_lock http://www.openmeetings.de http://www.webbase-design.de http://www.wagner-sebastian.com [email protected]
