Well I was able to find the PDF 1.3 reference (@ pdfzone - google is Great) and I'm just running a huge test. I was able to process 100 pages with a flat memory profile and 2Mb of heap so it's looking pretty good, but my Big Test has destroyed all pretenders to date so we'll see -- it's running now.
However I now believe that the PDF I'm generating is not quite 100% of the kosher variety. I delay writing the /Pages object until the end of the document (I have a list of PDFReferences as the kids) and that's all fine and seems to work OK in xpdf. However the problem is that each /Page object is apparently supposed to refer to it's parent /Pages object. I can't see how I can have a link to a Pages object that's not written yet, with an object reference I don't know yet, but apparently /Parent is mandatory. How can I avoid writing the list of Pages until the end of the document if each page has to refer to that list? At the moment there is a /Parent but the object ID is -1. (What is the correct term for the number in an indirect reference?)
The spec says that what I want to do is possible but not how (that I can find); I was wondering if someone with more experience with PDF might be able to help me. As I understand it there can be multiple /Pages objects and I was wondering if that's somehow what I'm supposed to use. But as Keiron suggested, I read the bits of the spec that seemed important and now I have a headache. >:-}
P.S. my test document has written 11K pages of 18K so far.. looking good!