Of course, the counter argument to that is: Use a format that's designed to handle that reasonably. XML isn't, so why use it if its not an optimal (or even reasonable) format to use for this kind of thing?
-------------------------- Dean Roddey The Charmed Quark Controller Charmed Quark Software [EMAIL PROTECTED] http://www.charmedquark.com "If it don't have a control port, don't buy it!" ----- Original Message ----- From: "Murphy, James" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, April 23, 2002 2:31 PM Subject: RE: RE: how to access the raw text that generated a sax event > Fair enough Dean - I'm sympathetic to your point that Xerces was designed > from an InfoSet perspective. That's cool - but when you are writing for > performance we are willing to make some Faustian bargains. Especially > since, like Jason our environment stipulates single entities anyway. > > Jim > > > -----Original Message----- > > From: Dean Roddey [mailto:[EMAIL PROTECTED]] > > Sent: Tuesday, April 23, 2002 3:34 PM > > To: [EMAIL PROTECTED] > > Subject: Re: RE: how to access the raw text that generated a sax event > > > > > > The source offset stuff is always relative to the entity, so > > if you have > > internal or external entity references and such, you are > > going to have to > > keep up with that fact. So if a entity reference to an > > internal general > > entity contains elements (and it pretty much has to contain > > whole elements), > > those offsets will be relative to that entity's base, so > > you'd have to have > > access to the entity definitions to get back to the raw stuff. > > > > Also, if you store away raw offsets, it might not make much > > sense to you > > becuase there will be no expansion of entities of any sort in > > that raw data, > > so if you go back and look at it, you will have to be > > reponsible for entity > > expansion if you want to know what it's real content is. And > > what if those > > entities are external? You'll have to go through the DTD ifno > > and find the > > refereences to those external entities and expand those, > > (which have entity > > references in them, etc...) i.e. you'll end up writing a > > small XML parser if > > the data contains much in the way of entity references. > > > > Anyway, the whole concept of getting back to the original raw > > XML text is > > counter to what an XML parser is supposed to do, so its never > > going to be > > easy because it wasn't designed to make that easy or useful > > to do. I always > > argued that we never even put any of that stuff in there, but some big > > customers forced us to at least allow it. But its kludey and it adds > > overhead to the input streaming system to have to keep track > > of how many > > bytes each transcoded XML character eat from the input. > > > > -------------------------- > > Dean Roddey > > The Charmed Quark Controller > > Charmed Quark Software > > [EMAIL PROTECTED] > > http://www.charmedquark.com > > > > "If it don't have a control port, don't buy it!" > > > > > > ----- Original Message ----- > > From: <[EMAIL PROTECTED]> > > To: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> > > Sent: Tuesday, April 23, 2002 11:13 AM > > Subject: Re: RE: how to access the raw text that generated a sax event > > > > > > > > > > The other potential solution I've found is the XMLScanner's > > "getSrcOffset" > > method. My only fear in using it is that it will give weird > > results if an > > XML document is comprised of more than 1 entity. > > > > > > Does "getSrcOffset" treat the document as a continuous > > sequence of bytes, > > or is it more low-level than that? > > > > > > -ted > > > > > > > > > > > > > > From: "Murphy, James" <[EMAIL PROTECTED]> > > > > Date: 2002/04/23 Tue PM 02:01:40 EDT > > > > To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]> > > > > Subject: RE: how to access the raw text that generated a sax event > > > > > > > > Looking through the source... > > > > > > > > BinInputStream::curPos() const; looks promising since the > > built in input > > > > sources actually implement it! So you should be able to > > call this in > > your > > > > SAX event handler methods if you provide your event > > handler class with > > the > > > > InputSource you use to parse. > > > > > > > > I haven't tried it yet but I think it will just work. > > > > > > > > Jim > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED]] > > > > > Sent: Tuesday, April 23, 2002 1:33 PM > > > > > To: [EMAIL PROTECTED] > > > > > Subject: Re: how to access the raw text that generated > > a sax event > > > > > > > > > > > > > > > "Murphy, James" <[EMAIL PROTECTED]> writes: > > > > > > > > > > > I thought this would be really handy when parsing from a > > > > > continuous buffer > > > > > > like a MemBufInputSource or a LocalFileInputSource. I have > > > > > a situation > > > > > > where I SAX parse _very_ large XML instances looking for > > > > > small repeating > > > > > > fragments. These fragments are operated on individually by > > > > > making a DOM to > > > > > > operating on those nodes in all sorts of application > > defined ways. > > > > > > > > > > > > If I had the functionality described by Ted, I could SAX > > > > > the file and save > > > > > > off the starting and ending offsets into the large > > > > > document. Post that info > > > > > > to a thread pool to process the fragments asynchronously. > > > > > In fact, I can > > > > > > use my Win32 memory mapped file input source to SAX the > > > > > original large file > > > > > > and serve as a source to the DOM parser during the per work > > > > > item processing. > > > > > > The way I'm doing it now involved _way_ too many buffer > > > > > copies to be really > > > > > > fast - but it could be. > > > > > > > > > > Hey Jim, > > > > > > > > > > I agree. For the MAGE object model we're going to be > > routinely parsing > > > > > big chunks of scientific data, maybe 0.5 Gb => 2.0 Gb, > > and looking for > > > > > certain pieces of the data. I'd like to be able to do > > lazy parsing, > > > > > and just store the byte offsets to the bits that I want. > > > > > > > > > > There *has* to be some easy modification that we can > > make to subclass > > > > > InputSource or XMLScanner to get this working. I don't > > know enough > > > > > about the internals of how the scanner works, but if > > someone can clue > > > > > me in a bit, I'd be happy to implement this. > > > > > > > > > > jas. > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
