>At one point I actually rewrote the Shapefile parser to be streaming as well...
I guess I hadn't actually realized it wasn't until now. Do you remember what happened to the code, or why you didn't stay with the streaming version? A very large shape file seems like a more likely scenario that I actually care about. Larry On 8/30/07, Martin Davis <[EMAIL PROTECTED]> wrote: > This would be especially important if you had multiple > FeatureCollections stored in one file, and you wanted to load only one > of them. > > Sunburned Surveyor wrote: > > Paul is correct. The pull parser does not reduce the memory of the > > parsing results, but it does reduce the memory used during the parsing > > process. That is because an in-memory representation of the entire XML > > document is not constructed. > > > > One advantage of this is using the parser to select only data within > > the XML file that meets specific criteria. For example, if we had a > > 50MB SGF file representing the city of Stockton, I could parse the > > file and create only building features, even thought the file might > > contain road features, landmark features, park features. etc. > > In fact, I could even parse the file and only create features for > > buildings whose "building type" attribute was set to "Public". This > > allows me to extract the information I want without reading all 50 MB > > into memory. > > > > The Sunburned Surveyor > > > > On 8/30/07, Paul Austin <[EMAIL PROTECTED]> wrote: > > > >> Hi Larry, > >> > >> You are correct that the resulting data set will take up a lot of memory > >> at the end, the advantage with the pull parser is that you don't take up > >> a whole bunch of extra memory for the XML DOM structures which typically > >> get loaded into memory for the whole document. So with the pull parser > >> there is little memory overhead where as for DOM you have probably at > >> least 2x memory required to load if not more > >> > >> Paul > >> > >> Larry Becker wrote: > >> > >>> It isn't the parser that takes up the memory except temporarily), but > >>> the memory resident dataset after loading. This will still limit the > >>> size. > >>> > >>> Larry > >>> > >>> On 8/30/07, Sunburned Surveyor <[EMAIL PROTECTED]> wrote: > >>> > >>> > >>>> Yup. It makes you wonder why they didn't use pull parsers from the > >>>> very beginning, doesn't it. > >>>> > >>>> SS > >>>> > >>>> On 8/30/07, Paul Austin <[EMAIL PROTECTED]> wrote: > >>>> > >>>> > >>>>> Agreed the pull parser is the only way to go for large XML files > >>>>> > >>>>> Paul > >>>>> > >>>>> Sunburned Surveyor wrote: > >>>>> > >>>>> > >>>>>> Martin, > >>>>>> > >>>>>> If we decide to support a restricted form of GML 2 we could build our > >>>>>> reader and writer on top of the XML Pull Parser from Sun. This would > >>>>>> help us to avoid memory problems when reading in large files. > >>>>>> > >>>>>> https://sjsxp.dev.java.net/ > >>>>>> > >>>>>> Just a thought. > >>>>>> > >>>>>> The Sunburned Surveyor > >>>>>> > >>>>>> ------------------------------------------------------------------------- > >>>>>> This SF.net email is sponsored by: Splunk Inc. > >>>>>> Still grepping through log files to find problems? Stop. > >>>>>> Now Search log events and configuration files using AJAX and a browser. > >>>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > >>>>>> _______________________________________________ > >>>>>> Jump-pilot-devel mailing list > >>>>>> Jump-pilot-devel@lists.sourceforge.net > >>>>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > >>>>>> > >>>>>> > >>>>>> > >>>>> ------------------------------------------------------------------------- > >>>>> This SF.net email is sponsored by: Splunk Inc. > >>>>> Still grepping through log files to find problems? Stop. > >>>>> Now Search log events and configuration files using AJAX and a browser. > >>>>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > >>>>> _______________________________________________ > >>>>> Jump-pilot-devel mailing list > >>>>> Jump-pilot-devel@lists.sourceforge.net > >>>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > >>>>> > >>>>> > >>>>> > >>>> ------------------------------------------------------------------------- > >>>> This SF.net email is sponsored by: Splunk Inc. > >>>> Still grepping through log files to find problems? Stop. > >>>> Now Search log events and configuration files using AJAX and a browser. > >>>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > >>>> _______________________________________________ > >>>> Jump-pilot-devel mailing list > >>>> Jump-pilot-devel@lists.sourceforge.net > >>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > >>>> > >>>> > >>>> > >>> > >>> > >> ------------------------------------------------------------------------- > >> This SF.net email is sponsored by: Splunk Inc. > >> Still grepping through log files to find problems? Stop. > >> Now Search log events and configuration files using AJAX and a browser. > >> Download your FREE copy of Splunk now >> http://get.splunk.com/ > >> _______________________________________________ > >> Jump-pilot-devel mailing list > >> Jump-pilot-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > >> > >> > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. > > Still grepping through log files to find problems? Stop. > > Now Search log events and configuration files using AJAX and a browser. > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jump-pilot-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > -- > Martin Davis > Senior Technical Architect > Refractions Research, Inc. > (250) 383-3022 > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > -- http://amusingprogrammer.blogspot.com/ ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
