>At one point I actually rewrote the Shapefile parser to be streaming as well...

I guess I hadn't actually realized it wasn't until now.  Do you
remember what happened to the code, or why you didn't stay with the
streaming version?  A very large shape file seems like a more likely
scenario that I actually care about.

Larry
On 8/30/07, Martin Davis <[EMAIL PROTECTED]> wrote:
> This would be especially important if you had multiple
> FeatureCollections stored in one file, and you wanted to load only one
> of them.
>
> Sunburned Surveyor wrote:
> > Paul is correct. The pull parser does not reduce the memory of the
> > parsing results, but it does reduce the memory used during the parsing
> > process. That is because an in-memory representation of the entire XML
> > document is not constructed.
> >
> > One advantage of this is using the parser to select only data within
> > the XML file that meets specific criteria. For example, if we had a
> > 50MB SGF file representing the city of Stockton, I could parse the
> > file and create only building features, even thought the file might
> > contain road features, landmark features, park features. etc.
> > In fact, I could even parse the file and only create features for
> > buildings whose "building type" attribute was set to "Public". This
> > allows me to extract the information I want without reading all 50 MB
> > into memory.
> >
> > The Sunburned Surveyor
> >
> > On 8/30/07, Paul Austin <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Larry,
> >>
> >> You are correct that the resulting data set will take up a lot of memory
> >> at the end, the advantage with the pull parser is that you don't take up
> >> a whole bunch of extra memory for the XML DOM structures which typically
> >> get loaded into memory for the whole document. So with the pull parser
> >> there is little memory overhead where as for DOM you have probably at
> >> least 2x memory required to load if not more
> >>
> >> Paul
> >>
> >> Larry Becker wrote:
> >>
> >>> It isn't the parser that takes up the memory except temporarily), but
> >>> the memory resident dataset after loading.  This will still limit the
> >>> size.
> >>>
> >>> Larry
> >>>
> >>> On 8/30/07, Sunburned Surveyor <[EMAIL PROTECTED]> wrote:
> >>>
> >>>
> >>>> Yup. It makes you wonder why they didn't use pull parsers from the
> >>>> very beginning, doesn't it.
> >>>>
> >>>> SS
> >>>>
> >>>> On 8/30/07, Paul Austin <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>
> >>>>> Agreed the pull parser is the only way to go for large XML files
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>> Sunburned Surveyor wrote:
> >>>>>
> >>>>>
> >>>>>> Martin,
> >>>>>>
> >>>>>> If we decide to support a restricted form of GML 2 we could build our
> >>>>>> reader and writer on top of the XML Pull Parser from Sun. This would
> >>>>>> help us to avoid memory problems when reading in large files.
> >>>>>>
> >>>>>> https://sjsxp.dev.java.net/
> >>>>>>
> >>>>>> Just a thought.
> >>>>>>
> >>>>>> The Sunburned Surveyor
> >>>>>>
> >>>>>> -------------------------------------------------------------------------
> >>>>>> This SF.net email is sponsored by: Splunk Inc.
> >>>>>> Still grepping through log files to find problems?  Stop.
> >>>>>> Now Search log events and configuration files using AJAX and a browser.
> >>>>>> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> >>>>>> _______________________________________________
> >>>>>> Jump-pilot-devel mailing list
> >>>>>> Jump-pilot-devel@lists.sourceforge.net
> >>>>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> -------------------------------------------------------------------------
> >>>>> This SF.net email is sponsored by: Splunk Inc.
> >>>>> Still grepping through log files to find problems?  Stop.
> >>>>> Now Search log events and configuration files using AJAX and a browser.
> >>>>> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> >>>>> _______________________________________________
> >>>>> Jump-pilot-devel mailing list
> >>>>> Jump-pilot-devel@lists.sourceforge.net
> >>>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> >>>>>
> >>>>>
> >>>>>
> >>>> -------------------------------------------------------------------------
> >>>> This SF.net email is sponsored by: Splunk Inc.
> >>>> Still grepping through log files to find problems?  Stop.
> >>>> Now Search log events and configuration files using AJAX and a browser.
> >>>> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> >>>> _______________________________________________
> >>>> Jump-pilot-devel mailing list
> >>>> Jump-pilot-devel@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >> -------------------------------------------------------------------------
> >> This SF.net email is sponsored by: Splunk Inc.
> >> Still grepping through log files to find problems?  Stop.
> >> Now Search log events and configuration files using AJAX and a browser.
> >> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> >> _______________________________________________
> >> Jump-pilot-devel mailing list
> >> Jump-pilot-devel@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> >>
> >>
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > _______________________________________________
> > Jump-pilot-devel mailing list
> > Jump-pilot-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> >
> >
>
> --
> Martin Davis
> Senior Technical Architect
> Refractions Research, Inc.
> (250) 383-3022
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Jump-pilot-devel mailing list
> Jump-pilot-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
>


-- 
http://amusingprogrammer.blogspot.com/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel

Reply via email to