[Catalyst] Generating a large XML document (KML file) from a database
I need to output a large XML file (actually, a KML file of points) from a database. I can't find any documentation for Catalyst::View::XML::Generator, or even decent documentation for XML::Generator, on how to do this. It would appear to expect me to put the entire structure in the stash, which I don't want to do, as it could contain thousands of points. Ideally I would pass an iterator function and it would do the rest. ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Generating a large XML document (KML file) from a database
Use the -write() methos in catalyst and XML::Witer Sent from my iPhone On Nov 26, 2012, at 8:58 AM, Robert Rothenberg rob...@gmail.com wrote: I need to output a large XML file (actually, a KML file of points) from a database. I can't find any documentation for Catalyst::View::XML::Generator, or even decent documentation for XML::Generator, on how to do this. It would appear to expect me to put the entire structure in the stash, which I don't want to do, as it could contain thousands of points. Ideally I would pass an iterator function and it would do the rest. ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Generating a large XML document (KML file) from a database
I once had a similar problem, and solved it by using an ordinary TT View which rendered the XML, and stashing the required resultset. It worked fine for me, but my xml structure was very simple. It just contained lots of data. I don't know if this is an option for you. If your xml is very complex or dynamic, creating the templates can be a pain! Lukas On 11/26/2012 05:58 PM, Robert Rothenberg wrote: I need to output a large XML file (actually, a KML file of points) from a database. I can't find any documentation for Catalyst::View::XML::Generator, or even decent documentation for XML::Generator, on how to do this. It would appear to expect me to put the entire structure in the stash, which I don't want to do, as it could contain thousands of points. Ideally I would pass an iterator function and it would do the rest. ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Generating a large XML document (KML file) from a database
XML::Writer Sent from my iPhone On Nov 26, 2012, at 9:23 AM, Francisco Obispo fobi...@isc.org wrote: Use the -write() methos in catalyst and XML::Witer Sent from my iPhone On Nov 26, 2012, at 8:58 AM, Robert Rothenberg rob...@gmail.com wrote: I need to output a large XML file (actually, a KML file of points) from a database. I can't find any documentation for Catalyst::View::XML::Generator, or even decent documentation for XML::Generator, on how to do this. It would appear to expect me to put the entire structure in the stash, which I don't want to do, as it could contain thousands of points. Ideally I would pass an iterator function and it would do the rest. ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Generating a large XML document (KML file) from a database
Ok, Now that I'm on my computer, I can write a decent response:
I had a similar problem with generating a fairly large CSV file.
you don't want to store this in a stash entry because it can get very large,
and
even it you were using a temp file to write the data to, the user
would have to wait until the file is fully written and then
transferred, which could translate to unresponsiveness.
I decided to use catalyst -write() method to output directly
to the socket, and generate the CSV on the fly.
On your example, you could use XML::Writer which allows you to
generate XML on the fly via:
$c-write( $writer-startTag('hello',(attribute='value')) );
$c-write( $writer-characters('World!') );
$c-write( $writer-endTag()) ;
Will generate to the output buffer:
hello attribute='value'World!/hello
You will need to generate a header with the right content type before you
start sending out stuff to your output buffer:
$c-response-content_type('text/comma-separated-values');
$c-res-header( 'Content-Disposition',
qq[attachment; filename=download.csv] );
This is how I generated mine for the CSV, you would have to fill the
right content_type: application/xxx?
And the name of the file in the Content-Disposition section.
Notice that you will not provide a Content-Length header, so the
client will not know how much data you will be sending until its
done.
And catalyst won't know that you actually sent anything so to avoid
Catalyst rendering the template, just set the body to something, in
my case I did:
$c-response-body(qq{\n});
I know there are some methods to tell Catalyst to end instead
of generating the template, but I couldn't get them to work properly.
Regards
Francisco Obispo
Director of Applications and Services - ISC
email: fobi...@isc.org
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE
On Nov 26, 2012, at 8:58 AM, Robert Rothenberg rob...@gmail.com wrote:
I need to output a large XML file (actually, a KML file of points) from a
database.
I can't find any documentation for Catalyst::View::XML::Generator, or even
decent documentation for XML::Generator, on how to do this.
It would appear to expect me to put the entire structure in the stash, which
I don't want to do, as it could contain thousands of points.
Ideally I would pass an iterator function and it would do the rest.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Generating a large XML document (KML file) from a database
Thanks. I guessed that was something I needed to do.
One note: when initializing a new XML::Writer object, you can pass $c-res
as the OUTPUT, and XML::Writer will do the right thing, apparently, since it
has a print method.
Also, sending an empty string to body
$c-res-body()
is enough.
On 26/11/12 17:54 Francisco Obispo wrote:
Ok, Now that I'm on my computer, I can write a decent response:
I had a similar problem with generating a fairly large CSV file.
you don't want to store this in a stash entry because it can get very large,
and
even it you were using a temp file to write the data to, the user
would have to wait until the file is fully written and then
transferred, which could translate to unresponsiveness.
I decided to use catalyst -write() method to output directly
to the socket, and generate the CSV on the fly.
On your example, you could use XML::Writer which allows you to
generate XML on the fly via:
$c-write( $writer-startTag('hello',(attribute='value')) );
$c-write( $writer-characters('World!') );
$c-write( $writer-endTag()) ;
Will generate to the output buffer:
hello attribute='value'World!/hello
You will need to generate a header with the right content type before you
start sending out stuff to your output buffer:
$c-response-content_type('text/comma-separated-values');
$c-res-header( 'Content-Disposition',
qq[attachment; filename=download.csv] );
This is how I generated mine for the CSV, you would have to fill the
right content_type: application/xxx?
And the name of the file in the Content-Disposition section.
Notice that you will not provide a Content-Length header, so the
client will not know how much data you will be sending until its
done.
And catalyst won't know that you actually sent anything so to avoid
Catalyst rendering the template, just set the body to something, in
my case I did:
$c-response-body(qq{\n});
I know there are some methods to tell Catalyst to end instead
of generating the template, but I couldn't get them to work properly.
Regards
Francisco Obispo
Director of Applications and Services - ISC
email: fobi...@isc.org
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE
On Nov 26, 2012, at 8:58 AM, Robert Rothenberg rob...@gmail.com wrote:
I need to output a large XML file (actually, a KML file of points) from a
database.
I can't find any documentation for Catalyst::View::XML::Generator, or even
decent documentation for XML::Generator, on how to do this.
It would appear to expect me to put the entire structure in the stash, which
I don't want to do, as it could contain thousands of points.
Ideally I would pass an iterator function and it would do the rest.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Generating a large XML document (KML file) from a database
Great! I'll see if I can do the same with the CSV module that I wrote. regards, Francisco Obispo Director of Applications and Services - ISC email: fobi...@isc.org Phone: +1 650 423 1374 || INOC-DBA *3557* NOC PGP KeyID = B38DB1BE On Nov 26, 2012, at 11:05 AM, Robert Rothenberg rob...@gmail.com wrote: Thanks. I guessed that was something I needed to do. One note: when initializing a new XML::Writer object, you can pass $c-res as the OUTPUT, and XML::Writer will do the right thing, apparently, since it has a print method. Also, sending an empty string to body $c-res-body() is enough. On 26/11/12 17:54 Francisco Obispo wrote: ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
