I should also say that the code really looks like this:
# in the controller:
$self->write_chunk($initial_json_stuff);
...
# In a model, invoked by controller
while (my $row = $query->hash) {
my $txt = ... # JSON encode hash
$controller->write_chunk($txt);
$count++;
$self->debug("Written $count/$row_count records") if ($count % 1000) ==
0;
}
...
# back in controller
$self->write_chunk($final_json_stuff);
I have tried a bit, without much success, to turn this into callback driven
code but cannot work it out!
As an aside, I don't think I need to actually write in chunks, since
Mojolicious' write() method also takes a callback to cause the output
buffer to flush.
Allan
On Friday, 5 June 2015 09:04:06 UTC-5, Allan Cochrane wrote:
>
> Hi,
>
> I'm having trouble understanding just how to make write_chunk work for me.
>
> My issue is that I'm querying a large database and emitting JSON.
> Sometimes there are hundreds of thousands of records to write.
>
> My algorithm looks like:
>
> // $query = DBIx::Simple query that return a DB row as a hash
> while (my $row = $query->hash) {
> my $txt = ... # JSON encode hash
> $controller->write_chunk($txt);
> $count++;
> $self->debug("Written $count/$row_count records") if ($count % 1000)
> == 0;
> }
>
> But that buffers the JSON strings and never seems to drain so my Perl
> process grows to a massive size. What I cannot understand is just how to
> restructure the code above to use a callback which is supposed to drain the
> buffer and prevent the huge memory usage.
>
> The documentation shows how to write "Hello world" using write_chunk with
> a callback but it's just a bit too terse for me to work out how to make use
> of it in the context above. I'm sure it's a simple concept but I just
> cannot grasp it at the moment.
>
> Thanks,
>
> Allan
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
