I'm new to node.js so forgive what's probably a very newbie question...
I've been trying to use writestreams in a few different use cases - one as
an HTTP request stream and one just writing to a plain file - and no matter
what, I always observe the same behavior, where nothing actually ever gets
written to the stream, until program execution is over.
So, for example, if I write this code:
var fs = require('fs');
var ws = fs.createWriteStream("/tmp/out", {
flags: 'w+'
});
var i = 0;
while (i < 100000) {
console.log("writing");
ws.write("random text\n");
i++;
}
ws.end();
I will see all the "writing" lines outputted, and only after all of the
"writing" lines are outputted to my terminal do any of the "random text"
lines get sent to my file. I could have set it to write 10 lines, or a
billion lines, and I see the same behavior.
My problem is, I'm trying to write a routine that generates JSON for
several million objects and writes those JSON objects to elasticsearch, via
the elasticsearchclient module, which sends data to elasticsearch via an
HTTP request (which is also a writestream). However, my routine always
fails, because node.js runs out of memory before any data actually gets
written to the stream. My routine works great if I only try to index 10
documents - once program execution ends, it sends all 10 documents over at
once, and they are indexed - but when I try to index the entire database,
it fails, even though I send the writes along 1000 at a time, and it has
ample time to start sending at least some of the documents . It would all
work great if it could just start sending the data as soon as it's
buffered, but nothing I've found gets it to do that. What I really need is
a "flush" command or something but there isn't one listed in the
documentation. The documentation would seem to indicate that this should
happen automatically, but it just doesn't.
Nothing in the documentation seems to indicate that writestreams should
work this way, so I find this very baffling and frustrating. Is there any
way to flush the writestream, to force it to start writing to the stream
before it runs out of memory? It seems like a pretty obvious thing, and in
other programming languages I've never had a problem like this, but I'm
just not finding the documentation where it explains this. Usually, in
most languages, you write to a stream, and it tries to output the data as
quickly as it can - it doesn't usually buffer until your program is done
executing. I try listening for the "drain" event but the "drain" event
never fires. The stream is always writable = false, right from the start -
the kernel buffer seems to be full right away. Nothing really seems to
work the way it's documented...
I'm running node 0.6.17. I'm pretty sure I'm missing something very
obvious here, but I've scoured the documentation and the forums for hours
and I can't find anything that helps me solve my problem. If anyone can
please help, I'd really appreciate it. Thanks.
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
