Hello,
I'm having memory problems with a string concatenation program.
I'm on Windows 7 64-bit with DMD32 v2.060 and cygwin gcc-3.4.4 32-bit.
Consider this example in C++:
#include <iostream>
int main(int argc, char **args) {
std::string s;
for (int i=0, j=0; i < 500000000; i++) {
s += "....";
if (i % 10000000 == 0) {
std::cout << "loop " << ++j << std::endl;
s = "";
}
}
return 0;
}
After the first "loop" (10 million iterations) the memory footprint is stable at
40 MB, which is expected. Now, rewriting that to D:
import std.stdio;
class C { string x; }
void main(string[] args) {
char[] s;
// s.reserve(40_000_000);
for (int i=0, j=0; i < 500_000_000; i++) {
s ~= "....";
if (i % 10_000_000 == 0) {
writeln("loop ", ++j); stdout.flush();
s = "".dup;
}
}
}
It runs 6 times faster thanks to a better string implementation than C++ (hats
off), but memory keeps growing long after the first "loop" message, reaching
around 350-430 MB, then sometimes near the end it starts growing again, reaching
even 700 MB. Why doesn't the GC take care of it? Interestingly, when I uncomment
the s.reserve(...) line, it's much worse - memory keeps growing until the
program dies with a core.exception.OutOfMemoryError.
Then I have rewritten this code to use an Appender:
import std.stdio, std.array;
class C { string x; }
void main(string[] args) {
char[] s;
s.reserve(40_000_000);
auto app = appender(s);
for (int i=0, j=0; i < 500_000_000; i++) {
app.put("....");
if (i % 10_000_000 == 0) {
writeln("loop ", ++j); stdout.flush();
app.clear();
}
}
}
This is what I was after - it runs 4 times faster than the first D code and uses
only 40 MB of memory. It's close to the efficiency of operating on the array
using indexing. I get, that it's by design much faster than s ~= "....", but why
is so much memory wasted in the first D example?
Am I doing something wrong? I thought s ~= a is rewritten as an equivalent of
s.push_back(a) (from C++'s vector) and not s = s ~ a.
best regards,
FG
