Hi there,
the task is simple: count word occurrences from stdin (around 150mb in
this case) and print sorted results to stdout in a somewhat idiomatic
fashion.
Now, d is quite elegant while maintaining high performance compared to
both c and c++, but I, as a complete beginner, can't identify where the
10x memory usage (~300mb, see results below) is coming from.
Unicode overhead? Internal buffer? Is something slurping the whole file?
Assoc array allocations? Couldn't find huge allocs with dmd -vgc and
-profile=gc either. What did I do wrong?
```d ===================================================================
void main()
{
import std.stdio, std.algorithm, std.range;
int[string] count;
foreach(const word; stdin.byLine.map!splitter.joiner) {
++count[word];
}
//or even:
//foreach(line; stdin.byLine) {
// foreach(const word; line.splitter) {
// ++count[word];
// }
//}
count.byKeyValue
.array
.sort!((a, b) => a.value > b.value)
.each!(a => writefln("%d %s", a.value, a.key));
}
```
```c++ (for reference) =================================================
#include <iostream>
#include <vector>
#include <unordered_map>
#include <algorithm>
using namespace std;
int main() {
string s;
unordered_map<string, int> count;
std::ios::sync_with_stdio(false);
while (cin >> s) {
count[s]++;
}
vector<pair<string, int>> temp {begin(count), end(count)};
sort(begin(temp), end(temp),
[](const auto& a, const auto& b) {return b.second < a.second;});
for (const auto& elem : temp) {
cout << elem.second << " " << elem.first << '\n';
}
}
```
Results on an old celeron dual core (wall clock and res mem):
0:08.78, 313732 kb <= d dmd
0:08.25, 318084 kb <= d ldc
0:08.38, 38512 kb <= c++ idiomatic (above)
0:07.76, 30276 kb <= c++ boost
0:08.42, 26756 kb <= c verbose, hand-rolled hashtable
Mem and time measured like so:
/usr/bin/time -v $cmd < input >/dev/null
Input words file creation (around 300k * 50 words):
tr '\n' ' ' < /usr/share/dict/$lang > joined
for i in {1..50}; do cat joined >> input; done
word count sample output:
[... snip ...]
50 ironsmith
50 gloried
50 quindecagon
50 directory's
50 hydrobiological
Compilation flags:
dmd -O -release -mcpu=native -ofwc-d-dmd wc.d
ldc2 -O3 -release -flto=full -mcpu=native -ofwc-d-ldc wc.d
clang -std=c11 -O3 -march=native -flto -o wp-c-clang wp.c
clang++ -std=c++17 -O3 -march=native -flto -o wp-cpp-clang wp-boost.cpp
Versions:
dmd: v2.082.1
ldc: 1.12.0 (based on DMD v2.082.1 and LLVM 6.0.1)
llvm/clang: 6.0.1
