In HashBackup (a Python app), planning a restore requires pre-traversal of all
data blocks to be restored to decide how to manage cache during the restore.
For multi-TB restores this can take something like 5 minutes, which isn't great.
This small test compares the creation and sorting of a 10M array of tuples in
Python and Nim. First is Python:
import random
import time
r = random.randint
a = [(r(0, 1000000),i) for i in xrange(10000000)]
print 'sorting'
t = time.time()
a.sort()
print 'done', time.time()-t
ms:nim jim$ /usr/bin/time -l py sort2.py
sorting
done 29.7032859325
47.61 real 46.82 user 0.76 sys
1389780992 maximum resident set size
374738 page reclaims
408 page faults
11 block input operations
19 block output operations
67 voluntary context switches
127 involuntary context switches
Run
Now Nim:
import random
import algorithm
const
els = 10_000_000
maxval = 1_000_000
type E = object
r: int32
i: int32
type A = object
a: array[els, E]
func cmp(a, b: E): int = cmp(a.r, b.r)
proc main() =
var
a: ref A
new a
for i in 0..high(a.a):
a.a[i].r = int32(rand(maxval))
a.a[i].i = int32(i)
a.a.sort(cmp)
echo a.a[0..<5], " ... ", a.a[^5..^1]
main()
ms:nim jim$ nim c -d:danger sort2
Hint: 32979 LOC; 0.850 sec; 47.047MiB peakmem; Dangerous Release build;
proj: /Users/jim/nim/sort2; out: /Users/jim/nim/sort2 [SuccessX]
ms:nim jim$ /usr/bin/time -l ./sort2
@[(r: 0, i: 861925), (r: 0, i: 1018886), (r: 0, i: 1112145), (r: 0, i:
3095791), (r: 0, i: 3127185)] ... @[(r: 1000000, i: 5801839), (r: 1000000, i:
6467043), (r: 1000000, i: 6768282), (r: 1000000, i: 7387067), (r: 1000000, i:
8454119)]
2.28 real 2.23 user 0.04 sys
120729600 maximum resident set size
29483 page reclaims
10 page faults
1 voluntary context switches
16 involuntary context switches
Run
This is a great example of where Nim shines. Not only is it ~21x faster, it
also used 11.5x less memory: 120M vs ~1.4G.
In the actual Python app, this huge memory use was a major issue to work around
because each unique integer in Python is a 24-byte structure (28 bytes for
64-bit integers). Instead of creating a list of tuples in memory, I had to
write the tuples to a file, sort the file, then read the sorted file to create
the data structure I actually needed, and it still used around 400MB. I could
have maybe used Python's array.array somehow, by bit-shifting 2 32-bit inteers
into a 64-bit array element, but arrays can't be sorted. I could have used
NumPy, and probably should have, but HashBackup is a static build and it
requires baking all 3rd-party code into a customized Python interpreter that is
used to build the final static executable. Based on the other, much smaller
things I've done this with, adding NumPy would probably be quite a challenge.
Here's the simpler Nim program I would have liked to have been able to write:
import random
import algorithm
const
els = 10_000_000
maxval = 1_000_000
type E = object
r: int32
i: int32
proc main() =
var
a: array[els, E]
for i in 0..high(a):
a[i].r = int32(rand(maxval))
a[i].i = int32(i)
a.sort()
echo a[0..<5], " ... ", a[^5..^1]
main()
Run
I'm a neoNim so maybe something like that is possible. Still, even with a
little extra complexity, I was able to put this Nim test together in a few
minutes and the performance is great, whereas I spent a day or 2 hacking the
Python code to get the memory usage down to 400M. And that's not even
mentioning that it was obviously much slower to write a disk file, sort it, and
read it back in (so much slower than this Python test program).
Great result for Nim!
