Incidentally, @lagerratrobe, once you save a lot of time by replacing sorting
by a perfect commutative hash, the next optimization is making the IO part
faster (you need cligen-0.9.43 for this):
import strutils, tables, cligen/[mfile, mslice]
const prime: array[26, uint64] = [ #9/267751 oflow
7'u64, 61, 41, 53, 2, 71, 47, 29, 3, 97, 89, 17, 59,
19, 5, 31, 101, 11, 13, 23, 37, 79, 73, 67, 43, 83 ]
proc product(word: MSlice): uint64 =
result = 1'u64 #Assumes whole file is uppercase
for ch in word: #iterator needs cligen>=0.9.43
result *= prime[ord(ch) - ord('A')]
proc pBuildQry(dict="words", query: seq[string]) =
var anas = initTable[uint64, seq[MSlice]]()
let mf = mopen(dict)
if mf == nil: return
defer: mf.close
for word in mf.mSlices():
anas.mgetOrPut(word.product, @[]).add word
for word in query:
let word = word.toUpperAscii
let prod = word.toMSlice.product
echo word, ":"
try:
for ana in anas[prod]: echo " ", ana
except: discard
import cligen; dispatch(pBuildQry)
Run
With that my 129 ms goes down by 57ms to 72 ms (1.8x better) on that SOWPODS
dictionary. Then if you Nim compile with `--gc:arc`, it goes down to about 47
ms (1.53x better). Then if you use gcc's PGO it goes down to 42 ms (1.12x
better).
I suspect with your dictionary and CPU that you would see a similar overall 4x
to 5x improvement taking you from 286 ms down to more like <70ms. Language
comparison-wise, I do not believe R or CPython would be able to be sped up
similarly. (Perhaps in PyPy the product might be able to become fast, but
perhaps not if they are worrying about overflow switching to arbitrary
precision ints.)
