BTW, the Unicode version of this with 65536 to 2e6 "letters" (depending upon if 
you can restrict to "one Unicode plane") is definitely trickier. You might 
think a 2 pass radix sort would help, but since most words are short most of 
your sorts are _very_ small. Indeed, even insertion sort might well beat the 
`algorithm.sort` merge sort in some average sense for this problem. Insertion 
sort tends to beat almost everything for N < 16..32 on modern CPUs due to cache 
effects. Almost any dictionary will have average, median, and mode word lengths 
below 16 just because long words are unpopular. So, just having a "wrapper 
sort" that switches to insertion for N < 20 and falls back to `algorithm.sort` 
for bigger is probably best. It would also be possibly valuable to the 
community if you looked into the stdlib `algorithm.sort` and had it switch to 
insertion at small N. It doesn't right now.

The trouble you run into with most array/table based approaches is that the 
alphabet is so much larger than the word length. So, a Fenwick Tree or anything 
with "implicit order" from the indices is just too big to iterate over 
effectively. Without implicit order you wind up still having to sort "used 
letters" and letter repeats are not usually very dramatic. About the only thing 
I can think of that _might_ help Unicode (beyond just in-place/d-danger tricks) 
in that counting-sort-style approach is another structure from Preston Briggs 
in a 1993 Rice University tech report/thesis. That sparse-dense thing can 
sometimes be useful to manage iteration over very sparse subsets of small-ish 
universes like this Unicode code point space. It's conceivable you could use 
that as a Unicode histogram, but you would still have to sort a small array 
(number of _unique_ letters). So, it's hard to say it'd be faster than the 
insertion-or-merge idea, but it's obscure enough to be worth mentioning as also 
neglected.

Of course, **if you find yourself running this calculation a lot on a small-ish 
fixed set of basically static dictionaries** , the single biggest optimization 
you might do is to simply **save the signature index** to a file. You could 
extend @juancarlospaco 's idea and have a `const Table` built into the binary 
executable on a per dictionary basis (e.g. 1 program file per dict). You would 
absolutely have to time several hundred or even thousand anagram queries to get 
a reading. Or if you wanted a generic program to work with many dictionary 
files then instead of a `Table` you could sort the signatures themselves paired 
with their words. Then you could just save that sorted list to a file and do 
lookup via binary search on the file with at most O(lg(Nwords)) disk probes per 
anagram query (plus time to open/mmap the file). You could also hash-structure 
that file to get that down to 1 probe at substantial code baggage. 
([https://github.com/c-blake/suggest](https://github.com/c-blake/suggest)/ has 
a fully worked out example of a much more complex persistent hash-structure 
store along those lines.) You _could_ try using Nim's `marshal` module to save 
your `Table` to disk and load it back whenever you want it, but in this case I 
think the loading of the table would be no faster than just building it from 
scratch. You could also "save in memory" by having a long-running server that 
processes each dictionary just once and then answers anagram queries over a 
local network or pipe or something.

Personally, I think _all_ of the above are good coding exercises.

Reply via email to