x64 build time 3x slower?

[Johnson Jones via Digitalmars-d-learn]

Why would that be. Program take about 4 seconds to compile and 12 
for x64. There is fundamentally no difference between the two 
versions. I do link in gtk x86 and gtk x64 depending on version, 
and that's it as far as I can tell.

Debug
x86         4
x64         12
Release
x86         3
x64         5
The timings are pretty steady.