Good luck with it! Like I said, distinctly non-trivial.
You might like to take a look at the debian-arm mailing list archives and wiki for details of how to navigate the dependency cycle in the fundamental packages. They've recently done a self hosted (i.e. GNU/Linux based) port to aarch64 (arm64) using only a simulator because there's no actual hardware yet. They'll have had all the problems you're likely to face as it's the first Debian non-x86 self host - they'll have had everything including the kitchen sink.
Also you might like to do some research if your 4 core has an AMD processor before you try to build using it. Certainly in 64 bit Intel processors feature more SSE vector instructions, which IMO is the most likely way you'll get better performance. But IMO you're unlikely to be able to host programs using them on an AMD box unless the kernel features some instruction emulation kludge. Since you'd only really need that for programs you don't have source for I doubt it'd be much of a priority even for LKML. It's more kvm and qemu territory.
Lastly big compilations are usually i/o bound, if you can afford to augment your 4 core assuming it's a desktop with a cheap smallish SSD do. I fitted one to my desktop for that purpose (putting the source on it) and the CPU fan started going during compilations. Had to clean the cooler :-).
Good luck again, and lots of fun.
