Reading .pem files for secured

[Dukc via Digitalmars-d-learn]

I need to manually sign and verify stuff with asymmetric crypto 
keys. I ended up using rsa from secured.

The key pair needs to be persistant between, so I made a 4096-bit 
private key with OpenSSL, stored in .pem file. Then I constructed 
a public key from the private one, again with OpenSSL. It seemed 
strange to me that I could generate a public key afterwards with 
the private key, instead of having to do both at the same time. I 
just thought that perhaps crypto is somehow cryptic enough to do 
that.

But then came a problem that I need to feed the key from .pem to 
initialize RSA class at 
https://github.com/LightBender/SecureD/blob/master/source/secured/rsa.d. Someone at the internet claimed that .pem files are Base64-encoding. "Great, there's a base64 handler in Phobos", was my reaction. It was easy to write a parser, but I think something must have went wrong.

The reason is that if I understand the logic of Base64, it's that 
each character stores 6 bits. My private key .pem has 49 lines of 
64 characters worth of Base64, though the sat line isn't full. 
Anyway, this is data worth of over 18000 bits. The RSA key is 
supposed to be 4096 bits, so this can't be correct.

What am I missing?