Re: Bad OIDs (was: Re: Verification of a x509 certificate signature)

2013-11-29 Thread Dr. Stephen Henson
On Thu, Nov 28, 2013, Erwann Abalea wrote:

 How nice, they're asking for a self-signed certificate to include a
 specific EKU to indicate it's a Trust Anchor, and the OID used for
 this has never been allocated. Crazy.
 
 I just looked at OpenSSL's objects.txt database, and found some OIDs
 that need some change:
 
 id-pkix-OCSP 8: extendedStatus: Extended OCSP Status
 should be id-pkix-ocsp-pref-sig-algs (RFC6960).
 
 id-pkix-OCSP 9: valid
 should be id-pkix-ocsp-extended-revoke (RFC6960).
 
 id-pkix-OCSP 10   : path
 id-pkix-OCSP 11   : trustRoot : Trust Root
 have never been defined by PKIX.
 

Weird.. I checked the OpenSSL OID history and those have been about since the
dawn of time... well July 2000 at any rate. They were added by Richard when
he created the scripts that handle objects.txt, no idea where they actually
came from.

Changing OIDs in the table is problematical. If anything uses them it could
break them in all sorts of ways. The NID_* entries would change and text based
lookup would no longer work. 

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   majord...@openssl.org


Re: Verification of a x509 certificate signature

2013-11-28 Thread Walter H.
Hi,

On Wed, November 27, 2013 16:02, Dereck Hurtubise wrote:
 X509v3 Extended Key Usage:
 Trust Root

what is this strange?
'Trust Root' as Extended Key Usage?

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   majord...@openssl.org


Re: Verification of a x509 certificate signature

2013-11-28 Thread Walter H.
the ASN.1 dump of this certificate ...

  0 470: SEQUENCE {
  4 319:   SEQUENCE {
  8   3: [0] {
 10   1:   INTEGER 2
   :   }
 13   5: INTEGER 00 D6 2D F4 34
 20  13: SEQUENCE {
 22   9:   OBJECT IDENTIFIER sha1WithRSAEncryption (1 2 840 113549 1 1 5)
 33   0:   NULL
   :   }
 35  20: SEQUENCE {
 37  18:   SET {
 39  16: SEQUENCE {
 41   3:   OBJECT IDENTIFIER commonName (2 5 4 3)
 46   9:   PrintableString 'NL1SPF002'
   :   }
   : }
   :   }
 57  30: SEQUENCE {
 59  13:   UTCTime 13/11/2013 12:51:00 GMT
 74  13:   UTCTime 13/11/2014 12:51:00 GMT
   :   }
 89  20: SEQUENCE {
 91  18:   SET {
 93  16: SEQUENCE {
 95   3:   OBJECT IDENTIFIER commonName (2 5 4 3)
100   9:   PrintableString 'NL1SPF002'
   :   }
   : }
   :   }
111 157: SEQUENCE {
114  13:   SEQUENCE {
116   9: OBJECT IDENTIFIER rsaEncryption (1 2 840 113549 1 1 1)
127   0: NULL
   : }
129 139:   BIT STRING, encapsulates {
133 135: SEQUENCE {
136 129:   INTEGER
   : 00 C7 42 A0 7F FF A8 1F 65 A0 39 DC 63 D9 8B 09
   : 7C F2 D3 59 6D 84 A6 4B 1F 05 DE 30 1B 6B FA 42
   : B0 86 8C 88 75 9F A9 57 5B B2 6E E6 60 79 D7 12
   : 1E 22 1B 91 18 D5 93 41 80 28 2C 4D F7 D5 46 A6
   : 3E 9D 55 E1 A2 89 86 ED DC 88 9D 1B DE B8 F2 03
   : 5A 56 5B 0E CB 97 3D B1 32 74 6A A8 3B 24 6C 45
   : E7 1A 69 EB 2C EF D7 FD C1 4C 60 2A 6D BA 4B A3
   : 34 3C D6 56 4A 3E CA 32 CD 6C 5C 47 A1 05 E6 E7
   : 8D
268   1:   INTEGER 3
   :   }
   : }
   :   }
271  54: [3] {
273  52:   SEQUENCE {
275  15: SEQUENCE {
277   3:   OBJECT IDENTIFIER basicConstraints (2 5 29 19)
282   1:   BOOLEAN TRUE
285   5:   OCTET STRING, encapsulates {
287   3: SEQUENCE {
289   1:   BOOLEAN TRUE
   :   }
   : }
   :   }
292  11: SEQUENCE {
294   3:   OBJECT IDENTIFIER keyUsage (2 5 29 15)
299   4:   OCTET STRING, encapsulates {
301   2: BIT STRING 2 unused bits
   :   '11'B
   : }
   :   }
305  20: SEQUENCE {
307   3:   OBJECT IDENTIFIER extKeyUsage (2 5 29 37)
312  13:   OCTET STRING, encapsulates {
314  11: SEQUENCE {
316   9:   OBJECT IDENTIFIER '1 3 6 1 5 5 7 48 1 11'
   :   }
   : }
   :   }
   : }
   :   }
   : }
327  13:   SEQUENCE {
329   9: OBJECT IDENTIFIER sha1WithRSAEncryption (1 2 840 113549 1 1 5)
340   0: NULL
   : }
342 129:   BIT STRING
   : C2 3A 8D 8E 2C A2 B5 46 7F CF 05 E2 01 38 C7 DF
   : 6F 6E 5F 4E E1 94 42 65 5A 67 BB 21 48 FE E1 FC
   : EB AB BE B2 34 65 AC 99 2E 2F 53 20 87 EC A5 0A
   : 14 5D 3A 08 DC 2B F2 1C 9E 46 F0 B3 E9 F9 26 FC
   : 6E 12 BD BF 95 4F E7 BC 11 CE 7C 22 05 B3 C7 28
   : E8 E9 67 A5 05 1B A0 47 C0 E1 DC B2 D1 96 9D 46
   : 90 97 66 C0 26 0F 88 90 A2 D1 6F 88 BB CB FE F4
   : BB A1 90 99 AB 82 A4 87 27 95 80 27 A4 59 69 87
   :   }


On Wed, November 27, 2013 16:02, Dereck Hurtubise wrote:
 The certificate is received in ASN.1 DER format. Not PEM.
 The only thing I want to do is verify the signature of the certificate,
 and
 thus verify the signature itself.
 It is self-signed so the public key in the certificate should be used to
 verify the signature, but it isn't working.

 Certificate:
 Data:
 Version: 3 (0x2)
 Serial Number: 3593335860 (0xd62df434)
 Signature Algorithm: sha1WithRSAEncryption
 Issuer: CN=NL1SPF002
 Validity
 Not Before: Nov 13 12:51:00 2013 GMT
 Not After : Nov 13 12:51:00 2014 GMT
 Subject: CN=NL1SPF002
 Subject Public Key Info:
 Public Key Algorithm: rsaEncryption
 Public-Key: (1024 bit)
 Modulus:
 00:c7:42:a0:7f:ff:a8:1f:65:a0:39:dc:63:d9:8b:
 09:7c:f2:d3:59:6d:84:a6:4b:1f:05:de:30:1b:6b:
 fa:42:b0:86:8c:88:75:9f:a9:57:5b:b2:6e:e6:60:
 79:d7:12:1e:22:1b:91:18:d5:93:41:80:28:2c:4d:
 f7:d5:46:a6:3e:9d:55:e1:a2:89:86:ed:dc:88:9d:
 1b:de:b8:f2:03:5a:56:5b:0e:cb:97:3d:b1:32:74:
 6a:a8:3b:24:6c:45:e7:1a:69:eb:2c:ef:d7:fd:c1:
 4c:60:2a:6d:ba:4b:a3:34:3c:d6:56:4a:3e:ca:32:
 cd:6c:5c:47:a1:05:e6:e7:8d
 Exponent: 3 (0x3)
 X509v3 extensions:
 X509v3 Basic Constraints: critical

Re: Verification of a x509 certificate signature

2013-11-28 Thread Dereck Hurtubise
It is NTP indicating that this certificate is held by a supposed trusted
root (authority).
This is NTP's way of figuring out if the certificate of the subject/issuer
should be trusted or not.

So they misuse X509 extensions for their own purposes.

This alone is not enough.
So they also implement a challenge/response scheme that they do after the
certificates are verified.

Read RFC 5906 (autokey) on the CERT message/exchange for more information
and why they do this.
The Trust Root is used in the identity exchange scheme after the CERT
exchange. Also in the RFC.


On Thu, Nov 28, 2013 at 2:07 PM, Walter H. walte...@mathemainzel.infowrote:

 Hi,

 On Wed, November 27, 2013 16:02, Dereck Hurtubise wrote:
  X509v3 Extended Key Usage:
  Trust Root

 what is this strange?
 'Trust Root' as Extended Key Usage?

 __
 OpenSSL Project http://www.openssl.org
 User Support Mailing Listopenssl-users@openssl.org
 Automated List Manager   majord...@openssl.org



Re: Verification of a x509 certificate signature

2013-11-28 Thread Dereck Hurtubise
I want to thank everyone who replied for the help.
I figured out what went wrong.

Two things.
The RSA public key wasn't loaded with the correct values.
Thank you for giving a hint about that.

The second thing was the data to verify somehow included the OID of the
signature.
So the second time the OID is in the file. This should've been omitted from
the data, but somehow didn't

Thank you all for the help.



On Thu, Nov 28, 2013 at 2:26 PM, Dereck Hurtubise djhurtub...@gmail.comwrote:

 It is NTP indicating that this certificate is held by a supposed trusted
 root (authority).
 This is NTP's way of figuring out if the certificate of the subject/issuer
 should be trusted or not.

 So they misuse X509 extensions for their own purposes.

 This alone is not enough.
 So they also implement a challenge/response scheme that they do after the
 certificates are verified.

 Read RFC 5906 (autokey) on the CERT message/exchange for more information
 and why they do this.
 The Trust Root is used in the identity exchange scheme after the CERT
 exchange. Also in the RFC.


 On Thu, Nov 28, 2013 at 2:07 PM, Walter H. walte...@mathemainzel.infowrote:

 Hi,

 On Wed, November 27, 2013 16:02, Dereck Hurtubise wrote:
  X509v3 Extended Key Usage:
  Trust Root

 what is this strange?
 'Trust Root' as Extended Key Usage?

 __
 OpenSSL Project http://www.openssl.org
 User Support Mailing Listopenssl-users@openssl.org
 Automated List Manager   majord...@openssl.org





Bad OIDs (was: Re: Verification of a x509 certificate signature)

2013-11-28 Thread Erwann Abalea
How nice, they're asking for a self-signed certificate to include a 
specific EKU to indicate it's a Trust Anchor, and the OID used for this 
has never been allocated. Crazy.


I just looked at OpenSSL's objects.txt database, and found some OIDs 
that need some change:


id-pkix-OCSP 8: extendedStatus: Extended OCSP Status
should be id-pkix-ocsp-pref-sig-algs (RFC6960).

id-pkix-OCSP 9: valid
should be id-pkix-ocsp-extended-revoke (RFC6960).

id-pkix-OCSP 10   : path
id-pkix-OCSP 11   : trustRoot : Trust Root
have never been defined by PKIX.

RFC5906 uses a trustRoot EKU, without any OID being proposed or 
referenced. Your certificate includes the later one in the EKU extension.


--
Erwann ABALEA

Le 28/11/2013 14:26, Dereck Hurtubise a écrit :
It is NTP indicating that this certificate is held by a supposed 
trusted root (authority).
This is NTP's way of figuring out if the certificate of the 
subject/issuer should be trusted or not.


So they misuse X509 extensions for their own purposes.

This alone is not enough.
So they also implement a challenge/response scheme that they do after 
the certificates are verified.


Read RFC 5906 (autokey) on the CERT message/exchange for more 
information and why they do this.
The Trust Root is used in the identity exchange scheme after the CERT 
exchange. Also in the RFC.



On Thu, Nov 28, 2013 at 2:07 PM, Walter H. walte...@mathemainzel.info 
mailto:walte...@mathemainzel.info wrote:


Hi,

On Wed, November 27, 2013 16:02, Dereck Hurtubise wrote:
 X509v3 Extended Key Usage:
 Trust Root

what is this strange?
'Trust Root' as Extended Key Usage?

__
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
mailto:openssl-users@openssl.org
Automated List Manager majord...@openssl.org
mailto:majord...@openssl.org






Re: Bad OIDs (was: Re: Verification of a x509 certificate signature)

2013-11-28 Thread Dereck Hurtubise
Welcome to the wonderful world of NTP Autokey.
Where they misuse X509v3 extensions for their own purposes.

Nothing I can do about it. It's in the specification of that RFC (5906)


On Thu, Nov 28, 2013 at 4:14 PM, Erwann Abalea
erwann.aba...@keynectis.comwrote:

  How nice, they're asking for a self-signed certificate to include a
 specific EKU to indicate it's a Trust Anchor, and the OID used for this has
 never been allocated. Crazy.

 I just looked at OpenSSL's objects.txt database, and found some OIDs that
 need some change:

 id-pkix-OCSP 8: extendedStatus: Extended OCSP Status
 should be id-pkix-ocsp-pref-sig-algs (RFC6960).

 id-pkix-OCSP 9: valid
 should be id-pkix-ocsp-extended-revoke (RFC6960).

 id-pkix-OCSP 10   : path
 id-pkix-OCSP 11   : trustRoot : Trust Root
 have never been defined by PKIX.

 RFC5906 uses a trustRoot EKU, without any OID being proposed or
 referenced. Your certificate includes the later one in the EKU extension.

 --
 Erwann ABALEA


 Le 28/11/2013 14:26, Dereck Hurtubise a écrit :

It is NTP indicating that this certificate is held by a supposed
 trusted root (authority).
  This is NTP's way of figuring out if the certificate of the
 subject/issuer should be trusted or not.

  So they misuse X509 extensions for their own purposes.

  This alone is not enough.
 So they also implement a challenge/response scheme that they do after the
 certificates are verified.

  Read RFC 5906 (autokey) on the CERT message/exchange for more information
 and why they do this.
  The Trust Root is used in the identity exchange scheme after the CERT
 exchange. Also in the RFC.


 On Thu, Nov 28, 2013 at 2:07 PM, Walter H. walte...@mathemainzel.infowrote:

 Hi,

 On Wed, November 27, 2013 16:02, Dereck Hurtubise wrote:
  X509v3 Extended Key Usage:
  Trust Root

  what is this strange?
 'Trust Root' as Extended Key Usage?

 __
 OpenSSL Project http://www.openssl.org
 User Support Mailing Listopenssl-users@openssl.org
 Automated List Manager   majord...@openssl.org






Verification of a x509 certificate signature

2013-11-27 Thread Dereck Hurtubise
Hello,

I'm trying to verify an x509 certificate with a custom library (other than
openssl)
The reason I'm writing to this mailing list is that I can't figure out what
is going wrong.
The library is checked and nothing is wrong so I must be missing something.

The program I'm writing has to be compatible with NTP,
which uses openssl to create its certificates and uses NIDs for algorithm
identification.

The certificate is created on a server by openssl with RSA/SHA1, or some
other algorithm.
The modulus is 1024 bit and the e-bits are 3.
The algorithm used to create the signature, is that RSA/SHA1 with PKCS1
v1.5?

The OID I'm getting is: 1.2.840.113549.1.1.5
The NID given, according to NTP, is 65.

The functions called to create this certificate, used by ntp-keygen, are
these:
- Create RSA key-pair
   - RSA_generate_key(1024, 3, cb, RSA);
   - So the modulus is 1024 bits, the exponent is 3, cb is some
callback function,
 and RSA is what?
   - RSA_check_key(rsa)
   - This checks the rsa key generated for validity, right?

- Create certificate
- The certificate is filled with all the necessary info
- The public key is set: X509_set_pubkey(cert, pkey)
 - cert is the internal openssl certificate format.
 - pkey is the key pair in the openssl internal format.
- The CA is set to TRUE, along with some other extensions for key
constraints etc.
- The certificate is signed: X509_sign(cert, pkey, md)
- md is the message digest algorithm to use, which is the NID of
RSA/SHA1 (65 correct?)
- The certificate is verified using: X509_verify(cert, pkey)


Then the certificate is transmitted from this server in DER format
(converted from the internal openssl format to DER with the i2d routine in
openssl)

This DER format is what I'm getting. The certificate is read correctly.
The only thing I can't figure out is how this certificate's signature
should be verified.

The certificate being verified is self-signed and created in the above
manner.
So the issuer and subject are the same and the extension CA is TRUE.
Also, there is an extension called trustRoot.

So the main questions are these:
- How should the RSA key format be interpreted?
   - The modulus is easy
   - Is the exponent transmitted the e-bits portion of the RSA format?
   - If so, how does this translate to the exponent portion?
 If the e-bits are 3, does this mean the exponent is (2^(e-bits
-1)) + 1?
 So, in the example: 2^2 + 1 = 5?
   - If not, then what?

- How do I verify the signature?
   - Is the algorithm to verify with RSA/SHA1 with PKCS1 v1.5?
   - Is the signature wrapped in any way and with which scheme if so?
   - Do I use the DER data up to and excluding the signature portion?
   - If any data of the DER format needs to be excluded from the hash,
which data?

Any help would be great and much appreciated.

Thanks,
Dereck


RE: Verification of a x509 certificate signature

2013-11-27 Thread Salz, Rich
NID is an internal openssl implementation detail; X509 data structures have 
OID's.

Post the PEM of the cert.

/r$

--
Principal Security Engineer
Akamai Technology
Cambridge, MA



Re: Verification of a x509 certificate signature

2013-11-27 Thread Dereck Hurtubise
The certificate is received in ASN.1 DER format. Not PEM.
The only thing I want to do is verify the signature of the certificate, and
thus verify the signature itself.
It is self-signed so the public key in the certificate should be used to
verify the signature, but it isn't working.

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3593335860 (0xd62df434)
Signature Algorithm: sha1WithRSAEncryption
Issuer: CN=NL1SPF002
Validity
Not Before: Nov 13 12:51:00 2013 GMT
Not After : Nov 13 12:51:00 2014 GMT
Subject: CN=NL1SPF002
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (1024 bit)
Modulus:
00:c7:42:a0:7f:ff:a8:1f:65:a0:39:dc:63:d9:8b:
09:7c:f2:d3:59:6d:84:a6:4b:1f:05:de:30:1b:6b:
fa:42:b0:86:8c:88:75:9f:a9:57:5b:b2:6e:e6:60:
79:d7:12:1e:22:1b:91:18:d5:93:41:80:28:2c:4d:
f7:d5:46:a6:3e:9d:55:e1:a2:89:86:ed:dc:88:9d:
1b:de:b8:f2:03:5a:56:5b:0e:cb:97:3d:b1:32:74:
6a:a8:3b:24:6c:45:e7:1a:69:eb:2c:ef:d7:fd:c1:
4c:60:2a:6d:ba:4b:a3:34:3c:d6:56:4a:3e:ca:32:
cd:6c:5c:47:a1:05:e6:e7:8d
Exponent: 3 (0x3)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage:
Digital Signature, Certificate Sign
X509v3 Extended Key Usage:
Trust Root
Signature Algorithm: sha1WithRSAEncryption
 c2:3a:8d:8e:2c:a2:b5:46:7f:cf:05:e2:01:38:c7:df:6f:6e:
 5f:4e:e1:94:42:65:5a:67:bb:21:48:fe:e1:fc:eb:ab:be:b2:
 34:65:ac:99:2e:2f:53:20:87:ec:a5:0a:14:5d:3a:08:dc:2b:
 f2:1c:9e:46:f0:b3:e9:f9:26:fc:6e:12:bd:bf:95:4f:e7:bc:
 11:ce:7c:22:05:b3:c7:28:e8:e9:67:a5:05:1b:a0:47:c0:e1:
 dc:b2:d1:96:9d:46:90:97:66:c0:26:0f:88:90:a2:d1:6f:88:
 bb:cb:fe:f4:bb:a1:90:99:ab:82:a4:87:27:95:80:27:a4:59:
 69:87

DER format:
30 82 01 d6 30 82 01 3f a0 03 02 01 02 02 05 00 d6 2d f4 34 30 0d 06 09 2a
86 48 86 f7 0d 01 01 05 05 00 30 14 31 12 30 10 06 03 55 04 03 13 09 4e 4c
31 53 50 46 30 30 32 30 1e 17 0d 31 33 31 31 31 33 31 32 35 31 30 30 5a 17
0d 31 34 31 31 31 33 31 32 35 31 30 30 5a 30 14 31 12 30 10 06 03 55 04 03
13 09 4e 4c 31 53 50 46 30 30 32 30 81 9d 30 0d 06 09 2a 86 48 86 f7 0d 01
01 01 05 00 03 81 8b 00 30 81 87 02 81 81 00 c7 42 a0 7f ff a8 1f 65 a0 39
dc 63 d9 8b 09 7c f2 d3 59 6d 84 a6 4b 1f 05 de 30 1b 6b fa 42 b0 86 8c 88
75 9f a9 57 5b b2 6e e6 60 79 d7 12 1e 22 1b 91 18 d5 93 41 80 28 2c 4d f7
d5 46 a6 3e 9d 55 e1 a2 89 86 ed dc 88 9d 1b de b8 f2 03 5a 56 5b 0e cb 97
3d b1 32 74 6a a8 3b 24 6c 45 e7 1a 69 eb 2c ef d7 fd c1 4c 60 2a 6d ba 4b
a3 34 3c d6 56 4a 3e ca 32 cd 6c 5c 47 a1 05 e6 e7 8d 02 01 03 a3 36 30 34
30 0f 06 03 55 1d 13 01 01 ff 04 05 30 03 01 01 ff 30 0b 06 03 55 1d 0f 04
04 03 02 02 84 30 14 06 03 55 1d 25 04 0d 30 0b 06 09 2b 06 01 05 05 07 30
01 0b 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 03 81 81 00 c2 3a 8d 8e
2c a2 b5 46 7f cf 05 e2 01 38 c7 df 6f 6e 5f 4e e1 94 42 65 5a 67 bb 21 48
fe e1 fc eb ab be b2 34 65 ac 99 2e 2f 53 20 87 ec a5 0a 14 5d 3a 08 dc 2b
f2 1c 9e 46 f0 b3 e9 f9 26 fc 6e 12 bd bf 95 4f e7 bc 11 ce 7c 22 05 b3 c7
28 e8 e9 67 a5 05 1b a0 47 c0 e1 dc b2 d1 96 9d 46 90 97 66 c0 26 0f 88 90
a2 d1 6f 88 bb cb fe f4 bb a1 90 99 ab 82 a4 87 27 95 80 27 a4 59 69 87



On Wed, Nov 27, 2013 at 3:45 PM, Salz, Rich rs...@akamai.com wrote:

 NID is an internal openssl implementation detail; X509 data structures
 have OID’s.



 Post the PEM of the cert.



 /r$



 --

 Principal Security Engineer

 Akamai Technology

 Cambridge, MA





RE: Verification of a x509 certificate signature

2013-11-27 Thread Salz, Rich
The point of posting PEM is that people can cut and paste from a mail message 
and decode it to get the DER or whatever.  (That's why PEM format was invented, 
to survive intact through email:)

You are generating a certificate, self-signing it, and your recipient cannot 
verify it.  Right?

Please post the PEM and maybe someone will find something wrong with the way 
you have used openssl's tools.  If not that type of user error, then it is most 
likely an error or limitation of the recipient software.  Openssl doesn't get 
these types of things wrong nowadays.

/r$

--
Principal Security Engineer
Akamai Technology
Cambridge, MA


RE: Verification of a x509 certificate signature

2013-11-27 Thread Dave Thompson
From: owner-openssl-users On Behalf Of Dereck Hurtubise
Sent: Wednesday, November 27, 2013 04:40

I'm trying to verify an x509 certificate with a custom library (other than
openssl)
The reason I'm writing to this mailing list is that I can't figure out what
is going wrong.
The certificate is created on a server by openssl with RSA/SHA1, or some
other algorithm.
The modulus is 1024 bit and the e-bits are 3.
The algorithm used to create the signature, is that RSA/SHA1 with PKCS1
v1.5?

By default yes, and confirmed by this OID:

The OID I'm getting is: 1.2.840.113549.1.1.5
The NID given, according to NTP, is 65.

The functions called to create this certificate, used by ntp-keygen, are
these:
- Create RSA key-pair
   - RSA_generate_key(1024, 3, cb, RSA);
   - So the modulus is 1024 bits, the exponent is 3, cb is some
callback function,
 and RSA is what?

A pointer to a structure (RSA is declared as a typedef for struct rsa_st in
openssl/rsa.h)
where the generated key is placed. In the same way the cert structure used
below 
is a pointer to X509 which is a typedef for struct x509_st.

   - RSA_check_key(rsa)
  - This checks the rsa key generated for validity, right?

Yes, although that's not necessary here.

- Create certificate
    - The certificate is filled with all the necessary info
    - The public key is set: X509_set_pubkey(cert, pkey)
 - cert is the internal openssl certificate format. 
     - pkey is the key pair in the openssl internal format.
    - The CA is set to TRUE, along with some other extensions for key
constraints etc.
    - The certificate is signed: X509_sign(cert, pkey, md)
    - md is the message digest algorithm to use, which is the NID of
RSA/SHA1 (65 correct?)
    - The certificate is verified using: X509_verify(cert, pkey)

Then the certificate is transmitted from this server in DER format 
(converted from the internal openssl format to DER with the i2d routine in
openssl)
This DER format is what I'm getting. The certificate is read correctly.
The only thing I can't figure out is how this certificate's signature
should be verified.

The certificate being verified is self-signed and created in the above
manner.
So the issuer and subject are the same and the extension CA is TRUE.
Also, there is an extension called trustRoot.

To be exact the BasicConstraints extension contains the field CA = TRUE.
(It can contain another field as well, but in this case doesn't.)
Similarly TrustRoot is the (only) value in ExtendedKeyUsage.

So the main questions are these:
- How should the RSA key format be interpreted?
   - The modulus is easy
   - Is the exponent transmitted the e-bits portion of the RSA format?
   - If so, how does this translate to the exponent portion?
 If the e-bits are 3, does this mean the exponent is (2^(e-bits
-1)) + 1?
 So, in the example: 2^2 + 1 = 5?
   - If not, then what?

The RSA key in (the bit string in) subjectPublicKeyInfo is encoded per 
PKCS#1 which is now RFC 3447. It is an ASN.1 (DER) SEQUENCE of 
two INTEGERs, the modulus n and the public exponent e. For the 
generate call above the exponent is 3. Not 3 bits, the number 3.
But you should use or at least compare the value in the key, so
you don't go off the rails if it is changed in the future.

- How do I verify the signature?
   - Is the algorithm to verify with RSA/SHA1 with PKCS1 v1.5?
   - Is the signature wrapped in any way and with which scheme if so?

Yes. The signature and verification scheme is also defined by PKCS#1.
(There are actually two schemes, v1_5 and PSS, but the openssl default,
everyone else's default I've seen, and the OID in the cert all say v1_5.
In brief: hash the TBS (see next), encode hash with an algorithmidentifier 
(which amounts to adding a fixed prefix, see PKCS#1), pad with 00 01 
a bunch of FFs and 00, and compute ^d mod n (often imprecisely called
'encrypt with private key', and usually done by the more complicated 
but more efficient CRT algorithm).

   - Do I use the DER data up to and excluding the signature portion?
   - If any data of the DER format needs to be excluded from the hash,
which data?

The to be signed or TBS is the first component in the outer SEQUENCE. 
I.e. the cert is SEQUENCE { cert_info SEQUENCE { ... good stuff ... },
algid AlgorithmIdentifier, sigval BIT STRING }. sigval is the signature 
using the algorithm specified by algid on the DER value of cert_info.
(This SEQUENCE{TBS,algid,sigval} pattern applies other places as well.)

Any help would be great and much appreciated.

Note that verifying the signature on a self-signed cert is nearly useless;
it only tells you the cert wasn't corrupted in transmission, or your
storage.
It does NOT provide any evidence at all that the server you got it from,
or the server identified in it, is legitimate in any way whatsoever.
If you got it from a legitimate source or authority by a means which 
prevents substitution, then you can trust it without