Hello,
I've recently come across a problem with openssl versions over 0.9.7a. I
have a network of approximately 100 servers using curl to access
different websites. Some of the servers are using openssl 0.9.7a and
some are using 0.9.8b. We recently encountered a problem accessing some
sites utilizing SSL that returns an error stating...
"Unknown SSL protocol error in connection"
This error only happens on servers running 0.9.8b. The 0.9.7a servers
can access the sites just fine. I tried upgrading one of the servers to
0.9.8i to see if there was a bug in openssl, but the same problem
happened. This issue appears to only happen if SSLv3 is attempted. TLS
or SSLv2 work. The problem is that curl uses sslv3 and fails out. This
only happens on a few sites.
This is a sample from 0.9.8b
$ openssl s_client -connect www.hottopic.com:443 -ssl3 -debug
CONNECTED(00000003)
write to 0x95a9bb0 [0x95b3968] (97 bytes => 97 (0x61))
0000 - 16 03 00 00 5c 01 00 00-58 03 00 49 64 e7 d8 f4
....\...X..Id...
0010 - 71 df 07 cb a3 1a f0 0c-e8 a9 95 48 3b 90 25 f7
q..........H;.%.
0020 - f4 00 b1 05 a7 ef 93 42-d7 46 5a 00 00 30 00 39
.......B.FZ..0.9
0030 - 00 38 00 35 00 16 00 13-00 0a 00 33 00 32 00 2f
.8.5.......3.2./
0040 - 00 66 00 05 00 04 00 63-00 62 00 15 00 12 00 09
.f.....c.b......
0050 - 00 65 00 64 00 14 00 11-00 08 00 06 00 03 02 01
.e.d............
0061 - <SPACES/NULS>
read from 0x95a9bb0 [0x95af158] (5 bytes => 0 (0x0))
5249:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake
failure:s3_pkt.c:530:
Here's a sample from 0.9.7a
> openssl s_client -connect www.hottopic.com:443 -ssl3 -debug
CONNECTED(00000003)
write to 080B2388 [080BC140] (100 bytes => 100 (0x64))
0000 - 16 03 00 00 5f 01 00 00-5b 03 00 49 64 e9 5a 69
...._...[..Id.Zi
0010 - 35 b8 92 66 d4 68 30 fb-ea 31 8d f2 d5 cd 3d aa
5..f.h0..1....=.
0020 - 0f 28 65 21 dc 0b 7c ad-e9 60 0c 00 00 34 00 39
.(e!..|..`...4.9
0030 - 00 38 00 35 00 16 00 13-00 0a 00 33 00 32 00 2f
.8.5.......3.2./
0040 - 00 66 00 05 00 04 00 63-00 62 00 61 00 15 00 12
.f.....c.b.a....
0050 - 00 09 00 65 00 64 00 60-00 14 00 11 00 08 00 06
...e.d.`........
0060 - 00 03 01 ...
0064 - <SPACES/NULS>
read from 080B2388 [080B7930] (5 bytes => 5 (0x5))
0000 - 16 03 00 06 a0 .....
read from 080B2388 [080B7935] (1696 bytes => 1696 (0x6A0))
0000 - 02 00 00 46 03 00 00 00-00 3a 72 7e 52 85 1d 38
...F.....:r~R..8
0010 - d9 80 11 b6 f3 24 0d ad-19 3a e9 83 a5 6e c6 a5
.....$...:...n..
0020 - 76 0b 67 95 5c 36 20 85-34 00 00 c5 38 ef df 6e v.g.\6
.4...8..n
0030 - 37 13 40 da 90 5d b9 a2-43 c0 ce 58 58 58 58 3a
7...@..]..c..xxxx:
0040 - e2 64 49 f0 00 00 00 00-0a 00 0b 00 06 4e 00 06
.dI..........N..
0050 - 4b 00 04 0d 30 82 04 09-30 82 03 76 a0 03 02 01
K...0...0..v....
0060 - 02 02 10 7b a8 95 b9 01-91 46 76 26 95 5e ef 67
...{.....Fv&.^.g
0070 - d3 6b 5a 30 0d 06 09 2a-86 48 86 f7 0d 01 01 05
.kZ0...*.H......
0080 - 05 00 30 5f 31 0b 30 09-06 03 55 04 06 13 02 55
..0_1.0...U....U
0090 - 53 31 20 30 1e 06 03 55-04 0a 13 17 52 53 41 20 S1 0...U....RSA
00a0 - 44 61 74 61 20 53 65 63-75 72 69 74 79 2c 20 49 Data Security,
I
00b0 - 6e 63 2e 31 2e 30 2c 06-03 55 04 0b 13 25 53 65
nc.1.0,..U...%Se
00c0 - 63 75 72 65 20 53 65 72-76 65 72 20 43 65 72 74 cure Server
Cert
00d0 - 69 66 69 63 61 74 69 6f-6e 20 41 75 74 68 6f 72 ification
Author
00e0 - 69 74 79 30 1e 17 0d 30-36 30 37 31 34 30 30 30
ity0...060714000
00f0 - 30 30 30 5a 17 0d 30 39-30 37 31 36 32 33 35 39
000Z..0907162359
0100 - 35 39 5a 30 81 be 31 0b-30 09 06 03 55 04 06 13
59Z0..1.0...U...
0110 - 02 55 53 31 13 30 11 06-03 55 04 08 13 0a 43 61
.US1.0...U....Ca
0120 - 6c 69 66 6f 72 6e 69 61-31 1b 30 19 06 03 55 04
lifornia1.0...U.
0130 - 07 14 12 43 69 74 79 20-6f 66 20 49 6e 64 75 73 ...City of
Indus
0140 - 74 72 79 2c 2c 31 16 30-14 06 03 55 04 0a 14 0d
try,,1.0...U....
0150 - 48 6f 74 20 54 6f 70 69-63 20 49 6e 63 31 15 30 Hot Topic
Inc1.0
0160 - 13 06 03 55 04 0b 14 0c-49 6e 74 65 72 6e 65 74
...U....Internet
0170 - 20 47 72 70 31 33 30 31-06 03 55 04 0b 14 2a 54
Grp1301..U...*T
0180 - 65 72 6d 73 20 6f 66 20-75 73 65 20 61 74 20 77 erms of use at
w
0190 - 77 77 2e 76 65 72 69 73-69 67 6e 2e 63 6f 6d 2f
ww.verisign.com/
01a0 - 72 70 61 20 28 63 29 30-35 31 19 30 17 06 03 55 rpa
(c)051.0...U
01b0 - 04 03 14 10 77 77 77 2e-68 6f 74 74 6f 70 69 63
....www.hottopic
01c0 - 2e 63 6f 6d 30 81 9f 30-0d 06 09 2a 86 48 86 f7
.com0..0...*.H..
01d0 - 0d 01 01 01 05 00 03 81-8d 00 30 81 89 02 81 81
..........0.....
01e0 - 00 90 f6 07 d2 75 9d 71-b4 ee ed 44 bb 90 5d f4
.....u.q...D..].
01f0 - 86 7b d0 e7 a3 a8 5d a5-3c a9 dc 6b f2 dd 1c 88
.{....].<..k....
0200 - a7 2e 19 ca 8c 45 27 b1-dc 42 63 3b ec 1c 6a 04
.....E'..Bc;..j.
0210 - 27 c0 03 6d e6 cb e6 27-47 cc fc 05 1d b2 4c 01
'..m...'G.....L.
0220 - 1a 14 5f 70 82 da 90 a2-42 ca fa 73 d7 a2 ad 4a
.._p....B..s...J
0230 - 6e 05 ac 80 b3 d1 64 19-19 fc e7 79 35 f4 74 cd
n.....d....y5.t.
0240 - 9c d2 81 f1 7b 23 5f da-4d 4a 09 4d 03 4c 7d fb
....{#_.MJ.M.L}.
0250 - 80 3f 83 26 16 38 14 e2-66 0c 33 2e ea 55 45 93
.?.&.8..f.3..UE.
0260 - 1f 02 03 01 00 01 a3 82-01 68 30 82 01 64 30 09
.........h0..d0.
0270 - 06 03 55 1d 13 04 02 30-00 30 0b 06 03 55 1d 0f
..U....0.0...U..
0280 - 04 04 03 02 05 a0 30 40-06 03 55 1d 1f 04 39 30
.......@..u...90
0290 - 37 30 35 a0 33 a0 31 86-2f 68 74 74 70 3a 2f 2f
705.3.1./http://
02a0 - 53 56 52 53 65 63 75 72-65 2d 63 72 6c 2e 76 65
SVRSecure-crl.ve
02b0 - 72 69 73 69 67 6e 2e 63-6f 6d 2f 53 56 52 53 65
risign.com/SVRSe
02c0 - 63 75 72 65 2e 63 72 6c-30 44 06 03 55 1d 20 04 cure.crl0D..U.
.
02d0 - 3d 30 3b 30 39 06 0b 60-86 48 01 86 f8 45 01 07
=0;09..`.H...E..
02e0 - 17 03 30 2a 30 28 06 08-2b 06 01 05 05 07 02 01
..0*0(..+.......
02f0 - 16 1c 68 74 74 70 73 3a-2f 2f 77 77 77 2e 76 65
..https://www.ve
0300 - 72 69 73 69 67 6e 2e 63-6f 6d 2f 72 70 61 30 1d
risign.com/rpa0.
0310 - 06 03 55 1d 25 04 16 30-14 06 08 2b 06 01 05 05
..U.%..0...+....
0320 - 07 03 01 06 08 2b 06 01-05 05 07 03 02 30 34 06
.....+.......04.
0330 - 08 2b 06 01 05 05 07 01-01 04 28 30 26 30 24 06
.+........(0&0$.
0340 - 08 2b 06 01 05 05 07 30-01 86 18 68 74 74 70 3a
.+.....0...http:
0350 - 2f 2f 6f 63 73 70 2e 76-65 72 69 73 69 67 6e 2e
//ocsp.verisign.
0360 - 63 6f 6d 30 6d 06 08 2b-06 01 05 05 07 01 0c 04
com0m..+........
0370 - 61 30 5f a1 5d a0 5b 30-59 30 57 30 55 16 09 69
a0_.].[0Y0W0U..i
0380 - 6d 61 67 65 2f 67 69 66-30 21 30 1f 30 07 06 05
mage/gif0!0.0...
0390 - 2b 0e 03 02 1a 04 14 8f-e5 d3 1a 86 ac 8d 8e 6b
+..............k
03a0 - c3 cf 80 6a d4 48 18 2c-7b 19 2e 30 25 16 23 68
...j.H.,{..0%.#h
03b0 - 74 74 70 3a 2f 2f 6c 6f-67 6f 2e 76 65 72 69 73
ttp://logo.veris
03c0 - 69 67 6e 2e 63 6f 6d 2f-76 73 6c 6f 67 6f 2e 67
ign.com/vslogo.g
03d0 - 69 66 30 0d 06 09 2a 86-48 86 f7 0d 01 01 05 05
if0...*.H.......
03e0 - 00 03 7e 00 80 b2 ee d7-ed ff 7d 99 26 c9 93 ba
..~.......}.&...
03f0 - 7a bc 19 34 6b fd ee 48-a4 3f 64 75 53 63 82 e8
z..4k..H.?duSc..
0400 - 86 e3 6b 5b fd 37 a6 2a-68 dd 3d 55 5f 00 ea 5c
..k[.7.*h.=U_..\
0410 - bb 6d ac 00 37 f3 2a 67-de f5 a5 0a 58 63 b7 b0
.m..7.*g....Xc..
0420 - fe 35 3a 51 10 a9 04 93-c1 f7 09 a2 a2 d9 68 22
.5:Q..........h"
0430 - 90 f8 bc 65 8f f3 f9 bd-58 3d fb 82 be b4 83 c3
...e....X=......
0440 - 10 e0 df 5b e3 4c 38 a4-6d 27 17 f4 9b 54 fb fa
...[.L8.m'...T..
0450 - 34 cf e3 d7 21 b3 b6 14-05 42 9c bb 78 9a 02 77
4...!....B..x..w
0460 - a2 00 02 38 30 82 02 34-30 82 01 a1 02 10 02 ad
...80..40.......
0470 - 66 7e 4e 45 fe 5e 57 6f-3c 98 19 5e dd c0 30 0d
f~NE.^Wo<..^..0.
0480 - 06 09 2a 86 48 86 f7 0d-01 01 02 05 00 30 5f 31
..*.H........0_1
0490 - 0b 30 09 06 03 55 04 06-13 02 55 53 31 20 30 1e .0...U....US1
0.
04a0 - 06 03 55 04 0a 13 17 52-53 41 20 44 61 74 61 20 ..U....RSA Data
04b0 - 53 65 63 75 72 69 74 79-2c 20 49 6e 63 2e 31 2e Security,
Inc.1.
04c0 - 30 2c 06 03 55 04 0b 13-25 53 65 63 75 72 65 20 0,..U...%Secure
04d0 - 53 65 72 76 65 72 20 43-65 72 74 69 66 69 63 61 Server
Certifica
04e0 - 74 69 6f 6e 20 41 75 74-68 6f 72 69 74 79 30 1e tion
Authority0.
04f0 - 17 0d 39 34 31 31 30 39-30 30 30 30 30 30 5a 17
..941109000000Z.
0500 - 0d 31 30 30 31 30 37 32-33 35 39 35 39 5a 30 5f
.100107235959Z0_
0510 - 31 0b 30 09 06 03 55 04-06 13 02 55 53 31 20 30 1.0...U....US1
0
0520 - 1e 06 03 55 04 0a 13 17-52 53 41 20 44 61 74 61 ...U....RSA
Data
0530 - 20 53 65 63 75 72 69 74-79 2c 20 49 6e 63 2e 31 Security,
Inc.1
0540 - 2e 30 2c 06 03 55 04 0b-13 25 53 65 63 75 72 65
.0,..U...%Secure
0550 - 20 53 65 72 76 65 72 20-43 65 72 74 69 66 69 63 Server
Certific
0560 - 61 74 69 6f 6e 20 41 75-74 68 6f 72 69 74 79 30 ation
Authority0
0570 - 81 9b 30 0d 06 09 2a 86-48 86 f7 0d 01 01 01 05
..0...*.H.......
0580 - 00 03 81 89 00 30 81 85-02 7e 00 92 ce 7a c1 ae
.....0...~...z..
0590 - 83 3e 5a aa 89 83 57 ac-25 01 76 0c ad ae 8e 2c
.>Z...W.%.v....,
05a0 - 37 ce eb 35 78 64 54 03-e5 84 40 51 c9 bf 8f 08
7..5xdt...@q....
05b0 - e2 8a 82 08 d2 16 86 37-55 e9 b1 21 02 ad 76 68
.......7U..!..vh
05c0 - 81 9a 05 a2 4b c9 4b 25-66 22 56 6c 88 07 8f f7
....K.K%f"Vl....
05d0 - 81 59 6d 84 07 65 70 13-71 76 3e 9b 77 4c e3 50
.Ym..ep.qv>.wL.P
05e0 - 89 56 98 48 b9 1d a7 29-1a 13 2e 4a 11 59 9c 1e
.V.H...)...J.Y..
05f0 - 15 d5 49 54 2c 73 3a 69-82 b1 97 39 9c 6d 70 67
..IT,s:i...9.mpg
0600 - 48 e5 dd 2d d6 c8 1e 7b-02 03 01 00 01 30 0d 06
H..-...{.....0..
0610 - 09 2a 86 48 86 f7 0d 01-01 02 05 00 03 7e 00 65
.*.H.........~.e
0620 - dd 7e e1 b2 ec b0 e2 3a-e0 ec 71 46 9a 19 11 b8
.~.....:..qF....
0630 - d3 c7 a0 b4 03 40 26 02-3e 09 9c e1 12 b3 d1 5a
.....@&.>......Z
0640 - f6 37 a5 b7 61 03 b6 5b-16 69 3b c6 44 08 0c 88
.7..a..[.i;.D...
0650 - 53 0c 6b 97 49 c7 3e 35-dc 6c b9 bb aa df 5c bb
S.k.I.>5.l....\.
0660 - 3a 2f 93 60 b6 a9 4b 4d-f2 20 f7 cd 5f 7f 64 7b :/.`..KM.
.._.d{
0670 - 8e dc 00 5c d7 fa 77 ca-39 16 59 6f 0e ea d3 b5
...\..w.9.Yo....
0680 - 83 7f 4d 4d 42 56 76 b4-c9 5f 04 f8 38 f8 eb d2
..MMBVv.._..8...
0690 - 5f 75 5f cd 7b fc e5 8e-80 7c fc 50 0e _u_.{....|.P.
06a0 - <SPACES/NULS>
depth=1 /C=US/O=RSA Data Security, Inc./OU=Secure Server Certification
Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
write to 080B2388 [080C1AC8] (137 bytes => 137 (0x89))
0000 - 16 03 00 00 84 10 00 00-80 71 3b 76 dc 3d c2 09
.........q;v.=..
0010 - f9 73 19 2d b0 a8 31 27-f8 d4 4c 67 d9 75 ee c5
.s.-..1'..Lg.u..
0020 - 35 38 60 97 d0 bc 51 6c-25 b3 81 15 3e b1 d1 bb
58`...Ql%...>...
0030 - 08 ab 2c 32 b6 ad 0a 4e-6b 78 3d f9 df cd cc 39
..,2...Nkx=....9
0040 - b1 2f 2f 0e a2 33 b7 51-2d 5e b8 f8 5e 0a 76 2d
.//..3.Q-^..^.v-
0050 - 31 8f e3 9d 2d 6c 44 b1-1d 6b 4d 7c a4 b8 c8 1e
1...-lD..kM|....
0060 - 09 56 5f 7f f5 7f 58 d1-f6 7d ff 6c 85 e7 18 dd
.V_...X..}.l....
0070 - 31 8c 2f 6d d0 17 c9 c8-37 e0 79 ca 01 bd 3c 1b
1./m....7.y...<.
0080 - aa a6 85 d8 33 87 02 d5-43 ....3...C
write to 080B2388 [080C1AC8] (6 bytes => 6 (0x6))
0000 - 14 03 00 00 01 01 ......
write to 080B2388 [080C1AC8] (69 bytes => 69 (0x45))
0000 - 16 03 00 00 40 7e de 7b-9a f2 a5 ff ff 6d da 3b
....@~.{.....m.;
0010 - 68 b2 ec bd c5 e3 40 2b-66 b0 10 76 ba 65 05 e6
h.....@+f..v.e..
0020 - d3 f7 09 3d ac 4c 43 f6-0b 6d ad 48 b3 dd 7e 63
...=.LC..m.H..~c
0030 - 1a 3c 79 29 43 83 e0 42-03 13 dd 2a 08 96 1c d2
.<y)C..B...*....
0040 - 4c d4 8f 7c 24 L..|$
read from 080B2388 [080B7930] (5 bytes => 5 (0x5))
0000 - 14 03 00 00 01 .....
read from 080B2388 [080B7935] (1 bytes => 1 (0x1))
0000 - 01 .
read from 080B2388 [080B7930] (5 bytes => 5 (0x5))
0000 - 16 03 00 00 40 ....@
read from 080B2388 [080B7935] (64 bytes => 64 (0x40))
0000 - 2f 55 31 b8 7f b4 98 b8-c9 18 34 b6 7c 7f dd e2
/U1.......4.|...
0010 - ac a2 40 0f b9 72 1f 3a-eb 4f 9b 09 3e 17 5c ac
....@..r.:.O..>.\.
0020 - e6 25 4b 7e 6f 28 4b d4-b6 60 6e 84 09 56 e4 02
.%K~o(K..`n..V..
0030 - d1 83 37 36 03 10 26 21-d5 6a 33 cd f3 17 b2 4e
..76..&!.j3....N
---
Certificate chain
0 s:/C=US/ST=California/L=City of Industry,,/O=Hot Topic Inc/OU=Internet
Grp/OU=Terms of use at www.verisign.com/rpa (c)05/CN=www.hottopic.com
i:/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification
Authority
1 s:/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification
Authority
i:/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification
Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIECTCCA3agAwIBAgIQe6iVuQGRRnYmlV7vZ9NrWjANBgkqhkiG9w0BAQUFADBf
MQswCQYDVQQGEwJVUzEgMB4GA1UEChMXUlNBIERhdGEgU2VjdXJpdHksIEluYy4x
LjAsBgNVBAsTJVNlY3VyZSBTZXJ2ZXIgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkw
HhcNMDYwNzE0MDAwMDAwWhcNMDkwNzE2MjM1OTU5WjCBvjELMAkGA1UEBhMCVVMx
EzARBgNVBAgTCkNhbGlmb3JuaWExGzAZBgNVBAcUEkNpdHkgb2YgSW5kdXN0cnks
LDEWMBQGA1UEChQNSG90IFRvcGljIEluYzEVMBMGA1UECxQMSW50ZXJuZXQgR3Jw
MTMwMQYDVQQLFCpUZXJtcyBvZiB1c2UgYXQgd3d3LnZlcmlzaWduLmNvbS9ycGEg
KGMpMDUxGTAXBgNVBAMUEHd3dy5ob3R0b3BpYy5jb20wgZ8wDQYJKoZIhvcNAQEB
BQADgY0AMIGJAoGBAJD2B9J1nXG07u1Eu5Bd9IZ70OejqF2lPKnca/LdHIinLhnK
jEUnsdxCYzvsHGoEJ8ADbebL5idHzPwFHbJMARoUX3CC2pCiQsr6c9eirUpuBayA
s9FkGRn853k19HTNnNKB8XsjX9pNSglNA0x9+4A/gyYWOBTiZgwzLupVRZMfAgMB
AAGjggFoMIIBZDAJBgNVHRMEAjAAMAsGA1UdDwQEAwIFoDBABgNVHR8EOTA3MDWg
M6Axhi9odHRwOi8vU1ZSU2VjdXJlLWNybC52ZXJpc2lnbi5jb20vU1ZSU2VjdXJl
LmNybDBEBgNVHSAEPTA7MDkGC2CGSAGG+EUBBxcDMCowKAYIKwYBBQUHAgEWHGh0
dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEwHQYDVR0lBBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEFBQcwAYYYaHR0cDovL29j
c3AudmVyaXNpZ24uY29tMG0GCCsGAQUFBwEMBGEwX6FdoFswWTBXMFUWCWltYWdl
L2dpZjAhMB8wBwYFKw4DAhoEFI/l0xqGrI2Oa8PPgGrUSBgsexkuMCUWI2h0dHA6
Ly9sb2dvLnZlcmlzaWduLmNvbS92c2xvZ28uZ2lmMA0GCSqGSIb3DQEBBQUAA34A
gLLu1+3/fZkmyZO6erwZNGv97kikP2R1U2OC6Ibja1v9N6YqaN09VV8A6ly7bawA
N/MqZ971pQpYY7ew/jU6URCpBJPB9wmiotloIpD4vGWP8/m9WD37gr60g8MQ4N9b
40w4pG0nF/SbVPv6NM/j1yGzthQFQpy7eJoCd6I=
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=City of Industry,,/O=Hot Topic
Inc/OU=Internet Grp/OU=Terms of use at www.verisign.com/rpa
(c)05/CN=www.hottopic.com
issuer=/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification
Authority
---
No client certificate CA names sent
---
SSL handshake has read 1776 bytes and written 312 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 1024 bit
SSL-Session:
Protocol : SSLv3
Cipher : DES-CBC3-SHA
Session-ID:
85340000C538EFDF6E371340DA905DB9A243C0CE585858583AE26449F0000000
Session-ID-ctx:
Master-Key:
D29F4EC25F6B9F6F8E1945F5017308D1DF1DD40AF3FB521721A3BED397D4E680BBE699BC
05127E45B42C7CE634C36E02
Key-Arg : None
Krb5 Principal: None
Start Time: 1231350106
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate
chain)
---
Any insight you can provide would be appreciated as I'd prefer not to
downgrade all my servers, but I also fear upgrading all of them.
-- Matt DeWald
