Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Thompson]

I think the people with the relevant skill are willing to fix it, when
they're show that what they did (cryptdev support) doesn't provide any
benefit.

read:  it's being taken care of.

On Mon, Nov 11, 2013 at 1:20 PM, Vick Khera  wrote:
> Did you get the sense people with the relevant skill were open to a bounty
> for implementing the necessary fixes?
>
>
> On Mon, Nov 11, 2013 at 1:36 PM, Jim Thompson  wrote:
>>
>> I was at the FreeBSD Vendor Summit last week, and raised the AES-NI
>> issue as "important to be solved in the next six months".
>>
>> The issue and fix are understood, it just needs someone to implement
>> it (and then, presumably, backport it to 8.3, so we can release an
>> update to 2.1 (2.1.1 or similar).
>>
>> Jim
>>
>> On Fri, Nov 8, 2013 at 12:33 PM, Thinker Rix 
>> wrote:
>> > Hi all,
>> >
>> >
>> > On 2013-11-06 07:53, Thinker Rix wrote:
>> >>
>> >> as I am planning to buy new hardware for pfSense, I was wondering if it
>> >> is
>> >> worthy to buy a CPU that supports "AES new instructions", i.e.
>> >> hardware-support for AES encyption.
>> >
>> >
>> > As I learned in this thread (big thanks to everybody participating),
>> > AES-NI
>> > is adding no value to pfSense currently, at all. So currently the only
>> > solution is to throw GHz at the problem.
>> >
>> > Searching myself through the web to learn what CPU speed I would need to
>> > achieve my desired 450 MBit/s VPN (or come at least somewhat close to
>> > this
>> > theoretical max), I found this:
>> >
>> > http://forums.freenas.org/threads/encryption-performance-benchmarks.12157/
>> > I copied those measurements found there into a spreadsheet so to analyze
>> > those values. If anybody is interested in this spreadsheet (.ods), I can
>> > send it to him via private mail (I guess binaries are not allowed in the
>> > mailing list). Just drop me a message.
>> >
>> >
>> > Regards
>> > Thinker Rix
>> > ___
>> > List mailing list
>> > List@lists.pfsense.org
>> > http://lists.pfsense.org/mailman/listinfo/list
>> ___
>> List mailing list
>> List@lists.pfsense.org
>> http://lists.pfsense.org/mailman/listinfo/list
>
>
>
> ___
> List mailing list
> List@lists.pfsense.org
> http://lists.pfsense.org/mailman/listinfo/list
>
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

Did you get the sense people with the relevant skill were open to a bounty
for implementing the necessary fixes?


On Mon, Nov 11, 2013 at 1:36 PM, Jim Thompson  wrote:

> I was at the FreeBSD Vendor Summit last week, and raised the AES-NI
> issue as "important to be solved in the next six months".
>
> The issue and fix are understood, it just needs someone to implement
> it (and then, presumably, backport it to 8.3, so we can release an
> update to 2.1 (2.1.1 or similar).
>
> Jim
>
> On Fri, Nov 8, 2013 at 12:33 PM, Thinker Rix 
> wrote:
> > Hi all,
> >
> >
> > On 2013-11-06 07:53, Thinker Rix wrote:
> >>
> >> as I am planning to buy new hardware for pfSense, I was wondering if it
> is
> >> worthy to buy a CPU that supports "AES new instructions", i.e.
> >> hardware-support for AES encyption.
> >
> >
> > As I learned in this thread (big thanks to everybody participating),
> AES-NI
> > is adding no value to pfSense currently, at all. So currently the only
> > solution is to throw GHz at the problem.
> >
> > Searching myself through the web to learn what CPU speed I would need to
> > achieve my desired 450 MBit/s VPN (or come at least somewhat close to
> this
> > theoretical max), I found this:
> >
> http://forums.freenas.org/threads/encryption-performance-benchmarks.12157/
> > I copied those measurements found there into a spreadsheet so to analyze
> > those values. If anybody is interested in this spreadsheet (.ods), I can
> > send it to him via private mail (I guess binaries are not allowed in the
> > mailing list). Just drop me a message.
> >
> >
> > Regards
> > Thinker Rix
> > ___
> > List mailing list
> > List@lists.pfsense.org
> > http://lists.pfsense.org/mailman/listinfo/list
> ___
> List mailing list
> List@lists.pfsense.org
> http://lists.pfsense.org/mailman/listinfo/list
>
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Thompson]

I was at the FreeBSD Vendor Summit last week, and raised the AES-NI
issue as "important to be solved in the next six months".

The issue and fix are understood, it just needs someone to implement
it (and then, presumably, backport it to 8.3, so we can release an
update to 2.1 (2.1.1 or similar).

Jim

On Fri, Nov 8, 2013 at 12:33 PM, Thinker Rix  wrote:
> Hi all,
>
>
> On 2013-11-06 07:53, Thinker Rix wrote:
>>
>> as I am planning to buy new hardware for pfSense, I was wondering if it is
>> worthy to buy a CPU that supports "AES new instructions", i.e.
>> hardware-support for AES encyption.
>
>
> As I learned in this thread (big thanks to everybody participating), AES-NI
> is adding no value to pfSense currently, at all. So currently the only
> solution is to throw GHz at the problem.
>
> Searching myself through the web to learn what CPU speed I would need to
> achieve my desired 450 MBit/s VPN (or come at least somewhat close to this
> theoretical max), I found this:
> http://forums.freenas.org/threads/encryption-performance-benchmarks.12157/
> I copied those measurements found there into a spreadsheet so to analyze
> those values. If anybody is interested in this spreadsheet (.ods), I can
> send it to him via private mail (I guess binaries are not allowed in the
> mailing list). Just drop me a message.
>
>
> Regards
> Thinker Rix
> ___
> List mailing list
> List@lists.pfsense.org
> http://lists.pfsense.org/mailman/listinfo/list
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Thinker Rix]

Hi all,

On 2013-11-06 07:53, Thinker Rix wrote:
as I am planning to buy new hardware for pfSense, I was wondering if 
it is worthy to buy a CPU that supports "AES new instructions", i.e. 
hardware-support for AES encyption.


As I learned in this thread (big thanks to everybody participating), 
AES-NI is adding no value to pfSense currently, at all. So currently the 
only solution is to throw GHz at the problem.


Searching myself through the web to learn what CPU speed I would need to 
achieve my desired 450 MBit/s VPN (or come at least somewhat close to 
this theoretical max), I found this: 
http://forums.freenas.org/threads/encryption-performance-benchmarks.12157/
I copied those measurements found there into a spreadsheet so to analyze 
those values. If anybody is interested in this spreadsheet (.ods), I can 
send it to him via private mail (I guess binaries are not allowed in the 
mailing list). Just drop me a message.


Regards
Thinker Rix
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Pingle]

On 11/7/2013 10:30 AM, Vick Khera wrote:
> On Thu, Nov 7, 2013 at 9:54 AM, Jim Pingle  > wrote:
> 
> The sheet could really use some more data, so anyone who has an AES-NI
> capable system, feel free to run through the tests and help fill out the
> sheet. :-)
> 
> 
> /usr/bin/openssl speed -evp aes-128-cbc -elapsed
[snip]

I added the data to the sheet. I picked one of the two runs, they were
similar enough I didn't see a significant difference for now. Not being
terribly scientific but going for a general idea at the moment.

> If I run without -elapsed, I can see the CPU time used by the tests. The
> selection of engine cryptodev makes no difference: openssl always
> offloads from the CPU to the AES-NI engine once it is loaded, as the CPU
> time is fraction of a second over the 3 second tests. True for both
> versions of openssl.

The automatic offload once loaded is expected. OpenSSL knows which
ciphers are supported by the available devices and will automatically
select one without being told. That's where this line helps:
/usr/local/bin/openssl engine -t -c

Using -elapsed uses real time rather than CPU time to calculate the
results. When you engage cryptodev the CPU time gets cut waaay down so
it's not quite as accurate a result to compare. Of course elapsed has
its own drawbacks as well. (Not accurate unless the system is idle,
doesn't account for time spent waiting on the crypto device to respond...)

> So it seems that the overall speed is lower with freebsd's AES-NI
> engine, but the offload from the CPU is significant.

That jives with other results we have had reported. Hopefully things
have improved in FreeBSD 10, but there may still be other blocking
factors that Jim Thompson mentioned elsewhere in this thread.

In the meantime, if you run with aesni.ko *unloaded*, you can at least
get a better benefit for OpenVPN and other OpenSSL operations that tie
into OpenSSL 1.0.1e. In some cases the difference between 1.0.1e and
cryptodev is significant, and you can tell they greatly improved the
AES-NI code between OpenSSL 0.9.x and 1.0.x.

Jim
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Thu, Nov 7, 2013 at 9:54 AM, Jim Pingle  wrote:

> The sheet could really use some more data, so anyone who has an AES-NI
> capable system, feel free to run through the tests and help fill out the
> sheet. :-)
>

/usr/bin/openssl speed -evp aes-128-cbc -elapsed

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  99399.17k   108488.44k   111490.64k   113100.76k
114121.28k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  97350.86k   108496.66k   111490.32k   112989.75k
114082.46k

/usr/local/bin/openssl speed -evp aes-128-cbc -elapsed

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc 397045.72k   422822.44k   430420.31k   432295.94k
432848.90k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc 396872.65k   423517.87k   430512.98k   432295.25k
432848.90k

cryptotest -va aes
cipher aes keylen 16
CIOCGSESSION: Invalid argument


Now we load aesni


/usr/bin/openssl speed -evp aes-128-cbc -elapsed

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  10898.07k40515.02k   122446.32k   249924.25k
20.00k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  11021.06k40772.30k   122881.76k   250294.41k
300729.52k

/usr/local/bin/openssl speed -evp aes-128-cbc -elapsed

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  11031.72k40436.22k   122958.08k   250292.91k
300296.87k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  10894.13k40242.52k   121988.61k   249543.34k
300127.57k



/usr/bin/openssl speed -evp aes-128-cbc -elapsed -engine cryptodev

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  11049.62k40428.86k   122644.60k   250343.91k
300689.88k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  11058.16k40830.55k   122402.15k   249873.61k
300421.51k


/usr/local/bin/openssl speed -evp aes-128-cbc -elapsed -engine cryptodev

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  10948.26k40138.15k   121933.23k   248702.14k
300463.45k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-128-cbc  11028.30k40161.71k   122037.42k   248784.53k
300324.18k


cryptotest -va aes
session = 0x0
device = aesni0
count = 1, size = 16
iv:
: 6f 38 32 75 74 74 6a 34 62 62 62 21 69 74 6e 6f
cleartext:
: 35 38 65 74 61 69 75 62 6e 31 6e 6a 6a 33 37 38
cleartext:
: 35 38 65 74 61 69 75 62 6e 31 6e 6a 6a 33 37 38
   0.000 sec,   2aes crypts,  16 bytes,  2461538 byte/sec,
 18.8 Mb/sec


If I run without -elapsed, I can see the CPU time used by the tests. The
selection of engine cryptodev makes no difference: openssl always offloads
from the CPU to the AES-NI engine once it is loaded, as the CPU time is
fraction of a second over the 3 second tests. True for both versions of
openssl.

So it seems that the overall speed is lower with freebsd's AES-NI engine,
but the offload from the CPU is significant.

CPU: Intel(R) Xeon(R) CPU E31220L @ 2.20GHz (2195.02-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping =
7

Features=0xbfebfbff

Features2=0x17bae3ff
  AMD Features=0x28100800
  AMD Features2=0x1
  TSC: P-state invariant
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Pingle]

On 11/7/2013 9:58 AM, Vick Khera wrote:
> 
> On Thu, Nov 7, 2013 at 9:54 AM, Jim Pingle  > wrote:
> 
> Also see the "How To Test" tab and other data here:
> 
> https://docs.google.com/spreadsheet/ccc?key=0AojFUXcbH0ROdE15eHB4dndHTXZYcU1mQm9Dc3V2elE&usp=sharing
> 
> The sheet could really use some more data, so anyone who has an AES-NI
> capable system, feel free to run through the tests and help fill out the
> sheet. :-)
> 
> 
> I'm running that now. Which numbers do I pull from the results? those on
> the bottom line of the openssl speed output? 

The output from the openssl speed line at various sizes match up with
the columns in the sheet, so if you paste the output of the commands on
the "How to Test" tab for each iteration that's good enough for now.
Note that you want the "-elapsed" parameter on the test command as shown
on the other tab. Also a good idea to run each test a few times and make
sure it doesn't vary a lot. Ideally we'd have a bunch of runs of each
type to get a good sampling of the data.

> Also, does it matter which
> CPU I have? That seems like it should be a column :)

Sure that can be added.

Jim
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Thu, Nov 7, 2013 at 9:54 AM, Jim Pingle  wrote:

> Also see the "How To Test" tab and other data here:
>
> https://docs.google.com/spreadsheet/ccc?key=0AojFUXcbH0ROdE15eHB4dndHTXZYcU1mQm9Dc3V2elE&usp=sharing
>
> The sheet could really use some more data, so anyone who has an AES-NI
> capable system, feel free to run through the tests and help fill out the
> sheet. :-)
>

I'm running that now. Which numbers do I pull from the results? those on
the bottom line of the openssl speed output? Also, does it matter which CPU
I have? That seems like it should be a column :)
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Pingle]

On 11/7/2013 8:51 AM, Vick Khera wrote:
> On Wed, Nov 6, 2013 at 8:29 AM, Jim Thompson  > wrote:
> 
> There are reports that FreeBSD doesn't support AES-NI very well.
> 
> 
> I'm thinking it is either zero gain, or negative gain.  On pfSense
> 2.1-RELEASE (aka FreeBSD 8.3 with OpenSSL 1.0.1e) we see:

Did you have aesni.ko loaded? If so, try again with aesni.ko unloaded
(not selected on System > Advanced, Misc tab)

Also see the "How To Test" tab and other data here:
https://docs.google.com/spreadsheet/ccc?key=0AojFUXcbH0ROdE15eHB4dndHTXZYcU1mQm9Dc3V2elE&usp=sharing

The sheet could really use some more data, so anyone who has an AES-NI
capable system, feel free to run through the tests and help fill out the
sheet. :-)

Jim
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Thu, Nov 7, 2013 at 9:44 AM, Vick Khera  wrote:

> CLEARLY it is killer fast for larger blocks.


I just pondered this for a few minutes... I think openssl's summary numbers
are misleading. They give you the time per CPU seconds used. So while the
CPU is not doing the computations, the number of computations done in 3
seconds is significantly lower with aesni loaded. Thus it only takes a
fraction of a second of CPU time over the 3 seconds elapsed, and they use
that as the divisor for the bytes per second processed. This is clearly
wrong, as all the CPU is doing is managing the work not the actual work.

So I fall back to my original thought that it may do more harm than good to
enable aesni.
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Thu, Nov 7, 2013 at 8:51 AM, Vick Khera  wrote:

> I'm thinking it is either zero gain, or negative gain.  On pfSense
> 2.1-RELEASE (aka FreeBSD 8.3 with OpenSSL 1.0.1e) we see:
>

Hm. So reading more, I learn that AES-NI will only be used with -evp on
openssl, and openvpn uses evp by default. So I re-run my tests. CLEARLY it
is killer fast for larger blocks. Doesn't seem like you need to specify
engine cryptodev to openssl at all.

I think I will see about configuring my openvpn tunnels to use larger MSS
and not fragment internally...



Before loading aesni into kernel:

/usr/local/bin/openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 54635117 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 14089042 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 3617963 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 907220 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 113504 aes-256-cbc's in 3.00s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Aug 26 08:47:16 EDT 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS
-pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
-DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O2 -pipe
-fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM
-DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-256-cbc 291387.29k   300566.23k   308732.84k   309664.43k
309941.59k


/usr/local/bin/openssl speed -evp aes-256-cbc -engine cryptodev
engine "cryptodev" set.
Doing aes-256-cbc for 3s on 16 size blocks: 54623793 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 14308715 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 3618613 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 906721 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 113508 aes-256-cbc's in 3.01s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Aug 26 08:47:16 EDT 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS
-pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
-DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O2 -pipe
-fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM
-DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-256-cbc 291326.90k   305252.59k   308788.31k   309494.10k
309147.44k


after loading into kernel:

/usr/local/bin/openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 2015840 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 64 size blocks: 1812883 aes-256-cbc's in 0.20s
Doing aes-256-cbc for 3s on 256 size blocks: 1277825 aes-256-cbc's in 0.10s
Doing aes-256-cbc for 3s on 1024 size blocks: 595015 aes-256-cbc's in 0.04s
Doing aes-256-cbc for 3s on 8192 size blocks: 86289 aes-256-cbc's in 0.06s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Aug 26 08:47:16 EDT 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS
-pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
-DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O2 -pipe
-fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM
-DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-256-cbc 242849.43k   571197.60k  3220905.35k 15597961.22k
11310071.81k

/usr/local/bin/openssl speed -evp aes-256-cbc -engine cryptodev
engine "cryptodev" set.
Doing aes-256-cbc for 3s on 16 size blocks: 2015797 aes-256-cbc's in 0.15s
Doing aes-256-cbc for 3s on 64 size blocks: 1820642 aes-256-cbc's in 0.20s
Doing aes-256-cbc for 3s on 256 size blocks: 1277035 aes-256-cbc's in 0.14s
Doing aes-256-cbc for 3s on 1024 size blocks: 596064 aes-256-cbc's in 0.06s
Doing aes-256-cbc for 3s on 8192 size blocks: 86177 aes-256-cbc's in 0.05s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Aug 26 08:47:16 EDT 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSS

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Mark Tinka]

On Thursday, November 07, 2013 04:44:54 PM Vick Khera wrote:

> I think I will see about configuring my openvpn tunnels
> to use larger MSS and not fragment internally...

Forwarding engines always prefer larger MSS's and MTU's. 
Increases throughput.

Mark.


signature.asc
Description: This is a digitally signed message part.
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Wed, Nov 6, 2013 at 11:04 AM, Thinker Rix wrote:

> What do you think is the reason for your VPN traffic maxing out at 20Mpbs
> (I assume that your connection is not the traffic bottle neck, right?),
> although your CPUs are almost idle?
>

I'm fairly sure it is the office Comcast connection. Even if I bypass the
VPN and user rsync over SSH, I max out at that rate. The data center can
*easily* push 150Mbps sustained (I have not found a way to push more data
even for testing) with barely a 0.1 load on the firewall.
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Wed, Nov 6, 2013 at 8:29 AM, Jim Thompson  wrote:

> There are reports that FreeBSD doesn't support AES-NI very well.
>

I'm thinking it is either zero gain, or negative gain.  On pfSense
2.1-RELEASE (aka FreeBSD 8.3 with OpenSSL 1.0.1e) we see:

% /usr/local/bin/openssl speed aes-256-cbc
Doing aes-256 cbc for 3s on 16 size blocks: 9065243 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 64 size blocks: 2411846 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 610745 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 151191 aes-256 cbc's in 2.99s
Doing aes-256 cbc for 3s on 8192 size blocks: 19202 aes-256 cbc's in 3.00s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Aug 26 08:47:16 EDT 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS
-pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
-DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O2 -pipe
-fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM
-DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-256 cbc  48347.96k51452.71k52116.91k51741.27k
 52434.26k


% /usr/local/bin/openssl speed aes-256-cbc -engine cryptodev
engine "cryptodev" set.
Doing aes-256 cbc for 3s on 16 size blocks: 9070243 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 64 size blocks: 2412033 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 610660 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 153469 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 8192 size blocks: 19207 aes-256 cbc's in 2.99s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Aug 26 08:47:16 EDT 2013
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS
-pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
-DL_ENDIAN -DTERMIOS -O3 -DMD32_REG_T=int -Wall -O2 -pipe
-fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT
-DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM
-DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
aes-256 cbc  48374.63k51456.70k52109.65k52384.09k
 52584.85k


We know the cryptodev device supports aes256 by this:

% cryptotest -v -a aes256 512 2048
session = 0x0
device = aesni0
count = 512, size = 2048
iv:
: 35 32 21 69 36 62 65 61 6e 39 69 31 33 6f 30 69
cleartext:
: 6f 38 32 75 74 74 6a 34 62 62 62 21 69 74 6e 6f
0010: 61 38 32 39 6a 6f 6f 73 6e 31 65 74 73 62 6f 75
0020: 69 37 39 73 74 37 35 75 6f 73 6e 75 31 6f 68 6e
0030: 33 30 35 31 6f 30 68 61 31 33 35 35 6f 30 6a 65
cleartext:
: 6f 38 32 75 74 74 6a 34 62 62 62 21 69 74 6e 6f
0010: 61 38 32 39 6a 6f 6f 73 6e 31 65 74 73 62 6f 75
0020: 69 37 39 73 74 37 35 75 6f 73 6e 75 31 6f 68 6e
0030: 33 30 35 31 6f 30 68 61 31 33 35 35 6f 30 6a 65
   0.007 sec,1024 aes256 crypts,2048 bytes, 313991915 byte/sec,
 2395.6 Mb/sec
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Thompson]

The Xeon CPUs are "almost idle". 

The "old Intel 32-bit Pentium 4 2.4GHz dual core server", however is the other 
end of that IPSEC tunnel. It's unlikely to be as idle as the Xeon. 

-- Jim

> On Nov 6, 2013, at 8:04, Thinker Rix  wrote:
> 
>> On 2013-11-06 15:22, Vick Khera wrote:
>> 
>> On Wed, Nov 6, 2013 at 12:53 AM, Thinker Rix  
>> wrote:
>>> Would pfSense use this CPU instructions so to hardware-encrypt/decrypt all 
>>> VPN traffic (openVPN)?
>>> Woud pfSense benefit from this in any other way, too?
>> 
>> 
>> pfSense lists the AES-NI as a supported option for crypto acceleration.  
>> pfSense will use it for OpenVPN and IPsec if you tell it to. There's a 
>> config setting for it.
>> 
>> As to your question of is it worth the cost, that depends on how much VPN 
>> traffic you have. The Xeon will handle a damn lot of traffic all on its own. 
>> If you are pushing more than 40Mbps on the VPN, then perhaps consider the 
>> extra cost. If it is low, like under 5 or 10Mbps, then I'd probably suggest 
>> that it is not worth the cost.
>> 
>> As a reference, between my data center and my primary office, I have an 
>> IPsec tunnel.  The office runs on an old Intel 32-bit Pentium 4 2.4GHz dual 
>> core server.  The data center runs on Intel Xeon E31220L @ 2.20GHz 
>> quad-core. Neither one has any built-in cryptodev supported devices. The 
>> IPsec tunnel maxes out at about 20Mbps during large file backups. I don't 
>> think it would go any faster with hardware acceleration, and the load on 
>> these boxes hovers around 0 still. The data center firewall is also busy 
>> pushing over 100Mpbs of regular traffic to hundreds of clients as well.
> 
> Hi Vick,
> 
> Thank you for your reference, it is very valuable for me!
> I guess I will go with a Pentium (Ivy Bridge) 2x 3.0 GHz CPU.
> 
> What do you think is the reason for your VPN traffic maxing out at 20Mpbs (I 
> assume that your connection is not the traffic bottle neck, right?), although 
> your CPUs are almost idle?
> 
> Best regards
> Thinker Rix
> ___
> List mailing list
> List@lists.pfsense.org
> http://lists.pfsense.org/mailman/listinfo/list
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Thompson]

The issue may not be that easy to fix. 
Current theory is that it's is a structural issue in cryptdev. 

-- Jim

> On Nov 6, 2013, at 20:59, Chris Buechler  wrote:
> 
> I have done some brief testing of AES-NI a few months back, though I
> can't seem to find the results at the moment and that test environment
> isn't online currently. It doesn't give the performance benefit that
> it should at this time. So the immediate benefit is minimal (except
> for the fact the Xeon proc would be faster than the Pentium), but it
> will be properly supported in the future, hopefully in 2.2 with its
> FreeBSD 10 base, but we haven't done any testing there yet.
> 
>> On Tue, Nov 5, 2013 at 11:53 PM, Thinker Rix  
>> wrote:
>> Hello all,
>> 
>> as I am planning to buy new hardware for pfSense, I was wondering if it is
>> worthy to buy a CPU that supports "AES new instructions", i.e.
>> hardware-support for AES encyption.
>> 
>> Would pfSense use this CPU instructions so to hardware-encrypt/decrypt all
>> VPN traffic (openVPN)?
>> Woud pfSense benefit from this in any other way, too?
>> 
>> The motherboards that I want to buy unfortunately support AES-NI only with
>> Xeons that currently start from approx 170 €. If I would take a CPU without
>> AES-IN, I could go with a dual-Pentium for 40€. What impact would you expect
>> from AES-IN, in regards to the fact tht I will be having traffic from VPN
>> secured WLAN with approx 300-450 Mpbs and VPN to/from the internet, 1-2
>> users at a time max. Do you think the AES-IN would be worthy the price
>> premium of the Xeon for my case, e.g. because it would reduce VPN latency,
>> etc., or is it just a pure waste of money in my case?
>> 
>> Best regards
>> Thinker Rix
>> 
>> 
>> 
>> ___
>> List mailing list
>> List@lists.pfsense.org
>> http://lists.pfsense.org/mailman/listinfo/list
> ___
> List mailing list
> List@lists.pfsense.org
> http://lists.pfsense.org/mailman/listinfo/list
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Chris Buechler]

I have done some brief testing of AES-NI a few months back, though I
can't seem to find the results at the moment and that test environment
isn't online currently. It doesn't give the performance benefit that
it should at this time. So the immediate benefit is minimal (except
for the fact the Xeon proc would be faster than the Pentium), but it
will be properly supported in the future, hopefully in 2.2 with its
FreeBSD 10 base, but we haven't done any testing there yet.

On Tue, Nov 5, 2013 at 11:53 PM, Thinker Rix  wrote:
> Hello all,
>
> as I am planning to buy new hardware for pfSense, I was wondering if it is
> worthy to buy a CPU that supports "AES new instructions", i.e.
> hardware-support for AES encyption.
>
> Would pfSense use this CPU instructions so to hardware-encrypt/decrypt all
> VPN traffic (openVPN)?
> Woud pfSense benefit from this in any other way, too?
>
> The motherboards that I want to buy unfortunately support AES-NI only with
> Xeons that currently start from approx 170 €. If I would take a CPU without
> AES-IN, I could go with a dual-Pentium for 40€. What impact would you expect
> from AES-IN, in regards to the fact tht I will be having traffic from VPN
> secured WLAN with approx 300-450 Mpbs and VPN to/from the internet, 1-2
> users at a time max. Do you think the AES-IN would be worthy the price
> premium of the Xeon for my case, e.g. because it would reduce VPN latency,
> etc., or is it just a pure waste of money in my case?
>
> Best regards
> Thinker Rix
>
>
>
> ___
> List mailing list
> List@lists.pfsense.org
> http://lists.pfsense.org/mailman/listinfo/list
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Paul Mather]

On Nov 6, 2013, at 1:43 PM, Jim Thompson  wrote:

> 
> On Nov 6, 2013, at 8:06 AM, Thinker Rix  wrote:
> 
>> On 2013-11-06 15:29, Jim Thompson wrote:
 On Nov 6, 2013, at 7:22, Vick Khera  wrote:
 
 pfSense lists the AES-NI as a supported option for crypto acceleration.  
 pfSense will use it for OpenVPN and IPsec if you tell it to. There's a 
 config setting for it.
>>> I'm not aware if any performance testing for AES-NI on pfSense.
>>> 
>>> There are reports that FreeBSD doesn't support AES-NI very well.
>> 
>> Thank you for this information, Jim. So I figure, that buying the Xeon just 
>> for it's AES functions would (currently) be a waste of money.
> 
> I can’t answer this, because I’ve not tested it.
> 
> I know that the linux kernel, and openbsd both take full advantage of AES-NI 
> instructions.
> 
> http://ibatanov.blogspot.com/2012/04/ipsec-performance-benchmarking-is-end.html
> http://comments.gmane.org/gmane.os.openbsd.misc/199639
> 
> I know there is an implementation of AES-NI for cryptdev, but **I HAVE NOT 
> TESTED IT (nor has anyone else on the pfSense team, AFAIK).
> 
> There seems to be an issue:
> http://forum.pfsense.org/index.php/topic,54008.30.html
> http://lists.freebsd.org/pipermail/freebsd-hackers/2012-May/038762.html
> 
> In the meantime, it might be possible to use OpenVPN with a patched openssl 
> library to achieve the results you desire (but now you’re off into DIY land.) 
>  https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux
> 
> That all said, we will find and fix the issue at some point.   (I’m actually 
> in San Jose for the FreeBSD Vendor Summit, and plan to bring it up as a 
> potential issue.)


Well, there's this thread from late August this year about improving AES-NI 
support that eventually kicked off into an epic kerfuffle and bike shed about 
the status of gcc in FreeBSD 10: 
http://lists.freebsd.org/pipermail/freebsd-toolchain/2013-August/000920.html

Cheers,

Paul.
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Thompson]

On Nov 6, 2013, at 8:06 AM, Thinker Rix  wrote:

> On 2013-11-06 15:29, Jim Thompson wrote:
>>> On Nov 6, 2013, at 7:22, Vick Khera  wrote:
>>> 
>>> pfSense lists the AES-NI as a supported option for crypto acceleration.  
>>> pfSense will use it for OpenVPN and IPsec if you tell it to. There's a 
>>> config setting for it.
>> I'm not aware if any performance testing for AES-NI on pfSense.
>> 
>> There are reports that FreeBSD doesn't support AES-NI very well.
> 
> Thank you for this information, Jim. So I figure, that buying the Xeon just 
> for it's AES functions would (currently) be a waste of money.

I can’t answer this, because I’ve not tested it.

I know that the linux kernel, and openbsd both take full advantage of AES-NI 
instructions.

http://ibatanov.blogspot.com/2012/04/ipsec-performance-benchmarking-is-end.html
http://comments.gmane.org/gmane.os.openbsd.misc/199639

I know there is an implementation of AES-NI for cryptdev, but **I HAVE NOT 
TESTED IT (nor has anyone else on the pfSense team, AFAIK).

There seems to be an issue:
http://forum.pfsense.org/index.php/topic,54008.30.html
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-May/038762.html

In the meantime, it might be possible to use OpenVPN with a patched openssl 
library to achieve the results you desire (but now you’re off into DIY land.)  
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

That all said, we will find and fix the issue at some point.   (I’m actually in 
San Jose for the FreeBSD Vendor Summit, and plan to bring it up as a potential 
issue.)

Jim

___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Thinker Rix]

On 2013-11-06 15:29, Jim Thompson wrote:

On Nov 6, 2013, at 7:22, Vick Khera  wrote:

pfSense lists the AES-NI as a supported option for crypto acceleration.  
pfSense will use it for OpenVPN and IPsec if you tell it to. There's a config 
setting for it.

I'm not aware if any performance testing for AES-NI on pfSense.

There are reports that FreeBSD doesn't support AES-NI very well.


Thank you for this information, Jim. So I figure, that buying the Xeon 
just for it's AES functions would (currently) be a waste of money.


Best regards
Thinker Rix
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Thinker Rix]

On 2013-11-06 15:22, Vick Khera wrote:


On Wed, Nov 6, 2013 at 12:53 AM, Thinker Rix > wrote:


Would pfSense use this CPU instructions so to
hardware-encrypt/decrypt all VPN traffic (openVPN)?
Woud pfSense benefit from this in any other way, too?


pfSense lists the AES-NI as a supported option for crypto 
acceleration.  pfSense will use it for OpenVPN and IPsec if you tell 
it to. There's a config setting for it.


As to your question of is it worth the cost, that depends on how much 
VPN traffic you have. The Xeon will handle a damn lot of traffic all 
on its own. If you are pushing more than 40Mbps on the VPN, then 
perhaps consider the extra cost. If it is low, like under 5 or 10Mbps, 
then I'd probably suggest that it is not worth the cost.


As a reference, between my data center and my primary office, I have 
an IPsec tunnel.  The office runs on an old Intel 32-bit Pentium 4 
2.4GHz dual core server.  The data center runs on Intel Xeon E31220L @ 
2.20GHz quad-core. Neither one has any built-in cryptodev supported 
devices. The IPsec tunnel maxes out at about 20Mbps during large file 
backups. I don't think it would go any faster with hardware 
acceleration, and the load on these boxes hovers around 0 still. The 
data center firewall is also busy pushing over 100Mpbs of regular 
traffic to hundreds of clients as well.




Hi Vick,

Thank you for your reference, it is very valuable for me!
I guess I will go with a Pentium (Ivy Bridge) 2x 3.0 GHz CPU.

What do you think is the reason for your VPN traffic maxing out at 
20Mpbs (I assume that your connection is not the traffic bottle neck, 
right?), although your CPUs are almost idle?


Best regards
Thinker Rix
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Jim Thompson]

> On Nov 6, 2013, at 7:22, Vick Khera  wrote:
> 
> pfSense lists the AES-NI as a supported option for crypto acceleration.  
> pfSense will use it for OpenVPN and IPsec if you tell it to. There's a config 
> setting for it.

I'm not aware if any performance testing for AES-NI on pfSense. 

There are reports that FreeBSD doesn't support AES-NI very well. 

Jim 
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

Re: [pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Vick Khera]

On Wed, Nov 6, 2013 at 12:53 AM, Thinker Rix wrote:

> Would pfSense use this CPU instructions so to hardware-encrypt/decrypt all
> VPN traffic (openVPN)?
> Woud pfSense benefit from this in any other way, too?
>

pfSense lists the AES-NI as a supported option for crypto acceleration.
 pfSense will use it for OpenVPN and IPsec if you tell it to. There's a
config setting for it.

As to your question of is it worth the cost, that depends on how much VPN
traffic you have. The Xeon will handle a damn lot of traffic all on its
own. If you are pushing more than 40Mbps on the VPN, then perhaps consider
the extra cost. If it is low, like under 5 or 10Mbps, then I'd probably
suggest that it is not worth the cost.

As a reference, between my data center and my primary office, I have an
IPsec tunnel.  The office runs on an old Intel 32-bit Pentium 4 2.4GHz dual
core server.  The data center runs on Intel Xeon E31220L @ 2.20GHz
quad-core. Neither one has any built-in cryptodev supported devices. The
IPsec tunnel maxes out at about 20Mbps during large file backups. I don't
think it would go any faster with hardware acceleration, and the load on
these boxes hovers around 0 still. The data center firewall is also busy
pushing over 100Mpbs of regular traffic to hundreds of clients as well.
___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list

[pfSense] AES-NI support of the CPU: Does it make sense for VPN with pfSense?

[Thinker Rix]

Hello all,

as I am planning to buy new hardware for pfSense, I was wondering if it 
is worthy to buy a CPU that supports "AES new instructions", i.e. 
hardware-support for AES encyption.


Would pfSense use this CPU instructions so to hardware-encrypt/decrypt 
all VPN traffic (openVPN)?

Woud pfSense benefit from this in any other way, too?

The motherboards that I want to buy unfortunately support AES-NI only 
with Xeons that currently start from approx 170 €. If I would take a CPU 
without AES-IN, I could go with a dual-Pentium for 40€. What impact 
would you expect from AES-IN, in regards to the fact tht I will be 
having traffic from VPN secured WLAN with approx 300-450 Mpbs and VPN 
to/from the internet, 1-2 users at a time max. Do you think the AES-IN 
would be worthy the price premium of the Xeon for my case, e.g. because 
it would reduce VPN latency, etc., or is it just a pure waste of money 
in my case?


Best regards
Thinker Rix



___
List mailing list
List@lists.pfsense.org
http://lists.pfsense.org/mailman/listinfo/list