Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-23 Thread Neel Natu
Hi Markiyan,

Fixed in r259737:
http://svnweb.freebsd.org/changeset/base/259737

best
Neel

On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-23 Thread Markiyan Kushnir
Works for me (I'm at r259742), no panic. Thanks!

--
Markiyan.

2013/12/23 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 Fixed in r259737:
 http://svnweb.freebsd.org/changeset/base/259737

 best
 Neel

 On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-14 Thread Markiyan Kushnir
2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing


 Thanks.

 It looks like something funky going on with the vcpu state. Do you
 know if there was any access to the VM via 'bhyvectl' close to the
 time of the panic?


 Well, I don't know if there was. I would set up the same scenario
 again + a script running on the host querying bhyvectl. May be I would
 catch it again. Please let me know if all it makes sense, and if so,
 how it can be made better.


To make it clear -- I didn't run bhyvectl when the VM was running, so
I can tell for sure that nobody was trying to interact with the VM
during its run. My first thought was that it would make sense to
(periodically) query state of the VM using bhyvectl to get info about
what was going on when it comes close to the crash...

--
Markiyan.

 --
 Markiyan.

 best
 Neel

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-14 Thread Markiyan Kushnir
2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing


 Thanks.

 It looks like something funky going on with the vcpu state. Do you
 know if there was any access to the VM via 'bhyvectl' close to the
 time of the panic?


 Well, I don't know if there was. I would set up the same scenario
 again + a script running on the host querying bhyvectl. May be I would
 catch it again. Please let me know if all it makes sense, and if so,
 how it can be made better.


 To make it clear -- I didn't run bhyvectl when the VM was running, so
 I can tell for sure that nobody was trying to interact with the VM
 during its run. My first thought was that it would make sense to
 (periodically) query state of the VM using bhyvectl to get info about
 what was going on when it comes close to the crash...

 --
 Markiyan.

Ok, I was able to hit a similarly looking panic exactly when trying to
run bhyvectl --vm=altroot-bhyve --get-all while the VM was busy with
its own pkg delete -af. When the VM is mostly idle, bhyvectl
--get-all goes smoothly.

Now I'm thinking I might inadvertently run bhyvectl over my heavily
running VM once when I was not at the desktop and wanted to see how
things were going remotely from my office...

Here are core.txt of the two panics in a row:

https://drive.google.com/file/d/0B9Q-zpUXxqCnMUxVdGwxSEs1dE0/edit?usp=sharing

https://drive.google.com/file/d/0B9Q-zpUXxqCnREtuTXpabnQ5QXM/edit?usp=sharing

--
Markiyan.


 --
 Markiyan.

 best
 Neel

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to 
 freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-14 Thread Neel Natu
Hi Markiyan,

On Sat, Dec 14, 2013 at 6:00 AM, Markiyan Kushnir
markiyan.kush...@gmail.com wrote:
 2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing


 Thanks.

 It looks like something funky going on with the vcpu state. Do you
 know if there was any access to the VM via 'bhyvectl' close to the
 time of the panic?


 Well, I don't know if there was. I would set up the same scenario
 again + a script running on the host querying bhyvectl. May be I would
 catch it again. Please let me know if all it makes sense, and if so,
 how it can be made better.


 To make it clear -- I didn't run bhyvectl when the VM was running, so
 I can tell for sure that nobody was trying to interact with the VM
 during its run. My first thought was that it would make sense to
 (periodically) query state of the VM using bhyvectl to get info about
 what was going on when it comes close to the crash...

 --
 Markiyan.

 Ok, I was able to hit a similarly looking panic exactly when trying to
 run bhyvectl --vm=altroot-bhyve --get-all while the VM was busy with
 its own pkg delete -af. When the VM is mostly idle, bhyvectl
 --get-all goes smoothly.

 Now I'm thinking I might inadvertently run bhyvectl over my heavily
 running VM once when I was not at the desktop and wanted to see how
 things were going remotely from my office…


Thanks, that helps a lot because I can see how bhyvectl could perturb
the state of the vcpu and eventually lead to a panic.

I'll submit a fix for this ASAP.

best
Neel

 Here are core.txt of the two panics in a row:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnMUxVdGwxSEs1dE0/edit?usp=sharing

 https://drive.google.com/file/d/0B9Q-zpUXxqCnREtuTXpabnQ5QXM/edit?usp=sharing

 --
 Markiyan.


 --
 Markiyan.

 best
 Neel

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to 
 freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


11.0-CURRENT panic while running a bhyve instance

2013-12-13 Thread Markiyan Kushnir
Forgot to fill the Subject: header, re-posting it fixed.

--
Markiyan


-- Forwarded message --
From: Markiyan Kushnir markiyan.kush...@gmail.com
Date: 2013/12/13
Subject:
To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


I started some ports to compile inside a bhyve instance:

root@vm:~ # uname -a
FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
r259250: Thu Dec 12 14:17:32 EET 2013
r...@vm.mkushnir.zapto.org:/
usr/obj/usr/src.svnup/sys/MAREK  amd64

and left it running unattended. Approx. 2 hours later the host went to
panic. The bhyve instance survived after the panic and I could be able
to complete my ports compilation.

core.txt attached (gzipped)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-13 Thread John Baldwin
On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

The mailing lists strips attachments, can you post it at a URL?

 --
 Markiyan
 
 
 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org
 
 
 I started some ports to compile inside a bhyve instance:
 
 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64
 
 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.
 
 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
 

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-13 Thread Markiyan Kushnir
2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


Shared here:

https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing

--
Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-13 Thread Neel Natu
Hi Markiyan,

On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing


Thanks.

It looks like something funky going on with the vcpu state. Do you
know if there was any access to the VM via 'bhyvectl' close to the
time of the panic?

best
Neel

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: 11.0-CURRENT panic while running a bhyve instance

2013-12-13 Thread Markiyan Kushnir
2013/12/14 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing


 Thanks.

 It looks like something funky going on with the vcpu state. Do you
 know if there was any access to the VM via 'bhyvectl' close to the
 time of the panic?


Well, I don't know if there was. I would set up the same scenario
again + a script running on the host querying bhyvectl. May be I would
catch it again. Please let me know if all it makes sense, and if so,
how it can be made better.

--
Markiyan.

 best
 Neel

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-current@freebsd.org, freebsd-virtualizat...@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org