MDB (Memory-Mapped Database), a read-optimized database library
Hi, I wonder if this is an interesting library for FreeBSD for some userland applications. Perhaps an embedded app running on very low end hardware might benefit. Interesting use of an old idea in any case. http://www.openldap.org/pub/hyc/mdm-slides.pdf - Mark ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
SMP Version of tar
I would be willing to work on a SMP version of tar (initially just gzip or something). I don't have the best experience in compression, and how to multi-thread it, but I think I would be able to learn and help out. Note: I would like to make this for *BSD under the BSD license. I am aware that there are already tools to do this (under GPL), but I would really like to see this existent in the FreeBSD base. Anyone interested? -Brandon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: SMP Version of tar
On 1 October 2012 12:51, Brandon Falk bfalk_...@brandonfa.lk wrote: I would be willing to work on a SMP version of tar (initially just gzip or something). I don't have the best experience in compression, and how to multi-thread it, but I think I would be able to learn and help out. Note: I would like to make this for *BSD under the BSD license. I am aware that there are already tools to do this (under GPL), but I would really like to see this existent in the FreeBSD base. Anyone interested? Tim Kientzle (cced) is the person to talk with. Try to keep questions on the mailing list with him CCed so we all learn. :) -- Eitan Adler ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: system hangs during dump + compress usb2-drive
From: Xin Li delp...@delphij.net To: Jin Guojun jguo...@sbcglobal.net Cc: questi...@freebsd.org; hack...@freebsd.org Sent: Sun, September 30, 2012 1:07:40 PM Subject: Re: system hangs during dump + compress usb2-drive On 9/29/12 10:49 PM, Jin Guojun wrote: In FreeBSD 8.3 release (possibly in earlier release), dump a file system has 2-3GB or more content can cause system hang in a specific case (pipe to compression): dump FS-on-SATA-drive usb-drive OK dump FS-on-SATA-drive | anyCompress sata-drive OK mv a-large-dump-file from STAT drive to a USB drive OK dump small-FS-on-SATA-drive | anyCompress usb-drive OK small -- 1.8GB or less dump large-FS-on-SATA-drive | anyCompress usb-drive hang content is 3GB or larger (did not try around 2GB yet) When system hangs, no sub system, such video, network, etc, will function. Typically, the unfinished compressed dump file is around 1.5-2.7GB, so guessing dumped file content is close to or over 2GB when failure occurred. Has anyone encountered the same problem? Because this usually takes a few hours to occur, this is hard to watch how/when it happens. Is any way to debug or determine what status the system is? For starters I'd use a different console for doing procstat -kk -a and see what the system is doing. (Perhaps also top) I *think* that if it's just hanging for some time, it's probably because the system is trying to take a snapshot? It takes time on UFS when creating and removing the snapshot. Just a guess... Cheers, --- Not sure how to use a different console. No tty is functioning (neither ttyv? nor over network). You are right on a different case -- mount /dev/da0s4d /mnt# mount a usb drive cd /mnt ssh remote-liux-host tar -cf - 8GB_FS | tar -xf - In this case, doing ls -l /mnt or df will hangs, but system is still alive. The network is 45Mbps. I have no idea how long it took the tar to finish since machine is 60 km away. When I left there last Friday, only 400MB was done in one hour. I will get the processing time tomorrow. The problem we can see now is that tar (probably the pipe) process only finish with 4GB. # df Filesystem1K-blocks Used Avail Capacity Mounted on /dev/ad4s3a 1012974355348 57659038%/ devfs 1 1 0 100%/dev ... /dev/da0s4d 1027486774 4198246 941089588 0%/mnt So, I suspect this is a pipe problem, not a compress issue. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re[2]: How to claim only some of USB interfaces of a composite USB device
Tested the patch with FreeBSD 9.1. It works. Thanks. Sun, 30 Sep 2012 21:33:31 +0200 от Hans Petter Selasky hsela...@c2i.net: On Saturday 29 September 2012 11:25:21 geoffrey levand wrote: geoffrey levand Can you verify this patch: http://svn.freebsd.org/changeset/base/241078 --HPS ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Wednesday, June 6, 2012 8:36:04 PM UTC-5, Mark Felder wrote: Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to my PC tonight, but this hopefully is enough to replicate the crash for any curious followers: ESXi 5 9 or 9-STABLE HAST 1 cpu is fine 1GB of ram UFS SUJ on HAST device No special loader.conf, sysctl, etc No need for VMWare tools Run Bonnie++ on the HAST device We can get the crash to happen on the first run of bonnie++ right now. I'll post the exact specs and precise command run in the PR. We found an old post from 2004 when we looked up the process state obtained from CTRL+T -- flswai -- which describes the symptoms nearly perfectly. http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2004-02/0250.html Hopefully this gets us closer to a fix... Is this a crash or a hang? Over the past couple of weeks, I've been working with a FreeBSD 9.1RC1 system under VMware ESXi 5.0 with a 64GB UFS root FS and 2TB ZFS filesystem mounted via a virtual LSI SAS interface. Sometimes during heavy I/O load (rsync from other servers) on the ZFS FS, this shows up in /var/log/messages: Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 5 ee 60 16 0 1 0 0 Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 42 51 0 1 0 0 Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 64 51 0 1 0 0 Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 66 51 0 1 0 0 Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy ... Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 41 f3 94 99 0 1 0 0 Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): Retrying command These have been happening roughly every other day. mpt0 and em0 were sharing int 18, so today I put hint.mpt.0.msi_enable=1 into /boot/devices.hints and rebooted; now mpt0 is using int 256. I'll see if it helps. Guy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
On Mon, 01 Oct 2012 15:00:40 -0500, guy.hel...@gmail.com wrote: Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 5 ee 60 16 0 1 0 0 Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 42 51 0 1 0 0 Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 64 51 0 1 0 0 Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): Retrying command Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 66 51 0 1 0 0 Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy ... Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 41 f3 94 99 0 1 0 0 Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): Retrying command Sometimes you'll see this before a crash, but not every time. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Fwd: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program
Hi Garrett, From: Garrett Cooper yaneg...@gmail.com Subject: [CFT/RFC]: refactor bsd.prog.mk to understand multiple = programs instead of a singular program Date: September 2, 2012 11:01:09 PM PDT To: freebsd-hackers@freebsd.org Cc: freebsd-a...@freebsd.org Arch freebsd-a...@freebsd.org =20 Hello, I've been a bit busy working on porting over ATF from NetBSD, and Thanks, we've been using ATF in Junos for a while and glad to see it being imported to FreeBSD. one of the pieces that's currently not available in FreeBSD that's available in NetBSD is the ability to understand and compile multiple programs. In order to do this I had to refactor bsd.prog.mk (a lot). A change like this to bsd.prog.mk can have considerable fallout. Eg. any makefile that tweaks OBJS is suddenly out of luck. Not to mention the fact that bsd.prog.mk goes from being relatively simple, to unspeakably hard to read, and all for rather limited return. Apart from ATF, is there any huge demand for building multiple progs in the same directory? FWIW we use progs.mk (as bsd.progs.mk) from ftp://ftp.netbsd.org/pub/NetBSD/misc/sjg/mk-*.tar.gz It isn't ideal, but it certainly avoids a lot of churn and complexity for what is essentially a corner case. We have an atf.test.mk which leverages that. It also handles automatically running the tests if building for the host. This allows us to fail the build if any unit-tests do not pass. Note, atf.test.mk is loosely derrived from NetBSD's bsd.tests.mk, but named for what it is (ATF specific tests) since we wanted the flexibility to have more than one test framework. --sjg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program
Hi Simon! On Oct 1, 2012, at 3:31 PM, Simon J. Gerraty wrote: Hi Garrett, From: Garrett Cooper yaneg...@gmail.com Subject: [CFT/RFC]: refactor bsd.prog.mk to understand multiple = programs instead of a singular program Date: September 2, 2012 11:01:09 PM PDT To: freebsd-hackers@freebsd.org Cc: freebsd-a...@freebsd.org Arch freebsd-a...@freebsd.org =20 Hello, I've been a bit busy working on porting over ATF from NetBSD, and Thanks, we've been using ATF in Junos for a while and glad to see it being imported to FreeBSD. Thanks (and I appreciate the help marcel@ has given by taking the first step in importing it). one of the pieces that's currently not available in FreeBSD that's available in NetBSD is the ability to understand and compile multiple programs. In order to do this I had to refactor bsd.prog.mk (a lot). A change like this to bsd.prog.mk can have considerable fallout. Eg. any makefile that tweaks OBJS is suddenly out of luck. Well… any Makefiles attempting to be clever (e.g. crunchgen) are out of luck and would need to change. Most [include] Makefiles will see virtually no change. Not to mention the fact that bsd.prog.mk goes from being relatively simple, to unspeakably hard to read, and all for rather limited return. I sort of tried to match bsd.prog.mk in NetBSD in the beginning, but things diverged from there… I'm sure things could be made more readable so any comments would be much appreciated :)! Apart from ATF, is there any huge demand for building multiple progs in the same directory? Well, I noticed that there are a couple ugly messes in the gnu/... directory that attempt to work around the fact that bsd.prog.mk doesn't support more than one program by being tricky and emulating bsd.prog.mk instead of solving the generic problem (and once I got over that compatibility issue I stopped tracking this class of consumer). Most consumers don't care (they split up programs into different directories or use hardlink hacks/basename detection to differentiate one program functionally from another). Getting back to the idea at hand, the enhancement goal was to get the testcases to live [and optionally be built/installed] with the source code to avoid bitrot; I know this isn't the current NetBSD design, but this is what was requested be done by gnn, marcel, and mdf, and I agree. It order to do this, I need to be able to build multiple programs so things are as transparent. On the flipside, bsd.prog[s].mk can live on its own, be pulled in automatically by bsd.test.mk, and that would be it. This encourages code duplication though and bugs can persist in either Makefile, when I'd rather work out all the kinks in whatever succeeds the legacy bsd.prog.mk file. FWIW we use progs.mk (as bsd.progs.mk) from ftp://ftp.netbsd.org/pub/NetBSD/misc/sjg/mk-*.tar.gz It isn't ideal, but it certainly avoids a lot of churn and complexity for what is essentially a corner case. This requires pulling bmake into FreeBSD proper in order for things to function the last I checked as bsd.prog.mk depends upon bmake directives in order to function properly 100% of the time (and there are external dependencies and assumptions one has to deal with when using bsd.prog.mk, etc from NetBSD that made that a bit of a no-go without refactoring/pulling in bits from NetBSD). I could be wrong though, so please let me know if I am :). Ideally however, I would like doing this compared to running a custom build infrastructure, but (as you probably know) this requires rototilling the current FreeBSD build system to a large degree (definitely not impossible… just needs time and effort). We have an atf.test.mk which leverages that. It also handles automatically running the tests if building for the host. This allows us to fail the build if any unit-tests do not pass. Hmm… that wasn't really the end-goal of bsd.test.mk based on my reading, but I'm sure Julio would be interested in it. I need to do adjusting in order to allow automatically executing testcases compatible to the host architecture, but that isn't hard to do (no more than a couple hours work). Note, atf.test.mk is loosely derrived from NetBSD's bsd.tests.mk, but named for what it is (ATF specific tests) since we wanted the flexibility to have more than one test framework. bsd.test.mk in NetBSD isn't completely test-agnostic, but it isn't entirely ATF centric either. I'm trying not to stray too far from NetBSD in this area because there really isn't any reason to do so. FWIW I'd like to see other test frameworks (lua, unittest//nose, etc) just snap into bsd.test.mk as easily as possible as it would make some groups lives easier, but that requires a bit more thought for another day. Thanks for the feedback! -Garrett___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to
Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program
Not to mention the fact that bsd.prog.mk goes from being relatively simple, to unspeakably hard to read, and all for rather limited = return. This btw I think is the more important issue. I was looking at bsd.prog.mk in netbsd the other day. It has no business being that complex. Getting back to the idea at hand, the enhancement goal was to get the = testcases to live [and optionally be built/installed] with the source = code to avoid bitrot; I know this isn't the current NetBSD design, but = this is what was requested be done by gnn, marcel, and mdf, and I agree. = I agree, that's what we do too. It order to do this, I need to be able to build multiple programs so = things are as transparent. On the flipside, bsd.prog[s].mk can live on = We put the test cases in a subdir of the lib/prog This has multiple benefits, and eliminates any impact on the normal build of said libs/progs. Also a number of our teams find it necessary to create mock classes etc to adequately test their libs. Keeping all this in a tests/ subdir makes it easy, to keep things neat up to date, and ensure that the tests pass. Trying to do all that in the same dir as the lib would be ugly. FWIW we use progs.mk (as bsd.progs.mk) from ftp://ftp.netbsd.org/pub/NetBSD/misc/sjg/mk-*.tar.gz=20 It isn't ideal, but it certainly avoids a lot of churn and complexity for what is essentially a corner case. This requires pulling bmake into FreeBSD proper in order for things to = function the last I checked as bsd.prog.mk depends upon bmake directives = This is already happening ;-) Ideally however, I would like doing this compared to running a custom = build infrastructure, but (as you probably know) this requires = rototilling the current FreeBSD build system to a large degree = Actually building FreeBSD-current using bmake requires only modest changes. We have an atf.test.mk which leverages that. It also handles automatically running the tests if building for the host. This allows us to fail the build if any unit-tests do not pass. Hmm=85 that wasn't really the end-goal of bsd.test.mk based on my = reading, I know, but it is a very useful thing to be able to do. You can add knobs to controll such things of course. bsd.test.mk in NetBSD isn't completely test-agnostic, but it isn't = entirely ATF centric either. I'm trying not to stray too far from NetBSD = in this area because there really isn't any reason to do so. FWIW I'd = Sure - we imported ATF directly from NetBSD to minimize the churn and not had to change much at all. bsd.test.mk was a point worth diverging on though. like to see other test frameworks (lua, unittest//nose, etc) just snap = into bsd.test.mk as easily as possible as it would make some groups = Yes, but making one test.mk handle potentially lots of frameworks will get rather ugly quite quickly. Better to add per-framework .mk files, and perhaps have them include a bsd.test.mk which only handles high level logic rather than the details of the frameworks. That's laregly why we didn't use bsd.test.mk --sjg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
NFS server bottlenecks
I had an email conversation with Rick Macklem about six months ago about NFS server bottlenecks. I'm now in a position to observe my large-scale NFS server under an actual production load, so I thought I would update folks on what it looks like. This is a 9.1 prerelease kernel (I hope 9.1 will be released soon as I have four moe of these servers to deploy!). When under nearly 100% load on an 8-core (16-thread) Quanta QSSC-S99Q storage server, with a 10G network interface, pmcstat tells me this: PMC: [INST_RETIRED.ANY_P] Samples: 2727105 (100.0%) , 27 unresolved Key: q = exiting... %SAMP IMAGE FUNCTION CALLERS 29.3 kernel _mtx_lock_sleep nfsrvd_updatecache:10.0 nfsrvd_getcache:7.4 ... 9.5 kernel cpu_search_highest cpu_search_highest:8.1 sched_idletd:1.4 7.4 zfs.ko lzjb_decompress zio_decompress 4.3 kernel _mtx_lock_spin turnstile_trywait:2.2 pmclog_reserve:1.0 ... 4.0 zfs.ko fletcher_4_nativezio_checksum_error:3.1 zio_checksum_compute:0.8 3.6 kernel cpu_search_lowestcpu_search_lowest 3.3 kernel nfsrc_trimcache nfsrvd_getcache:1.6 nfsrvd_updatecache:1.6 2.3 kernel ipfw_chk ipfw_check_hook 2.1 pmcstat_init 1.1 kernel _sx_xunlock 0.9 kernel _sx_xlock 0.9 kernel spinlock_exit This does seem to confirm my original impression that the NFS replay cache is quite expensive. Running a gprof(1) analysis on the same PMC data reveals a bit more detail (I've removed some uninteresting parts of the call graph): called/total parents index %timeself descendents called+selfname index called/total children 4881.00 2004642.70 932627/932627 svc_run_internal [2] [4] 45.1 4881.00 2004642.70 932627 nfssvc_program [4] 13199.00 504436.33 584319/584319 nfsrvd_updatecache [9] 23075.00 403396.18 468009/468009 nfsrvd_getcache [14] 1032.25 416249.442239/2284svc_sendreply_mbuf [15] 6168.00 381770.44 11618/11618 nfsrvd_dorpc [24] 3526.8786869.88 112478/112514 nfsrvd_sentcache [74] 890.0050540.894252/4252svc_getcred [101] 14876.6032394.264177/24500 crfree cycle 3 [263] 11550.1125150.733243/24500 free cycle 3 [102] 1348.8815451.662716/16831 m_freem [59] 4066.61 216.811434/1456svc_freereq [321] 2342.15 677.40 557/1459malloc_type_freed [265] 59.14 1916.84 134/2941crget [113] 1602.250.00 322/9682bzero [105] 690.930.00 43/44 getmicrotime [571] 287.227.33 138/1205prison_free [384] 233.610.00 60/798 PHYS_TO_VM_PAGE [358] 203.120.00 94/230 nfsrv_mallocmget_limit [632] 151.760.00 51/1723pmap_kextract [309] 0.78 70.28 9/3281_mtx_unlock_sleep [154] 19.22 16.88 38/400403 nfsrc_trimcache [26] 11.05 21.74 7/197 crsetgroups [532] 30.370.00 11/6592critical_enter [190] 25.500.00 9/36 turnstile_chain_unlock [844] 24.860.00 3/7 nfsd_errmap [913] 12.368.57 8/2145in_cksum_skip [298] 9.103.59 5/12455 mb_free_ext [140] 1.844.85 2/2202VOP_UNLOCK_APV [269] --- 0.490.15 1/1129009 uhub_explore [1581] 0.490.15 1/1129009 tcp_output [10] 0.490.15 1/1129009 pmap_remove_all [1141] 0.490.15 1/1129009 vm_map_insert [236] 0.490.15 1/1129009 vnode_create_vobject [281] 0.490.15 1/1129009 biodone [351] 0.490.15 1/1129009 vm_object_madvise [670] 0.490.15 1/1129009 xpt_done [483] 0.490.15 1/1129009 vputx [80] 0.490.15 1/1129009 vm_map_delete cycle 3 [49] 0.490.15 1/1129009 vm_object_deallocate cycle 3 [356] 0.490.15 1/1129009 vm_page_unwire [338] 0.490.15 1/1129009 pmap_change_wiring [318] 0.980.31 2/1129009 getnewvnode [227] 0.98
Re: SMP Version of tar
On Oct 1, 2012, at 9:51 AM, Brandon Falk wrote: I would be willing to work on a SMP version of tar (initially just gzip or something). I don't have the best experience in compression, and how to multi-thread it, but I think I would be able to learn and help out. Note: I would like to make this for *BSD under the BSD license. I am aware that there are already tools to do this (under GPL), but I would really like to see this existent in the FreeBSD base. Anyone interested? Great! First rule: be skeptical. In particular, tar is so entirely disk-bound that many performance optimizations have no impact whatsoever. If you don't do a lot of testing, you can end up wasting a lot of time. There are a few different parallel command-line compressors and decompressors in ports; experiment a lot (with large files being read from and/or written to disk) and see what the real effect is. In particular, some decompression algorithms are actually faster than memcpy() when run on a single processor. Parallelizing such algorithms is not likely to help much in the real world. The two popular algorithms I would expect to benefit most are bzip2 compression and lzma compression (targeting xz or lzip format). For decompression, bzip2 is block-oriented so fits SMP pretty naturally. Other popular algorithms are stream-oriented and less amenable to parallelization. Take a careful look at pbzip2, which is a parallelized bzip2/bunzip2 implementation that's already under a BSD license. You should be able to get a lot of ideas about how to implement a parallel compression algorithm. Better yet, you might be able to reuse a lot of the existing pbzip2 code. Mark Adler's pigz is also worth studying. It's also license-friendly, and is built on top of regular zlib, which is a nice technique when it's feasible. There are three fundamentally different implementation approaches with different complexity/performance issues: * Implement as a stand-alone executable similar to pbzip2. This makes your code a lot simpler and makes it reasonably easy for people to reuse your work. This could work with tar, though it could be slightly slower than the in-process version due to the additional data-copying and process-switch overhead. * Implement within libarchive directly. This would benefit tar and a handful of other programs that use libarchive, but may not be worth the complexity. * Implement as a standalone library with an interface similar to zlib or libbz2 or liblzma. The last would be my personal preference, though it's probably the most complex of all. That would easily support libarchive and you could create a simple stand-alone wrapper around it as well, giving you the best of all worlds. If you could extend the pigz technique, you might be able to build a multi-threaded compression library where the actual compression was handled by an existing single-threaded library. Since zlib, bzlib, and liblzma already have similar interfaces, your layer might require only a thin adapter to handle any of those three. *That* would be very interesting, indeed. Sounds like a fun project. I wish I had time to work on something like this. Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
