Hi, I've gotten this kernel oops after running the system continuously for about 14-15 hours. This only happens when the system is under load most of the time (load is around 95% of CPU most of the time). If the system load is lower (idle being 40-50% load) this doesn't happen.
Kernel source: https://github.com/SolidRun/linux-linaro-stable-mx6 Kernel version: 3.10.30 (patched with aufs3.10.x) Kernel arch: armv7 dual processor (SMP) - cubox-i Aufs branches: br0:/aufs/pup_rw=rw br1:/aufs/pup_save=ro+wh br2:/aufs/pup_ro=rr br3:/aufs/kernel-modules=rr br4:/aufs/pup_init=rr br0 is tmpfs, br1 is loop-mounted ext3, br2 and br3 are loop-mounted squashfs, br4 is tmpfs. I've already compiled the kernel with DEBUG_INFO, AUFS_DEBUG, extended BUG_ON, and DYNAMIC_PRINTK (but I don't see aufs debug statements in dmesg although I have already echo "module aufs +p" > /sys/kernel/debug/dynamic_debug/control at the beginning). AUFS configuration: CONFIG_AUFS_FS=y CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y CONFIG_AUFS_HNOTIFY=y CONFIG_AUFS_HFSNOTIFY=y # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SHWH is not set # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set CONFIG_AUFS_BDEV_LOOP=y CONFIG_AUFS_DEBUG=y CONFIG_AUFS_MAGIC_SYSRQ=y I have only used the first 3 aufs patches (aufs3-kbuild, aufs3-base, aufs3-mmap). I didn't use aufs3-loopback (actually, in earlier builds, I used that too - and got the same bug). The version of aufs used is the latest (as of yesterday) commit for 3.10.x branch: b72117ef0c528c4f6013845e6e704d92682fc913 My apology for the large email size: I'm attaching the oops detail at the end of this email since I'm not sure whether the list accept attachments). >From what I can see, it is the first kernel null pointer that is the problem. >The kernel doesn't immediately crash and continues to run, and all subsequent >access to aufs fails and results in more BUG_ON later on. I'm happy to supply any other details as needed. cheers! -- James B <jamesbond3...@gmail.com> ---------- [55096.152331] Unable to handle kernel NULL pointer dereference at virtual address 0000003f [55096.159197] pgd = a853c000 [55096.160610] [0000003f] *pgd=380db831, *pte=00000000, *ppte=00000000 [55096.165701] Internal error: Oops: 17 [#1] SMP ARM [55096.169105] Modules linked in: iptable_filter ip_tables x_tables snd_soc_omap_hdmi_codec brcmfmac mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc brcmutil snd_soc_imx_hdmi snd_soc_fsl_hdmi ipu_fg_overlay_sdc snd_soc_fsl_spdif [55096.189965] CPU: 0 PID: 31816 Comm: mv Not tainted 3.10.30 #10 [55096.194501] task: 8e3f9c00 ti: a8cdc000 task.ti: a8cdc000 [55096.198608] PC is at dput+0x30/0x214 [55096.200889] LR is at aufs_rename+0x1ea4/0x1ee4 [55096.204034] pc : [<800d6da8>] lr : [<8022f908>] psr: a0000013 sp : a8cddde0 ip : 00000000 fp : aa3814c8 [55096.212908] r10: ac5a9600 r9 : a8570d20 r8 : a8570d00 [55096.216833] r7 : 809aef48 r6 : 00000000 r5 : ffffffef r4 : ffffffef [55096.222059] r3 : 00000001 r2 : 00000000 r1 : 00000001 r0 : ffffffef [55096.227289] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [55096.233125] Control: 10c53c7d Table: 3853c04a DAC: 00000015 [55096.237572] Process mv (pid: 31816, stack limit = 0xa8cdc238) [55096.242018] Stack: (0xa8cddde0 to 0xa8cde000) [55096.245079] dde0: a8570dc4 ffffffef 00000000 8022f908 add2bb40 adc32810 ade19400 00000008 [55096.251959] de00: a8cdc018 00000001 aa2d2a00 00004000 000041ed 00004000 ade19400 00000003 [55096.258840] de20: 00000001 80229340 adc32810 aa2d2a38 aa2d0500 aa2d2a38 adc32810 aa2d2a38 [55096.265720] de40: ab0f9699 0000000d 000000d0 800bf064 ab0f9699 a8786100 ab0f968c ab0f9660 [55096.272599] de60: 00000001 aa2d2a38 aa2d2a38 8022da64 a8786100 00000000 aa3814c8 800d059c [55096.279494] de80: ffffff9c 800cc0a8 00000800 00000000 00000000 8e33d000 8e33b000 00000000 [55096.286383] dea0: ab0f6880 7e8c8e86 ffffff9c 7e8c8eab ffffff9c 800d131c 090276d9 00000008 [55096.293267] dec0: ab0f9660 ab0f6880 00000000 aa3814c8 add2b9d0 ab0f6880 1782a13f 0000000c [55096.300146] dee0: 8e33d028 00000001 00000000 aa10ae58 aa2d2a38 00000000 00000002 00000000 [55096.307026] df00: 00000000 7e8c8e86 7e8c8ac4 800084e8 ffffff9c 7e8c8eab a8cddf50 800d0a28 [55096.313905] df20: 00000000 00000000 add2b9d0 ab0f6880 090276d9 00000008 8e33b028 00000001 [55096.320784] df40: 00000000 aa10ae58 aa2d2a38 00000800 00000002 00000000 00000000 ac3c41ed [55096.327663] df60: 00000002 00000000 00000000 00000000 0000003c 00000000 53c45451 09896828 [55096.334543] df80: 53c3aa31 0000000e 00000000 00000000 00000026 8000ec84 a8cdc000 00000000 [55096.341422] dfa0: 7e8c8ac4 8000eb00 0000000e 00000000 7e8c8e86 7e8c8eab 00000001 00000001 [55096.348301] dfc0: 0000000e 00000000 00000000 00000026 00004000 7e8c8b6c 7e8c8e86 7e8c8ac4 [55096.355180] dfe0: 76e33300 7e8c8814 0000c6ec 76e3330c 60000010 7e8c8e86 00000000 00000000 [55096.362071] [<800d6da8>] (dput) from [<8022f908>] (aufs_rename+0x1ea4/0x1ee4) [55096.367915] [<8022f908>] (aufs_rename) from [<800d059c>] (vfs_rename+0x170/0x454) [55096.374106] [<800d059c>] (vfs_rename) from [<800d131c>] (SyS_renameat+0x244/0x280) [55096.380385] [<800d131c>] (SyS_renameat) from [<8000eb00>] (ret_fast_syscall+0x0/0x30) [55096.386918] Code: f57ff04f e320f004 e3540000 0a000039 (e5943050) [55096.391797] ---[ end trace 35f9c046959b15ac ]--- [58332.888762] ------------[ cut here ]------------ [58332.892108] kernel BUG at fs/aufs/super.h:414! [58332.895273] Internal error: Oops - BUG: 0 [#2] SMP ARM [58332.899115] Modules linked in: iptable_filter ip_tables x_tables snd_soc_omap_hdmi_codec brcmfmac mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc brcmutil snd_soc_imx_hdmi snd_soc_fsl_hdmi ipu_fg_overlay_sdc snd_soc_fsl_spdif [58332.920036] CPU: 1 PID: 31816 Comm: confirm_browser Tainted: G D 3.10.30 #10 [58332.926662] task: 8ed3c000 ti: ac784000 task.ti: ac784000 [58332.930781] PC is at au_do_flush+0x280/0x288 [58332.933771] LR is at __down_read+0xe4/0xe8 [58332.936595] pc : [<802213d0>] lr : [<8066a304>] psr: 200f0013 sp : ac785f68 ip : 00000000 fp : 000abd50 [58332.945492] r10: a9f6ec00 r9 : 80223b54 r8 : ac784028 [58332.949437] r7 : ade19400 r6 : ac9d8300 r5 : aa577e38 r4 : adc32810 [58332.954678] r3 : 00000001 r2 : 00000080 r1 : ac0d4000 r0 : 00007c47 [58332.959921] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [58332.965772] Control: 10c53c7d Table: 1fb9004a DAC: 00000015 [58332.970242] Process confirm_browser (pid: 31816, stack limit = 0xac784238) [58332.975836] Stack: (0xac785f68 to 0xac786000) [58332.978912] 5f60: 80222c48 ac9d8300 a9f6ec00 00000002 00000006 8000ec84 [58332.985811] 5f80: ac784000 00000000 000abd50 800c0dec 000a4558 00007c48 00000000 800c21cc [58332.992711] 5fa0: 00c504f0 8000eb00 000a4558 00007c48 000000ff 00000003 00000000 00c504f0 [58332.999608] 5fc0: 000a4558 00007c48 00000000 00000006 00c52b70 000abf9c 000a494c 000abd50 [58333.006502] 5fe0: 00000000 7ebcf65c 0001bf00 76e951dc 600f0010 000000ff 00000000 00000000 [58333.013409] [<802213d0>] (au_do_flush) from [<800c0dec>] (filp_close+0x34/0x88) [58333.019431] [<800c0dec>] (filp_close) from [<800c21cc>] (SyS_close+0x20/0x48) [58333.025276] [<800c21cc>] (SyS_close) from [<8000eb00>] (ret_fast_syscall+0x0/0x30) [58333.031550] Code: e3a01068 e348007d ebf81909 eaffffb2 (e7f001f2) [58333.036346] ---[ end trace 35f9c046959b15ad ]--- [60454.305612] ------------[ cut here ]------------ [60454.308949] kernel BUG at fs/aufs/super.h:414! [60454.312095] Internal error: Oops - BUG: 0 [#3] SMP ARM [60454.315933] Modules linked in: iptable_filter ip_tables x_tables snd_soc_omap_hdmi_codec brcmfmac mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc brcmutil snd_soc_imx_hdmi snd_soc_fsl_hdmi ipu_fg_overlay_sdc snd_soc_fsl_spdif [60454.336799] CPU: 1 PID: 31816 Comm: verify_daemons. Tainted: G D 3.10.30 #10 [60454.343419] task: ac331500 ti: 893ec000 task.ti: 893ec000 [60454.347536] PC is at au_do_flush+0x280/0x288 [60454.350537] LR is at __down_read+0xe4/0xe8 [60454.353354] pc : [<802213d0>] lr : [<8066a304>] psr: 200f0013 sp : 893edf68 ip : 00000000 fp : 00dcda59 [60454.362246] r10: a87cd400 r9 : 80223b54 r8 : 893ec028 [60454.366194] r7 : ade19400 r6 : ac474480 r5 : aa92e838 r4 : adc32810 [60454.371443] r3 : 00000001 r2 : 00000080 r1 : ac0d4000 r0 : 00007c47 [60454.376693] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [60454.382554] Control: 10c53c7d Table: 12a7404a DAC: 00000015 [60454.387022] Process verify_daemons. (pid: 31816, stack limit = 0x893ec238) [60454.392621] Stack: (0x893edf68 to 0x893ee000) [60454.395708] df60: 80222c48 ac474480 a87cd400 00000002 00000006 8000ec84 [60454.402600] df80: 893ec000 00000000 00dcda59 800c0dec 000a4558 00007c48 00000000 800c21cc [60454.409487] dfa0: 00dcc4f0 8000eb00 000a4558 00007c48 000000ff 00000003 00000000 00dcc4f0 [60454.416383] dfc0: 000a4558 00007c48 00000000 00000006 00000000 000abf9c 00000028 00dcda59 [60454.423270] dfe0: 00000000 7eb5a13c 0001bf00 76e171dc 600f0010 000000ff 00000000 00000000 [60454.430169] [<802213d0>] (au_do_flush) from [<800c0dec>] (filp_close+0x34/0x88) [60454.436189] [<800c0dec>] (filp_close) from [<800c21cc>] (SyS_close+0x20/0x48) [60454.442037] [<800c21cc>] (SyS_close) from [<8000eb00>] (ret_fast_syscall+0x0/0x30) [60454.448315] Code: e3a01068 e348007d ebf81909 eaffffb2 (e7f001f2) [60454.453112] ---[ end trace 35f9c046959b15ae ]--- [62617.310073] ------------[ cut here ]------------ [62617.313411] kernel BUG at fs/aufs/super.h:414! [62617.316560] Internal error: Oops - BUG: 0 [#4] SMP ARM [62617.320406] Modules linked in: iptable_filter ip_tables x_tables snd_soc_omap_hdmi_codec brcmfmac mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc brcmutil snd_soc_imx_hdmi snd_soc_fsl_hdmi ipu_fg_overlay_sdc snd_soc_fsl_spdif [62617.341296] CPU: 1 PID: 31816 Comm: verify_daemons. Tainted: G D 3.10.30 #10 [62617.347919] task: ac187b80 ti: 89fc6000 task.ti: 89fc6000 [62617.352031] PC is at si_read_lock+0x1cc/0x1f8 [62617.355101] LR is at __down_read+0xe4/0xe8 [62617.357904] pc : [<80206dcc>] lr : [<8066a304>] psr: 200f0013 sp : 89fc7d80 ip : 00000000 fp : 000abf54 [62617.366778] r10: 00004000 r9 : aa2a3000 r8 : 00000000 [62617.370719] r7 : 89fc6000 r6 : 00000008 r5 : ade19400 r4 : adc32810 [62617.375952] r3 : 00007c47 r2 : 00000001 r1 : ac0d4000 r0 : 00000080 [62617.381182] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [62617.387018] Control: 10c53c7d Table: 19b6c04a DAC: 00000015 [62617.391467] Process verify_daemons. (pid: 31816, stack limit = 0x89fc6238) [62617.397060] Stack: (0x89fc7d80 to 0x89fc8000) [62617.400146] 7d80: 00000000 00000001 000abf54 80008454 4f87559f 0004f875 aa2a3038 aa2a3038 [62617.407054] 7da0: ade19400 00000001 00000000 80229480 80229444 aa2a3038 aa2a3038 89fc7eb0 [62617.413965] 7dc0: ffffff9c 89fc6000 89817010 89fc7eb0 000abf54 800ccdb8 ffffff9c 89fc6000 [62617.420877] 7de0: 89fc7eb0 800cd294 00000000 89b6ddb0 95c788a0 89fc6028 000000d1 00000028 [62617.427772] 7e00: 00000000 89fc6020 89fc7f58 8cbc08c0 00000817 89b6c000 80290a40 200f0013 [62617.434657] 7e20: ffffffff 00000041 89817010 89fc7eb0 ffffff9c 89fc6008 89fc6000 89fc7eb0 [62617.441545] 7e40: 000abf54 800ce0b8 89fc7e58 00000ff0 00000000 00000000 00000000 000abf54 [62617.448433] 7e60: 8989fcf8 89fc7e78 800cccd0 80290a40 200f0013 00000001 89817000 89fc7eb0 [62617.455322] 7e80: ffffff9c ffffff9c 89fc6000 00000000 000abf54 800ce844 89817000 00000001 [62617.462205] 7ea0: ffffff9c 89fc7f30 ffffff9c 800d09ec add2b9d0 aa2717f8 00000000 00000000 [62617.469084] 7ec0: 00000000 80900ac0 00000000 8095027d aa2a3038 00001011 00000002 00000001 [62617.475965] 7ee0: 00000000 ac12f9d0 94bf60c0 00000007 80019d38 8090f688 76df7ec0 89fc7fb0 [62617.482845] 7f00: 00000024 00000040 7edc0e28 00000001 ffffff9c 0008dcf8 89fc7f50 800d0a28 [62617.489727] 7f20: 00000000 80904040 00000000 800c6db0 89fc7f38 00000001 7edc0e28 00888960 [62617.496607] 7f40: 000ab5d0 000000c3 8000ec84 800c7498 76e75000 000000ae 8000ec84 7edc0bdc [62617.503491] 7f60: 00000014 00000000 00000000 ac9b7300 ac9b7300 ade7f500 00000000 893e4840 [62617.510374] 7f80: 893e4840 ade7f500 00000000 800c0e1c ffffffff 00000003 76df7ec0 200f0010 [62617.517254] 7fa0: 00000000 8000eb00 00000000 00888960 0008dcf8 7edc0e28 7edc0e28 0000002f [62617.524135] 7fc0: 00000000 00888960 000ab5d0 000000c3 0088bd70 00000024 00000040 000abf54 [62617.531015] 7fe0: 000a4500 7edc0e14 00062134 76df7ed4 200f0010 0008dcf8 3ff7e821 3ff7ec21 [62617.537919] [<80206dcc>] (si_read_lock) from [<80229480>] (aufs_permission+0x3c/0x504) [62617.544556] [<80229480>] (aufs_permission) from [<800ccdb8>] (__inode_permission+0x60/0x9c) [62617.551622] [<800ccdb8>] (__inode_permission) from [<800cd294>] (link_path_walk+0x21c/0x8c4) [62617.558768] [<800cd294>] (link_path_walk) from [<800ce0b8>] (path_lookupat+0x54/0x7c0) [62617.565392] [<800ce0b8>] (path_lookupat) from [<800ce844>] (filename_lookup.isra.53+0x20/0x60) [62617.572712] [<800ce844>] (filename_lookup.isra.53) from [<800d09ec>] (user_path_at_empty+0x50/0x78) [62617.580466] [<800d09ec>] (user_path_at_empty) from [<800d0a28>] (user_path_at+0x14/0x1c) [62617.587263] [<800d0a28>] (user_path_at) from [<800c6db0>] (vfs_fstatat+0x44/0x98) [62617.593469] [<800c6db0>] (vfs_fstatat) from [<800c7498>] (SyS_stat64+0x14/0x30) [62617.599497] [<800c7498>] (SyS_stat64) from [<8000eb00>] (ret_fast_syscall+0x0/0x30) [62617.605876] Code: ba000003 e1a00005 ebf90b01 eaffffbb (e7f001f2) [62617.610689] ---[ end trace 35f9c046959b15af ]--- [64812.992276] ------------[ cut here ]------------ [64812.995619] kernel BUG at fs/aufs/super.h:414! [64812.998774] Internal error: Oops - BUG: 0 [#5] SMP ARM [64813.002616] Modules linked in: iptable_filter ip_tables x_tables snd_soc_omap_hdmi_codec brcmfmac mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc brcmutil snd_soc_imx_hdmi snd_soc_fsl_hdmi ipu_fg_overlay_sdc snd_soc_fsl_spdif [64813.023503] CPU: 0 PID: 31816 Comm: sync.sh Tainted: G D 3.10.30 #10 [64813.029432] task: ac526d80 ti: 91cce000 task.ti: 91cce000 [64813.033546] PC is at au_do_flush+0x280/0x288 [64813.036528] LR is at __down_read+0xe4/0xe8 [64813.039330] pc : [<802213d0>] lr : [<8066a304>] psr: 200f0013 sp : 91ccff68 ip : 00000000 fp : 01b85049 [64813.048207] r10: a9b9be00 r9 : 80223b54 r8 : 91cce028 [64813.052133] r7 : ade19400 r6 : ac994b40 r5 : aaf0ea38 r4 : adc32810 [64813.057361] r3 : 00000001 r2 : 00000080 r1 : ac0d4000 r0 : 00007c47 [64813.062590] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [64813.068427] Control: 10c53c7d Table: 122d804a DAC: 00000015 [64813.072877] Process sync.sh (pid: 31816, stack limit = 0x91cce238) [64813.077758] Stack: (0x91ccff68 to 0x91cd0000) [64813.080820] ff60: 80222c48 ac994b40 a9b9be00 00000002 00000006 8000ec84 [64813.087701] ff80: 91cce000 00000000 01b85049 800c0dec 000a4558 00007c48 00000000 800c21cc [64813.094581] ffa0: 01b72340 8000eb00 000a4558 00007c48 000000ff 00000002 00000000 01b72340 [64813.101461] ffc0: 000a4558 00007c48 00000000 00000006 00000000 000abf9c 00000028 01b85049 [64813.108360] ffe0: 00000000 7e992ee4 0001bf00 76e4f1dc 600f0010 000000ff ff344781 ff364983 [64813.115270] [<802213d0>] (au_do_flush) from [<800c0dec>] (filp_close+0x34/0x88) [64813.121326] [<800c0dec>] (filp_close) from [<800c21cc>] (SyS_close+0x20/0x48) [64813.127197] [<800c21cc>] (SyS_close) from [<8000eb00>] (ret_fast_syscall+0x0/0x30) [64813.133479] Code: e3a01068 e348007d ebf81909 eaffffb2 (e7f001f2) [64813.138279] ---[ end trace 35f9c046959b15b0 ]--- [67081.939370] ------------[ cut here ]------------ [67081.942703] kernel BUG at fs/aufs/super.h:414! [67081.945848] Internal error: Oops - BUG: 0 [#6] SMP ARM [67081.949686] Modules linked in: iptable_filter ip_tables x_tables snd_soc_omap_hdmi_codec brcmfmac mxc_v4l2_capture ipu_bg_overlay_sdc ipu_still ipu_prp_enc ipu_csi_enc brcmutil snd_soc_imx_hdmi snd_soc_fsl_hdmi ipu_fg_overlay_sdc snd_soc_fsl_spdif [67081.970554] CPU: 0 PID: 31816 Comm: get_browser.sh Tainted: G D 3.10.30 #10 [67081.977089] task: adee3480 ti: 893d6000 task.ti: 893d6000 [67081.981199] PC is at au_do_flush+0x280/0x288 [67081.984179] LR is at __down_read+0xe4/0xe8 [67081.986978] pc : [<802213d0>] lr : [<8066a304>] psr: 200f0013 sp : 893d7f68 ip : 00000000 fp : 000abf54 [67081.995852] r10: a9f6e300 r9 : 80223b54 r8 : 893d6028 [67081.999778] r7 : ade19400 r6 : 8279ad80 r5 : aae38e38 r4 : adc32810 [67082.005005] r3 : 00000001 r2 : 00000080 r1 : ac0d4000 r0 : 00007c47 [67082.010234] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [67082.016069] Control: 10c53c7d Table: 19edc04a DAC: 00000015 [67082.020519] Process get_browser.sh (pid: 31816, stack limit = 0x893d6238) [67082.026009] Stack: (0x893d7f68 to 0x893d8000) [67082.029070] 7f60: 80222c48 8279ad80 a9f6e300 00000002 00000006 8000ec84 [67082.035951] 7f80: 893d6000 00000000 000abf54 800c0dec 000a4558 00007c48 00000000 800c21cc [67082.042831] 7fa0: 01d2e4f0 8000eb00 000a4558 00007c48 000000ff ffffffff 00000000 01d2e4f0 [67082.049712] 7fc0: 000a4558 00007c48 00000000 00000006 01d2fc78 000abf9c 00000000 000abf54 [67082.056593] 7fe0: 00000000 7eaa56c4 0001bf00 76e2c1dc 600f0010 000000ff 00000000 00000000 [67082.063494] [<802213d0>] (au_do_flush) from [<800c0dec>] (filp_close+0x34/0x88) [67082.069515] [<800c0dec>] (filp_close) from [<800c21cc>] (SyS_close+0x20/0x48) [67082.075365] [<800c21cc>] (SyS_close) from [<8000eb00>] (ret_fast_syscall+0x0/0x30) [67082.081643] Code: e3a01068 e348007d ebf81909 eaffffb2 (e7f001f2) [67082.086443] ---[ end trace 35f9c046959b15b1 ]--- ------------------------------------------------------------------------------ Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds
