Re: Fwd: Fwd: upgrade to jessie from wheezy with cuda problems
I set PCIe 3.0 permanently. With a system of 150K atoms there is no acceleration at all of molecular dynamics with ivy with respect to sandy bridge. At the end of this exercise, given the very meager acceleration with 500K atoms (which is a large system under any respect, even for supercomputers), one might wonder whether changing from sandy bridge to ivy bridge is worth the money. Of course, this the viewpoint of number crunching. One could test if PCIe 3.0 is more advantageous with CUDA-accelerated viewers like VMD (same house as the code NAMD for molecular dynamics). cheers francesco pietra On Nov 18, 2013 8:59 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Mon, Nov 18, 2013 at 10:39:53PM +0100, Francesco Pietra wrote: I forgot to mention that LnkSta 8GT/s is obtained only when actually carrying out the MD simulation. I believe to save power the link speed changes on the fly based on demand. -- Len Sorensen
Fwd: upgrade to jessie from wheezy with cuda problems
It is getting hard, unless I mistaken what was suggested by nvidia . Thus, following what was suggested by nvidia as a no-barrier solution, https://devtalk.nvidia.com/default/topic/545186/enabling-pcie-3-0-with-nvreg_enablepciegen3-on-titan/?offset=10#4021328 I updated the kernel boot string by 1) typing 'e' at grub prompt, 2) adding the option to the linux line, 3) Ctrl-x to boot If that procedure is correct (probably it is, as francesco@gig64:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg1-root ro 1. nvidia.NVreg_EnablePCIeGen3=1 quiet francesco@gig64:~$ ) then, no luck. Both LnkCap and LnkSta were at at 5GT/s, as for PCIe 2.0. Molecular dynamics, accordingly, was not accelerated. I wonder whether 1. preceding nvidia... is what is needed for a grub bootloader option. I did not find any other instance about that nvidia suggestion on internet. francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Sun, Nov 17, 2013 at 4:09 PM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org, Lennart Sorensen lsore...@csclub.uwaterloo.ca This addendum to let you know that simply adding 1. options nvidia NVreg_EnablePCIeGen3=1 to /etc/modprobe.d/nvidia.conf as suggested in https://devtalk.nvidia.com/default/topic/545186/enabling-pcie-3-0-with-nvreg_enablepciegen3-on-titan/ had no effect. Also, please note that what should be added to the kernel boot string, according to the same source, is nvidia.NVreg_EnablePCIeGen3=1 unlike I wrote before (i.e., no options, while a dot between nvidia and NVreg francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Sun, Nov 17, 2013 at 11:42 AM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org, Lennart Sorensen lsore...@csclub.uwaterloo.ca Very sorry, forget about previous post. There, I had started MD from the gnome terminal, without activating the GPUUs. When carrying out regularly MD from the Linux prompt, without X-server, while activating the cards with #nvidia-smi -L #nvidia-smi -pm 1 as in all previous tests, both LnkCap and LnkSta are 5GT/s, as from PCIe 2.0. Thus, the problem seems to be activating PCIe 3.0, as before said. francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Sun, Nov 17, 2013 at 11:29 AM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org, Lennart Sorensen lsore...@csclub.uwaterloo.ca I have to correct my previous report. When running molecular dynamics, the capability of the GPU is 5GT/, but the actual speed link is at 2.5GT/s, i.e., below PCIe 2.o #lspci - 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 0969 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at fa00 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c800 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at fb00 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap:MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl:Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta:CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap:Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 512ns, L1 4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl:ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta:Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin
Re: Fwd: upgrade to jessie from wheezy with cuda problems
Em 18-11-2013 13:13, Francesco Pietra escreveu: It is getting hard, unless I mistaken what was suggested by nvidia . Thus, following what was suggested by nvidia as a no-barrier solution, https://devtalk.nvidia.com/default/topic/545186/enabling-pcie-3-0-with-nvreg_enablepciegen3-on-titan/?offset=10#4021328 I updated the kernel boot string by 1) typing 'e' at grub prompt, 2) adding the option to the linux line, 3) Ctrl-x to boot If that procedure is correct (probably it is, as francesco@gig64:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg1-root ro 1. nvidia.NVreg_EnablePCIeGen3=1 quiet francesco@gig64:~$ ) then, no luck. Both LnkCap and LnkSta were at at 5GT/s, as for PCIe 2.0. Molecular dynamics, accordingly, was not accelerated. I wonder whether 1. preceding nvidia... is what is needed for a grub bootloader option. I did not find any other instance about that nvidia suggestion on internet. I may be wrong, but it seems that there is a hardware bump in your pci-express 3.0 road , Francesco : http://www.anandtech.com/show/7521/nvidia-launches-tesla-k40 Close to the end it says that nvidia ... is finally enabling full pci-express 3.0 speeds ... , so it may be that your card suffers from this issue too . Hope it helps. -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/528a3d42.1070...@gmail.com
Re: Fwd: upgrade to jessie from wheezy with cuda problems
Em 18-11-2013 13:13, Francesco Pietra escreveu: It is getting hard, unless I mistaken what was suggested by nvidia . Thus, following what was suggested by nvidia as a no-barrier solution, https://devtalk.nvidia.com/default/topic/545186/enabling-pcie-3-0-with-nvreg_enablepciegen3-on-titan/?offset=10#4021328 I updated the kernel boot string by 1) typing 'e' at grub prompt, 2) adding the option to the linux line, 3) Ctrl-x to boot If that procedure is correct (probably it is, as francesco@gig64:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg1-root ro 1. nvidia.NVreg_EnablePCIeGen3=1 quiet francesco@gig64:~$ ) then, no luck. Both LnkCap and LnkSta were at at 5GT/s, as for PCIe 2.0. Molecular dynamics, accordingly, was not accelerated. I wonder whether 1. preceding nvidia... is what is needed for a grub bootloader option. I did not find any other instance about that nvidia suggestion on internet. I may be wrong, but it seems that there is a hardware bump in your pci-express 3.0 road , Francesco : http://www.anandtech.com/show/7521/nvidia-launches-tesla-k40 Close to the end it says that nvidia ... is finally enabling full pci-express 3.0 speeds ... , so it may be that your card suffers from this issue too . Hope it helps. -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/528a3dcd.1090...@gmail.com
Re: Fwd: upgrade to jessie from wheezy with cuda problems
On Sun, Nov 17, 2013 at 10:45:58AM +0100, Francesco Pietra wrote: I am attacking the problem from another side, directly from within the OS itself: #lspi - tells that the link speed (= link status) LnkSta is at 5Gb/s, no matter whether the system is at number crunching or not. I.e., my system is at PCIe 2.0. This might explain why upgrading from sandy bridge to ivy bridge gave no speed gain of molecular dynamics. PCIe 3.0 was not achieved. As far as I could investigate, nvidia suggests to either: (1) Modify /etc/modprobe.d/local.conf (which does not exist on jessie) or create a new /etc/modprobe.d/nvidia.conf, adding to that 1. options nvidia NVreg_EnablePCIeGen3=1 Might need nvidia-current instead of nvidia. Actually, on my jessie, nvidia.conf reads alias nvidia nvidia-current remove nvidia-current rm mod nvidia Some guys found that useless, even when both grub-efi and initramfs are edited accordingly, so that nvidia offered a different move, updating the kernel boot string, by appending this: 1. options nvidia NVreg_EnablePCIeGen3=1 *** That is NOT the syntax for a kernel command line. It is the syntax for the modprobe config. Something like nvidia.NVreg_EnablePCIeGen3=1 or nvidia-current.NVreg_EnablePCIeGen3=1 (depending on the name of the module as far as the module is concerned). I did nothing, as I hope that the best adaptation to jessie may be suggested by those who know the OS better than me. The kind of information about links includes: LnkSta: the actual speed LnkCap: the capacity of the specific port, as far as I can understand. LnkCtl: ?? One could also run #lspci -vt to determine the bus where the GPU card is located, then running # lspci -vv -s ## where ## is the location. ** So, it is a tricky matter, but perhaps not so much when one knows where to put the hands. At any event, being unable to go to 8GT/s, as from PCIe 3.0, means loosing time and energy (=money and pollution), at least when the GPUs are used for long number crunching. Well it means slower transfers of data to and from the card. If the data set fits in the card entirely during a long number crunch, then bandwidth would not matter much at all. So depends on the size of the data set and how often data has to be moved in and out of the card. I'll continue investigating. The above seems to be promising. Hope to get help. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131118170238.gm20...@csclub.uwaterloo.ca
Re: Fwd: upgrade to jessie from wheezy with cuda problems
Might need nvidia-current instead of nvidia. It failed to bring to PCIe 3.0 when inserted into nvidia.conf francesco@gig64:/etc/modprobe.d$ cat nvidia.conf alias nvidia nvidia-current remove nvidia-current rmmod nvidia # 1. options nvidia-current NVreg_EnablePCIeGen3=1 (of course it was not commented when the test was carried out) However, it brought to PCIe 3.0 when in the kernel GREAT SUGGESTION Thus, I added (temporarily) to GRUB by 1) typing 'e' at grub prompt, 2) adding the option to the END OF the linux line, 3) Ctrl-x to boot verifying that it was taken into accout ~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg1-root ro quiet 1. nvidia-current.NVreg_EnablePCIeGen3=1 #lspci - ... 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 0969 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at fa00 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c800 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at fb00 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap:MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl:Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta:CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap:Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 1us, L1 4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl:ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta:Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+ Capabilities: [b4] Vendor Specific Information: Len=14 ? Capabilities: [100 v1] Virtual Channel Caps:LPEVC=0 RefClk=100ns PATEntryBits=1 Arb:Fixed- WRR32- WRR64- WRR128- Ctrl:ArbSelect=Fixed Status:InProgress- VC0:Caps:PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb:Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl:Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status:NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting ? Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 ? Capabilities: [900 v1] #19 Kernel driver in use: nvidia ... The same for the other GPU. * Well, the surprise was that molecular dynamics for a large system (500K atoms) was very modestly accelerated. From the simulation log file: Info: Benchmark time: 6 CPUs 0.123387 s/step 1.4281 days/ns 1171.53 MB memory From the same simulation with same motherboard and GTX-680, but with sansy bridge i7-3930 and 1066MHz RAM: Info: Benchmark time: 6 CPUs 0.138832 s/step 1.60686 days/ns 1161.23 MB memory The better performance of the ivy bridge might be the result from the higher clock of both the CPU and RAM (1866MHz). A variety of interpretaions of these observations are possible, taking into account, however, that with simple machines as the one used here, it would be difficult to run MD with much bigger systems than 500K atoms. Finally we succeeded to get PCIe 3.0 and now the PCIe 3.0 setting can be passed permanently to the kernel. I have to learn how. Thanks a lot francesco pietra On Mon, Nov 18, 2013 at 6:02 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Sun, Nov 17, 2013 at
Fwd: Fwd: upgrade to jessie from wheezy with cuda problems
I forgot to mention that LnkSta 8GT/s is obtained only when actually carrying out the MD simulation. fp -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Mon, Nov 18, 2013 at 10:37 PM Subject: Re: Fwd: upgrade to jessie from wheezy with cuda problems To: Lennart Sorensen lsore...@csclub.uwaterloo.ca Cc: amd64 Debian debian-amd64@lists.debian.org Might need nvidia-current instead of nvidia. It failed to bring to PCIe 3.0 when inserted into nvidia.conf francesco@gig64:/etc/modprobe.d$ cat nvidia.conf alias nvidia nvidia-current remove nvidia-current rmmod nvidia # 1. options nvidia-current NVreg_EnablePCIeGen3=1 (of course it was not commented when the test was carried out) However, it brought to PCIe 3.0 when in the kernel GREAT SUGGESTION Thus, I added (temporarily) to GRUB by 1) typing 'e' at grub prompt, 2) adding the option to the END OF the linux line, 3) Ctrl-x to boot verifying that it was taken into accout ~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg1-root ro quiet 1. nvidia-current.NVreg_EnablePCIeGen3=1 #lspci - ... 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 0969 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at fa00 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c800 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at fb00 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap:MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl:Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta:CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap:Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 1us, L1 4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl:ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta:Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+ Capabilities: [b4] Vendor Specific Information: Len=14 ? Capabilities: [100 v1] Virtual Channel Caps:LPEVC=0 RefClk=100ns PATEntryBits=1 Arb:Fixed- WRR32- WRR64- WRR128- Ctrl:ArbSelect=Fixed Status:InProgress- VC0:Caps:PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb:Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl:Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status:NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting ? Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 ? Capabilities: [900 v1] #19 Kernel driver in use: nvidia ... The same for the other GPU. * Well, the surprise was that molecular dynamics for a large system (500K atoms) was very modestly accelerated. From the simulation log file: Info: Benchmark time: 6 CPUs 0.123387 s/step 1.4281 days/ns 1171.53 MB memory From the same simulation with same motherboard and GTX-680, but with sansy bridge i7-3930 and 1066MHz RAM: Info: Benchmark time: 6 CPUs 0.138832 s/step 1.60686 days/ns 1161.23 MB memory The better performance of the ivy bridge might be the result from the higher clock of both the CPU and RAM (1866MHz). A variety of interpretaions of these observations are possible, taking into account, however
Re: Fwd: Fwd: upgrade to jessie from wheezy with cuda problems
On Mon, Nov 18, 2013 at 10:39:53PM +0100, Francesco Pietra wrote: I forgot to mention that LnkSta 8GT/s is obtained only when actually carrying out the MD simulation. I believe to save power the link speed changes on the fly based on demand. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131118215951.go20...@csclub.uwaterloo.ca
Re: Fwd: upgrade to jessie from wheezy with cuda problems
I am attacking the problem from another side, directly from within the OS itself: #lspi - tells that the link speed (= link status) LnkSta is at 5Gb/s, no matter whether the system is at number crunching or not. I.e., my system is at PCIe 2.0. This might explain why upgrading from sandy bridge to ivy bridge gave no speed gain of molecular dynamics. PCIe 3.0 was not achieved. As far as I could investigate, nvidia suggests to either: (1) Modify /etc/modprobe.d/local.conf (which does not exist on jessie) or create a new /etc/modprobe.d/nvidia.conf, adding to that 1. options nvidia NVreg_EnablePCIeGen3=1 Actually, on my jessie, nvidia.conf reads alias nvidia nvidia-current remove nvidia-current rm mod nvidia Some guys found that useless, even when both grub-efi and initramfs are edited accordingly, so that nvidia offered a different move, updating the kernel boot string, by appending this: 1. options nvidia NVreg_EnablePCIeGen3=1 *** I did nothing, as I hope that the best adaptation to jessie may be suggested by those who know the OS better than me. The kind of information about links includes: LnkSta: the actual speed LnkCap: the capacity of the specific port, as far as I can understand. LnkCtl: ?? One could also run #lspci -vt to determine the bus where the GPU card is located, then running # lspci -vv -s ## where ## is the location. ** So, it is a tricky matter, but perhaps not so much when one knows where to put the hands. At any event, being unable to go to 8GT/s, as from PCIe 3.0, means loosing time and energy (=money and pollution), at least when the GPUs are used for long number crunching. I'll continue investigating. The above seems to be promising. Hope to get help. francesco pietra PS With my jessie /etc/modprobe.d includes the following files: alsa-base.conf alsa-case-blacklist.conf dkms.conf (which has no active statemente) fbdev-blacklist.conf i915-kms.conf nvidia.conf nvidia-blacklist-nouveau.conf radeon-kms.conf ** On Thu, Nov 14, 2013 at 3:33 AM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Nov 13, 2013 at 05:43:47PM -0500, Lennart Sorensen wrote: On Wed, Nov 13, 2013 at 10:53:30PM +0100, Francesco Pietra wrote: francesco@gig64:~/tmp$ file ./CUDA-Z-0.7.189.run ./CUDA-Z-0.7.189.run: data francesco@gig64:~/tmp$ OK that's weird. I expected to see x86 32 or 64bit binary. Seems to be a shell scripts with compressed code in it. Yuck. :) OK I got it running. It is a 32bit binary. I had to install these: ii libcuda1:i386 331.20-1 i386 NVIDIA CUDA runtime library ii libcudart5.0:i386 5.0.35-8 i386 NVIDIA CUDA runtime library ii libgl1-nvidia-glx:i386331.20-1 i386 NVIDIA binary OpenGL libraries ii libstdc++6:i386 4.8.2-4 i386 GNU Standard C++ Library v3 ii libxrender1:i386 1:0.9.8-1 i386 X Rendering Extension client library ii zlib1g:i386 1:1.2.8.dfsg-1 i386 compression library - runtime Then I was able to run it. No messing with LD_LIBRARY_PATH or anything. To install :i386 packages you first have to enable multiarch support with dpkg and run apt-get update. So something like: dpkg --add-architecture i386 apt-get update apt-get install libcuda1:i386 libcudart5.0:i386 libgl1-nvidia-glx:i386 libstdc++6:i386 libxrender1:i386 zlib1g:i386 Don't worry about the exact versions, since I am running unstable+experimental. You don;t need to do that to get it working. For your 64bit code you probably need libcuda1 libcudart5.0 and such installed in the 64bit version. -- Len Sorensen
Fwd: upgrade to jessie from wheezy with cuda problems
I have to correct my previous report. When running molecular dynamics, the capability of the GPU is 5GT/, but the actual speed link is at 2.5GT/s, i.e., below PCIe 2.o #lspci - 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 0969 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at fa00 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c800 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at fb00 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap:MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl:Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta:CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap:Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 512ns, L1 4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl:ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta:Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Tue, Nov 12, 2013 at 3:54 PM Subject: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org Hello: I decided to try jessie to get PCIe 3.0 with a recent nvidia driver, thus upgrading from wheezy. wheezy was uname -r 3.2.0-4-amd64 nvidia-smi 304.88 nvcc --version 4.2 (the latter is also the version at which the molecular dynamics code was compiled, and used without calling the X-server) Following aptitude update aptitude-upgrade a number of dependecies related to gnome were not met (evolution-common lbfolks25 gnome-panel gnome-shell gnome-theme-extras gnome-theme-standard libreoffice-evolution). This notwithstanding, I decided to upgrade. After rebooting to get linux matching with nvidia: nvcc --version 5.0 uname -r 3.10-3-amd64 nvidia-smi the nvidia kernel module has version 304.108 but the nvidia driver component has version 319.60. Driver 319.6 is just what I wanted. Now, how best fix the problems? Install linux image 3.2? In the past I tried dist-upgrade, getting into devastating problems. thanks francesco pietra PS I was advised that debian is getting bounces from my address above. If so, please try my institutional address francesco.pie...@accademialucchese.it
Fwd: upgrade to jessie from wheezy with cuda problems
Very sorry, forget about previous post. There, I had started MD from the gnome terminal, without activating the GPUUs. When carrying out regularly MD from the Linux prompt, without X-server, while activating the cards with #nvidia-smi -L #nvidia-smi -pm 1 as in all previous tests, both LnkCap and LnkSta are 5GT/s, as from PCIe 2.0. Thus, the problem seems to be activating PCIe 3.0, as before said. francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Sun, Nov 17, 2013 at 11:29 AM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org, Lennart Sorensen lsore...@csclub.uwaterloo.ca I have to correct my previous report. When running molecular dynamics, the capability of the GPU is 5GT/, but the actual speed link is at 2.5GT/s, i.e., below PCIe 2.o #lspci - 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 0969 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at fa00 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c800 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at fb00 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap:MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl:Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta:CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap:Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 512ns, L1 4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl:ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta:Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Tue, Nov 12, 2013 at 3:54 PM Subject: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org Hello: I decided to try jessie to get PCIe 3.0 with a recent nvidia driver, thus upgrading from wheezy. wheezy was uname -r 3.2.0-4-amd64 nvidia-smi 304.88 nvcc --version 4.2 (the latter is also the version at which the molecular dynamics code was compiled, and used without calling the X-server) Following aptitude update aptitude-upgrade a number of dependecies related to gnome were not met (evolution-common lbfolks25 gnome-panel gnome-shell gnome-theme-extras gnome-theme-standard libreoffice-evolution). This notwithstanding, I decided to upgrade. After rebooting to get linux matching with nvidia: nvcc --version 5.0 uname -r 3.10-3-amd64 nvidia-smi the nvidia kernel module has version 304.108 but the nvidia driver component has version 319.60. Driver 319.6 is just what I wanted. Now, how best fix the problems? Install linux image 3.2? In the past I tried dist-upgrade, getting into devastating problems. thanks francesco pietra PS I was advised that debian is getting bounces from my address above. If so, please try my institutional address francesco.pie...@accademialucchese.it
Fwd: upgrade to jessie from wheezy with cuda problems
This addendum to let you know that simply adding 1. options nvidia NVreg_EnablePCIeGen3=1 to /etc/modprobe.d/nvidia.conf as suggested in https://devtalk.nvidia.com/default/topic/545186/enabling-pcie-3-0-with-nvreg_enablepciegen3-on-titan/ had no effect. Also, please note that what should be added to the kernel boot string, according to the same source, is nvidia.NVreg_EnablePCIeGen3=1 unlike I wrote before (i.e., no options, while a dot between nvidia and NVreg francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Sun, Nov 17, 2013 at 11:42 AM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org, Lennart Sorensen lsore...@csclub.uwaterloo.ca Very sorry, forget about previous post. There, I had started MD from the gnome terminal, without activating the GPUUs. When carrying out regularly MD from the Linux prompt, without X-server, while activating the cards with #nvidia-smi -L #nvidia-smi -pm 1 as in all previous tests, both LnkCap and LnkSta are 5GT/s, as from PCIe 2.0. Thus, the problem seems to be activating PCIe 3.0, as before said. francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Sun, Nov 17, 2013 at 11:29 AM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org, Lennart Sorensen lsore...@csclub.uwaterloo.ca I have to correct my previous report. When running molecular dynamics, the capability of the GPU is 5GT/, but the actual speed link is at 2.5GT/s, i.e., below PCIe 2.o #lspci - 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 0969 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast TAbort- TAbort- MAbort- SERR- PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at fa00 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c800 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at fb00 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap:MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl:Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta:CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap:Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 512ns, L1 4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl:ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta:Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Tue, Nov 12, 2013 at 3:54 PM Subject: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org Hello: I decided to try jessie to get PCIe 3.0 with a recent nvidia driver, thus upgrading from wheezy. wheezy was uname -r 3.2.0-4-amd64 nvidia-smi 304.88 nvcc --version 4.2 (the latter is also the version at which the molecular dynamics code was compiled, and used without calling the X-server) Following aptitude update aptitude-upgrade a number of dependecies related to gnome were not met (evolution-common lbfolks25 gnome-panel gnome-shell gnome-theme-extras gnome-theme-standard libreoffice-evolution). This notwithstanding, I decided to upgrade. After rebooting to get linux matching with nvidia: nvcc --version 5.0 uname -r
Re: upgrade to jessie from wheezy with cuda problems
I think it was renamed. No idea why. modinfo nvidia-current should work though. Yes, it does. Do you have the cuda libraries for the 319 version installed? Yes I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. 500,000 atoms, as in my test, is a large system for unbiased molecular dynamics. At any event, I looked at the the nvidia-cuda-toolkit version 5.0. nvidia for GPU Computing SDK, to build examples that should include a bandwidth test, offers linux packages for Fedora RHEL Ubuntu OpenSUSE and SUSE. No Debian. I had unpleasant experiences with Ubuntu packages, and it is well known that Ubuntu, unlike LinuxMint, is not compatible with Debian. Therefore, I did not try the cuda toolkit. I wonder why Ubuntu has so widely replaced Debian among the mass. Sad, and somewhat irritating, for me. I tried francesco@gig64:~/tmp$ ls CUDA-Z-0.7.189.run francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-657a-580e-a8aa-0faa/cuda-z: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or directory francesco@gig64:~/tmp$ ls CUDA-Z-0.7.189.run libXrender.so.1 francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-a3db-49bf-8cb7-059d/cuda-z: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or directory francesco@gig64:~/tmp$ Actually the required lib is available, as shown by my copy into tmp. I don't remember the source of this GNU CUDA-Z tool. Any experience with? I have also met reports of unexciting experience with PCIe 3.0, that is meager or no gain over PCIe 2.0, however it deals of people carrying out games, which is different from NAMD molecular dynamics, where most is done by the GPUs but AT EACH STEP energy has to be calculated by the CPU. thanks francesco pietra On Tue, Nov 12, 2013 at 11:37 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Tue, Nov 12, 2013 at 10:35:53PM +0100, Francesco Pietra wrote: # apt-get --purge remove *legacy* did the job. I wonder how these legacy packages entered the scene while updating/upgrading from a clean wheezy. The bad news are that with the new driver 319.60 there was no acceleration of molecular dynamics for a job of modest size (150K atoms) and slight acceleration (0.12 s/step vs 0.14 s/step) for a heavy job (500K atoms). Weather bringing from PCIe 2.0 (with the 304.xx driver of wheezy) to PCIe 3.0 (with driver 319.60 of jessie) (increasing the bandwidth from GPUs to RAM from 5 to 8GB/s) has not the effect that I hoped on the calculations, or PCIe is still 2.0 with jessie. Now, with cuda 5.0, it should be easy to measure the bandwidth directly. I have to learn how and I'll report about in due course. Now nvidia-smi activates the GPUs for normal work, nvidia-smi -L tells about the GPUs, dpkg -l |grep nvidia shows all 319.60 or 5.0.35-8, the X-server can be started and gnome loaded (startx, gnome-session), nvcc --version gives 5.0, however # modinfo nvidia ERROR: module nvidia not found In analogy with wheezy 3.2.0-4, I expected /lib/modules/3.10-3-amd64/updates/dkms/nvidia.ko Instead, there is /lib/modules/3.10-3-amd64/nvidia/nvidia-current.ko is that a feature of jessie or something wrong? I think it was renamed. No idea why. modinfo nvidia-current should work though. Do you have the cuda libraries for the 319 version installed? I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. -- Len Sorensen
Re: upgrade to jessie from wheezy with cuda problems
My answer seems to have disappeared. I summarize here. modinfo nvidia-curred works well. CUDA libraries are installed. For nvidia-cuda-toolkit, nvidia offers SDK packages for Ubuntu, not for Debian. I don't like to get into troubles with Ubuntu, which, unlike LinuxMINT, is not compatible with Debian. I tried GNU CUDA-Z-07.189.run (don't remember from where it was downloaded). However it does not find the shared libXrender.so.1, even if made available into the same folder of CUDA-Z. Actually root@gig64:/home/francesco# apt-file search libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 libxrender1-dbg: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 root@gig64:/home/francesco# francesco@gig64:~$ echo $PATH /opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2 francesco@gig64:~$ Should /usr/lib/x86_64-linux-gnu be put on my path explicitly? Thanks francesco pietra On Tue, Nov 12, 2013 at 11:37 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Tue, Nov 12, 2013 at 10:35:53PM +0100, Francesco Pietra wrote: # apt-get --purge remove *legacy* did the job. I wonder how these legacy packages entered the scene while updating/upgrading from a clean wheezy. The bad news are that with the new driver 319.60 there was no acceleration of molecular dynamics for a job of modest size (150K atoms) and slight acceleration (0.12 s/step vs 0.14 s/step) for a heavy job (500K atoms). Weather bringing from PCIe 2.0 (with the 304.xx driver of wheezy) to PCIe 3.0 (with driver 319.60 of jessie) (increasing the bandwidth from GPUs to RAM from 5 to 8GB/s) has not the effect that I hoped on the calculations, or PCIe is still 2.0 with jessie. Now, with cuda 5.0, it should be easy to measure the bandwidth directly. I have to learn how and I'll report about in due course. Now nvidia-smi activates the GPUs for normal work, nvidia-smi -L tells about the GPUs, dpkg -l |grep nvidia shows all 319.60 or 5.0.35-8, the X-server can be started and gnome loaded (startx, gnome-session), nvcc --version gives 5.0, however # modinfo nvidia ERROR: module nvidia not found In analogy with wheezy 3.2.0-4, I expected /lib/modules/3.10-3-amd64/updates/dkms/nvidia.ko Instead, there is /lib/modules/3.10-3-amd64/nvidia/nvidia-current.ko is that a feature of jessie or something wrong? I think it was renamed. No idea why. modinfo nvidia-current should work though. Do you have the cuda libraries for the 319 version installed? I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. -- Len Sorensen
Re: upgrade to jessie from wheezy with cuda problems
On Wed, Nov 13, 2013 at 10:13:15AM +0100, Francesco Pietra wrote: I think it was renamed. No idea why. modinfo nvidia-current should work though. Yes, it does. Do you have the cuda libraries for the 319 version installed? Yes I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. 500,000 atoms, as in my test, is a large system for unbiased molecular dynamics. At any event, I looked at the the nvidia-cuda-toolkit version 5.0. nvidia for GPU Computing SDK, to build examples that should include a bandwidth test, offers linux packages for Fedora RHEL Ubuntu OpenSUSE and SUSE. No Debian. I had unpleasant experiences with Ubuntu packages, and it is well known that Ubuntu, unlike LinuxMint, is not compatible with Debian. Therefore, I did not try the cuda toolkit. I wonder why Ubuntu has so widely replaced Debian among the mass. Sad, and somewhat irritating, for me. I tried francesco@gig64:~/tmp$ ls CUDA-Z-0.7.189.run francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-657a-580e-a8aa-0faa/cuda-z: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or directory Try: LD_LIBRARY_PATH=. ./CUDA-Z-0.7.189.run See if it finds that lirary then. francesco@gig64:~/tmp$ ls CUDA-Z-0.7.189.run libXrender.so.1 francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-a3db-49bf-8cb7-059d/cuda-z: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or directory francesco@gig64:~/tmp$ Actually the required lib is available, as shown by my copy into tmp. I don't remember the source of this GNU CUDA-Z tool. Any experience with? I have also met reports of unexciting experience with PCIe 3.0, that is meager or no gain over PCIe 2.0, however it deals of people carrying out games, which is different from NAMD molecular dynamics, where most is done by the GPUs but AT EACH STEP energy has to be calculated by the CPU. I see a package in Debian named 'nvidia-cuda-toolkit'. Does that include that you were looking for? I guess the bandwidthtest isn't built normally. -- Len Sroensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131113171328.gg20...@csclub.uwaterloo.ca
Re: upgrade to jessie from wheezy with cuda problems
On Wed, Nov 13, 2013 at 10:32:26AM +0100, Francesco Pietra wrote: My answer seems to have disappeared. I summarize here. modinfo nvidia-curred works well. CUDA libraries are installed. For nvidia-cuda-toolkit, nvidia offers SDK packages for Ubuntu, not for Debian. I don't like to get into troubles with Ubuntu, which, unlike LinuxMINT, is not compatible with Debian. I tried GNU CUDA-Z-07.189.run (don't remember from where it was downloaded). However it does not find the shared libXrender.so.1, even if made available into the same folder of CUDA-Z. Actually root@gig64:/home/francesco# apt-file search libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 libxrender1-dbg: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 root@gig64:/home/francesco# francesco@gig64:~$ echo $PATH /opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2 francesco@gig64:~$ Should /usr/lib/x86_64-linux-gnu be put on my path explicitly? The PATH is not for libraries. LD_LIBRARY_PATH is, as is /etc/ld.so.conf stuff. Also is what you downloaded 32 or 64 bit? Try: ldd CUDA-Z-07.189.run See what it is looking for. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131113171504.gh20...@csclub.uwaterloo.ca
Fwd: upgrade to jessie from wheezy with cuda problems
francesco@gig64:~/tmp$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: error while loading shared libraries: libXrender.so.1: wrong ELF class: ELFCLASS64 francesco@gig64:~/tmp$ -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Wed, Nov 13, 2013 at 10:32 AM Subject: Re: upgrade to jessie from wheezy with cuda problems To: Lennart Sorensen lsore...@csclub.uwaterloo.ca Cc: amd64 Debian debian-amd64@lists.debian.org My answer seems to have disappeared. I summarize here. modinfo nvidia-curred works well. CUDA libraries are installed. For nvidia-cuda-toolkit, nvidia offers SDK packages for Ubuntu, not for Debian. I don't like to get into troubles with Ubuntu, which, unlike LinuxMINT, is not compatible with Debian. I tried GNU CUDA-Z-07.189.run (don't remember from where it was downloaded). However it does not find the shared libXrender.so.1, even if made available into the same folder of CUDA-Z. Actually root@gig64:/home/francesco# apt-file search libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 libxrender1-dbg: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 root@gig64:/home/francesco# francesco@gig64:~$ echo $PATH /opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2 francesco@gig64:~$ Should /usr/lib/x86_64-linux-gnu be put on my path explicitly? Thanks francesco pietra On Tue, Nov 12, 2013 at 11:37 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Tue, Nov 12, 2013 at 10:35:53PM +0100, Francesco Pietra wrote: # apt-get --purge remove *legacy* did the job. I wonder how these legacy packages entered the scene while updating/upgrading from a clean wheezy. The bad news are that with the new driver 319.60 there was no acceleration of molecular dynamics for a job of modest size (150K atoms) and slight acceleration (0.12 s/step vs 0.14 s/step) for a heavy job (500K atoms). Weather bringing from PCIe 2.0 (with the 304.xx driver of wheezy) to PCIe 3.0 (with driver 319.60 of jessie) (increasing the bandwidth from GPUs to RAM from 5 to 8GB/s) has not the effect that I hoped on the calculations, or PCIe is still 2.0 with jessie. Now, with cuda 5.0, it should be easy to measure the bandwidth directly. I have to learn how and I'll report about in due course. Now nvidia-smi activates the GPUs for normal work, nvidia-smi -L tells about the GPUs, dpkg -l |grep nvidia shows all 319.60 or 5.0.35-8, the X-server can be started and gnome loaded (startx, gnome-session), nvcc --version gives 5.0, however # modinfo nvidia ERROR: module nvidia not found In analogy with wheezy 3.2.0-4, I expected /lib/modules/3.10-3-amd64/updates/dkms/nvidia.ko Instead, there is /lib/modules/3.10-3-amd64/nvidia/nvidia-current.ko is that a feature of jessie or something wrong? I think it was renamed. No idea why. modinfo nvidia-current should work though. Do you have the cuda libraries for the 319 version installed? I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. -- Len Sorensen
Fwd: upgrade to jessie from wheezy with cuda problems
Sorry, I had not noticed the suggestion, however I had already what inthe meantime became obvious. The executable is said for both 64 and 32 bit but apparently the lib has to be 32. I have no 32, nor multiarch, to avoid complications for a box used for MD only. No luck for me on investigating the bandwidth. thanks francesco pietra -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Wed, Nov 13, 2013 at 7:40 PM Subject: Fwd: upgrade to jessie from wheezy with cuda problems To: amd64 Debian debian-amd64@lists.debian.org francesco@gig64:~/tmp$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: error while loading shared libraries: libXrender.so.1: wrong ELF class: ELFCLASS64 francesco@gig64:~/tmp$ -- Forwarded message -- From: Francesco Pietra chiendar...@gmail.com Date: Wed, Nov 13, 2013 at 10:32 AM Subject: Re: upgrade to jessie from wheezy with cuda problems To: Lennart Sorensen lsore...@csclub.uwaterloo.ca Cc: amd64 Debian debian-amd64@lists.debian.org My answer seems to have disappeared. I summarize here. modinfo nvidia-curred works well. CUDA libraries are installed. For nvidia-cuda-toolkit, nvidia offers SDK packages for Ubuntu, not for Debian. I don't like to get into troubles with Ubuntu, which, unlike LinuxMINT, is not compatible with Debian. I tried GNU CUDA-Z-07.189.run (don't remember from where it was downloaded). However it does not find the shared libXrender.so.1, even if made available into the same folder of CUDA-Z. Actually root@gig64:/home/francesco# apt-file search libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1 libxrender1: /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 libxrender1-dbg: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0 root@gig64:/home/francesco# francesco@gig64:~$ echo $PATH /opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2:/opt/amber12/bin:/opt/amber10/bin:/opt/UCSF/Chimera64-2012-10-10/bin:/opt/namd2.9_cuda4.0_2012-09-26/bin/namd2 francesco@gig64:~$ Should /usr/lib/x86_64-linux-gnu be put on my path explicitly? Thanks francesco pietra On Tue, Nov 12, 2013 at 11:37 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Tue, Nov 12, 2013 at 10:35:53PM +0100, Francesco Pietra wrote: # apt-get --purge remove *legacy* did the job. I wonder how these legacy packages entered the scene while updating/upgrading from a clean wheezy. The bad news are that with the new driver 319.60 there was no acceleration of molecular dynamics for a job of modest size (150K atoms) and slight acceleration (0.12 s/step vs 0.14 s/step) for a heavy job (500K atoms). Weather bringing from PCIe 2.0 (with the 304.xx driver of wheezy) to PCIe 3.0 (with driver 319.60 of jessie) (increasing the bandwidth from GPUs to RAM from 5 to 8GB/s) has not the effect that I hoped on the calculations, or PCIe is still 2.0 with jessie. Now, with cuda 5.0, it should be easy to measure the bandwidth directly. I have to learn how and I'll report about in due course. Now nvidia-smi activates the GPUs for normal work, nvidia-smi -L tells about the GPUs, dpkg -l |grep nvidia shows all 319.60 or 5.0.35-8, the X-server can be started and gnome loaded (startx, gnome-session), nvcc --version gives 5.0, however # modinfo nvidia ERROR: module nvidia not found In analogy with wheezy 3.2.0-4, I expected /lib/modules/3.10-3-amd64/updates/dkms/nvidia.ko Instead, there is /lib/modules/3.10-3-amd64/nvidia/nvidia-current.ko is that a feature of jessie or something wrong? I think it was renamed. No idea why. modinfo nvidia-current should work though. Do you have the cuda libraries for the 319 version installed? I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. -- Len Sorensen
Re: Fwd: upgrade to jessie from wheezy with cuda problems
On Wed, Nov 13, 2013 at 07:40:10PM +0100, Francesco Pietra wrote: francesco@gig64:~/tmp$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ That is unnecesary. That is already in the library path. The local directory is not. Windows implicitly looks in the current directory for files, linux (and almost all other systems) does not. hence: export LD_LIBRARY_PATH=. (. for current directory), or LD_LIBRARY_PATH=/tmp if that is where you put the library you were trying. francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: error while loading shared libraries: libXrender.so.1: wrong ELF class: ELFCLASS64 francesco@gig64:~/tmp$ What does 'file ./CUDA-Z-0.7.189.run' say? -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131113185253.gi20...@csclub.uwaterloo.ca
Re: Fwd: upgrade to jessie from wheezy with cuda problems
Em 13-11-2013 16:40, Francesco Pietra escreveu: francesco@gig64:~/tmp$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: error while loading shared libraries: libXrender.so.1: wrong ELF class: ELFCLASS64 francesco@gig64:~/tmp$ Hi Francesco . is CUDA-Z a 32-bit binary ? what is the output of this command : $ file /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/5283d078.6010...@gmail.com
Re: Fwd: upgrade to jessie from wheezy with cuda problems
francesco@gig64:~$ file /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: ERROR: cannot open `/home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z' (No such file or directory) francesco@gig64:~$ On Wed, Nov 13, 2013 at 8:18 PM, Fabricio Cannini fcann...@gmail.comwrote: Em 13-11-2013 16:40, Francesco Pietra escreveu: francesco@gig64:~/tmp$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: error while loading shared libraries: libXrender.so.1: wrong ELF class: ELFCLASS64 francesco@gig64:~/tmp$ Hi Francesco . is CUDA-Z a 32-bit binary ? what is the output of this command : $ file /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/5283d078.6010...@gmail.com
Re: Fwd: upgrade to jessie from wheezy with cuda problems
What does 'file ./CUDA-Z-0.7.189.run' say? francesco@gig64:~/tmp$ file ./CUDA-Z-0.7.189.run ./CUDA-Z-0.7.189.run: data francesco@gig64:~/tmp$ On Wed, Nov 13, 2013 at 7:52 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Nov 13, 2013 at 07:40:10PM +0100, Francesco Pietra wrote: francesco@gig64:~/tmp$ export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ That is unnecesary. That is already in the library path. The local directory is not. Windows implicitly looks in the current directory for files, linux (and almost all other systems) does not. hence: export LD_LIBRARY_PATH=. (. for current directory), or LD_LIBRARY_PATH=/tmp if that is where you put the library you were trying. francesco@gig64:~/tmp$ ./CUDA-Z-0.7.189.run CUDA-Z 0.7.189 Container Starting CUDA-Z... /home/francesco/tmp/CUDA-Z-95b0-7943-3edd-827e/cuda-z: error while loading shared libraries: libXrender.so.1: wrong ELF class: ELFCLASS64 francesco@gig64:~/tmp$ What does 'file ./CUDA-Z-0.7.189.run' say? -- Len Sorensen
Re: Fwd: upgrade to jessie from wheezy with cuda problems
On Wed, Nov 13, 2013 at 10:53:30PM +0100, Francesco Pietra wrote: francesco@gig64:~/tmp$ file ./CUDA-Z-0.7.189.run ./CUDA-Z-0.7.189.run: data francesco@gig64:~/tmp$ OK that's weird. I expected to see x86 32 or 64bit binary. Seems to be a shell scripts with compressed code in it. Yuck. :) -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131113224347.gj20...@csclub.uwaterloo.ca
Re: Fwd: upgrade to jessie from wheezy with cuda problems
On Wed, Nov 13, 2013 at 05:43:47PM -0500, Lennart Sorensen wrote: On Wed, Nov 13, 2013 at 10:53:30PM +0100, Francesco Pietra wrote: francesco@gig64:~/tmp$ file ./CUDA-Z-0.7.189.run ./CUDA-Z-0.7.189.run: data francesco@gig64:~/tmp$ OK that's weird. I expected to see x86 32 or 64bit binary. Seems to be a shell scripts with compressed code in it. Yuck. :) OK I got it running. It is a 32bit binary. I had to install these: ii libcuda1:i386 331.20-1i386 NVIDIA CUDA runtime library ii libcudart5.0:i386 5.0.35-8i386 NVIDIA CUDA runtime library ii libgl1-nvidia-glx:i386331.20-1i386 NVIDIA binary OpenGL libraries ii libstdc++6:i386 4.8.2-4 i386 GNU Standard C++ Library v3 ii libxrender1:i386 1:0.9.8-1 i386 X Rendering Extension client library ii zlib1g:i386 1:1.2.8.dfsg-1 i386 compression library - runtime Then I was able to run it. No messing with LD_LIBRARY_PATH or anything. To install :i386 packages you first have to enable multiarch support with dpkg and run apt-get update. So something like: dpkg --add-architecture i386 apt-get update apt-get install libcuda1:i386 libcudart5.0:i386 libgl1-nvidia-glx:i386 libstdc++6:i386 libxrender1:i386 zlib1g:i386 Don't worry about the exact versions, since I am running unstable+experimental. You don;t need to do that to get it working. For your 64bit code you probably need libcuda1 libcudart5.0 and such installed in the 64bit version. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131114023334.gk20...@csclub.uwaterloo.ca
upgrade to jessie from wheezy with cuda problems
Hello: I decided to try jessie to get PCIe 3.0 with a recent nvidia driver, thus upgrading from wheezy. wheezy was uname -r 3.2.0-4-amd64 nvidia-smi 304.88 nvcc --version 4.2 (the latter is also the version at which the molecular dynamics code was compiled, and used without calling the X-server) Following aptitude update aptitude-upgrade a number of dependecies related to gnome were not met (evolution-common lbfolks25 gnome-panel gnome-shell gnome-theme-extras gnome-theme-standard libreoffice-evolution). This notwithstanding, I decided to upgrade. After rebooting to get linux matching with nvidia: nvcc --version 5.0 uname -r 3.10-3-amd64 nvidia-smi the nvidia kernel module has version 304.108 but the nvidia driver component has version 319.60. Driver 319.6 is just what I wanted. Now, how best fix the problems? Install linux image 3.2? In the past I tried dist-upgrade, getting into devastating problems. thanks francesco pietra PS I was advised that debian is getting bounces from my address above. If so, please try my institutional address francesco.pie...@accademialucchese.it
Re: upgrade to jessie from wheezy with cuda problems
On Tue, Nov 12, 2013 at 03:54:32PM +0100, Francesco Pietra wrote: Hello: I decided to try jessie to get PCIe 3.0 with a recent nvidia driver, thus upgrading from wheezy. wheezy was uname -r 3.2.0-4-amd64 nvidia-smi 304.88 nvcc --version 4.2 (the latter is also the version at which the molecular dynamics code was compiled, and used without calling the X-server) Following aptitude update aptitude-upgrade a number of dependecies related to gnome were not met (evolution-common lbfolks25 gnome-panel gnome-shell gnome-theme-extras gnome-theme-standard libreoffice-evolution). This notwithstanding, I decided to upgrade. After rebooting to get linux matching with nvidia: nvcc --version 5.0 uname -r 3.10-3-amd64 nvidia-smi the nvidia kernel module has version 304.108 but the nvidia driver component has version 319.60. Driver 319.6 is just what I wanted. Now, how best fix the problems? Install linux image 3.2? In the past I tried dist-upgrade, getting into devastating problems. Do you have nvidia-kernel-dkms installed? -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131112145937.gc20...@csclub.uwaterloo.ca
Re: upgrade to jessie from wheezy with cuda problems
On Tue, Nov 12, 2013 at 05:22:18PM +0100, Francesco Pietra wrote: Yes. Also, # apt-get remove nvidia-kernel-dkms # apt-get install nvidia-kernel-dkms (which, in the year 2011, served to clear the driver at /lib/modules/2.6.38-2-amd64/updates/dkms. But now the kernel was 3.2.) left the issue unaltered. # modinfo nvidia ERROR: module nvidia not found $ dpkg -l |grep nvidia |less shows libl1-nvidia-glx:amd64 319.60 and libg1-nvidia-legacy-304xx--glx:amd64 304.108-4 NVIDIA metapackage rc nvidia-glx 304.88-1-deb7u1 nvidia-legacy-304xx-driver 304.108-4 nvidia-legacy-304xx-kernel-dkms 304.108-4 nvidia-settings-legacy-303xx 304.108-2 xserver-xorg-video-nvidia-legacy-304xx304.108-4 Everything else 319.60-1 and cuda 5.0 I don't understand why these 304xx are threatening. I had also run # nvidia-xconfig I think you should remove all packages with legacy-304xx in the name, and install the current ones (nvidia-kernel-dkms, nvidia-glx, etc). legacy-304xx will never move beyond version 304.xx after all as the name implies. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131112165947.ge20...@csclub.uwaterloo.ca
Re: upgrade to jessie from wheezy with cuda problems
# apt-get --purge remove *legacy* did the job. I wonder how these legacy packages entered the scene while updating/upgrading from a clean wheezy. The bad news are that with the new driver 319.60 there was no acceleration of molecular dynamics for a job of modest size (150K atoms) and slight acceleration (0.12 s/step vs 0.14 s/step) for a heavy job (500K atoms). Weather bringing from PCIe 2.0 (with the 304.xx driver of wheezy) to PCIe 3.0 (with driver 319.60 of jessie) (increasing the bandwidth from GPUs to RAM from 5 to 8GB/s) has not the effect that I hoped on the calculations, or PCIe is still 2.0 with jessie. Now, with cuda 5.0, it should be easy to measure the bandwidth directly. I have to learn how and I'll report about in due course. Now nvidia-smi activates the GPUs for normal work, nvidia-smi -L tells about the GPUs, dpkg -l |grep nvidia shows all 319.60 or 5.0.35-8, the X-server can be started and gnome loaded (startx, gnome-session), nvcc --version gives 5.0, however # modinfo nvidia ERROR: module nvidia not found In analogy with wheezy 3.2.0-4, I expected /lib/modules/3.10-3-amd64/updates/dkms/nvidia.ko Instead, there is /lib/modules/3.10-3-amd64/nvidia/nvidia-current.ko is that a feature of jessie or something wrong? Thanks a lot for advice. francesco pietra. On Tue, Nov 12, 2013 at 5:59 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Tue, Nov 12, 2013 at 05:22:18PM +0100, Francesco Pietra wrote: Yes. Also, # apt-get remove nvidia-kernel-dkms # apt-get install nvidia-kernel-dkms (which, in the year 2011, served to clear the driver at /lib/modules/2.6.38-2-amd64/updates/dkms. But now the kernel was 3.2.) left the issue unaltered. # modinfo nvidia ERROR: module nvidia not found $ dpkg -l |grep nvidia |less shows libl1-nvidia-glx:amd64 319.60 and libg1-nvidia-legacy-304xx--glx:amd64 304.108-4 NVIDIA metapackage rc nvidia-glx 304.88-1-deb7u1 nvidia-legacy-304xx-driver 304.108-4 nvidia-legacy-304xx-kernel-dkms 304.108-4 nvidia-settings-legacy-303xx 304.108-2 xserver-xorg-video-nvidia-legacy-304xx304.108-4 Everything else 319.60-1 and cuda 5.0 I don't understand why these 304xx are threatening. I had also run # nvidia-xconfig I think you should remove all packages with legacy-304xx in the name, and install the current ones (nvidia-kernel-dkms, nvidia-glx, etc). legacy-304xx will never move beyond version 304.xx after all as the name implies. -- Len Sorensen
Re: upgrade to jessie from wheezy with cuda problems
On Tue, Nov 12, 2013 at 10:35:53PM +0100, Francesco Pietra wrote: # apt-get --purge remove *legacy* did the job. I wonder how these legacy packages entered the scene while updating/upgrading from a clean wheezy. The bad news are that with the new driver 319.60 there was no acceleration of molecular dynamics for a job of modest size (150K atoms) and slight acceleration (0.12 s/step vs 0.14 s/step) for a heavy job (500K atoms). Weather bringing from PCIe 2.0 (with the 304.xx driver of wheezy) to PCIe 3.0 (with driver 319.60 of jessie) (increasing the bandwidth from GPUs to RAM from 5 to 8GB/s) has not the effect that I hoped on the calculations, or PCIe is still 2.0 with jessie. Now, with cuda 5.0, it should be easy to measure the bandwidth directly. I have to learn how and I'll report about in due course. Now nvidia-smi activates the GPUs for normal work, nvidia-smi -L tells about the GPUs, dpkg -l |grep nvidia shows all 319.60 or 5.0.35-8, the X-server can be started and gnome loaded (startx, gnome-session), nvcc --version gives 5.0, however # modinfo nvidia ERROR: module nvidia not found In analogy with wheezy 3.2.0-4, I expected /lib/modules/3.10-3-amd64/updates/dkms/nvidia.ko Instead, there is /lib/modules/3.10-3-amd64/nvidia/nvidia-current.ko is that a feature of jessie or something wrong? I think it was renamed. No idea why. modinfo nvidia-current should work though. Do you have the cuda libraries for the 319 version installed? I don't play around with GPU computations, but from what I have read it does need a certain size job before the overhead of transfering the data and managing the GPU makse it worthwhile, but for large jobs the high core count and memory bandwidth makes a big difference. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131112223724.gf20...@csclub.uwaterloo.ca
