Got it. Thank you. When I increase the voxel size to twice and image dimension to half, it runs without any memory error.
I was wondering if I’d be able to work with float16 images and projections, since the default datatype is float32 for the filters I am using. On Jun 19, 2023, at 1:07 AM, Simon Rit <simon....@creatis.insa-lyon.fr> wrote: Hi, Yes, RTK only uses one GPU. We haven't implemented multi-GPU capability. Any contribution in that direction is welcome! If you really have 32 GB available for GPU0, then I don't understand what is the issue but you should be able to find out. Simon On Fri, Jun 16, 2023 at 6:32 PM Rahman, Obaid <rahm...@ornl.gov<mailto:rahm...@ornl.gov>> wrote: Thank you, Simon and Nils, for the suggestions. This is what I am doing: a. I read a (Xray CT) projection data (~1.5 GB) b. Then perform FDK reconstruction (~10 GB) c. Then forward project that reconstruction I am using the cuda version of itk-rtk. Steps a and b work fine and the reconstruction looks as expected. Then these are the lines of code I use for forward projection: ForwardProj = rtk.CudaForwardProjectionImageFilter[CudaImageType].New() ForwardProj.SetGeometry( geometry ) ForwardProj.SetInput(0, constantImageSource.GetOutput()) ForwardProj.SetInput(1, recon_image.GetOutput()) ForwardProj.Update() I get this error: Traceback (most recent call last): File "projection_test.py", line 223, in <module> ForwardProj.Update() RuntimeError: /work/src/rtkCudaUtilities.cu:115: ITK ERROR: CUDA ERROR: out of memory I also try to delete the variables I don’t need using (del variable), but I still get that error. The reconstruction is supposed to be ~10GB, projection is supposed to be 1.5 GB. GPU0 memory is 32 GB. I can see that before I get the error, GPU0 memory reaches about its capacity. The weird thing is that GPU1, GPU2, GPU3 that have the same memory capacity are basically unused (~4MB each). Is the forward projection not performed over all 4 GPUs? To try to force the code to use all 4 GPUs, I have also included these lines in the code: os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2,3" # specify which GPU(s) to be used But I still get that error, and only GPU0 gets used. Best, Obaid On Jun 16, 2023, at 8:58 AM, krah <nils.k...@creatis.insa-lyon.fr<mailto:nils.k...@creatis.insa-lyon.fr>> wrote: Just to add to Simon's answer: Objects are deleted by python's garbage collection mechanism once there are no references left to the object. (keyword: reference counting). The del variable statement Simon mention deletes the reference variable, but the memory is freed only when python runs a garbage collection. That might immediately afterwards, or slightly later (scheduled). Usually, you do not need to worry about that and deleting the reference as Simon said is sufficient. If for some reason you need to be absolutely sure that the object is garbage collected immediately, you can trigger a garbage collection by import gc gc.collect() Again, that should not be necessary, but it is useful to bear in mind how the memory is actually handled in python. Cheers, Nils On Jun 16 2023, at 9:10 am, Simon Rit <simon....@creatis.insa-lyon.fr<mailto:simon....@creatis.insa-lyon.fr>> wrote: Hi, If you have a Python variable holding memory, you can simply call del variable_name If you want to clear the GPU memory for an image, you can access the CPU buffer pointer to have it moved to the computer RAM. Examples below. Simon >>> import os >>> os.system('nvidia-smi') Fri Jun 16 08:49:38 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A | | N/A 52C P8 N/A / N/A | 0MiB / 4096MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ 0 >>> import itk >>> from itk import RTK as rtk >>> img=itk.CudaImage[itk.F, 3].New() >>> img.SetRegions([500]*3) >>> img.Allocate() >>> os.system('nvidia-smi') Fri Jun 16 08:51:28 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A | | N/A 58C P8 N/A / N/A | 45MiB / 4096MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 10890 C python 43MiB | +-----------------------------------------------------------------------------+ 0 >>> img.GetCudaDataManager().GetGPUBufferPointer() <Swig Object of type 'void *' at 0x7f20d3327f60> >>> os.system('nvidia-smi') Fri Jun 16 08:52:05 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A | | N/A 54C P0 N/A / N/A | 525MiB / 4096MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 10890 C python 523MiB | +-----------------------------------------------------------------------------+ 0 >>> del img >>> os.system('nvidia-smi') Fri Jun 16 08:52:21 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A | | N/A 56C P0 N/A / N/A | 2MiB / 4096MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ 0 >>> img=itk.CudaImage[itk.F, 3].New() >>> img.SetRegions([500]*3) >>> img.GetCudaDataManager().GetGPUBufferPointer() <Swig Object of type 'void *' at 0x7f20b2c30870> >>> os.system('nvidia-smi') Fri Jun 16 08:54:10 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A | | N/A 58C P0 N/A / N/A | 525MiB / 4096MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 10890 C python 523MiB | +-----------------------------------------------------------------------------+ 0 >>> img.GetCudaDataManager().GetCPUBufferPointer() >>> os.system('nvidia-smi') Fri Jun 16 08:54:20 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A | | N/A 58C P0 N/A / N/A | 47MiB / 4096MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 10890 C python 45MiB | +-----------------------------------------------------------------------------+ 0 On Thu, Jun 15, 2023 at 5:33 PM Rahman, Obaid <rahm...@ornl.gov<mailto:rahm...@ornl.gov>> wrote: Hi, I am using ink-rtk-cuda116 with python. I have too many cuda variables (images and filters). I would like to clear some of these variables. I am getting the following error: ITK ERROR: CUDA ERROR: out of memory Does anyone know how to clear Cuda variables in python? Thanks. Best, Obaidullah Rahman Oak Ridge National Laboratory, TN, United States _______________________________________________ Rtk-users mailing list rtk-us...@openrtk.org<mailto:rtk-us...@openrtk.org> https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users<https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=> _______________________________________________ Rtk-users mailing list rtk-us...@openrtk.org<mailto:rtk-us...@openrtk.org> https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users<https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwQFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
_______________________________________________ Rtk-users mailing list rtk-us...@openrtk.org https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users
