Re: [Freesurfer] GPU compile cuda8/centos7

2017-11-30 Thread R Edgar 
On 30 November 2017 at 09:42, Dr. M. Shahid  wrote:

>> > Has all of the GPU code also been updated in the new FS6.0 version or
>> > the
>> > code is still from the old 5.x but can be compiled with new cuda
>> > toolkits?
>>
>> I made a small tweak for GCAMorph (primarily ca_register) which might
>> have made it in to v6; I'm not aware of any other changes. However,
>> being compiled with a new compiler for the new architectures should
>> give a small speed boost.
>
> Thanks for the info.
>
> However if the cpu code in FS6 is changed a lot, but the cuda code not, then
> the results would be different even if we get speed boost.
> Then how would we take the different results?

I don't know if there have been changes in the relevant portions of
the CPU code. And unfortunately, there aren't any automated
comparisons. If there have, then there will obviously be larger
deviations between the CPU and GPU streams (they will never be
identical anyway).

Richard
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] GPU compile cuda8/centos7

2017-11-30 Thread Dr. M. Shahid 
On Wed, Nov 29, 2017 at 5:27 AM, R Edgar  wrote:

> On 28 November 2017 at 09:12, Dr. M. Shahid 
> wrote:
>
> > Has all of the GPU code also been updated in the new FS6.0 version or the
> > code is still from the old 5.x but can be compiled with new cuda
> toolkits?
>
> I made a small tweak for GCAMorph (primarily ca_register) which might
> have made it in to v6; I'm not aware of any other changes. However,
> being compiled with a new compiler for the new architectures should
> give a small speed boost.
>
>
Thanks for the info.

However if the cpu code in FS6 is changed a lot, but the cuda code not,
then the results would be different even if we get speed boost.
Then how would we take the different results?


Best regards,

--
Shahid.




> Richard
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] GPU compile cuda8/centos7

2017-11-28 Thread R Edgar 
On 28 November 2017 at 09:12, Dr. M. Shahid  wrote:

> Has all of the GPU code also been updated in the new FS6.0 version or the
> code is still from the old 5.x but can be compiled with new cuda toolkits?

I made a small tweak for GCAMorph (primarily ca_register) which might
have made it in to v6; I'm not aware of any other changes. However,
being compiled with a new compiler for the new architectures should
give a small speed boost.

Richard
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] GPU compile cuda8/centos7

2017-11-28 Thread Dr. M. Shahid 
Hi,

Has all of the GPU code also been updated in the new FS6.0 version or the
code is still from the old 5.x but can be compiled with new cuda toolkits?

Best regards,

--

Shahid.



On Mon, Nov 27, 2017 at 9:29 PM, Meij, Henk  wrote:

> Well, it looks the *_cuda files for mri_glmfit and mri_robust_register and
> some others were dropped from the repository. Downloaded fresh copy today
> but the files are not there. So no errors.
>
>
> -Henk
> --
> *From:* Meij, Henk
> *Sent:* Monday, November 27, 2017 11:02:44 AM
> *To:* freesurfer@nmr.mgh.harvard.edu
> *Subject:* GPU compile cuda8/centos7
>
>
> Just wanted to report back a success. I had to do some fixes so hope this
> helps anybody.
>
> My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this
> server.
>
> The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3
> hours.
>
>
> Q: I seem to not have any errors or indications why mri_glmfit or
> mri_robust_register cuda binaries did not compile. Any ideas?
>
>
> -Henk
>
>
> In configure file (from previous post) comment out the with_cuda="" line
> https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/
> 2016-June/046236.html
> CUDA_DIR=""
> with_cuda=""
>
>
> In utils/mrifilter.c uncomment line defining 'int width, weight, height'
> above the test if (width <=1 || height <=1 || depth <=1) on line 3005
>
>
> then find Makefiles with incorrect gpu arch for the gpu packages
> change architecture from sm_?? to sm_61 (multiple times per file)
>
>
> which nvcc
>
> export PATH=/usr/local/cuda/bin:$PATH
> locate libcuda.so
> export LD_LIBRARY_PATH=/usr/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH
>
>
> ./configure --disable-Werror 
> --with-pkgs-dir=/usr/local/centos6-x86_64-packages
> --prefix=/usr/local/freesurfer-6.0.0 --with-cuda=/usr/local/cuda
>  make clean
> make -j 4
> make install
>
>
> yields
>
>
> ./bin/mris_flatten_cuda
> ./bin/mris_sphere_cuda
> ./bin/mris_fix_topology_cuda
> ./bin/mri_segstats_cuda
> ./bin/mri_em_register_cuda
> ./bin/mri_vol2vol_cuda
> ./bin/mris_inflate_cuda
> ./bin/mri_ca_register_cuda
>
>
>
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] GPU compile cuda8/centos7

2017-11-28 Thread R Edgar 
On 27 November 2017 at 11:02, Meij, Henk  wrote:
> Just wanted to report back a success. I had to do some fixes so hope this
> helps anybody.
>
> My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this
> server.
>
> The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3
> hours.

Thanks for the update. Can you give the CPU vs GPU times for
individual programs in the pipeline? I'd be interested to know those
(especially for a machine with as many cores as yours).

TIA,

Richard
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] GPU compile cuda8/centos7

2017-11-27 Thread Meij, Henk 
Well, it looks the *_cuda files for mri_glmfit and mri_robust_register and some 
others were dropped from the repository. Downloaded fresh copy today but the 
files are not there. So no errors.


-Henk


From: Meij, Henk
Sent: Monday, November 27, 2017 11:02:44 AM
To: freesurfer@nmr.mgh.harvard.edu
Subject: GPU compile cuda8/centos7


Just wanted to report back a success. I had to do some fixes so hope this helps 
anybody.

My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this 
server.

The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3 
hours.


Q: I seem to not have any errors or indications why mri_glmfit or 
mri_robust_register cuda binaries did not compile. Any ideas?


-Henk


In configure file (from previous post) comment out the with_cuda="" line
https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/2016-June/046236.html
CUDA_DIR=""
with_cuda=""


In utils/mrifilter.c uncomment line defining 'int width, weight, height'
above the test if (width <=1 || height <=1 || depth <=1) on line 3005


then find Makefiles with incorrect gpu arch for the gpu packages
change architecture from sm_?? to sm_61 (multiple times per file)


which nvcc

export PATH=/usr/local/cuda/bin:$PATH
locate libcuda.so
export LD_LIBRARY_PATH=/usr/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH


./configure --disable-Werror --with-pkgs-dir=/usr/local/centos6-x86_64-packages 
--prefix=/usr/local/freesurfer-6.0.0 --with-cuda=/usr/local/cuda
 make clean
make -j 4
make install


yields


./bin/mris_flatten_cuda
./bin/mris_sphere_cuda
./bin/mris_fix_topology_cuda
./bin/mri_segstats_cuda
./bin/mri_em_register_cuda
./bin/mri_vol2vol_cuda
./bin/mris_inflate_cuda
./bin/mri_ca_register_cuda

___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

[Freesurfer] GPU compile cuda8/centos7

2017-11-27 Thread Meij, Henk 
Just wanted to report back a success. I had to do some fixes so hope this helps 
anybody.

My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this 
server.

The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3 
hours.


Q: I seem to not have any errors or indications why mri_glmfit or 
mri_robust_register cuda binaries did not compile. Any ideas?


-Henk


In configure file (from previous post) comment out the with_cuda="" line
https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/2016-June/046236.html
CUDA_DIR=""
with_cuda=""


In utils/mrifilter.c uncomment line defining 'int width, weight, height'
above the test if (width <=1 || height <=1 || depth <=1) on line 3005


then find Makefiles with incorrect gpu arch for the gpu packages
change architecture from sm_?? to sm_61 (multiple times per file)


which nvcc

export PATH=/usr/local/cuda/bin:$PATH
locate libcuda.so
export LD_LIBRARY_PATH=/usr/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH


./configure --disable-Werror --with-pkgs-dir=/usr/local/centos6-x86_64-packages 
--prefix=/usr/local/freesurfer-6.0.0 --with-cuda=/usr/local/cuda
 make clean
make -j 4
make install


yields


./bin/mris_flatten_cuda
./bin/mris_sphere_cuda
./bin/mris_fix_topology_cuda
./bin/mri_segstats_cuda
./bin/mri_em_register_cuda
./bin/mri_vol2vol_cuda
./bin/mris_inflate_cuda
./bin/mri_ca_register_cuda

___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.