Re: GPU Support
Spark itself does not use GPUs, but you can write and run code on Spark that uses GPUs. You'd typically use software like Tensorflow that uses CUDA to access the GPU. On Thu, Jan 5, 2023 at 7:05 AM K B M Kaala Subhikshan < kbmkaalasubhiks...@gmail.com> wrote: > Is Gigabyte GeForce RTX 3080 GPU support for running machine learning in > Spark? >
GPU Support
Is Gigabyte GeForce RTX 3080 GPU support for running machine learning in Spark?
Re: Spark/Mesos with GPU support
Would it be possible to share that communication? I am interested in this thread. 2016-12-30 11:02 GMT-08:00 Ji Yan <ji...@drive.ai>: > Thanks Michael, Tim and I have touched base and thankfully the issue has > already been resolved > > On Fri, Dec 30, 2016 at 9:20 AM, Michael Gummelt <mgumm...@mesosphere.io> > wrote: > >> I've cc'd Tim and Kevin, who worked on GPU support. >> >> On Wed, Dec 28, 2016 at 11:22 AM, Ji Yan <ji...@drive.ai> wrote: >> >>> Dear Spark Users, >>> >>> Has anyone had successful experience running Spark on Mesos with GPU >>> support? We have a Mesos cluster that can see and offer nvidia GPU >>> resources. With Spark, it seems that the GPU support with Mesos ( >>> https://github.com/apache/spark/pull/14644) has only recently been >>> merged into Spark Master which is not found in 2.0.2 release yet. We have a >>> custom built Spark from 2.1-rc5 which is confirmed to have the above >>> change. However when we try to run any code from Spark on this Mesos setup, >>> the spark program hangs and keeps saying >>> >>> “WARN TaskSchedulerImpl: Initial job has not accepted any resources; >>> check your cluster UI to ensure that workers are registered and have >>> sufficient resources” >>> >>> We are pretty sure that the cluster has enough resources as there is >>> nothing running on it. If we disable the GPU support in configuration and >>> restart mesos and retry the same program, it would work. >>> >>> Any comment/advice on this greatly appreciated >>> >>> Thanks, >>> Ji >>> >>> >>> The information in this email is confidential and may be legally >>> privileged. It is intended solely for the addressee. Access to this email >>> by anyone else is unauthorized. If you are not the intended recipient, any >>> disclosure, copying, distribution or any action taken or omitted to be >>> taken in reliance on it, is prohibited and may be unlawful. >>> >> >> >> >> -- >> Michael Gummelt >> Software Engineer >> Mesosphere >> > > > The information in this email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. >
Re: Spark/Mesos with GPU support
Thanks Michael, Tim and I have touched base and thankfully the issue has already been resolved On Fri, Dec 30, 2016 at 9:20 AM, Michael Gummelt <mgumm...@mesosphere.io> wrote: > I've cc'd Tim and Kevin, who worked on GPU support. > > On Wed, Dec 28, 2016 at 11:22 AM, Ji Yan <ji...@drive.ai> wrote: > >> Dear Spark Users, >> >> Has anyone had successful experience running Spark on Mesos with GPU >> support? We have a Mesos cluster that can see and offer nvidia GPU >> resources. With Spark, it seems that the GPU support with Mesos ( >> https://github.com/apache/spark/pull/14644) has only recently been >> merged into Spark Master which is not found in 2.0.2 release yet. We have a >> custom built Spark from 2.1-rc5 which is confirmed to have the above >> change. However when we try to run any code from Spark on this Mesos setup, >> the spark program hangs and keeps saying >> >> “WARN TaskSchedulerImpl: Initial job has not accepted any resources; >> check your cluster UI to ensure that workers are registered and have >> sufficient resources” >> >> We are pretty sure that the cluster has enough resources as there is >> nothing running on it. If we disable the GPU support in configuration and >> restart mesos and retry the same program, it would work. >> >> Any comment/advice on this greatly appreciated >> >> Thanks, >> Ji >> >> >> The information in this email is confidential and may be legally >> privileged. It is intended solely for the addressee. Access to this email >> by anyone else is unauthorized. If you are not the intended recipient, any >> disclosure, copying, distribution or any action taken or omitted to be >> taken in reliance on it, is prohibited and may be unlawful. >> > > > > -- > Michael Gummelt > Software Engineer > Mesosphere > -- The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.
Re: Spark/Mesos with GPU support
I've cc'd Tim and Kevin, who worked on GPU support. On Wed, Dec 28, 2016 at 11:22 AM, Ji Yan <ji...@drive.ai> wrote: > Dear Spark Users, > > Has anyone had successful experience running Spark on Mesos with GPU > support? We have a Mesos cluster that can see and offer nvidia GPU > resources. With Spark, it seems that the GPU support with Mesos ( > https://github.com/apache/spark/pull/14644) has only recently been merged > into Spark Master which is not found in 2.0.2 release yet. We have a custom > built Spark from 2.1-rc5 which is confirmed to have the above change. > However when we try to run any code from Spark on this Mesos setup, the > spark program hangs and keeps saying > > “WARN TaskSchedulerImpl: Initial job has not accepted any resources; > check your cluster UI to ensure that workers are registered and have > sufficient resources” > > We are pretty sure that the cluster has enough resources as there is > nothing running on it. If we disable the GPU support in configuration and > restart mesos and retry the same program, it would work. > > Any comment/advice on this greatly appreciated > > Thanks, > Ji > > > The information in this email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. > -- Michael Gummelt Software Engineer Mesosphere
Spark/Mesos with GPU support
Dear Spark Users, Has anyone had successful experience running Spark on Mesos with GPU support? We have a Mesos cluster that can see and offer nvidia GPU resources. With Spark, it seems that the GPU support with Mesos (https://github.com/apache/spark/pull/14644 <https://github.com/apache/spark/pull/14644>) has only recently been merged into Spark Master which is not found in 2.0.2 release yet. We have a custom built Spark from 2.1-rc5 which is confirmed to have the above change. However when we try to run any code from Spark on this Mesos setup, the spark program hangs and keeps saying “WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources” We are pretty sure that the cluster has enough resources as there is nothing running on it. If we disable the GPU support in configuration and restart mesos and retry the same program, it would work. Any comment/advice on this greatly appreciated Thanks, Ji -- The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.
