RE: Do we anything for Deep Learning in Spark?
Thank you. Best Regards, Hari On 7 Jul 2017 3:59 a.m., "Roope Astala" wrote: > You can use an attached GPU VM for DNN training, and do other processing > on regular CPU nodes. You can even deallocate the GPU VM to save costs when > not using it. The GPU branch has instructions how to set up such compute > environment: > > > > https://github.com/Azure/mmlspark/tree/gpu#gpu-vm-setup > > > > Cheers, > > Roope – Microsoft Cloud AI Team > > > > *From:* hosur narahari [mailto:hnr1...@gmail.com] > *Sent:* Thursday, July 6, 2017 1:54 AM > *To:* Gaurav1809 > *Cc:* user > *Subject:* Re: Do we anything for Deep Learning in Spark? > > > > Hi Roope, > > > > Does this mmlspark project uses GPGPU for processing and just CPU cores > since DL models are computationally very intensive. > > > > Best Regards, > > Hari > > > > On 6 Jul 2017 9:33 a.m., "Gaurav1809" wrote: > > Thanks Roope for the inputs. > > > > On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden > email] <http://user/SendEmail.jtp?type=node&node=28826&i=0>> wrote: > > Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep > learning models on Spark. > > https://github.com/Azure/mmlspark > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fmmlspark&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=1FqnrRsLY6JrPb%2BdkG4AlLBN%2F8f2O6w3Ehi9DYvu6lc%3D&reserved=0> > > The library APIs are focused on image processing scenarios, and are > compatible with SparkML Pipelines. > > Cheers, > Roope - Microsoft Cloud AI Team > -- > > *If you reply to this email, your message will be added to the discussion > below:* > > http://apache-spark-user-list.1001560.n3.nabble.com/Do-we- > anything-for-Deep-Learning-in-Spark-tp28772p28824.html > <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=zFajYYH16xwzDRwgP1se9m%2BAA5fIufUS7FFxHjzTOl0%3D&reserved=0> > > To start a new topic under Apache Spark User List, email [hidden email] > <http://user/SendEmail.jtp?type=node&node=28826&i=1> > To unsubscribe from Do we anything for Deep Learning in Spark?, click here. > NAML > <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=wzDT2AoJRfwqxlumkdoRlGq6b0PaeoypZtvvLgFlR1M%3D&reserved=0> > > > > > -- > > View this message in context: Re: Do we anything for Deep Learning in > Spark? > <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=dDprNB1TSm9WkAY1iuN1NjkG7vzx6JaiXB42DcXqUt4%3D&reserved=0> > Sent from the Apache Spark User List mailing list archive > <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2F&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=bI5METH7hpg6xCjUQF0%2BiIKY7qFy709YGC4g0n9MjW0%3D&reserved=0> > at Nabble.com. > >
RE: Do we anything for Deep Learning in Spark?
You can use an attached GPU VM for DNN training, and do other processing on regular CPU nodes. You can even deallocate the GPU VM to save costs when not using it. The GPU branch has instructions how to set up such compute environment: https://github.com/Azure/mmlspark/tree/gpu#gpu-vm-setup Cheers, Roope – Microsoft Cloud AI Team From: hosur narahari [mailto:hnr1...@gmail.com] Sent: Thursday, July 6, 2017 1:54 AM To: Gaurav1809 Cc: user Subject: Re: Do we anything for Deep Learning in Spark? Hi Roope, Does this mmlspark project uses GPGPU for processing and just CPU cores since DL models are computationally very intensive. Best Regards, Hari On 6 Jul 2017 9:33 a.m., "Gaurav1809" mailto:gauravhpan...@gmail.com>> wrote: Thanks Roope for the inputs. On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden email]<http://user/SendEmail.jtp?type=node&node=28826&i=0>> wrote: Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep learning models on Spark. https://github.com/Azure/mmlspark<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fmmlspark&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=1FqnrRsLY6JrPb%2BdkG4AlLBN%2F8f2O6w3Ehi9DYvu6lc%3D&reserved=0> The library APIs are focused on image processing scenarios, and are compatible with SparkML Pipelines. Cheers, Roope - Microsoft Cloud AI Team If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=zFajYYH16xwzDRwgP1se9m%2BAA5fIufUS7FFxHjzTOl0%3D&reserved=0> To start a new topic under Apache Spark User List, email [hidden email]<http://user/SendEmail.jtp?type=node&node=28826&i=1> To unsubscribe from Do we anything for Deep Learning in Spark?, click here. NAML<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=wzDT2AoJRfwqxlumkdoRlGq6b0PaeoypZtvvLgFlR1M%3D&reserved=0> ____ View this message in context: Re: Do we anything for Deep Learning in Spark?<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=dDprNB1TSm9WkAY1iuN1NjkG7vzx6JaiXB42DcXqUt4%3D&reserved=0> Sent from the Apache Spark User List mailing list archive<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2F&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=bI5METH7hpg6xCjUQF0%2BiIKY7qFy709YGC4g0n9MjW0%3D&reserved=0> at Nabble.com.
Re: Do we anything for Deep Learning in Spark?
Hi Roope, Does this mmlspark project uses GPGPU for processing and just CPU cores since DL models are computationally very intensive. Best Regards, Hari On 6 Jul 2017 9:33 a.m., "Gaurav1809" wrote: > Thanks Roope for the inputs. > > On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=28826&i=0>> wrote: > >> Microsoft Machine Learning Library for Apache Spark lets you run CNTK >> deep learning models on Spark. >> >> https://github.com/Azure/mmlspark >> >> The library APIs are focused on image processing scenarios, and are >> compatible with SparkML Pipelines. >> >> Cheers, >> Roope - Microsoft Cloud AI Team >> >> -- >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-an >> ything-for-Deep-Learning-in-Spark-tp28772p28824.html >> To start a new topic under Apache Spark User List, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=28826&i=1> >> To unsubscribe from Do we anything for Deep Learning in Spark?, click >> here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > -- > View this message in context: Re: Do we anything for Deep Learning in > Spark? > <http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html> > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >
Re: Do we anything for Deep Learning in Spark?
Thanks Roope for the inputs. On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] < ml+s1001560n2882...@n3.nabble.com> wrote: > Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep > learning models on Spark. > > https://github.com/Azure/mmlspark > > The library APIs are focused on image processing scenarios, and are > compatible with SparkML Pipelines. > > Cheers, > Roope - Microsoft Cloud AI Team > > -- > If you reply to this email, your message will be added to the discussion > below: > http://apache-spark-user-list.1001560.n3.nabble.com/Do-we- > anything-for-Deep-Learning-in-Spark-tp28772p28824.html > To start a new topic under Apache Spark User List, email > ml+s1001560n1...@n3.nabble.com > To unsubscribe from Do we anything for Deep Learning in Spark?, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=28772&code=Z2F1cmF2aHBhbmR5YUBnbWFpbC5jb218Mjg3NzJ8NTQ0MzUyNjA4> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Do we anything for Deep Learning in Spark?
There is a BigDL project: https://github.com/intel-analytics/BigDL On 20 June 2017 at 16:17, Jules Damji wrote: > And we will having a webinar on July 27 going into some more details. Stay > tuned. > > Cheers > Jules > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Jun 20, 2017, at 7:00 AM, Michael Mior wrote: > > It's still in the early stages, but check out Deep Learning Pipelines from > Databricks > > https://github.com/databricks/spark-deep-learning > > -- > Michael Mior > mm...@apache.org > > 2017-06-20 0:36 GMT-04:00 Gaurav1809 : >> >> Hi All, >> >> Similar to how we have machine learning library called ML, do we have >> anything for deep learning? >> If yes, please share the details. If not then what should be the approach? >> >> Thanks and regards, >> Gaurav Pandya >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> > - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Do we anything for Deep Learning in Spark?
And we will having a webinar on July 27 going into some more details. Stay tuned. Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Jun 20, 2017, at 7:00 AM, Michael Mior wrote: > > It's still in the early stages, but check out Deep Learning Pipelines from > Databricks > > https://github.com/databricks/spark-deep-learning > > -- > Michael Mior > mm...@apache.org > > 2017-06-20 0:36 GMT-04:00 Gaurav1809 : >> Hi All, >> >> Similar to how we have machine learning library called ML, do we have >> anything for deep learning? >> If yes, please share the details. If not then what should be the approach? >> >> Thanks and regards, >> Gaurav Pandya >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >
Re: Do we anything for Deep Learning in Spark?
It's still in the early stages, but check out Deep Learning Pipelines from Databricks https://github.com/databricks/spark-deep-learning -- Michael Mior mm...@apache.org 2017-06-20 0:36 GMT-04:00 Gaurav1809 : > Hi All, > > Similar to how we have machine learning library called ML, do we have > anything for deep learning? > If yes, please share the details. If not then what should be the approach? > > Thanks and regards, > Gaurav Pandya > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in- > Spark-tp28772.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Do we anything for Deep Learning in Spark?
Hi All, Similar to how we have machine learning library called ML, do we have anything for deep learning? If yes, please share the details. If not then what should be the approach? Thanks and regards, Gaurav Pandya -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
