Hi, the SPARK community should have been able to build distributed ML capabilities, and as far as I remember that was the idea initially behind SPARK 3.x roadmap (barrier execution mode, https://issues.apache.org/jira/browse/SPARK-24579).
Ray, another Berkeley Labs output like SPARK, is trying to capture that market space. I am not sure whether there is any steer by the SPARK community leaders to seriously prioritise building those capabilities at all. But I am sure if the brilliant and fantastic minds behind SPARK did actually want to allow building those capabilities, they can easily do so, and achieve that :) I would sincerely request the open source SPARK community to prioritise building the SPARK capabilities to scale ML applications. Thanks and Regards, Gourav Sengupta On Wed, Feb 23, 2022 at 3:53 AM Bitfox <bit...@bitfox.top> wrote: > tensorflow itself can implement the distributed computing via a > parameter server. Why did you want spark here? > > regards. > > On Wed, Feb 23, 2022 at 11:27 AM Vijayant Kumar > <vijayant.ku...@mavenir.com.invalid> wrote: > >> Thanks Sean for your response. !! >> >> >> >> Want to add some more background here. >> >> >> >> I am using Spark3.0+ version with Tensorflow 2.0+. >> >> My use case is not for the image data but for the Time-series data where >> I am using LSTM and transformers to forecast. >> >> >> >> I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and >> there has been no major development recently on those libraries. I faced >> the issue of version dependencies on those and had a hard time fixing the >> library compatibilities. Hence a couple of below doubts:- >> >> >> >> - Does *Horovod* have any dependencies? >> - Any other library which is suitable for my use case.? >> - Any example code would really be of great help to understand. >> >> >> >> Thanks, >> >> Vijayant >> >> >> >> *From:* Sean Owen <sro...@gmail.com> >> *Sent:* Wednesday, February 23, 2022 8:40 AM >> *To:* Vijayant Kumar <vijayant.ku...@mavenir.com.invalid> >> *Cc:* user @spark <user@spark.apache.org> >> *Subject:* [E] COMMERCIAL BULK: Re: TensorFlow on Spark >> >> >> >> *Email is from a Free Mail Service (Gmail/Yahoo/Hotmail….) *: Beware of >> Phishing Scams, Report questionable emails to s...@mavenir.com >> >> Sure, Horovod is commonly used on Spark for this: >> >> https://horovod.readthedocs.io/en/stable/spark_include.html >> >> >> >> On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar < >> vijayant.ku...@mavenir.com.invalid> wrote: >> >> Hi All, >> >> >> >> Anyone using Apache spark with TensorFlow for building models. My >> requirement is to use TensorFlow distributed model training across the >> Spark executors. >> >> Please help me with some resources or some sample code. >> >> >> >> Thanks, >> >> Vijayant >> ------------------------------ >> >> This e-mail message may contain confidential or proprietary information >> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >> use of the intended recipient(s). If you are not the intended recipient of >> this message, you are hereby notified that any review, use or distribution >> of this information is absolutely prohibited and we request that you delete >> all copies in your control and contact us by e-mailing to >> secur...@mavenir.com. This message contains the views of its author and >> may not necessarily reflect the views of Mavenir Systems, Inc. or its >> affiliates, who employ systems to monitor email messages, but make no >> representation that such messages are authorized, secure, uncompromised, or >> free from computer viruses, malware, or other defects. Thank You >> >> ------------------------------ >> >> This e-mail message may contain confidential or proprietary information >> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >> use of the intended recipient(s). If you are not the intended recipient of >> this message, you are hereby notified that any review, use or distribution >> of this information is absolutely prohibited and we request that you delete >> all copies in your control and contact us by e-mailing to >> secur...@mavenir.com. This message contains the views of its author and >> may not necessarily reflect the views of Mavenir Systems, Inc. or its >> affiliates, who employ systems to monitor email messages, but make no >> representation that such messages are authorized, secure, uncompromised, or >> free from computer viruses, malware, or other defects. Thank You >> >
