Hi, I do not think that there is any reason for using over engineered platforms like Petastorm and Ray, except for certain use cases.
What Ray is doing, except for certain use cases, could have been easily done by SPARK, I think, had the open source community got that steer. But maybe I am wrong and someone should be able to explain why the SPARK open source community cannot develop the capabilities which are so natural to almost all use cases of data processing in SPARK where the data gets consumed by deep learning frameworks and we are asked to use Ray or Petastorm? For those of us who are asking what does native integrations means please try to compare delta between release 2.x and 3.x and koalas before 3.2 and after 3.2. I am sure that the SPARK community can push for extending the dataframes from SPARK to deep learning and other frameworks by natively integrating them. Regards, Gourav Sengupta On Wed, Feb 23, 2022 at 4:42 PM Dennis Suhari <d.suh...@icloud.com.invalid> wrote: > Currently we are trying AnalyticsZoo and Ray > > > Von meinem iPhone gesendet > > Am 23.02.2022 um 04:53 schrieb Bitfox <bit...@bitfox.top>: > > > tensorflow itself can implement the distributed computing via a > parameter server. Why did you want spark here? > > regards. > > On Wed, Feb 23, 2022 at 11:27 AM Vijayant Kumar > <vijayant.ku...@mavenir.com.invalid> wrote: > >> Thanks Sean for your response. !! >> >> >> >> Want to add some more background here. >> >> >> >> I am using Spark3.0+ version with Tensorflow 2.0+. >> >> My use case is not for the image data but for the Time-series data where >> I am using LSTM and transformers to forecast. >> >> >> >> I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and >> there has been no major development recently on those libraries. I faced >> the issue of version dependencies on those and had a hard time fixing the >> library compatibilities. Hence a couple of below doubts:- >> >> >> >> - Does *Horovod* have any dependencies? >> - Any other library which is suitable for my use case.? >> - Any example code would really be of great help to understand. >> >> >> >> Thanks, >> >> Vijayant >> >> >> >> *From:* Sean Owen <sro...@gmail.com> >> *Sent:* Wednesday, February 23, 2022 8:40 AM >> *To:* Vijayant Kumar <vijayant.ku...@mavenir.com.invalid> >> *Cc:* user @spark <user@spark.apache.org> >> *Subject:* [E] COMMERCIAL BULK: Re: TensorFlow on Spark >> >> >> >> *Email is from a Free Mail Service (Gmail/Yahoo/Hotmail….) *: Beware of >> Phishing Scams, Report questionable emails to s...@mavenir.com >> >> Sure, Horovod is commonly used on Spark for this: >> >> https://horovod.readthedocs.io/en/stable/spark_include.html >> >> >> >> On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar < >> vijayant.ku...@mavenir.com.invalid> wrote: >> >> Hi All, >> >> >> >> Anyone using Apache spark with TensorFlow for building models. My >> requirement is to use TensorFlow distributed model training across the >> Spark executors. >> >> Please help me with some resources or some sample code. >> >> >> >> Thanks, >> >> Vijayant >> ------------------------------ >> >> This e-mail message may contain confidential or proprietary information >> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >> use of the intended recipient(s). If you are not the intended recipient of >> this message, you are hereby notified that any review, use or distribution >> of this information is absolutely prohibited and we request that you delete >> all copies in your control and contact us by e-mailing to >> secur...@mavenir.com. This message contains the views of its author and >> may not necessarily reflect the views of Mavenir Systems, Inc. or its >> affiliates, who employ systems to monitor email messages, but make no >> representation that such messages are authorized, secure, uncompromised, or >> free from computer viruses, malware, or other defects. Thank You >> >> ------------------------------ >> >> This e-mail message may contain confidential or proprietary information >> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >> use of the intended recipient(s). If you are not the intended recipient of >> this message, you are hereby notified that any review, use or distribution >> of this information is absolutely prohibited and we request that you delete >> all copies in your control and contact us by e-mailing to >> secur...@mavenir.com. This message contains the views of its author and >> may not necessarily reflect the views of Mavenir Systems, Inc. or its >> affiliates, who employ systems to monitor email messages, but make no >> representation that such messages are authorized, secure, uncompromised, or >> free from computer viruses, malware, or other defects. Thank You >> >
