com> wrote:
>
>
> You don't want to use CPUs with Tensorflow.
> If it's not scaling, you may have a problem that is far too small to
> distribute.
>
> On Sat, Apr 29, 2023 at 7:30 AM second_co...@yahoo.com.INVALID
> wrote:
>
> Anyone successfully run
at 7:30 AM second_co...@yahoo.com.INVALID
wrote:
Anyone successfully run native tensorflow on Spark ? i tested example at
https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-distributor
on Kubernetes CPU . By running in on multiple workers CPUs. I do not see any
speed up i
You don't want to use CPUs with Tensorflow.
If it's not scaling, you may have a problem that is far too small to
distribute.
On Sat, Apr 29, 2023 at 7:30 AM second_co...@yahoo.com.INVALID
wrote:
> Anyone successfully run native tensorflow on Spark ? i tested example at
> ht
Anyone successfully run native tensorflow on Spark ? i tested example at
https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-distributor
on Kubernetes CPU . By running in on multiple workers CPUs. I do not see any
speed up in training time by setting number of slot from1
Dear Sean,
I do agree with you to a certain extent, makes sense. Perhaps I am wrong in
asking for native integrations and not depending on over engineered
external solutions which have their own performance issues, and bottlenecks
in live production environment. But asking and stating ones opinion
ibraries. I faced
>>>> the issue of version dependencies on those and had a hard time fixing the
>>>> library compatibilities. Hence a couple of below doubts:-
>>>>
>>>>
>>>>
>>>>- Does *Horovod* have any de
On the contrary, distributed deep learning is not data parallel. It's
dominated by the need to share parameters across workers.
Gourav, I don't understand what you're looking for. Have you looked at
Petastorm and Horovod? they _use Spark_, not another platform like Ray. Why
recreate this which has
t;>> - Does *Horovod* have any dependencies?
>>>- Any other library which is suitable for my use case.?
>>>- Any example code would really be of great help to understand.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Vijayant
&
e any dependencies?
>>- Any other library which is suitable for my use case.?
>>- Any example code would really be of great help to understand.
>>
>>
>>
>> Thanks,
>>
>> Vijayant
>>
>>
>>
>> *From:* Sean Owen
>&
.?
>> Any example code would really be of great help to understand.
>>
>>
>> Thanks,
>>
>> Vijayant
>>
>>
>>
>> From: Sean Owen
>> Sent: Wednesday, February 23, 2022 8:40 AM
>> To: Vijayant Kumar
>> Cc: user
es data
>>>>> where I am using LSTM and transformers to forecast.
>>>>>
>>>>>
>>>>>
>>>>> I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and
>>>>> there has been no major development recent
t; I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and
>>>> there has been no major development recently on those libraries. I faced
>>>> the issue of version dependencies on those and had a hard time fixing the
>>>> library comp
gt;>
>>>- Does *Horovod* have any dependencies?
>>>- Any other library which is suitable for my use case.?
>>>- Any example code would really be of great help to understand.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Vijaya
Horovod* have any dependencies?
>>- Any other library which is suitable for my use case.?
>>- Any example code would really be of great help to understand.
>>
>>
>>
>> Thanks,
>>
>> Vijayant
>>
>>
>>
>> *From:
is suitable for my use case.?
>- Any example code would really be of great help to understand.
>
>
>
> Thanks,
>
> Vijayant
>
>
>
> *From:* Sean Owen
> *Sent:* Wednesday, February 23, 2022 8:40 AM
> *To:* Vijayant Kumar
> *Cc:* user @spark
> *S
ve any dependencies?
>- Any other library which is suitable for my use case.?
>- Any example code would really be of great help to understand.
>
>
>
> Thanks,
>
> Vijayant
>
>
>
> *From:* Sean Owen
> *Sent:* Wednesday, February 23, 2022 8:40 AM
> *T
is suitable for my use case.?
* Any example code would really be of great help to understand.
Thanks,
Vijayant
From: Sean Owen
Sent: Wednesday, February 23, 2022 8:40 AM
To: Vijayant Kumar
Cc: user @spark
Subject: [E] COMMERCIAL BULK: Re: TensorFlow on Spark
Email is from a Free Mail
Sure, Horovod is commonly used on Spark for this:
https://horovod.readthedocs.io/en/stable/spark_include.html
On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar
wrote:
> Hi All,
>
>
>
> Anyone using Apache spark with TensorFlow for building models. My
> requirement is to use TensorFlow distributed m
Hi All,
Anyone using Apache spark with TensorFlow for building models. My requirement
is to use TensorFlow distributed model training across the Spark executors.
Please help me with some resources or some sample code.
Thanks,
Vijayant
This e-mail message may cont
19 matches
Mail list logo
