isn't also overridden by the master...)
How would you suggest setting this up?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/heterogeneous-cluster-hardware-tp11567p12609.html
Sent from the Apache Spark User List mailing list archiv
>> alg. operations...
>>> >>
>>> >> My Idea has been:
>>> >>
>>> >> 1) to find a way to configure Spark to allocate different resources
>>> >> per-machine, per-job. -- at least have a "standard executor"... and
>
>> >>
>> >> My Idea has been:
>> >>
>> >> 1) to find a way to configure Spark to allocate different resources
>> >> per-machine, per-job. -- at least have a "standard executor"... and
>> allow
>> >> different machi
-- at least have a "standard executor"... and
> allow
> >> different machines to have different numbers of executors.
> >>
> >> 2) make (using vanilla spark) a pre-run optimization phase which
> >> benchmarks
> >> the throughput of each node
spark) a pre-run optimization phase which
> >> benchmarks
> >> the throughput of each node (per hardware), and repartition the dataset
> to
> >> more efficiently use the hardware rather than rely on Spark
> Speculation--
> >> which has always seemed a di
rk) a pre-run optimization phase which
>> benchmarks
>> the throughput of each node (per hardware), and repartition the dataset to
>> more efficiently use the hardware rather than rely on Spark Speculation--
>> which has always seemed a dis-optimal way to balance the load across
ering machines.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/heterogeneous-cluster-hardware-tp11567p12581.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> --
.com/heterogeneous-cluster-hardware-tp11567p12581.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
look into
attempting to introduce this functionality if it is lacking.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/heterogeneous-cluster-hardware-tp11567.html
Sent from the Apache Spark User List mailing list archive at Nabbl
