Hi Starch, It also depends on the applications behavior, some might not be properly able to utilize the network. If you are using say Kafka, then one thing that you should keep in mind is the Size of the individual message and the number of partitions that you are having. The higher the message size and higher number of partitions (in kafka) will utilize the network properly. With this combination, we have operated few pipelines running at 10Gb/s (~ 1GB/s ).
Thanks Best Regards On Tue, Aug 11, 2015 at 12:24 AM, Starch, Michael D (398M) < michael.d.sta...@jpl.nasa.gov> wrote: > All, > > I am trying to get data moving in and out of spark at 10Gb/s. I currently > have a very powerful cluster to work on, offering 40Gb/s inifiniband links > so I believe the network pipe should be fast enough. > > Has anyone gotten spark operating at high data rates before? Any advice > would be appreciated. > > -Michael Starch > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
