Hi,
We've recently started testing spark on kubernetes, and have found some odd
performance decreases. In particular its almost an order of magnitude
slower pulling data from kafka than it is in our mesos cluster.
We've tested a few set-ups:
Baseline: Spark 2.3.0 on Mesos host networking
Thanks for putting a comprehensive observation about Spark on Kubernetes. In
mesos Spark deployment, it has a property called spark.mesos.extra.cores.
The property means:
*
Set the extra number of cores for an executor to advertise. This does not
result in more cores allocated. It instead means
Thank you so much TD, Matt, Anirudh and Oz,
Really appropriate this.
On Fri, Apr 13, 2018 at 9:54 PM, Oz Ben-Ami <ozzi...@gmail.com> wrote:
> I can confirm that Structured Streaming works on Kubernetes, though we're
> not quite on production with that yet. Issues w
lover behavior right now.
>
>
>
> -Matt Cheah
>
>
>
> *From: *Tathagata Das <t...@databricks.com>
> *Date: *Friday, April 13, 2018 at 1:27 AM
> *To: *Krishna Kalyan <krishnakaly...@gmail.com>
> *Cc: *user <user@spark.apache.org>
> *Subject:
:27 AM
To: Krishna Kalyan <krishnakaly...@gmail.com>
Cc: user <user@spark.apache.org>
Subject: Re: Structured Streaming on Kubernetes
Structured streaming is stable in production! At Databricks, we and our
customers collectively process almost 100s of billions of records per day usin
and you are checkpointing to HDFS (for
fault-tolerance guarantees).
TD
On Fri, Apr 13, 2018, 12:28 AM Krishna Kalyan <krishnakaly...@gmail.com>
wrote:
> Hello All,
> We were evaluating Spark Structured Streaming on Kubernetes (Running on
> GCP). It would be awesome if the spark
Hello All,
We were evaluating Spark Structured Streaming on Kubernetes (Running on
GCP). It would be awesome if the spark community could share their
experience around this. I would like to know more about you production
experience and the monitoring tools you are using.
Since spark on kubernetes
