Hi,
We've recently started testing spark on kubernetes, and have found some odd
performance decreases. In particular its almost an order of magnitude
slower pulling data from kafka than it is in our mesos cluster.
We've tested a few set-ups:
Baseline: Spark 2.3.0 on Mesos host networking (~5mill
Thanks for putting a comprehensive observation about Spark on Kubernetes. In
mesos Spark deployment, it has a property called spark.mesos.extra.cores.
The property means:
*
Set the extra number of cores for an executor to advertise. This does not
result in more cores allocated. It instead means tha
.
>>>
>>>
>>>
>>> However, I’m unaware of any specific use of streaming with the Spark on
>>> Kubernetes integration right now. Would be curious to get feedback on the
>>> failover behavior right now.
>>>
>>>
>>>
>
now.
>
>
>
> -Matt Cheah
>
>
>
> *From: *Tathagata Das
> *Date: *Friday, April 13, 2018 at 1:27 AM
> *To: *Krishna Kalyan
> *Cc: *user
> *Subject: *Re: Structured Streaming on Kubernetes
>
>
>
> Structured streaming is stable in production! At Data
: user
Subject: Re: Structured Streaming on Kubernetes
Structured streaming is stable in production! At Databricks, we and our
customers collectively process almost 100s of billions of records per day using
SS. However, we are not using kubernetes :)
Though I don't think it will matte
gured and you are checkpointing to HDFS (for
fault-tolerance guarantees).
TD
On Fri, Apr 13, 2018, 12:28 AM Krishna Kalyan
wrote:
> Hello All,
> We were evaluating Spark Structured Streaming on Kubernetes (Running on
> GCP). It would be awesome if the spark community could share thei
Hello All,
We were evaluating Spark Structured Streaming on Kubernetes (Running on
GCP). It would be awesome if the spark community could share their
experience around this. I would like to know more about you production
experience and the monitoring tools you are using.
Since spark on kubernetes
