Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-05 Thread kant kodali
te and get alerts >>> if the state is growing too large. >>> >>> Read more in the programming guide. >>> Watermarking - http://spark.apache.org/docs >>> /latest/structured-streaming-programming-guide.html#handling >>> -late-data-and-watermarking

Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-05 Thread kant kodali
king >> Monitoring - http://spark.apache.org/docs/latest/structured-streaming- >> programming-guide.html#monitoring-streaming-queries >> >> In case you were referring to something else, please give us more context >> details - what query, what are the symptoms you are o

Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-05 Thread kant kodali
ermarking > Monitoring - http://spark.apache.org/docs/latest/structured- > streaming-programming-guide.html#monitoring-streaming-queries > > In case you were referring to something else, please give us more context > details - what query, what are the symptoms you are observing.

Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-04 Thread Tathagata Das
kodali wrote: > Why do we ever run out of memory in Spark Structured Streaming especially > when Memory can always spill to disk ? until the disk is full we shouldn't > be out of memory.isn't it? sure thrashing will happen more frequently and > degrades performance but we do we ev

Why do we ever run out of memory in Spark Structured Streaming?

2017-04-04 Thread kant kodali
Why do we ever run out of memory in Spark Structured Streaming especially when Memory can always spill to disk ? until the disk is full we shouldn't be out of memory.isn't it? sure thrashing will happen more frequently and degrades performance but we do we ever run out Memory even