HI TD,
Some observations:
1. If I submit the application using spark-submit tool with *client as
deploy mode* it works fine with single master and worker (driver, master
and worker are running in same machine)
2. If I submit the application using spark-submit tool with client as
deploy mode it
*bump*
On Thu, Apr 23, 2015 at 3:46 PM, Sourav Chandra
sourav.chan...@livestream.com wrote:
HI TD,
Some observations:
1. If I submit the application using spark-submit tool with *client as
deploy mode* it works fine with single master and worker (driver, master
and worker are running in
It could very well be that your executor memory is not enough to store the
state RDDs AND operate on the data. 1G per executor is quite low.
Definitely give more memory. And have you tried increasing the number of
partitions (specify number of partitions in updateStateByKey) ?
On Wed, Apr 22,
Anyone?
On Wed, Apr 22, 2015 at 12:29 PM, Sourav Chandra
sourav.chan...@livestream.com wrote:
Hi Olivier,
*the update function is as below*:
*val updateFunc = (values: Seq[IConcurrentUsers], state: Option[(Long,
Long)]) = {*
* val previousCount = state.getOrElse((0L, 0L))._2*
*
Hi Sourav,
Can you post your updateFunc as well please ?
Regards,
Olivier.
Le mar. 21 avr. 2015 à 12:48, Sourav Chandra sourav.chan...@livestream.com
a écrit :
Hi,
We are building a spark streaming application which reads from kafka, does
updateStateBykey based on the received message type
