Hi All, In my application I have a serializable class which is taking InputDStream from Kafka. The inputDStream contains JSON which is stored in serializable case class. Transformations are applied and saveToCassandra() is executed. I was getting task not serializable exception , so I made the class serializable.
Now ,the application is working fine in standalone mode, but not able to receive data in local mode with the below mentioned log. What is internally happening?, if anyone have some insights Please share! Thank You in advance Regards, Prateek From: Prateek . Sent: Friday, October 09, 2015 6:55 PM To: user@spark.apache.org Subject: Streaming Application Unable to get Stream from Kafka Hi, I have Spark Streaming application running with the following log on console, I don’t get any exception but I am not able to receive the data from Kafka Stream. Can anyone please provide any insight what is happening with Spark Streaming. Is Receiver is not able to read the stream? How shall I debug it? JobScheduler: Added jobs for time 1444396043000 ms 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(922) called with curMem=1009312, maxMem=278302556 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043000 stored as bytes in memory (estimated size 922.0 B, free 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043000 in memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043000 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043000 already exists on this machine; not re-adding it 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043000 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(926) called with curMem=1010234, maxMem=278302556 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043200 stored as bytes in memory (estimated size 926.0 B, free 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043200 in memory on webanalytics03:51843 (size: 926.0 B, free: 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043200 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043200 already exists on this machine; not re-adding it 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043200 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(923) called with curMem=1011160, maxMem=278302556 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043400 stored as bytes in memory (estimated size 923.0 B, free 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043400 in memory on webanalytics03:51843 (size: 923.0 B, free: 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043400 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043400 already exists on this machine; not re-adding it 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043400 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(917) called with curMem=1012083, maxMem=278302556 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043600 stored as bytes in memory (estimated size 917.0 B, free 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043600 in memory on webanalytics03:51843 (size: 917.0 B, free: 264.4 MB) 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043600 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043600 already exists on this machine; not re-adding it 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043600 15/10/09 18:37:24 INFO ReceiverTracker: Stream 0 received 5 blocks 15/10/09 18:37:24 INFO MemoryStore: ensureFreeSpace(922) called with curMem=1013000, maxMem=278302556 15/10/09 18:37:24 INFO MemoryStore: Block input-0-1444396043800 stored as bytes in memory (estimated size 922.0 B, free 264.4 MB) 15/10/09 18:37:24 INFO BlockManagerInfo: Added input-0-1444396043800 in memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB) 15/10/09 18:37:24 INFO BlockManagerMaster: Updated info of block input-0-1444396043800 15/10/09 18:37:24 WARN BlockManager: Block input-0-1444396043800 already exists on this machine; not re-adding it 15/10/09 18:37:24 INFO BlockGenerator: Pushed block input-0-1444396043800 Thanks in advance Prateek "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."
