Hi, I have created a Custom Receiver in Java which receives data from Websphere MQ and I am only writing the received records on HDFS.
I have referred many forums for optimizing speed of spark streaming application. Here I am listing a few: * Spark Official<http://spark.apache.org/docs/latest/streaming-programming-guide.html#performance-tuning> * VIrdata<http://www.virdata.com/tuning-spark/> * TD's Slide (A bit Old but Useful)<http://www.slideshare.net/spark-project/deep-divewithsparkstreaming-tathagatadassparkmeetup20130617> I got mainly two point for my applicability : * giving batch interval as 1 sec * Controlling "spark.streaming.blockInterval" =200ms * inputStream.repartition(3) But that did not improve my actual speed (records/sec) of receiver which is MAX 5-10 records /sec. This is way less from my expectation. Am I missing something? Regards, Umesh Chaudhary This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
