@All There is a strange problem,I had been running a spark streaming application for long time,follow is the application info:
1) Fetch data from kafka use dricet api 2) Use sql to write each rdd data of Dstream into redis 3) Read data from redis Everything seems ok during one week ,after one week,this application don't generate jobs any more ,only print follow info in driver log after last jobs generated,the 16/04/14 10:37:49 INFO JobScheduler: Added jobs for time 1460600680000 ms 16/04/14 10:37:51 INFO MetadataCleaner: Ran metadata cleaner for MAP_OUTPUT_TRACKER 16/04/14 10:37:51 INFO MetadataCleaner: Ran metadata cleaner for SPARK_CONTEXT 16/04/14 10:37:51 INFO BlockManager: Dropping non broadcast blocks older than 1460601351512 16/04/14 10:37:51 INFO BlockManager: Dropping broadcast blocks older than 1460601351512 16/04/14 10:37:51 INFO MetadataCleaner: Ran metadata cleaner for BROADCAST_VARS 16/04/14 10:37:51 INFO MetadataCleaner: Ran metadata cleaner for BLOCK_MANAGER 16/04/14 10:37:51 INFO SparkContext: Starting job: transform at BindCard.scala:44 16/04/14 10:38:03 INFO BlockManager: Dropping non broadcast blocks older than 1460601363512 16/04/14 10:38:03 INFO MetadataCleaner: Ran metadata cleaner for BLOCK_MANAGER 16/04/14 10:38:03 INFO MetadataCleaner: Ran metadata cleaner for MAP_OUTPUT_TRACKER 16/04/14 10:38:03 INFO MetadataCleaner: Ran metadata cleaner for SPARK_CONTEXT 16/04/14 10:38:03 INFO BlockManager: Dropping broadcast blocks older than 1460601363513 16/04/14 10:38:03 INFO MetadataCleaner: Ran metadata cleaner for BROADCAST_VARS 16/04/14 10:38:15 INFO BlockManager: Dropping non broadcast blocks older than 1460601375512 16/04/14 10:38:15 INFO MetadataCleaner: Ran metadata cleaner for BLOCK_MANAGER 16/04/14 10:38:15 INFO MetadataCleaner: Ran metadata cleaner for MAP_OUTPUT_TRACKER 16/04/14 10:38:15 INFO MetadataCleaner: Ran metadata cleaner for SPARK_CONTEXT 16/04/14 10:38:15 INFO BlockManager: Dropping broadcast blocks older than 1460601375513 16/04/14 10:38:15 INFO MetadataCleaner: Ran metadata cleaner for BROADCAST_VARS 16/04/14 10:38:27 INFO BlockManager: Dropping non broadcast blocks older than 1460601387513 Anyone had met this problem? Can anyone give me some advise for this issue,any possible reasons? I suspect wether the cpu for driver is all used for Full GC,no time for job generate. 岳猛(Rick) 00277916 大数据技术开发部 ***************************************************************************** [cid:image012.jpg@01D0D9C8.DDEDCC20]文档包<http://platformdoc.huawei.com/hedex/hwdc/doc/docInfo.jsp?productId=2472&type=doc> [cid:image009.png@01D0DA69.58E5C9A0]培训中心<http://3ms.huawei.com/hi/index.php?app=group&mod=Core&act=showSectionData&gid=2031037&id=1250433> [cid:image010.png@01D0DA69.58E5C9A0]案例库<http://3ms.huawei.com/hi/group/2031037/threads.html#category=1179741> 中软大数据3ms团队： http://3ms.huawei.com/hi/group/2031037
