Hi Start by running it locally. See how it compares. Debug Then move to cluster Debugging stuff running on cluster is a pain as there can be tons of reasons Isolate the problem locally
On 20 Nov 2016 5:04 pm, "Kappaganthu, Sivaram (ES)" < sivaram.kappagan...@adp.com> wrote: > Thank You for helping Mistroni. > > > > I took this example from the link http://stdatalabs.blogspot.in/ > 2016/09/spark-streaming-part-1-real-time.html . > > > > I also took top 10 elements in the row. I just ran this example with Yarn > master. It ran fine and giving me good results. But in Spark master, it is > taking so much time and went on hold. Please help. > > > > Thanks, > > SIvaram > > > > *From:* Marco Mistroni [mailto:mmistr...@gmail.com] > *Sent:* Sunday, November 20, 2016 6:34 PM > *To:* Kappaganthu, Sivaram (ES) > *Cc:* user@spark.apache.org > *Subject:* Re: Error in running twitter streaming job > > > > HI > > is that from Spark Streaming Course by Frank Kane? anyway, i have nearly > same example (where i only pick up top 10) and works for me > > so i am guessing perhaps you are not setting up twitter credentials? > > Did you try to see if in the output you are fetching them correctly? > > hth > > marco > > > > > > On Sun, Nov 20, 2016 at 11:09 AM, Kappaganthu, Sivaram (ES) < > sivaram.kappagan...@adp.com> wrote: > > Hi, > > > > I am trying to run a twitter streaming job with the below code and not > getting any output. Could someone please help in resolving the issue. > > > > > > package org.test.scala > > > > import org.apache.spark.SparkConf > > import org.apache.spark.SparkContext > > import org.apache.spark.streaming._ > > import org.apache.spark.SparkContext._ > > import org.apache.spark.streaming.twitter._ > > > > > > > > object twitterAnalysis { > > val conf = new SparkConf().setAppName("twitter sentiment analysis") > > val sc = new SparkContext(conf) > > > > def main(args: Array[String]): Unit = { > > > > sc.setLogLevel("WARN") > > // val Array(consumerKey,consumerSecret,accessToken,accessSecret) = > args.take(4) > > val filters = args > > > > for(i <- filters) { println("recieved arg" + i) } > > > > /* setting up the Twitter4j library parameters */ > > > > > > System.setProperty("twitter4j.oauth.consumerKey", mykey) > > System.setProperty("twitter4j.oauth.consumerSecret", mysecret) > > System.setProperty("twitter4j.oauth.accessToken", acesskey) > > System.setProperty("twitter4j.oauth.accessTokenSecret", accesssecret) > > > > > > val ssc = new StreamingContext(sc,Seconds(5)) > > > > val stream = TwitterUtils.createStream(ssc,None,filters) > > > > //stream.print() > > > > val hashtags = stream.flatMap( status => status.getText.split(" > ").filter( _.startsWith("#")) ) > > > > println("here are the elements of HashTags") > > // hashtags.print() > > > > val topcounts60 = hashtags.map((_,1)).reduceByKeyAndWindow(_ + _ > ,Seconds(60)) > > .map { case (topic,count) => (count,topic) } > > .transform(_.sortByKey(false)) > > > > > > val topcount10 = hashtags.map((_,1)).reduceByKeyAndWindow(_+_, > Seconds(10)) > > .map{ case(topic,count) => (count,topic) } > > .transform(_.sortByKey(false)) > > > > > > topcounts60.foreachRDD(rdd => { > > val toplist = rdd.take(10) > > println("\n Popular topics in last 10 seconds ( %s total) > ".format(rdd.count())) > > toplist.foreach { case (count,tag) => println("%s (%s > tweets)".format(tag, count)) } > > > > }) > > ssc.start > > ssc.awaitTermination > > > > } > > > > } > > > > > > Upon hitting this with command bin/spark-submit –class > org.test.scala.twitterAnalysis –master spark://localhost.localdomain:7077 > –jars <dependency jarlocation> jar name, the job is getting hold and not > giving me any output. Below are the terminal output and STDERR log > > > > Terminal output > > > > [edureka@localhost spark-1.5.2]$ bin/spark-submit --class > org.test.scala.twitterAnalysis --master spark://localhost.localdomain:7077 > --jars file:///home/edureka/TransferredJars/twitter-0.0.1- > SNAPSHOT-jar-with-dependencies.jar > /home/edureka/TransferredJars/twitter-0.0.1-SNAPSHOT.jar > #apple #spark #currency #iphone > > 16/11/20 15:48:31 INFO spark.SparkContext: Running Spark version 1.5.2 > > 16/11/20 15:48:32 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 16/11/20 15:48:32 WARN util.Utils: Your hostname, localhost.localdomain > resolves to a loopback address: 127.0.0.1; using 192.168.0.107 instead (on > interface eth3) > > 16/11/20 15:48:32 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind > to another address > > 16/11/20 15:48:32 INFO spark.SecurityManager: Changing view acls to: > edureka > > 16/11/20 15:48:32 INFO spark.SecurityManager: Changing modify acls to: > edureka > > 16/11/20 15:48:32 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(edureka); users with modify permissions: Set(edureka) > > 16/11/20 15:48:34 INFO slf4j.Slf4jLogger: Slf4jLogger started > > 16/11/20 15:48:34 INFO Remoting: Starting remoting > > 16/11/20 15:48:35 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@192.168.0.107:38060] > > 16/11/20 15:48:35 INFO util.Utils: Successfully started service > 'sparkDriver' on port 38060. > > 16/11/20 15:48:35 INFO spark.SparkEnv: Registering MapOutputTracker > > 16/11/20 15:48:35 INFO spark.SparkEnv: Registering BlockManagerMaster > > 16/11/20 15:48:35 INFO storage.DiskBlockManager: Created local directory > at /tmp/blockmgr-521ff84c-473d-4c91-b675-933a02fc6cf4 > > 16/11/20 15:48:35 INFO storage.MemoryStore: MemoryStore started with > capacity 534.5 MB > > 16/11/20 15:48:35 INFO spark.HttpFileServer: HTTP File server directory is > /tmp/spark-1e434766-f32a-4608-92a0-2207b4a52727/httpd- > 29639e5c-b536-432a-888a-396eb60a60cb > > 16/11/20 15:48:35 INFO spark.HttpServer: Starting HTTP Server > > 16/11/20 15:48:35 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 16/11/20 15:48:36 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:56724 > > 16/11/20 15:48:36 INFO util.Utils: Successfully started service 'HTTP file > server' on port 56724. > > 16/11/20 15:48:36 INFO spark.SparkEnv: Registering OutputCommitCoordinator > > 16/11/20 15:48:36 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 16/11/20 15:48:36 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > > 16/11/20 15:48:36 INFO util.Utils: Successfully started service 'SparkUI' > on port 4040. > > 16/11/20 15:48:36 INFO ui.SparkUI: Started SparkUI at > http://192.168.0.107:4040 > > 16/11/20 15:48:44 INFO spark.SparkContext: Added JAR file:///home/edureka/ > TransferredJars/twitter-0.0.1-SNAPSHOT-jar-with-dependencies.jar at > http://192.168.0.107:56724/jars/twitter-0.0.1-SNAPSHOT- > jar-with-dependencies.jar with timestamp 1479637124496 > > 16/11/20 15:48:44 INFO spark.SparkContext: Added JAR file:/home/edureka/ > TransferredJars/twitter-0.0.1-SNAPSHOT.jar at http://192.168.0.107:56724/ > jars/twitter-0.0.1-SNAPSHOT.jar with timestamp 1479637124520 > > 16/11/20 15:48:44 WARN metrics.MetricsSystem: Using default name > DAGScheduler for source because spark.app.id is not set. > > 16/11/20 15:48:44 INFO client.AppClient$ClientEndpoint: Connecting to > master spark://localhost.localdomain:7077... > > 16/11/20 15:48:45 INFO cluster.SparkDeploySchedulerBackend: Connected to > Spark cluster with app ID app-20161120154845-0000 > > 16/11/20 15:48:45 INFO client.AppClient$ClientEndpoint: Executor added: > app-20161120154845-0000/0 on worker-20161120154741-192.168.0.107-42431 ( > 192.168.0.107:42431) with 1 cores > > 16/11/20 15:48:45 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20161120154845-0000/0 on hostPort 192.168.0.107:42431 > with 1 cores, 1024.0 MB RAM > > 16/11/20 15:48:45 INFO client.AppClient$ClientEndpoint: Executor updated: > app-20161120154845-0000/0 is now RUNNING > > 16/11/20 15:48:46 INFO client.AppClient$ClientEndpoint: Executor updated: > app-20161120154845-0000/0 is now LOADING > > 16/11/20 15:48:47 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38621. > > 16/11/20 15:48:47 INFO netty.NettyBlockTransferService: Server created on > 38621 > > 16/11/20 15:48:47 INFO storage.BlockManagerMaster: Trying to register > BlockManager > > 16/11/20 15:48:47 INFO storage.BlockManagerMasterEndpoint: Registering > block manager 192.168.0.107:38621 with 534.5 MB RAM, > BlockManagerId(driver, 192.168.0.107, 38621) > > 16/11/20 15:48:47 INFO storage.BlockManagerMaster: Registered BlockManager > > 16/11/20 15:48:48 INFO cluster.SparkDeploySchedulerBackend: > SchedulerBackend is ready for scheduling beginning after reached > minRegisteredResourcesRatio: 0.0 > > here are the elements of HashTags > > 16/11/20 15:48:52 WARN util.SizeEstimator: Failed to check whether > UseCompressedOops is set; assuming yes > > > > *STDERR LOG* > > > > > > 16/11/20 15:52:36 INFO receiver.BlockGenerator: Pushed block > input-0-1479637356400 > > 16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(15589) called > with curMem=12488978, maxMem=560497950 > > 16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637356800 > stored as bytes in memory (estimated size 15.2 KB, free 522.6 MB) > > 16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637356800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block > input-0-1479637356800 > > 16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(7069) called > with curMem=12504567, maxMem=560497950 > > 16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357000 > stored as bytes in memory (estimated size 6.9 KB, free 522.6 MB) > > 16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block > input-0-1479637357000 > > 16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(47183) called > with curMem=12511636, maxMem=560497950 > > 16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357200 > stored as bytes in memory (estimated size 46.1 KB, free 522.6 MB) > > 16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block > input-0-1479637357200 > > 16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(3424) called > with curMem=12558819, maxMem=560497950 > > 16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357400 > stored as bytes in memory (estimated size 3.3 KB, free 522.6 MB) > > 16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block > input-0-1479637357400 > > 16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(4630) called > with curMem=12562243, maxMem=560497950 > > 16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357600 > stored as bytes in memory (estimated size 4.5 KB, free 522.5 MB) > > 16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357600 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block > input-0-1479637357600 > > 16/11/20 15:52:38 INFO storage.MemoryStore: ensureFreeSpace(10421) called > with curMem=12566873, maxMem=560497950 > > 16/11/20 15:52:38 INFO storage.MemoryStore: Block input-0-1479637357800 > stored as bytes in memory (estimated size 10.2 KB, free 522.5 MB) > > 16/11/20 15:52:38 WARN storage.BlockManager: Block input-0-1479637357800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:38 INFO receiver.BlockGenerator: Pushed block > input-0-1479637357800 > > 16/11/20 15:52:38 INFO storage.MemoryStore: ensureFreeSpace(44965) called > with curMem=12577294, maxMem=560497950 > > 16/11/20 15:52:38 INFO storage.MemoryStore: Block input-0-1479637358000 > stored as bytes in memory (estimated size 43.9 KB, free 522.5 MB) > > 16/11/20 15:52:38 WARN storage.BlockManager: Block input-0-1479637358000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:38 INFO receiver.BlockGenerator: Pushed block > input-0-1479637358000 > > 16/11/20 15:52:38 INFO storage.MemoryStore: ensureFreeSpace(6376) called > with curMem=12622259, maxMem=560497950 > > 16/11/20 15:52:38 INFO storage.MemoryStore: Block input-0-1479637358200 > stored as bytes in memory (estimated size 6.2 KB, free 522.5 MB) > > 16/11/20 15:52:38 WARN storage.BlockManager: Block input-0-1479637358200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:38 INFO receiver.BlockGenerator: Pushed block > input-0-1479637358200 > > 16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(10971) called > with curMem=12628635, maxMem=560497950 > > 16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637358800 > stored as bytes in memory (estimated size 10.7 KB, free 522.5 MB) > > 16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637358800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:39 INFO receiver.BlockGenerator: Pushed block > input-0-1479637358800 > > 16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(2507) called > with curMem=12639606, maxMem=560497950 > > 16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637359000 > stored as bytes in memory (estimated size 2.4 KB, free 522.5 MB) > > 16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637359000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:39 INFO receiver.BlockGenerator: Pushed block > input-0-1479637359000 > > 16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(45512) called > with curMem=12642113, maxMem=560497950 > > 16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637359200 > stored as bytes in memory (estimated size 44.4 KB, free 522.4 MB) > > 16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637359200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:39 INFO receiver.BlockGenerator: Pushed block > input-0-1479637359200 > > 16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(3027) called > with curMem=12687625, maxMem=560497950 > > 16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637359400 > stored as bytes in memory (estimated size 3.0 KB, free 522.4 MB) > > 16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637359400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:39 INFO receiver.BlockGenerator: Pushed block > input-0-1479637359400 > > 16/11/20 15:52:40 INFO storage.MemoryStore: ensureFreeSpace(11076) called > with curMem=12690652, maxMem=560497950 > > 16/11/20 15:52:40 INFO storage.MemoryStore: Block input-0-1479637359800 > stored as bytes in memory (estimated size 10.8 KB, free 522.4 MB) > > 16/11/20 15:52:40 WARN storage.BlockManager: Block input-0-1479637359800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:40 INFO receiver.BlockGenerator: Pushed block > input-0-1479637359800 > > 16/11/20 15:52:40 INFO storage.MemoryStore: ensureFreeSpace(25581) called > with curMem=12701728, maxMem=560497950 > > 16/11/20 15:52:40 INFO storage.MemoryStore: Block input-0-1479637360000 > stored as bytes in memory (estimated size 25.0 KB, free 522.4 MB) > > 16/11/20 15:52:40 WARN storage.BlockManager: Block input-0-1479637360000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:40 INFO receiver.BlockGenerator: Pushed block > input-0-1479637360000 > > 16/11/20 15:52:40 INFO storage.MemoryStore: ensureFreeSpace(13282) called > with curMem=12727309, maxMem=560497950 > > 16/11/20 15:52:40 INFO storage.MemoryStore: Block input-0-1479637360200 > stored as bytes in memory (estimated size 13.0 KB, free 522.4 MB) > > 16/11/20 15:52:40 WARN storage.BlockManager: Block input-0-1479637360200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:40 INFO receiver.BlockGenerator: Pushed block > input-0-1479637360200 > > 16/11/20 15:52:41 INFO storage.MemoryStore: ensureFreeSpace(10337) called > with curMem=12740591, maxMem=560497950 > > 16/11/20 15:52:41 INFO storage.MemoryStore: Block input-0-1479637360800 > stored as bytes in memory (estimated size 10.1 KB, free 522.4 MB) > > 16/11/20 15:52:41 WARN storage.BlockManager: Block input-0-1479637360800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:41 INFO receiver.BlockGenerator: Pushed block > input-0-1479637360800 > > 16/11/20 15:52:41 INFO storage.MemoryStore: ensureFreeSpace(22085) called > with curMem=12750928, maxMem=560497950 > > 16/11/20 15:52:41 INFO storage.MemoryStore: Block input-0-1479637361000 > stored as bytes in memory (estimated size 21.6 KB, free 522.4 MB) > > 16/11/20 15:52:41 WARN storage.BlockManager: Block input-0-1479637361000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:41 INFO receiver.BlockGenerator: Pushed block > input-0-1479637361000 > > 16/11/20 15:52:41 INFO storage.MemoryStore: ensureFreeSpace(32643) called > with curMem=12773013, maxMem=560497950 > > 16/11/20 15:52:41 INFO storage.MemoryStore: Block input-0-1479637361200 > stored as bytes in memory (estimated size 31.9 KB, free 522.3 MB) > > 16/11/20 15:52:41 WARN storage.BlockManager: Block input-0-1479637361200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:41 INFO receiver.BlockGenerator: Pushed block > input-0-1479637361200 > > 16/11/20 15:52:41 INFO storage.MemoryStore: ensureFreeSpace(5317) called > with curMem=12805656, maxMem=560497950 > > 16/11/20 15:52:41 INFO storage.MemoryStore: Block input-0-1479637361400 > stored as bytes in memory (estimated size 5.2 KB, free 522.3 MB) > > 16/11/20 15:52:41 WARN storage.BlockManager: Block input-0-1479637361400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:41 INFO receiver.BlockGenerator: Pushed block > input-0-1479637361400 > > 16/11/20 15:52:42 INFO storage.MemoryStore: ensureFreeSpace(8229) called > with curMem=12810973, maxMem=560497950 > > 16/11/20 15:52:42 INFO storage.MemoryStore: Block input-0-1479637361800 > stored as bytes in memory (estimated size 8.0 KB, free 522.3 MB) > > 16/11/20 15:52:42 WARN storage.BlockManager: Block input-0-1479637361800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:42 INFO receiver.BlockGenerator: Pushed block > input-0-1479637361800 > > 16/11/20 15:52:42 INFO storage.MemoryStore: ensureFreeSpace(56457) called > with curMem=12819202, maxMem=560497950 > > 16/11/20 15:52:42 INFO storage.MemoryStore: Block input-0-1479637362200 > stored as bytes in memory (estimated size 55.1 KB, free 522.3 MB) > > 16/11/20 15:52:42 WARN storage.BlockManager: Block input-0-1479637362200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:42 INFO receiver.BlockGenerator: Pushed block > input-0-1479637362200 > > 16/11/20 15:52:42 INFO storage.MemoryStore: ensureFreeSpace(2703) called > with curMem=12875659, maxMem=560497950 > > 16/11/20 15:52:42 INFO storage.MemoryStore: Block input-0-1479637362400 > stored as bytes in memory (estimated size 2.6 KB, free 522.3 MB) > > 16/11/20 15:52:42 WARN storage.BlockManager: Block input-0-1479637362400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:42 INFO receiver.BlockGenerator: Pushed block > input-0-1479637362400 > > 16/11/20 15:52:42 INFO storage.MemoryStore: ensureFreeSpace(3665) called > with curMem=12878362, maxMem=560497950 > > 16/11/20 15:52:42 INFO storage.MemoryStore: Block input-0-1479637362600 > stored as bytes in memory (estimated size 3.6 KB, free 522.2 MB) > > 16/11/20 15:52:42 WARN storage.BlockManager: Block input-0-1479637362600 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:42 INFO receiver.BlockGenerator: Pushed block > input-0-1479637362600 > > 16/11/20 15:52:43 INFO storage.MemoryStore: ensureFreeSpace(6013) called > with curMem=12882027, maxMem=560497950 > > 16/11/20 15:52:43 INFO storage.MemoryStore: Block input-0-1479637362800 > stored as bytes in memory (estimated size 5.9 KB, free 522.2 MB) > > 16/11/20 15:52:43 WARN storage.BlockManager: Block input-0-1479637362800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:43 INFO receiver.BlockGenerator: Pushed block > input-0-1479637362800 > > 16/11/20 15:52:43 INFO storage.MemoryStore: ensureFreeSpace(46940) called > with curMem=12888040, maxMem=560497950 > > 16/11/20 15:52:43 INFO storage.MemoryStore: Block input-0-1479637363000 > stored as bytes in memory (estimated size 45.8 KB, free 522.2 MB) > > 16/11/20 15:52:43 WARN storage.BlockManager: Block input-0-1479637363000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:43 INFO receiver.BlockGenerator: Pushed block > input-0-1479637363000 > > 16/11/20 15:52:43 INFO storage.MemoryStore: ensureFreeSpace(23876) called > with curMem=12934980, maxMem=560497950 > > 16/11/20 15:52:43 INFO storage.MemoryStore: Block input-0-1479637363200 > stored as bytes in memory (estimated size 23.3 KB, free 522.2 MB) > > 16/11/20 15:52:43 WARN storage.BlockManager: Block input-0-1479637363200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:43 INFO receiver.BlockGenerator: Pushed block > input-0-1479637363200 > > 16/11/20 15:52:43 INFO storage.MemoryStore: ensureFreeSpace(14806) called > with curMem=12958856, maxMem=560497950 > > 16/11/20 15:52:43 INFO storage.MemoryStore: Block input-0-1479637363400 > stored as bytes in memory (estimated size 14.5 KB, free 522.2 MB) > > 16/11/20 15:52:43 WARN storage.BlockManager: Block input-0-1479637363400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:43 INFO receiver.BlockGenerator: Pushed block > input-0-1479637363400 > > 16/11/20 15:52:43 INFO storage.MemoryStore: ensureFreeSpace(4873) called > with curMem=12973662, maxMem=560497950 > > 16/11/20 15:52:43 INFO storage.MemoryStore: Block input-0-1479637363600 > stored as bytes in memory (estimated size 4.8 KB, free 522.2 MB) > > 16/11/20 15:52:43 WARN storage.BlockManager: Block input-0-1479637363600 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:43 INFO receiver.BlockGenerator: Pushed block > input-0-1479637363600 > > 16/11/20 15:52:44 INFO storage.MemoryStore: ensureFreeSpace(7554) called > with curMem=12978535, maxMem=560497950 > > 16/11/20 15:52:44 INFO storage.MemoryStore: Block input-0-1479637363800 > stored as bytes in memory (estimated size 7.4 KB, free 522.1 MB) > > 16/11/20 15:52:44 WARN storage.BlockManager: Block input-0-1479637363800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:44 INFO receiver.BlockGenerator: Pushed block > input-0-1479637363800 > > 16/11/20 15:52:44 INFO storage.MemoryStore: ensureFreeSpace(15541) called > with curMem=12986089, maxMem=560497950 > > 16/11/20 15:52:44 INFO storage.MemoryStore: Block input-0-1479637364000 > stored as bytes in memory (estimated size 15.2 KB, free 522.1 MB) > > 16/11/20 15:52:44 WARN storage.BlockManager: Block input-0-1479637364000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:44 INFO receiver.BlockGenerator: Pushed block > input-0-1479637364000 > > 16/11/20 15:52:44 INFO storage.MemoryStore: ensureFreeSpace(50543) called > with curMem=13001630, maxMem=560497950 > > 16/11/20 15:52:44 INFO storage.MemoryStore: Block input-0-1479637364200 > stored as bytes in memory (estimated size 49.4 KB, free 522.1 MB) > > 16/11/20 15:52:44 WARN storage.BlockManager: Block input-0-1479637364200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:44 INFO receiver.BlockGenerator: Pushed block > input-0-1479637364200 > > 16/11/20 15:52:44 INFO storage.MemoryStore: ensureFreeSpace(5843) called > with curMem=13052173, maxMem=560497950 > > 16/11/20 15:52:44 INFO storage.MemoryStore: Block input-0-1479637364400 > stored as bytes in memory (estimated size 5.7 KB, free 522.1 MB) > > 16/11/20 15:52:44 WARN storage.BlockManager: Block input-0-1479637364400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:44 INFO receiver.BlockGenerator: Pushed block > input-0-1479637364400 > > 16/11/20 15:52:44 INFO storage.MemoryStore: ensureFreeSpace(3474) called > with curMem=13058016, maxMem=560497950 > > 16/11/20 15:52:44 INFO storage.MemoryStore: Block input-0-1479637364600 > stored as bytes in memory (estimated size 3.4 KB, free 522.1 MB) > > 16/11/20 15:52:44 WARN storage.BlockManager: Block input-0-1479637364600 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:44 INFO receiver.BlockGenerator: Pushed block > input-0-1479637364600 > > 16/11/20 15:52:45 INFO storage.MemoryStore: ensureFreeSpace(11141) called > with curMem=13061490, maxMem=560497950 > > 16/11/20 15:52:45 INFO storage.MemoryStore: Block input-0-1479637364800 > stored as bytes in memory (estimated size 10.9 KB, free 522.1 MB) > > 16/11/20 15:52:45 WARN storage.BlockManager: Block input-0-1479637364800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:45 INFO receiver.BlockGenerator: Pushed block > input-0-1479637364800 > > 16/11/20 15:52:45 INFO storage.MemoryStore: ensureFreeSpace(21278) called > with curMem=13072631, maxMem=560497950 > > 16/11/20 15:52:45 INFO storage.MemoryStore: Block input-0-1479637365000 > stored as bytes in memory (estimated size 20.8 KB, free 522.0 MB) > > 16/11/20 15:52:45 WARN storage.BlockManager: Block input-0-1479637365000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:45 INFO receiver.BlockGenerator: Pushed block > input-0-1479637365000 > > 16/11/20 15:52:45 INFO storage.MemoryStore: ensureFreeSpace(34514) called > with curMem=13093909, maxMem=560497950 > > 16/11/20 15:52:45 INFO storage.MemoryStore: Block input-0-1479637365200 > stored as bytes in memory (estimated size 33.7 KB, free 522.0 MB) > > 16/11/20 15:52:45 WARN storage.BlockManager: Block input-0-1479637365200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:45 INFO receiver.BlockGenerator: Pushed block > input-0-1479637365200 > > 16/11/20 15:52:45 INFO storage.MemoryStore: ensureFreeSpace(5259) called > with curMem=13128423, maxMem=560497950 > > 16/11/20 15:52:45 INFO storage.MemoryStore: Block input-0-1479637365400 > stored as bytes in memory (estimated size 5.1 KB, free 522.0 MB) > > 16/11/20 15:52:45 WARN storage.BlockManager: Block input-0-1479637365400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:45 INFO receiver.BlockGenerator: Pushed block > input-0-1479637365400 > > 16/11/20 15:52:46 INFO storage.MemoryStore: ensureFreeSpace(13296) called > with curMem=13133682, maxMem=560497950 > > 16/11/20 15:52:46 INFO storage.MemoryStore: Block input-0-1479637365800 > stored as bytes in memory (estimated size 13.0 KB, free 522.0 MB) > > 16/11/20 15:52:46 WARN storage.BlockManager: Block input-0-1479637365800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:46 INFO receiver.BlockGenerator: Pushed block > input-0-1479637365800 > > 16/11/20 15:52:46 INFO storage.MemoryStore: ensureFreeSpace(29508) called > with curMem=13146978, maxMem=560497950 > > 16/11/20 15:52:46 INFO storage.MemoryStore: Block input-0-1479637366000 > stored as bytes in memory (estimated size 28.8 KB, free 522.0 MB) > > 16/11/20 15:52:46 WARN storage.BlockManager: Block input-0-1479637366000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:46 INFO receiver.BlockGenerator: Pushed block > input-0-1479637366000 > > 16/11/20 15:52:46 INFO storage.MemoryStore: ensureFreeSpace(8692) called > with curMem=13176486, maxMem=560497950 > > 16/11/20 15:52:46 INFO storage.MemoryStore: Block input-0-1479637366200 > stored as bytes in memory (estimated size 8.5 KB, free 522.0 MB) > > 16/11/20 15:52:46 WARN storage.BlockManager: Block input-0-1479637366200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:46 INFO receiver.BlockGenerator: Pushed block > input-0-1479637366200 > > 16/11/20 15:52:46 INFO storage.MemoryStore: ensureFreeSpace(4744) called > with curMem=13185178, maxMem=560497950 > > 16/11/20 15:52:46 INFO storage.MemoryStore: Block input-0-1479637366400 > stored as bytes in memory (estimated size 4.6 KB, free 522.0 MB) > > 16/11/20 15:52:46 WARN storage.BlockManager: Block input-0-1479637366400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:46 INFO receiver.BlockGenerator: Pushed block > input-0-1479637366400 > > 16/11/20 15:52:47 INFO storage.MemoryStore: ensureFreeSpace(11335) called > with curMem=13189922, maxMem=560497950 > > 16/11/20 15:52:47 INFO storage.MemoryStore: Block input-0-1479637366800 > stored as bytes in memory (estimated size 11.1 KB, free 521.9 MB) > > 16/11/20 15:52:47 WARN storage.BlockManager: Block input-0-1479637366800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:47 INFO receiver.BlockGenerator: Pushed block > input-0-1479637366800 > > 16/11/20 15:52:47 INFO storage.MemoryStore: ensureFreeSpace(34087) called > with curMem=13201257, maxMem=560497950 > > 16/11/20 15:52:47 INFO storage.MemoryStore: Block input-0-1479637367000 > stored as bytes in memory (estimated size 33.3 KB, free 521.9 MB) > > 16/11/20 15:52:47 WARN storage.BlockManager: Block input-0-1479637367000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:47 INFO receiver.BlockGenerator: Pushed block > input-0-1479637367000 > > 16/11/20 15:52:47 INFO storage.MemoryStore: ensureFreeSpace(5401) called > with curMem=13235344, maxMem=560497950 > > 16/11/20 15:52:47 INFO storage.MemoryStore: Block input-0-1479637367200 > stored as bytes in memory (estimated size 5.3 KB, free 521.9 MB) > > 16/11/20 15:52:47 WARN storage.BlockManager: Block input-0-1479637367200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:47 INFO receiver.BlockGenerator: Pushed block > input-0-1479637367200 > > 16/11/20 15:52:47 INFO storage.MemoryStore: ensureFreeSpace(3631) called > with curMem=13240745, maxMem=560497950 > > 16/11/20 15:52:47 INFO storage.MemoryStore: Block input-0-1479637367600 > stored as bytes in memory (estimated size 3.5 KB, free 521.9 MB) > > 16/11/20 15:52:47 WARN storage.BlockManager: Block input-0-1479637367600 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:47 INFO receiver.BlockGenerator: Pushed block > input-0-1479637367600 > > 16/11/20 15:52:48 INFO storage.MemoryStore: ensureFreeSpace(16114) called > with curMem=13244376, maxMem=560497950 > > 16/11/20 15:52:48 INFO storage.MemoryStore: Block input-0-1479637367800 > stored as bytes in memory (estimated size 15.7 KB, free 521.9 MB) > > 16/11/20 15:52:48 WARN storage.BlockManager: Block input-0-1479637367800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:48 INFO receiver.BlockGenerator: Pushed block > input-0-1479637367800 > > 16/11/20 15:52:48 INFO storage.MemoryStore: ensureFreeSpace(31073) called > with curMem=13260490, maxMem=560497950 > > 16/11/20 15:52:48 INFO storage.MemoryStore: Block input-0-1479637368000 > stored as bytes in memory (estimated size 30.3 KB, free 521.9 MB) > > 16/11/20 15:52:48 WARN storage.BlockManager: Block input-0-1479637368000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:48 INFO receiver.BlockGenerator: Pushed block > input-0-1479637368000 > > 16/11/20 15:52:48 INFO storage.MemoryStore: ensureFreeSpace(5415) called > with curMem=13291563, maxMem=560497950 > > 16/11/20 15:52:48 INFO storage.MemoryStore: Block input-0-1479637368200 > stored as bytes in memory (estimated size 5.3 KB, free 521.9 MB) > > 16/11/20 15:52:48 WARN storage.BlockManager: Block input-0-1479637368200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:48 INFO receiver.BlockGenerator: Pushed block > input-0-1479637368200 > > 16/11/20 15:52:48 INFO storage.MemoryStore: ensureFreeSpace(3651) called > with curMem=13296978, maxMem=560497950 > > 16/11/20 15:52:48 INFO storage.MemoryStore: Block input-0-1479637368400 > stored as bytes in memory (estimated size 3.6 KB, free 521.8 MB) > > 16/11/20 15:52:48 WARN storage.BlockManager: Block input-0-1479637368400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:48 INFO receiver.BlockGenerator: Pushed block > input-0-1479637368400 > > 16/11/20 15:52:49 INFO storage.MemoryStore: ensureFreeSpace(16052) called > with curMem=13300629, maxMem=560497950 > > 16/11/20 15:52:49 INFO storage.MemoryStore: Block input-0-1479637368800 > stored as bytes in memory (estimated size 15.7 KB, free 521.8 MB) > > 16/11/20 15:52:49 WARN storage.BlockManager: Block input-0-1479637368800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:49 INFO receiver.BlockGenerator: Pushed block > input-0-1479637368800 > > 16/11/20 15:52:49 INFO storage.MemoryStore: ensureFreeSpace(18325) called > with curMem=13316681, maxMem=560497950 > > 16/11/20 15:52:49 INFO storage.MemoryStore: Block input-0-1479637369000 > stored as bytes in memory (estimated size 17.9 KB, free 521.8 MB) > > 16/11/20 15:52:49 WARN storage.BlockManager: Block input-0-1479637369000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:49 INFO receiver.BlockGenerator: Pushed block > input-0-1479637369000 > > 16/11/20 15:52:49 INFO storage.MemoryStore: ensureFreeSpace(3968) called > with curMem=13335006, maxMem=560497950 > > 16/11/20 15:52:49 INFO storage.MemoryStore: Block input-0-1479637369400 > stored as bytes in memory (estimated size 3.9 KB, free 521.8 MB) > > 16/11/20 15:52:49 WARN storage.BlockManager: Block input-0-1479637369400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:49 INFO receiver.BlockGenerator: Pushed block > input-0-1479637369400 > > 16/11/20 15:52:49 INFO storage.MemoryStore: ensureFreeSpace(3551) called > with curMem=13338974, maxMem=560497950 > > 16/11/20 15:52:49 INFO storage.MemoryStore: Block input-0-1479637369600 > stored as bytes in memory (estimated size 3.5 KB, free 521.8 MB) > > 16/11/20 15:52:49 WARN storage.BlockManager: Block input-0-1479637369600 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:49 INFO receiver.BlockGenerator: Pushed block > input-0-1479637369600 > > 16/11/20 15:52:50 INFO storage.MemoryStore: ensureFreeSpace(14364) called > with curMem=13342525, maxMem=560497950 > > 16/11/20 15:52:50 INFO storage.MemoryStore: Block input-0-1479637369800 > stored as bytes in memory (estimated size 14.0 KB, free 521.8 MB) > > 16/11/20 15:52:50 WARN storage.BlockManager: Block input-0-1479637369800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:50 INFO receiver.BlockGenerator: Pushed block > input-0-1479637369800 > > 16/11/20 15:52:50 INFO storage.MemoryStore: ensureFreeSpace(5972) called > with curMem=13356889, maxMem=560497950 > > 16/11/20 15:52:50 INFO storage.MemoryStore: Block input-0-1479637370000 > stored as bytes in memory (estimated size 5.8 KB, free 521.8 MB) > > 16/11/20 15:52:50 WARN storage.BlockManager: Block input-0-1479637370000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:50 INFO receiver.BlockGenerator: Pushed block > input-0-1479637370000 > > 16/11/20 15:52:50 INFO storage.MemoryStore: ensureFreeSpace(39382) called > with curMem=13362861, maxMem=560497950 > > 16/11/20 15:52:50 INFO storage.MemoryStore: Block input-0-1479637370200 > stored as bytes in memory (estimated size 38.5 KB, free 521.8 MB) > > 16/11/20 15:52:50 WARN storage.BlockManager: Block input-0-1479637370200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:50 INFO receiver.BlockGenerator: Pushed block > input-0-1479637370200 > > 16/11/20 15:52:51 INFO storage.MemoryStore: ensureFreeSpace(16798) called > with curMem=13402243, maxMem=560497950 > > 16/11/20 15:52:51 INFO storage.MemoryStore: Block input-0-1479637370800 > stored as bytes in memory (estimated size 16.4 KB, free 521.7 MB) > > 16/11/20 15:52:51 WARN storage.BlockManager: Block input-0-1479637370800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:51 INFO receiver.BlockGenerator: Pushed block > input-0-1479637370800 > > 16/11/20 15:52:51 INFO storage.MemoryStore: ensureFreeSpace(20782) called > with curMem=13419041, maxMem=560497950 > > 16/11/20 15:52:51 INFO storage.MemoryStore: Block input-0-1479637371000 > stored as bytes in memory (estimated size 20.3 KB, free 521.7 MB) > > 16/11/20 15:52:51 WARN storage.BlockManager: Block input-0-1479637371000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:51 INFO receiver.BlockGenerator: Pushed block > input-0-1479637371000 > > 16/11/20 15:52:51 INFO storage.MemoryStore: ensureFreeSpace(11410) called > with curMem=13439823, maxMem=560497950 > > 16/11/20 15:52:51 INFO storage.MemoryStore: Block input-0-1479637371200 > stored as bytes in memory (estimated size 11.1 KB, free 521.7 MB) > > 16/11/20 15:52:51 WARN storage.BlockManager: Block input-0-1479637371200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:51 INFO receiver.BlockGenerator: Pushed block > input-0-1479637371200 > > 16/11/20 15:52:51 INFO storage.MemoryStore: ensureFreeSpace(3651) called > with curMem=13451233, maxMem=560497950 > > 16/11/20 15:52:51 INFO storage.MemoryStore: Block input-0-1479637371400 > stored as bytes in memory (estimated size 3.6 KB, free 521.7 MB) > > 16/11/20 15:52:51 WARN storage.BlockManager: Block input-0-1479637371400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:51 INFO receiver.BlockGenerator: Pushed block > input-0-1479637371400 > > 16/11/20 15:52:52 INFO storage.MemoryStore: ensureFreeSpace(17534) called > with curMem=13454884, maxMem=560497950 > > 16/11/20 15:52:52 INFO storage.MemoryStore: Block input-0-1479637371800 > stored as bytes in memory (estimated size 17.1 KB, free 521.7 MB) > > 16/11/20 15:52:52 WARN storage.BlockManager: Block input-0-1479637371800 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:52 INFO receiver.BlockGenerator: Pushed block > input-0-1479637371800 > > 16/11/20 15:52:52 INFO storage.MemoryStore: ensureFreeSpace(26989) called > with curMem=13472418, maxMem=560497950 > > 16/11/20 15:52:52 INFO storage.MemoryStore: Block input-0-1479637372000 > stored as bytes in memory (estimated size 26.4 KB, free 521.7 MB) > > 16/11/20 15:52:52 WARN storage.BlockManager: Block input-0-1479637372000 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:52 INFO receiver.BlockGenerator: Pushed block > input-0-1479637372000 > > 16/11/20 15:52:52 INFO storage.MemoryStore: ensureFreeSpace(3621) called > with curMem=13499407, maxMem=560497950 > > 16/11/20 15:52:52 INFO storage.MemoryStore: Block input-0-1479637372200 > stored as bytes in memory (estimated size 3.5 KB, free 521.7 MB) > > 16/11/20 15:52:52 WARN storage.BlockManager: Block input-0-1479637372200 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:52 INFO receiver.BlockGenerator: Pushed block > input-0-1479637372200 > > 16/11/20 15:52:52 INFO storage.MemoryStore: ensureFreeSpace(3479) called > with curMem=13503028, maxMem=560497950 > > 16/11/20 15:52:52 INFO storage.MemoryStore: Block input-0-1479637372400 > stored as bytes in memory (estimated size 3.4 KB, free 521.7 MB) > > 16/11/20 15:52:52 WARN storage.BlockManager: Block input-0-1479637372400 > replicated to only 0 peer(s) instead of 1 peers > > 16/11/20 15:52:52 INFO receiver.BlockGenerator: Pushed block > input-0-1479637372400 > > 16/11/20 15:52:53 INFO storage.MemoryStore: ensureFreeSpace(7137) called > with curMem=13506507, maxMem=560497950 > > 16/11/20 15:52:53 INFO storage.MemoryStore: Block input-0-1479637372800 > stored as bytes in memory (estimated > ...
