Thanks :) I found the Flume, Kafka, Twitter, ZeroMQ examples confusing because they need a third-party product which I have no clue about.
Please let me know if you have some experience in all of these stuff. And I am currently interested in collecting all sorts of streaming apps for my research, please feel free to discuss with me about those things :) On Sun, Feb 2, 2014 at 7:59 PM, OlegYch <[email protected]> wrote: > Thanks, i'll look into that. > as for kafka, i've just used the simplest configuration, you can create it > using their quickstart and code like this for consumer > val kafkaParams = Map[String, String]( > "zookeeper.connect" -> "localhost:2181", "group.id" -> > "test-consumer-group1", > "zookeeper.connection.timeout.ms" -> "10000", "auto.offset.reset" -> > "smallest") > > val textStream = KafkaUtils.createStream[String, String, StringDecoder, > StringDecoder](ssc, kafkaParams, Map("test" -> 1), > StorageLevel.MEMORY_AND_DISK_SER_2).map(_._2) > > and like this for producer > > val Array(brokers, topic, messagesPerSec, wordsPerMessage) = > Array("localhost:9092", "test", "10", "10") > > // Zookeper connection properties > val props = new Properties() > props.put("metadata.broker.list", brokers) > props.put("serializer.class", "kafka.serializer.StringEncoder") > > val config = new ProducerConfig(props) > val producer = new Producer[String, String](config) > > // Send some messages > while(true) { > producer.send(new KeyedMessage(topic, getNextClickEvent())) > Thread.sleep(10) > } > > (i took that from > org.apache.spark.streaming.examples.clickstream.PageViewGenerator > org.apache.spark.streaming.examples.clickstream.PageViewStream and > org.apache.spark.streaming.examples.KafkaWordCount) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/PageView-streaming-sample-lost-page-views-tp1126p1149.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Dachuan Huang Cellphone: 614-390-7234 2015 Neil Avenue Ohio State University Columbus, Ohio U.S.A. 43210
