[
https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659014#comment-14659014
]
Anuj Ojha commented on SPARK-1022:
----------------------------------
[~tdas] and [~jerryshao2015] I was working on writing Integration test for my
application, which reads data from kafka, performs transformation on the data
and then writes the Dstream back to other kafka topic. To do this I had to get
KafkaTestUtils and some other dependent classes locally as I needed embedded
zookeeper and kafa queue. I also had to modify sendMessage to accept my
specific type of ket and value. I was wondering if it is possible to make the
KafkaTestUtils generic? so that others can use it for in-memory
unit/integration testing. I have created generic sendMessage method hopefully
it's of some use.
{code}
class KafkaTestUtils [T,U]{
.......
// Kafka producer
private var producer: Producer[T, U] = _
..........
def sendMessages(topic: String, messageToFreq: JMap[T, U]): Unit = {
import scala.collection.JavaConversions._
producer = new Producer[T, U](new ProducerConfig(producerConfiguration))
for ((k,v) <- messageToFreq) {
var message = new KeyedMessage(topic, k,v)
producer.send(message)
}
producer.close()
producer = null
}
.........
}
{code}
I also had to change producer properties to provide key and value serialization
class. It might be worth making possible to set these values from user test. I
did not create JIRA for the request as I wanted to see if this is even a
acceptable request. Please let me know.
> Add unit tests for kafka streaming
> ----------------------------------
>
> Key: SPARK-1022
> URL: https://issues.apache.org/jira/browse/SPARK-1022
> Project: Spark
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Saisai Shao
>
> It would be nice if we could add unit tests to verify elements of kafka's
> stream. Right now we do integration tests only which makes it hard to upgrade
> versions of kafka. The place to start here would be to look at how kafka
> tests itself and see if the functionality can be exposed to third party users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
