Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5918#discussion_r29820932
--- Diff:
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
---
@@ -167,26 +167,24 @@ class FlumePollingStreamSuite extends FunSuite with
BeforeAndAfter with Logging
}
}
- def writeAndVerify(channels: Seq[MemoryChannel], ssc: StreamingContext,
+ def writeAndVerify(sinks: Seq[SparkSink], channels: Seq[MemoryChannel],
ssc: StreamingContext,
outputBuffer: ArrayBuffer[Seq[SparkFlumeEvent]]) {
val clock = ssc.scheduler.clock.asInstanceOf[ManualClock]
val executor = Executors.newCachedThreadPool()
val executorCompletion = new ExecutorCompletionService[Void](executor)
- channels.map(channel => {
+
+ val latch = new CountDownLatch(batchCount * channels.size)
+ sinks.foreach(_.countdownWhenBatchReceived(latch))
+
+ channels.foreach(channel => {
executorCompletion.submit(new TxnSubmitter(channel, clock))
})
+
for (i <- 0 until channels.size) {
executorCompletion.take()
}
- val startTime = System.currentTimeMillis()
- while (outputBuffer.size < batchCount * channels.size &&
- System.currentTimeMillis() - startTime < 15000) {
- logInfo("output.size = " + outputBuffer.size)
- Thread.sleep(100)
- }
- val timeTaken = System.currentTimeMillis() - startTime
- assert(timeTaken < 15000, "Operation timed out after " + timeTaken + "
ms")
- logInfo("Stopping context")
+
+ latch.await(15, TimeUnit.SECONDS)
--- End diff --
We can add the old code checking for the buffer back, so we wait till the
data was sent and then wait till it has been processed. That should be
enough, correct?
On Wednesday, May 6, 2015, Tathagata Das <[email protected]> wrote:
> In
>
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
> <https://github.com/apache/spark/pull/5918#discussion_r29816484>:
>
> > for (i <- 0 until channels.size) {
> > executorCompletion.take()
> > }
> > - val startTime = System.currentTimeMillis()
> > - while (outputBuffer.size < batchCount * channels.size &&
> > - System.currentTimeMillis() - startTime < 15000) {
> > - logInfo("output.size = " + outputBuffer.size)
> > - Thread.sleep(100)
> > - }
> > - val timeTaken = System.currentTimeMillis() - startTime
> > - assert(timeTaken < 15000, "Operation timed out after " + timeTaken
+ " ms")
> > - logInfo("Stopping context")
> > +
> > + latch.await(15, TimeUnit.SECONDS)
>
> Does this only guarantee that the data has been successfully received by
> the PollingReceiver? The data may not have been processed and added to the
> buffer yet. Stopping the context before the guarantee that it has been
> processed is wrong, isnt it?
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5918/files#r29816484>.
>
--
Thanks,
Hari
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
