Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5918#discussion_r29822154
--- Diff:
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
---
@@ -167,26 +167,24 @@ class FlumePollingStreamSuite extends FunSuite with
BeforeAndAfter with Logging
}
}
- def writeAndVerify(channels: Seq[MemoryChannel], ssc: StreamingContext,
+ def writeAndVerify(sinks: Seq[SparkSink], channels: Seq[MemoryChannel],
ssc: StreamingContext,
outputBuffer: ArrayBuffer[Seq[SparkFlumeEvent]]) {
val clock = ssc.scheduler.clock.asInstanceOf[ManualClock]
val executor = Executors.newCachedThreadPool()
val executorCompletion = new ExecutorCompletionService[Void](executor)
- channels.map(channel => {
+
+ val latch = new CountDownLatch(batchCount * channels.size)
+ sinks.foreach(_.countdownWhenBatchReceived(latch))
+
+ channels.foreach(channel => {
executorCompletion.submit(new TxnSubmitter(channel, clock))
})
+
for (i <- 0 until channels.size) {
executorCompletion.take()
}
- val startTime = System.currentTimeMillis()
- while (outputBuffer.size < batchCount * channels.size &&
- System.currentTimeMillis() - startTime < 15000) {
- logInfo("output.size = " + outputBuffer.size)
- Thread.sleep(100)
- }
- val timeTaken = System.currentTimeMillis() - startTime
- assert(timeTaken < 15000, "Operation timed out after " + timeTaken + "
ms")
- logInfo("Stopping context")
+
+ latch.await(15, TimeUnit.SECONDS)
--- End diff --
Then why even wait for it to be sent?
On May 6, 2015 7:50 PM, "Hari Shreedharan" <[email protected]> wrote:
> In
>
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
> <https://github.com/apache/spark/pull/5918#discussion_r29820932>:
>
> > for (i <- 0 until channels.size) {
> > executorCompletion.take()
> > }
> > - val startTime = System.currentTimeMillis()
> > - while (outputBuffer.size < batchCount * channels.size &&
> > - System.currentTimeMillis() - startTime < 15000) {
> > - logInfo("output.size = " + outputBuffer.size)
> > - Thread.sleep(100)
> > - }
> > - val timeTaken = System.currentTimeMillis() - startTime
> > - assert(timeTaken < 15000, "Operation timed out after " + timeTaken
+ " ms")
> > - logInfo("Stopping context")
> > +
> > + latch.await(15, TimeUnit.SECONDS)
>
> We can add the old code checking for the buffer back, so we wait till the
> data was sent and then wait till it has been processed. That should be
> enough, correct?
> ⦠<#14d2c485c280d1d4_>
> On Wednesday, May 6, 2015, Tathagata Das <[email protected]>
> wrote: In
>
external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala
> <https://github.com/apache/spark/pull/5918#discussion_r29816484>: > for
> (i <- 0 until channels.size) { > executorCompletion.take() > } > - val
> startTime = System.currentTimeMillis() > - while (outputBuffer.size <
> batchCount * channels.size && > - System.currentTimeMillis() - startTime <
> 15000) { > - logInfo("output.size = " + outputBuffer.size) > -
> Thread.sleep(100) > - } > - val timeTaken = System.currentTimeMillis() -
> startTime > - assert(timeTaken < 15000, "Operation timed out after " +
> timeTaken + " ms") > - logInfo("Stopping context") > + > + latch.await(15,
> TimeUnit.SECONDS) Does this only guarantee that the data has been
> successfully received by the PollingReceiver? The data may not have been
> processed and added to the buffer yet. Stopping the context before the
> guarantee that it has been processed is wrong, isnt it? â Reply to this
> email directly or view it on GitHub <
> https://github.com/apache/spark/pull/5918/files#r29816484>.
> -- Thanks, Hari
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5918/files#r29820932>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
