Not exactly sure what you were looking for. I'm relatively new to akka streams. This is the minimal amount of code that reproduces the problem, involving:
a custom source an flow with AsyncCallback and parallelizing only deps are akka and joda time On Friday, December 16, 2016 at 3:31:37 PM UTC-7, √ wrote: > > Is this really a *minimized* reproducer? > > On Fri, Dec 16, 2016 at 11:09 PM, <[email protected] <javascript:>> wrote: > >> Here's the code to reproduce. The issue only seems to occur with my >> custom Source and using a callback in a Flow and while parallelizing. If >> this code is run with .via(parallelizeFlow(parallelize = 1, asyncFlow)) it >> drops the last minute, but when run with .via(asyncFlow) it does not. >> >> So with parallelize it prints all but the last minute: >> >> seq is Vector(2016/12/14/23/50, 2016/12/14/23/51, 2016/12/14/23/52, >> 2016/12/14/23/53, 2016/12/14/23/54, 2016/12/14/23/55, 2016/12/14/23/56, >> 2016/12/14/23/57, 2016/12/14/23/58) >> >> and without it prints the expected value, with all minutes flowing >> through the graph: >> >> seq is Vector(2016/12/14/23/50, 2016/12/14/23/51, 2016/12/14/23/52, >> 2016/12/14/23/53, 2016/12/14/23/54, 2016/12/14/23/55, 2016/12/14/23/56, >> 2016/12/14/23/57, 2016/12/14/23/58, 2016/12/14/23/59) >> >> >> >> >> >> package hack.streams >> >> import akka.NotUsed >> import akka.stream._ >> import akka.stream.scaladsl.{Balance, Flow, GraphDSL, Merge, Sink, Source} >> import akka.stream.stage._ >> import org.joda.time.{DateTime, Duration, Interval} >> >> import scala.collection.immutable.Seq >> import scala.concurrent.{Await, Future} >> >> object AsyncIssue { >> import StreamsMaterializer._ >> >> def minuteSource(interval: Interval) = new >> GraphStage[SourceShape[DateTime]]() { >> val out = Outlet[DateTime]("keys") >> val shape = SourceShape(out) >> >> def zeroToMinute(date: DateTime) = >> date.withMillisOfSecond(0).withSecondOfMinute(0) >> >> override def createLogic(inheritedAttributes: Attributes): >> GraphStageLogic = new GraphStageLogic(shape) with StageLogging { >> var isDone: Boolean = false >> var current: DateTime = new DateTime(0) >> >> override def preStart() = { >> current = zeroToMinute(interval.getStart) >> } >> >> setHandler(out, >> new OutHandler { >> override def onPull(): Unit = { >> if (!isDone) { >> push(out, current) >> current = current.plusMinutes(1) >> >> if (current.isEqual(zeroToMinute(interval.getEnd))) { >> isDone = true >> } >> } else { >> complete(out) >> } >> } >> }) >> } >> } >> >> def futureCallbackFlow = new GraphStage[FlowShape[DateTime, String]]() { >> val in = Inlet[DateTime]("minute") >> val out = Outlet[String]("string") >> >> val formatter = >> org.joda.time.format.DateTimeFormat.forPattern("yyyy/MM/dd/HH/mm") >> >> val shape = FlowShape.of(in, out) >> >> override def createLogic(inheritedAttributes: Attributes): >> GraphStageLogic = new GraphStageLogic(shape) with StageLogging { >> val pushCallback = getAsyncCallback[String] { seq => >> push(out, seq) >> } >> >> setHandler(in, new InHandler { >> override def onPush(): Unit = { >> val minute = grab(in) >> >> val fMin: Future[DateTime] = Future {minute} >> >> fMin.foreach { min => >> pushCallback.invoke(formatter.print(min)) >> } >> } >> }) >> >> setHandler(out, >> new OutHandler { >> override def onPull(): Unit = { >> pull(in) >> } >> } >> ) >> } >> } >> >> def parallelizeFlow[In, Out](parallelize: Int, flow: >> Flow[In,Out,NotUsed]): Flow[In, Out, NotUsed] = >> Flow.fromGraph(GraphDSL.create() { implicit builder => >> import GraphDSL.Implicits._ >> >> val dispatcher = builder.add(Balance[In](parallelize)) >> val merger = builder.add(Merge[Out](parallelize)) >> >> for (i <- 0 to parallelize - 1) { >> dispatcher.out(i) ~> flow.async ~> merger.in(i) >> } >> >> FlowShape(dispatcher.in, merger.out) >> }) >> >> def run() = { >> import StreamsMaterializer._ >> >> val asyncFlow = Flow[DateTime].via(futureCallbackFlow) >> >> val source: Source[DateTime, NotUsed] = Source(11 to >> 20).via(Flow[Int].map { num => >> val formatter = >> org.joda.time.format.DateTimeFormat.forPattern("yyyy/MM/dd/HH/mm") >> formatter.parseDateTime(s"2016/12/14/14/$num") >> }) >> >> val mat: Future[Seq[String]] = Source.fromGraph(minuteSource(new >> Interval(Duration.standardMinutes(10), new >> DateTime().dayOfMonth().roundFloorCopy().minusDays(1)))) >> .via(parallelizeFlow(parallelize = 1, asyncFlow)) >> // .via(asyncFlow) >> .runWith(Sink.seq) >> >> val seq: Seq[String] = Await.result(mat, >> scala.concurrent.duration.Duration.Inf) >> println(s"seq is $seq") >> } >> } >> >> >> >> >> On Friday, December 16, 2016 at 8:03:45 AM UTC-7, √ wrote: >>> >>> Plase submit a miminized reproducer so readers have a chance of running >>> the code. >>> >>> On Fri, Dec 16, 2016 at 3:44 PM, <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I'm seeing an issue where the graph completes while there is still data >>>> in one of the flows. The last element emitted by the source enters a >>>> custom >>>> GraphStageLogic flow, where it is sent to a function that returns a >>>> Future. >>>> That Future has a callback which invokes getAsyncCallback and then >>>> push(out). For the last element, the Future callback fires ( >>>> pushCallback.invoke(xml)) but the AsyncCallback is never invoked and >>>> the graph stops. >>>> >>>> For more context, this is what I have going on inside the >>>> GraphStageLogic: >>>> >>>> val s3ListBucket: Source[ByteString, NotUsed] = >>>> s3Client.listBucket(bucket, Some(currentPrefix), maxKeys, nextMarker) >>>> >>>> val bucketListingXml: Future[String] = s3ListBucket >>>> .map(_.utf8String) >>>> .runWith(Sink.seq)(materializer) >>>> .map(_.mkString)(materializer.executionContext) >>>> >>>> >>>> bucketListingXml.foreach { >>>> xml => >>>> >>>> println(s"This gets called. prefix $currentPrefix") >>>> >>>> pushCallback.invoke(xml) >>>> }(materializer.executionContext) >>>> >>>> >>>> And the callback >>>> >>>> >>>> val pushCallback = getAsyncCallback[String] { xml => >>>> log.info(s"This is never called for last element in graph!") >>>> push(out, xml) >>>> } >>>> >>>> >>>> I don't see any errors and this issue consistently occurs on the last >>>> element. Thanks >>>> >>>> Andrew >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Cheers, >>> √ >>> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Cheers, > √ > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
