Hi
I am developing a simple batch application which is using akka stream 1.0.
All is running smootly if i avoid using mapAsync method on flow step.
When on complete is called, i finalize my result file and shutdown the
agent system through an actor using Reaper pattern:
val file = new File(inputFile)
val run: Future[Int] = source(file)
.via(parse)
.via(enrich)
.via(writeEnriched)
.runWith(printProgress)
run.onComplete { result: Try[Int] =>
context.system.log.info(s"Nb elements processed: ${result.get}")
writerActorRef ! FinalizeResults()
}
One of the step i want to accelerate is the part which enrich some data.
Sometimes, data cant be enriched and should be ignore for next steps
def enrich(implicit ec: ExecutionContext) : Flow[Data, EnrichedData, Unit]
= Flow[Data].map(enriched.enrich(_)).collect {
case Some(enrichedData) => enrichedData
}
All this code is running well and i have no lost elements when onComplete()
is called.
Input: 45639
Nb elements processed: 45639
When i try to accelerate thing using mapAync and Future instead of map for
enrichment step, the onComplete is called before all elements have been
processed.
def enrich(implicit ec: ExecutionContext) : Flow[Data, EnrichedData, Unit]
= Flow[Data].mapAsyncUnordered(8)(data =>
Future(enricher.enrich(data))).collect {
case Some(enrichedData) => enrichedData
}
I missed some elements at the end and never the same number
All this code is running well and i have no lost elements when onComplete()
is called.
Input: 45639
Nb elements processed: 45628
I cant find a way to find that everything have been processed...
Any idea what i am doing wrong?
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
