Hi,
I am new to Akka streams, I am currently trying to implement a Source that
takes continuous input from RocksDB. The idea is to use RocksDB as a queue
from which the source will constantly pull data, perform some stream
operations and finally store it into a database store like Cassandra or
MongoDB.
My code looks like this
Source.repeat(NotUsed.getInstance()).
.map(readRocksDB())
.runWith(storeMessages(), ActorMaterializer.create(system));
private Source<byte[], NotUsed> getMessageSource(RocksDB db) {
Source<byte[], NotUsed> s = Source.from(getValues(db));
return s;
}
private Flow<File, List<Document>, NotUsed> readRocksDB() {
return Flow.of(File.class).flatMapConcat(f -> {
RocksDB db = RocksDB.open(f.getAbsolutePath());
return this.getMessageSource(db)
.mapAsync(numOfThreads, this::parseMessageBytes);
});
}
storeMessage() is responsible for storing the processed data into a
different database (Cassandra for example).
I am expecting a workload of 1 mil writes per second, so I am using RocksDB
to store raw before processing instead of Akka mailbox. Since I am new, I
would like to get a feed back on this approach.
Thanks,
Jerry
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
