That is a one big map. :) Maybe it would be more efficient to use TCP instead of HTTP for such large data structures. Whichever transport you choose you still need to serialize your map to send it through a wire. You can serialize to json (spray-json or play-json support macro based serializer/deserializer generation) if you choose HTTP or you could try scala-pickling, which also supports JSON as well as binary format or some other serialization framework.
On Thu, Mar 19, 2015 at 11:25 AM, Ömer Faruk Gül <[email protected]> wrote: > Thanks, I actually make use of spray-can for external data exchange. So I > will use it internally too. I have one more question on that but it may be > out of scope. I receive POST calls that contains the serialised object > which I get as ByteString (extracted from spray.http.HttpEntity) and I > simply store it with a key (which I read it from the header). I basically > get the ByteString as follows: > > val bytes: ByteString = request.entity.data.toByteString > > > Now, lets say a node contains an immutable map with million unique keys > mapped to ByteStrings. When transferring data to another node, sending a > million POST request to the other node could be a problem in terms of > performance. Could there be a better way to manage this? > > On Thursday, March 19, 2015 at 7:48:57 PM UTC+2, Martynas Mickevičius > wrote: >> >> Hi Ömer, >> >> it is not recommended to send large messages via the remoting channel. >> Mainly because the same channel is used for the heartbeat and other system >> messages. If you are able to use milestone versions of akka-stream and >> akka-http I would suggest creating an HTTP endpoint where big data >> structures can be streamed as JSON. Then one would use remoting to exchange >> the address of the reasourse and akka-http to consume it. >> >> On Thu, Mar 19, 2015 at 8:41 AM, Ömer Faruk Gül <[email protected]> >> wrote: >> >>> Hi, >>> >>> What would be the best practice to transfer data (an immutable Map which >>> could be of size from 1MB to 100 MB) in between two nodes in a an Akka >>> Cluster? >>> >>> I initially thought of dividing the map into chunks and sending them as >>> a message to the new actor living in the other node, is that a good >>> practice, or should I try something else? >>> >>> My immutable variable is this: >>> >>> var store = Map.empty[String, ByteString] >>> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Martynas Mickevičius >> Typesafe <http://typesafe.com/> – Reactive >> <http://www.reactivemanifesto.org/> Apps on the JVM >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Martynas Mickevičius Typesafe <http://typesafe.com/> – Reactive <http://www.reactivemanifesto.org/> Apps on the JVM -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
