Hi Everyone, I'm coding a TCP Server that streams data to Spark. I'm hoping Akka TCP Streaming is good solution for what I want to do as it looks pretty compelling.
My Server will simply send data upon connection and disconnect once all data is all sent. To give some more detail: - Internally the data is coming from a chunked HTTP file upload. I want to forward each chunk to the TCP connection to Spark with as little overhead as possible as the data files are upwards of 65gb in size. I already got the chunking working with Akka HTTP and tested it with the 65gb file and it works flawlessly. - Each record in the file is delimited by a newline character. Records are not the same size. This sounds like a natural fit as Akka TCP Streams .via(Framing.Delimiter) looks like it might help here. I'm right now stuck as all the examples I've found seem to be things like echo servers ping-ponging the data around. What I really need is a unidirectional connection where the server does all the transmitting and the client just gets the data. Is it possible for me to build a solution using Akka TCP Streaming where the HTTP file upload Actor publishes a message to the Event Bus which is in turn subscribed to by the TCP server which in turn results in data being pushed to the TCP client? How do I make the TCP client just get the data and not send anything to the server? (I would like this for a test harness outside of Spark.) .dropWhile(b => true)? I'd gladly publish my code to a GitHub Gist and my blog once I get this solution working to further help out the community. There isn't a lot of stuff on Akka TCP Streaming out there yet. Any help would be greatly appreciated. Akka TCP Streaming looks very powerful and I'm still trying to wrap my head around it. Like I presume, many others. Thanks, Phil -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
