Hi Phil,
On Thu, Sep 10, 2015 at 6:00 AM, Phil Podlevsky <[email protected]> wrote: > Hi Everyone, > > I'm coding a TCP Server that streams data to Spark. I'm hoping Akka TCP > Streaming is good solution for what I want to do as it looks pretty > compelling. > > My Server will simply send data upon connection and disconnect once all > data is all sent. To give some more detail: > > - Internally the data is coming from a chunked HTTP file upload. I > want to forward each chunk to the TCP connection to Spark with as little > overhead as possible as the data files are upwards of 65gb in size. I > already got the chunking working with Akka HTTP and tested it with the 65gb > file and it works flawlessly. > > > - Each record in the file is delimited by a newline character. Records > are not the same size. This sounds like a natural fit as Akka TCP Streams > .via(Framing.Delimiter) looks like it might help here. > > Yes, I think Delimiter is the way to go, the only problem that you might encounter is that it only works with a static delimiter (i.e. it cannot work with mixed line endings). Your pipeline would so far roughly look like this (pseudocode, I might not remember all the APIs exactly) val linesAsBytes = chunkSource.map(chunksToByteStrings).via(Delimiter.framing("\n", maxLineSize = 1024 * 1024)) > I'm right now stuck as all the examples I've found seem to be things like > echo servers ping-ponging the data around. What I really need is a > unidirectional connection where the server does all the transmitting and > the client just gets the data. Is it possible for me to build a solution > using Akka TCP Streaming where the HTTP file upload Actor publishes a > message to the Event Bus which is in turn subscribed to by the TCP server > which in turn results in data being pushed to the TCP client? > I am a bit lost here what goes through what and how :) Maybe a simple ASCII diagram? > How do I make the TCP client just get the data and not send anything to > the server? (I would like this for a test harness outside of Spark.) > .dropWhile(b => true)? > val stopPromise = Source.lazyEmpty.via(Tcp.outoingConnection(...)).to(SinkOfYourChoice).run() I.e. you take a source that is empty, but will not signal completion until the Promise it returns is completed (in other words it is a source that does not emit anything, including completion until you say so, when it emits a complete). Then you can stop or abort the connection by one of: stopPromise.trySuccess() stopPromise.tryFailure(ex) > > I'd gladly publish my code to a GitHub Gist and my blog once I get this > solution working to further help out the community. There isn't a lot of > stuff on Akka TCP Streaming out there yet. > > Any help would be greatly appreciated. Akka TCP Streaming looks very > powerful and I'm still trying to wrap my head around it. Like I presume, > many others. > I have a couple of examples here, but they are not really documented: https://gist.github.com/drewhk/25bf7472db04b5699b80 This is the thread where I described what they do: https://groups.google.com/forum/#!searchin/akka-user/25bf7472db04b5699b80/akka-user/tng5CiUtfig/b3HSM0r9DAAJ -Endre > > Thanks, > > Phil > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - Reactive apps on the JVM Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
