Re: [akka-user] Streaming proxy/tunel on top of akka-http

[Jozsef Zsido]

You are completrly right. I have did something differenly somewhere when I 
tested last time with browser. My code also works fine without the test 
code.
Thank you very mutch!

On Monday, November 13, 2017 at 9:28:22 PM UTC+2, Jozsef Zsido wrote:
>
> yes, I try all the time with real browser. The code blows up the memory 
> when I download a 2GB file and try to stream it from one response to another
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Streaming proxy/tunel on top of akka-http

[Jozsef Zsido]

yes, I try all the time with real browser. The code blows up the memory 
when I download a 2GB file and try to stream it from one response to another

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Streaming proxy/tunel on top of akka-http

[johannes . rudolph]

I tried your code and it doesn't OOM for me. Have you tried it outside of a 
test suite? It might be that the test infrastructure is collecting all the 
data when you use something as `reponse.entity`.  If that doesn't 
help,  try capturing a heap dump on OOM and see where the memory is spent.

Johannes

On Wednesday, November 8, 2017 at 11:01:18 AM UTC+1, Jozsef Zsido wrote:
>
> Hi,
>
> Not much code I have, basically I took a sample for akka-http proxy: 
> https://gist.github.com/kovacshuni/afb7d53f40f501d0ab82
>
> trait BrowseServerRoutes3 {
>
>   implicit val system = ActorSystem("Browse")
>   implicit val materializer = ActorMaterializer()
>   implicit val ec = system.dispatcher
>
>   val proxy = Route { context =>
>
> val request = context.request
> println("Processing " + request)
>
> val flow = Http(system).outgoingConnection("10.66.0.4", 80)
>
> Source.single(context.request)
>   .map {
> _.withHeaders(HttpUtils.completeHeaderList(request))
>   .withUri(request.uri.path.toString())
>   }
>   .via(flow)
>   .runWith(Sink.head)
>   .flatMap(f => {
> context.complete(f)
>   })
>   }
> }
>
> object BrowseServer extends App with BrowseServerRoutes3 {
>
>   val binding = Http(system).bindAndHandle(handler = proxy, interface = 
> "0.0.0.0", port = 8080)
>   println(s"Server online.")
> }
>
> "Download" should "return OK" in {
>   Get("/test/huge_MP3WRAP.mp3") ~> proxy ~> check {
> response.status shouldEqual StatusCodes.OK
>   }
> }
>
>
> The first problem I have with this code is that I have a 2GB file at the 
> destination and I get the following exception:
> EntityStreamSizeException: actual entity size (Some(2272610895)) exceeded 
> content length limit (8388608 bytes)! You can configure this by setting 
> `akka.http.[server|client].parsing.max-content-length` or calling 
> `HttpEntity.withSizeLimit` before materializing the dataBytes stream.
>
> I see that the limit could be rised but this seems like akka wants to load 
> the entire data into memory. Actually I don't have a concrete value to put 
> there.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Streaming proxy/tunel on top of akka-http

[Jozsef Zsido]

Hi
After disabling the content-length limit I get OutOfMemory error. That 
clearly means that akka-http reads the entire response into memory which is 
not desired in my case.
Since I don't try to read the entity, this could be considered a bug?

I tried to reconstruct the response object in flatMap:

Source.single(proxyReq)
  .via(flow)
  .runWith(Sink.head)
  .flatMap(rsp => {
val source: Source[ByteString,_] = req.entity.dataBytes
val entity = HttpEntity(ContentTypes.`application/octet-stream`,source)
val response = HttpResponse(rsp.status,rsp.headers,entity)
context.complete(response)
  })

but doesn't help.

Somehow I need to instruct to stream the source, but just not getting the 
right code
I'm thinking on something like:

 val source: Source[ByteString,_] = 
Source.single(req.entity.dataBytes).runWith(Sink.head)

but unfortunately this doesn't compile.

As a test I tried to write the response into a file:

 .flatMap(f => {
f.entity.dataBytes.runWith(FileIO.toPath(new 
File("/tmp/example.out").toPath()))
val entity = HttpEntity.Empty
context.complete(entity)
  })

This works and properly streams the response into the file without 
exploding the memory.

Any idea?

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Streaming proxy/tunel on top of akka-http

[Jozsef Zsido]

Hi,

Not much code I have, basically I took a sample for akka-http proxy: 
https://gist.github.com/kovacshuni/afb7d53f40f501d0ab82

trait BrowseServerRoutes3 {

  implicit val system = ActorSystem("Browse")
  implicit val materializer = ActorMaterializer()
  implicit val ec = system.dispatcher

  val proxy = Route { context =>

val request = context.request
println("Processing " + request)

val flow = Http(system).outgoingConnection("10.66.0.4", 80)

Source.single(context.request)
  .map {
_.withHeaders(HttpUtils.completeHeaderList(request))
  .withUri(request.uri.path.toString())
  }
  .via(flow)
  .runWith(Sink.head)
  .flatMap(f => {
context.complete(f)
  })
  }
}

object BrowseServer extends App with BrowseServerRoutes3 {

  val binding = Http(system).bindAndHandle(handler = proxy, interface = 
"0.0.0.0", port = 8080)
  println(s"Server online.")
}

"Download" should "return OK" in {
  Get("/test/huge_MP3WRAP.mp3") ~> proxy ~> check {
response.status shouldEqual StatusCodes.OK
  }
}


The first problem I have with this code is that I have a 2GB file at the 
destination and I get the following exception:
EntityStreamSizeException: actual entity size (Some(2272610895)) exceeded 
content length limit (8388608 bytes)! You can configure this by setting 
`akka.http.[server|client].parsing.max-content-length` or calling 
`HttpEntity.withSizeLimit` before materializing the dataBytes stream.

I see that the limit could be rised but this seems like akka wants to load 
the entire data into memory. Actually I don't have a concrete value to put 
there.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Streaming proxy/tunel on top of akka-http

[Konrad “ktoso” Malawski]

Akka-HTTP does support completely streaming things, that’s pretty much it’s
main use case.
Please share code and or specifics about what does not work.

-- 
Cheers,
Konrad 'ktoso ' Malawski
Akka  @ Lightbend 

On November 8, 2017 at 12:05:15, Jozsef Zsido (zsjo...@gmail.com) wrote:

Hi,

I'm trying to write an akka-http based service which tunels some of the
requests through a given (ip,port).
The basics are working propely but I have problem when I try to download
large files. akka-http tries to load the complete response as Entity and
does not streams trully.
My expectation would be that the flow handler to read only the headers of
the response and to copy the content bytes in reactive way.
It should support large downloads, media streaming and chunked transfer.

Doeas have akka-http implementation support for these?
If yes, how should I set-up the flow?

Thanks,
Jozsef
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Streaming proxy/tunel on top of akka-http

[Jozsef Zsido]

Hi,

I'm trying to write an akka-http based service which tunels some of the 
requests through a given (ip,port).
The basics are working propely but I have problem when I try to download 
large files. akka-http tries to load the complete response as Entity and 
does not streams trully.
My expectation would be that the flow handler to read only the headers of 
the response and to copy the content bytes in reactive way.
It should support large downloads, media streaming and chunked transfer.

Doeas have akka-http implementation support for these?
If yes, how should I set-up the flow?

Thanks,
Jozsef

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.