Re: [akka-user] Marshalling FormData to multi-part mime with one field per part

[Johannes Rudolph]

On Wednesday, October 14, 2015 at 7:17:14 PM UTC+2, Eric Swenson wrote:
>
> AWS/S3 doesn’t want the form data parts, nor the whole HTTP POST to be 
> chunked. So those entities have to be made “strict”.  So I had to make each 
> MIME part be strict, as well as the final Multipart.FormData entity.
>

Have you seen/tried 
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html? 
It seems to suggest that chunked transfer-encoding is supported, though in 
a bit complicated format where on top of chunked transfer-encoding AWS 
seems to introduce their own chunked content-encoding layer where each 
chunk is signed separately.
 

> The point of my original post was that it seems the kaka-http library has 
> made a policy decision
>

WDYM? akka-http supports both `FormData` = www-url-encoding and 
`multipart/formdata` encodings, so, I wonder which "policy decision" you 
mean?
 

> that the only way to convert a FormData to an entity (in a multi-part MIME 
> body) is to create a single part, using www-url-encoding, and include all 
> fields in query string syntax. This doesn’t work for AWS and therefore it 
> seems a limiting policy decision.
>

Yes, because `scaladsl.model.FormData` models www-url-encoding and not 
multipart/formdata. You seem to suggest that there should be a more general 
encoding that would allow marshalling to either of both variants. In any 
case, there's no policy decision but, as explained above, two different 
models for two different kinds of modelling things.

It should be equally possible/easy to convert a FormData to a collection of 
> parts. 
>

Yes, that could be useful, or otherwise a more general representation of 
form data that unifies both kinds.
 

> The requirement that there should be a Content-Type header in these parts, 
> also should not be dictated by the framework.
>

 I see that you have to fight a bit to get akka-http to render exactly what 
you want, but the reason seems to be mostly that AWS has very specific 
rules and interpretations of how to use HTTP. akka-http is a general 
purpose HTTP library and cannot foresee all possible deviations or 
additional constraints HTTP APIs try to enforce. So, in the end, akka-http 
should make it possible to connect to these APIs (as long as they support 
HTTP to a large enough degree) but it may mean that the extra complexity 
the API enforces lies in your code. You could see that as a feature.

That said, I'm pretty sure that there's some sugar missing in the APIs that 
would help you build those requests more easily. If you can distill the 
generic helpers that are missing from your particular use case we could 
discuss how to improve things.

Here are some things I can see:

 * you can directly build a `Multipart.FormData.Strict` which can be 
directly converted into a Strict entity, I guess one problem that we could 
solve is that there's only a non-strict FormData.BodyPart.fromFile` which 
builds a streamed part to prevent loading the complete file into memory. 
There's no `FormData.BodyPart.fromFile` that would actually load the file 
and create a strict version of it. We could add that (even if it wouldn't 
be recommended to load files into memory...)
 * having to run marshalling and unmarshalling manually could be replaced 
by some sugar
 * dealing with those chains of `T => Future[U]` functions is cumbersome, 
for this and the previous point, we had the simple `pipelining` DSL in 
spray which seems to have been lost in parts in the transition to akka-http

Btw. I don't think your code is too bad, if you break it down into methods 
for every step and put it into a utility object, you can reuse it and don't 
need to deal with it any more.

Johannes

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Marshalling FormData to multi-part mime with one field per part

[Eric Swenson]

On Thursday, October 15, 2015 at 12:24:48 AM UTC-7, Johannes Rudolph wrote:
>
> Have you seen/tried 
> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html? It 
> seems to suggest that chunked transfer-encoding is supported, though in a 
> bit complicated format where on top of chunked transfer-encoding AWS seems 
> to introduce their own chunked content-encoding layer where each chunk is 
> signed separately.
>

Yes, but I don't know if this works for S3 HTTP POSTs, and I don't need 
each chunk separately signed.  

>  
>
>> The point of my original post was that it seems the kaka-http library has 
>> made a policy decision
>>
>
> WDYM? akka-http supports both `FormData` = www-url-encoding and 
> `multipart/formdata` encodings, so, I wonder which "policy decision" you 
> mean?
>

Perhaps I just couldn't find out how to do it (except very manually as you 
saw in my code sample). You can easily create a single HTTP entity via 
Marshal(formData).to[HttpEntity], where that entity will encode all form 
parameters/values as a query string. How do you, using existing 
marshallers, create a List[HttpEntity] given a FormData object? The bias is 
that there is no direct way to do this. I can clearly map over all the 
fields in the FormData myself and marshall each name/value pair to an 
HttpEntity. But by default, doing so uses ContentType.`text/plain`. You 
need your own custom logic to force the content type to NoContentType, 
which is what we need in this case. 

>  
>
>> that the only way to convert a FormData to an entity (in a multi-part 
>> MIME body) is to create a single part, using www-url-encoding, and include 
>> all fields in query string syntax. This doesn’t work for AWS and therefore 
>> it seems a limiting policy decision.
>>
>
> Yes, because `scaladsl.model.FormData` models www-url-encoding and not 
> multipart/formdata. You seem to suggest that there should be a more general 
> encoding that would allow marshalling to either of both variants. In any 
> case, there's no policy decision but, as explained above, two different 
> models for two different kinds of modelling things.
>

That is the bias I'm talking about. FormData is biased toward 
www-url-encoding rather than multipart/formdata. Both should be provided. 

>
> It should be equally possible/easy to convert a FormData to a collection 
>> of parts. 
>>
>
> Yes, that could be useful, or otherwise a more general representation of 
> form data that unifies both kinds.
>  
>
>> The requirement that there should be a Content-Type header in these 
>> parts, also should not be dictated by the framework.
>>
>
>  I see that you have to fight a bit to get akka-http to render exactly 
> what you want, but the reason seems to be mostly that AWS has very specific 
> rules and interpretations of how to use HTTP. 
>

Actually, AWS is expecting an entity that is exactly what all browsers 
would generate when you include one or more files in an HTTP form.  All the 
form fields including the files go in separate multi-part mime parts. The 
files have a Content-Type header and the non-files don't. Now I don't have 
any issue with the non-strict bias in akka-http. That is the more general, 
and more async-friendly approach. So it doesn't bother me that you have to 
do something special to get the various entities to be strict. Here, an 
expression that maps over the body parts converting each from non-strict to 
strict and returning a Future[Multipart.FormData] (or whatever it would be) 
would be fine.

akka-http is a general purpose HTTP library and cannot foresee all possible 
> deviations or additional constraints HTTP APIs try to enforce. So, in the 
> end, akka-http should make it possible to connect to these APIs (as long as 
> they support HTTP to a large enough degree) but it may mean that the extra 
> complexity the API enforces lies in your code. You could see that as a 
> feature.
>

I don't disagree with anything you've said there. But there is a bias 
toward www-url-encode form data. And it would be helpful if akka-http made 
it as easy to create Multipart.FormData entities. 

>
> That said, I'm pretty sure that there's some sugar missing in the APIs 
> that would help you build those requests more easily. If you can distill 
> the generic helpers that are missing from your particular use case we could 
> discuss how to improve things.
>
> Here are some things I can see:
>
>  * you can directly build a `Multipart.FormData.Strict` which can be 
> directly converted into a Strict entity, I guess one problem that we could 
> solve is that there's only a non-strict FormData.BodyPart.fromFile` which 
> builds a streamed part to prevent loading the complete file into memory. 
> There's no `FormData.BodyPart.fromFile` that would actually load the file 
> and create a strict version of it. We could add that (even if it wouldn't 
> be recommended to load files into memory...)
>  * having to run marshalling and unmarshalling manually could 

Re: [akka-user] Marshalling FormData to multi-part mime with one field per part

[Mathias]

Eric,

you can create a multipart/formdata request like this, for example:

val formData = {
  def utf8TextEntity(content: String) = {
val bytes = ByteString(content)
HttpEntity.Default(ContentTypes.`text/plain(UTF-8)`, bytes.length, 
Source.single(bytes))
  }
  
  import Multipart._
  FormData(Source(
FormData.BodyPart("foo", utf8TextEntity("FOO")) ::
  FormData.BodyPart("bar", utf8TextEntity("BAR")) :: Nil))
}

val request = HttpRequest(POST, entity = formData.toEntity())

This will render a chunked request that looks something like this:

POST / HTTP/1.1
Host: example.com
User-Agent: akka-http/test
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=o5bjIygPrPrCCYUp8FyfvsQe

71
--o5bjIygPrPrCCYUp8FyfvsQe
Content-Type: text/plain; charset=UTF-8
Content-Disposition: form-data; name=foo


3
FOO
73

--o5bjIygPrPrCCYUp8FyfvsQe
Content-Type: text/plain; charset=UTF-8
Content-Disposition: form-data; name=bar


3
BAR
1e

--o5bjIygPrPrCCYUp8FyfvsQe--
0

If you want to force the rendering of an unchunked request you can call 
`toStrict` on the `formData` first.
This would create a request looking something like this:

POST / HTTP/1.1
Host: example.com
User-Agent: akka-http/test
Content-Type: multipart/form-data; boundary=1QipXPrd9L25eJwcQYgiNbfA
Content-Length: 264

--1QipXPrd9L25eJwcQYgiNbfA
Content-Type: text/plain; charset=UTF-8
Content-Disposition: form-data; name=foo

FOO
--1QipXPrd9L25eJwcQYgiNbfA
Content-Type: text/plain; charset=UTF-8
Content-Disposition: form-data; name=bar

BAR
--1QipXPrd9L25eJwcQYgiNbfA--

There is no www-url-encoding happening anywhere.

Cheers,
Mathias


On Tuesday, October 13, 2015 at 11:33:43 PM UTC+2, Eric Swenson wrote:
>
> I have not been able to find any way to marshall a set of HTTP Post form 
> parameters AND a file using akka-http, where the form data parameters are 
> NOT www-url-encoded, but rather each placed in a separate MIME part, with 
> no Content-Type header. This appears to be the format required in order to 
> do an HTTP POST to AWS S3 with a policy.  Furthermore, it appears AWS/S3 is 
> not happy with Chunked HTTP POST requests, and therefore I've had to use 
> toStrict on all my entities.  I've resorted to some hacky code, which 
> works, but I find it surprising that there is no built-in way to generate a 
> multi-part MIME message where post parameters are not combined into one 
> www-url-encoded query string.  
>
> Here is my (admittedly ugly) code:
>
> val m = Map(
>
>   "key" -> "upload/quux.txt",
>
>   "acl" -> acl,
>
>   "policy" -> policy,
>
>   "X-amz-algorithm" -> "AWS4-HMAC-SHA256",
>
>   "X-amz-credential" -> s"$awsAccessKeyId/$shortDate/$region/$service
> /aws4_request",
>
>   "X-amz-date" -> date,
>
>   "X-amz-expires" -> expires.toString,
>
>   "X-amz-signature" -> signature
>
> )
>
> 
>
> implicit val timeout: FiniteDuration = 10.seconds
>
> 
>
> val uri = s"http://$bucket.$service-$region.amazonaws.com;
>
> 
>
> // create request entities for each of the POST parameters. Note that 
> generating Strict entities is an asynchronous
>
> // operation (returns a future). Strict entities are NOT chunked. It 
> appears AWS is not happy with chunked MIME
>
> // body parts.
>
> val strictFormDataEntityFuture = m.map { case (k, v) => (k, 
> HttpEntity(ContentTypes.NoContentType, 
> v).toStrict(timeout)) }
>
> 
>
> val responseFuture = for {
>
>   strictEntityMap <- Future.sequence(strictFormDataEntityFuture.map(
> entry => entry._2.map(i => (entry._1, i.map(_.toMap) recoverWith { 
> case t: Throwable => throw t }
>
>   formDataBodyPartList = strictEntityMap.map {e => 
> Multipart.FormData.BodyPart.Strict(e._1, e._2)}.toList
>
>   fileEntity <- HttpEntity(ContentTypes.`application/octet-stream`, 3, 
> Source.single(ByteString("foo"))).toStrict(timeout) recoverWith { case t: 
> Throwable => throw t }
>
>   multipartForm = Multipart.FormData(Source(formDataBodyPartList 
> ++List(Multipart.FormData.BodyPart("file", fileEntity, Map("filename" -> 
> "foo.bin")
>
>   requestEntity <- Marshal(multipartForm).to[RequestEntity] 
> recoverWith { case t: Throwable => throw t }
>
>   strictRequestEntity <- requestEntity.toStrict(timeout) recoverWith 
> { case t: Throwable => throw t }
>
>   response <- Http().singleRequest(HttpRequest(method = 
> HttpMethods.POST, uri = uri, entity = strictRequestEntity)) recoverWith { 
> case t: Throwable => throw t }
>
>   responseEntity <- Unmarshal(response.entity).to[String] recoverWith 
> { case t: Throwable => throw t }
>
> } yield (responseEntity, response)
>
>
> If anyone can suggest a better approach, I'd appreciate it.  Thanks. 
>
> On Sunday, October 11, 2015 at 10:45:37 PM UTC-7, 

Re: [akka-user] Marshalling FormData to multi-part mime with one field per part

[Eric Swenson]

Hi Mathias,

I think you are suggesting pretty much exactly what I did, but I did it in such 
a way that I didn’t have to make each body part (corresponding to a form 
parameter) manually (I used mapping).  

However, AWS/S3 will not accept the Content-Type: text/plain in the body parts. 
It thinks the request is trying to “upload” multiple files.  I had to use 
ContentTypes.NoContentType. 

See below for my approach (where m is a map of the parameter/value pairs):

val strictFormDataEntityFuture = m.map { case (k, v) => (k, 
HttpEntity(ContentTypes.NoContentType, v).toStrict(timeout)) }

That produces a Map[String, Future[HttpEntity.Strict]], which is then converted 
to a Future[Map[String,HttpEntity.Strict]] with:

for {
  strictEntityMap <- Future.sequence(strictFormDataEntityFuture.map(entry 
=> entry._2.map(i => (entry._1, i.map(_.toMap)
…

AWS/S3 doesn’t want the form data parts, nor the whole HTTP POST to be chunked. 
So those entities have to be made “strict”.  So I had to make each MIME part be 
strict, as well as the final Multipart.FormData entity.

The point of my original post was that it seems the kaka-http library has made 
a policy decision that the only way to convert a FormData to an entity (in a 
multi-part MIME body) is to create a single part, using www-url-encoding, and 
include all fields in query string syntax. This doesn’t work for AWS and 
therefore it seems a limiting policy decision. It should be equally 
possible/easy to convert a FormData to a collection of parts. The requirement 
that there should be a Content-Type header in these parts, also should not be 
dictated by the framework.

— Eric


> On Oct 14, 2015, at 5:21 AM, Mathias  wrote:
> 
> Eric,
> 
> you can create a multipart/formdata request like this, for example:
> 
> val formData = {
>   def utf8TextEntity(content: String) = {
> val bytes = ByteString(content)
> HttpEntity.Default(ContentTypes.`text/plain(UTF-8)`, bytes.length, 
> Source.single(bytes))
>   }
>   
>   import Multipart._
>   FormData(Source(
> FormData.BodyPart("foo", utf8TextEntity("FOO")) ::
>   FormData.BodyPart("bar", utf8TextEntity("BAR")) :: Nil))
> }
> 
> val request = HttpRequest(POST, entity = formData.toEntity())
> 
> This will render a chunked request that looks something like this:
> 
> POST / HTTP/1.1
> Host: example.com
> User-Agent: akka-http/test
> Transfer-Encoding: chunked
> Content-Type: multipart/form-data; boundary=o5bjIygPrPrCCYUp8FyfvsQe
> 
> 71
> --o5bjIygPrPrCCYUp8FyfvsQe
> Content-Type: text/plain; charset=UTF-8
> Content-Disposition: form-data; name=foo
> 
> 
> 3
> FOO
> 73
> 
> --o5bjIygPrPrCCYUp8FyfvsQe
> Content-Type: text/plain; charset=UTF-8
> Content-Disposition: form-data; name=bar
> 
> 
> 3
> BAR
> 1e
> 
> --o5bjIygPrPrCCYUp8FyfvsQe--
> 0
> 
> If you want to force the rendering of an unchunked request you can call 
> `toStrict` on the `formData` first.
> This would create a request looking something like this:
> 
> POST / HTTP/1.1
> Host: example.com
> User-Agent: akka-http/test
> Content-Type: multipart/form-data; boundary=1QipXPrd9L25eJwcQYgiNbfA
> Content-Length: 264
> 
> --1QipXPrd9L25eJwcQYgiNbfA
> Content-Type: text/plain; charset=UTF-8
> Content-Disposition: form-data; name=foo
> 
> FOO
> --1QipXPrd9L25eJwcQYgiNbfA
> Content-Type: text/plain; charset=UTF-8
> Content-Disposition: form-data; name=bar
> 
> BAR
> --1QipXPrd9L25eJwcQYgiNbfA--
> 
> There is no www-url-encoding happening anywhere.
> 
> Cheers,
> Mathias
> 
> 
> On Tuesday, October 13, 2015 at 11:33:43 PM UTC+2, Eric Swenson wrote:
> I have not been able to find any way to marshall a set of HTTP Post form 
> parameters AND a file using akka-http, where the form data parameters are NOT 
> www-url-encoded, but rather each placed in a separate MIME part, with no 
> Content-Type header. This appears to be the format required in order to do an 
> HTTP POST to AWS S3 with a policy.  Furthermore, it appears AWS/S3 is not 
> happy with Chunked HTTP POST requests, and therefore I've had to use toStrict 
> on all my entities.  I've resorted to some hacky code, which works, but I 
> find it surprising that there is no built-in way to generate a multi-part 
> MIME message where post parameters are not combined into one www-url-encoded 
> query string.  
> 
> Here is my (admittedly ugly) code:
> 
> val m = Map(
> 
>   "key" -> "upload/quux.txt",
> 
>   "acl" -> acl,
> 
>   "policy" -> policy,
> 
>   "X-amz-algorithm" -> "AWS4-HMAC-SHA256",
> 
>   "X-amz-credential" -> 
> s"$awsAccessKeyId/$shortDate/$region/$service/aws4_request",
> 
>   "X-amz-date" -> date,
> 
>   "X-amz-expires" -> expires.toString,
> 
>   "X-amz-signature" -> signature
> 
> )
> 
> 
> implicit val timeout: FiniteDuration 

Re: [akka-user] Marshalling FormData to multi-part mime with one field per part

[Scott Maher]

I can't test this as I am not at home but there are multiple form
marshallers, one for url encoding and one for multipart called, I think,
Multipart.FormData. sorry if you already know this.

Hi Eric! :P
On Oct 11, 2015 9:18 PM, "Eric Swenson"  wrote:

> AWS/S3 HTTP Post with a policy requires an HTTP post with multipart-mime
> -- one part for the file and one part, each, for various form parameters.
> When I use akka.http and attempt to marshal FormData to an Entity and use
> that to create a multi-part mime message, all the form data go into one
> mime-part using the query string format (foo=bar=quux). AWS rejects
> such a request as it doesn't want all form parameters in a single
> www-url-encoded-form-data part, but rather separate mime parts for each
> parameter. How to I cause a Form to be martialled in this way using
> akka.http?
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Marshalling FormData to multi-part mime with one field per part

[Eric Swenson]

Hey Scott!

When you get a chance, if you can point me to the right one, I'd appreciate 
it. I've been looking at the sources, tests, and examples, and I can't find 
how to do it.  What I'm after is a way to take a Form, and emit a 
multi-part MIME message with one part for each field in the form and one 
part for the single file I need to upload. Using code like this:

 val formBodyPartFuture = for {

  httpEntity <- Marshal(form).to[HttpEntity]

  strictEntity <- httpEntity.toStrict(timeout)

} yield Multipart.FormData.BodyPart("foo", strictEntity)


gives me one body part with Content-Type x-www-url-encoded, and the set of 
all fields are encoded as in a query string.If you can point me to the 
"other" way of marshalling a form to a set of mime body parts, I'd 
appreciate it. I had pretty much given up (and googling and using stack 
overflow have turned up nothing) and was going to generate the parts 
manually.  i'm sure there must be a correct way to do this.  Thanks. -- 
 Eric




On Sunday, October 11, 2015 at 9:43:20 PM UTC-7, Scott Maher wrote:
>
> I can't test this as I am not at home but there are multiple form 
> marshallers, one for url encoding and one for multipart called, I think, 
> Multipart.FormData. sorry if you already know this.
>
> Hi Eric! :P
> On Oct 11, 2015 9:18 PM, "Eric Swenson"  
> wrote:
>
>> AWS/S3 HTTP Post with a policy requires an HTTP post with multipart-mime 
>> -- one part for the file and one part, each, for various form parameters. 
>> When I use akka.http and attempt to marshal FormData to an Entity and use 
>> that to create a multi-part mime message, all the form data go into one 
>> mime-part using the query string format (foo=bar=quux). AWS rejects 
>> such a request as it doesn't want all form parameters in a single 
>> www-url-encoded-form-data part, but rather separate mime parts for each 
>> parameter. How to I cause a Form to be martialled in this way using 
>> akka.http?
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.