Re: [libmicrohttpd] POST data processing
Dear Austin, Please look at the code samples I pointed you towards. If you have XML or JSON POST data, you must not use the MHD_PostProcessor API at all. Happy hacking! Christian On 05/18/2018 09:25 AM, Austin Einter wrote: > Dear All > I am getting data in access content callback. > I can do my own parsing. > > To get body data, independent chunks through iterate callback (NOT > access content callback), I have issues. > > In some cases, post process create fails (returns null). > In some cases, post process is created, but when I call post process > api, iterate callback is not called. > > Are you telling that > 1) always post process create will succeed (irrespective of xml / json / > binary body) > 2) if post process is created, on calling post process api, iterate > callback is called. > > Once I get sometime (sorry, but really occupied), will put a small code > and share. > > Regards > Austin > > > > On Thu, May 17, 2018 at 8:24 PM, Kenneth Mastro >> wrote: > > Unless something broke in a very recent version, POST definitely > works with JSON data. I do it all the time. I could be wrong, but > I suspect MHD does not care about the data type at all. > > I very strongly suspect the problem is in your code or your test. > Since it works with form data, are you sure your request is getting > sent as a proper 'POST' when you're trying with XML or JSON? > > Just a thought - are you just trying to use MHD's post-processor to > process the data like you do with form data? In that case - no, MHD > does not PARSE the JSON or XML (like it can with form data). You > have to do that yourself or use a third party library (but either > way, you have to capture the data from MHD and do something with it > on your own). > > > Ken > > > On Thu, May 17, 2018 at 10:33 AM, Austin Einter > > wrote: > > Hello > When I tried, if content type is xml, post processor crate > failed (it returned null). > When content type is binary, it created post processor, however > when I call post process api, the iterate callback not called. > > That made me to search google. > Somewhere I read it does not support xml and json. > > Data I am getting in access content callback, however my iterate > callback is not called. > Please note that same code is working for text data (file upload > or form data). > > Thanks > Austin > > > > > On Thu, May 17, 2018 at 6:56 PM, silvioprog > > wrote: > > Hello Austin, > > As Christian explained, via "upload_data" you can receive > any payload kind. I'm writting a library which maps MHD > callbacks to "objects" and it needs to support the following > features: > > 1. receive one or more files *on demand*(to receive large > files about 5 GB+ without "frying" CPU/RAM) in same request > (files via form-data); > 2. receive form fields (HTML-form fields via > x-www-form-urlencoded); > 3. 1 and 2 (both files and fields via form-data); > 4. payload contents (raw-data, json/xml and so on via > application/); > 5. handlers to create customized 'body-parser' if none of > the above options fits. > > it seems the option 4 above is the one you are looking for. > For example, supposing you need to send a JSON > "{"foo":"bar"}" to the server, in the client: > > curl -X POST -H "Content-Type: application/json" -d > '{"foo":"bar"}' http://localhost:8080 > > and in a minimal snipped server example, you can receive it as: > > ... > > static void req_cb(void *cls, struct bk_httpreq *req, struct > bk_httpres *res) { > struct bk_str *payload = bk_httpreq_payload(req); > printf("Payload string: %s; Size: %zd", > bk_str_content(payload), bk_str_length(payload)); > ... > } > > int main(void) { > struct bk_httpsrv *srv = bk_httpsrv_new(req_cb, NULL); > bk_httpsrv_listen(srv, 8080, false); > ... > } > > ... > > then it prints "Payload string: {"foo":"bar"}; Size: 13" in > the terminal. The "payload" variable above is an instance of > the object "bk_str" which contains useful "methods" for > string handling like "content", "length", "clear", "printf" > and more. > > If you want to take a look how it was implemented, please >
Re: [libmicrohttpd] POST data processing
Dear All I am getting data in access content callback. I can do my own parsing. To get body data, independent chunks through iterate callback (NOT access content callback), I have issues. In some cases, post process create fails (returns null). In some cases, post process is created, but when I call post process api, iterate callback is not called. Are you telling that 1) always post process create will succeed (irrespective of xml / json / binary body) 2) if post process is created, on calling post process api, iterate callback is called. Once I get sometime (sorry, but really occupied), will put a small code and share. Regards Austin On Thu, May 17, 2018 at 8:24 PM, Kenneth Mastrowrote: > Unless something broke in a very recent version, POST definitely works > with JSON data. I do it all the time. I could be wrong, but I suspect MHD > does not care about the data type at all. > > I very strongly suspect the problem is in your code or your test. Since > it works with form data, are you sure your request is getting sent as a > proper 'POST' when you're trying with XML or JSON? > > Just a thought - are you just trying to use MHD's post-processor to > process the data like you do with form data? In that case - no, MHD does > not PARSE the JSON or XML (like it can with form data). You have to do > that yourself or use a third party library (but either way, you have to > capture the data from MHD and do something with it on your own). > > > Ken > > > On Thu, May 17, 2018 at 10:33 AM, Austin Einter > wrote: > >> Hello >> When I tried, if content type is xml, post processor crate failed (it >> returned null). >> When content type is binary, it created post processor, however when I >> call post process api, the iterate callback not called. >> >> That made me to search google. >> Somewhere I read it does not support xml and json. >> >> Data I am getting in access content callback, however my iterate callback >> is not called. >> Please note that same code is working for text data (file upload or form >> data). >> >> Thanks >> Austin >> >> >> >> >> On Thu, May 17, 2018 at 6:56 PM, silvioprog wrote: >> >>> Hello Austin, >>> >>> As Christian explained, via "upload_data" you can receive any payload >>> kind. I'm writting a library which maps MHD callbacks to "objects" and it >>> needs to support the following features: >>> >>> 1. receive one or more files *on demand* (to receive large files about >>> 5 GB+ without "frying" CPU/RAM) in same request (files via form-data); >>> 2. receive form fields (HTML-form fields via x-www-form-urlencoded); >>> 3. 1 and 2 (both files and fields via form-data); >>> 4. payload contents (raw-data, json/xml and so on via >>> application/); >>> 5. handlers to create customized 'body-parser' if none of the above >>> options fits. >>> >>> it seems the option 4 above is the one you are looking for. For example, >>> supposing you need to send a JSON "{"foo":"bar"}" to the server, in the >>> client: >>> >>> curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar"}' >>> http://localhost:8080 >>> >>> and in a minimal snipped server example, you can receive it as: >>> >>> ... >>> >>> static void req_cb(void *cls, struct bk_httpreq *req, struct bk_httpres >>> *res) { >>> struct bk_str *payload = bk_httpreq_payload(req); >>> printf("Payload string: %s; Size: %zd", bk_str_content(payload), >>> bk_str_length(payload)); >>> ... >>> } >>> >>> int main(void) { >>> struct bk_httpsrv *srv = bk_httpsrv_new(req_cb, NULL); >>> bk_httpsrv_listen(srv, 8080, false); >>> ... >>> } >>> >>> ... >>> >>> then it prints "Payload string: {"foo":"bar"}; Size: 13" in the >>> terminal. The "payload" variable above is an instance of the object >>> "bk_str" which contains useful "methods" for string handling like >>> "content", "length", "clear", "printf" and more. >>> >>> If you want to take a look how it was implemented, please use the branch >>> "*new_api*" and specifically the line 78 from bk_httpuplds.c file: >>> >>> https://github.com/risoflora/libbrook/blob/new_api/src/bk_ht >>> tpuplds.c#L78 >>> >>> hope this help you. >>> >>> (the library is under development, so there is parts undocumented yet >>> but feel free to send any questions via Github issues) >>> >>> On Thu, May 17, 2018 at 1:52 AM, Austin Einter >>> wrote: >>> Hello I am using libmicrohttpd and facing issue with POST processing. The POST message received contains xml or json data. libmicrohttpd does not support xml / json. So I can not use post process or iterate callbacks for body data processing. In fact I tried and it does not work (iterate callbacks not called even when I call post process). So I am left with two options. 1) Either write body parser, where I need to take care of content-length case, chunked data case etc. 2) OR use
Re: [libmicrohttpd] POST data processing
Unless something broke in a very recent version, POST definitely works with JSON data. I do it all the time. I could be wrong, but I suspect MHD does not care about the data type at all. I very strongly suspect the problem is in your code or your test. Since it works with form data, are you sure your request is getting sent as a proper 'POST' when you're trying with XML or JSON? Just a thought - are you just trying to use MHD's post-processor to process the data like you do with form data? In that case - no, MHD does not PARSE the JSON or XML (like it can with form data). You have to do that yourself or use a third party library (but either way, you have to capture the data from MHD and do something with it on your own). Ken On Thu, May 17, 2018 at 10:33 AM, Austin Einterwrote: > Hello > When I tried, if content type is xml, post processor crate failed (it > returned null). > When content type is binary, it created post processor, however when I > call post process api, the iterate callback not called. > > That made me to search google. > Somewhere I read it does not support xml and json. > > Data I am getting in access content callback, however my iterate callback > is not called. > Please note that same code is working for text data (file upload or form > data). > > Thanks > Austin > > > > > On Thu, May 17, 2018 at 6:56 PM, silvioprog wrote: > >> Hello Austin, >> >> As Christian explained, via "upload_data" you can receive any payload >> kind. I'm writting a library which maps MHD callbacks to "objects" and it >> needs to support the following features: >> >> 1. receive one or more files *on demand* (to receive large files about 5 >> GB+ without "frying" CPU/RAM) in same request (files via form-data); >> 2. receive form fields (HTML-form fields via x-www-form-urlencoded); >> 3. 1 and 2 (both files and fields via form-data); >> 4. payload contents (raw-data, json/xml and so on via application/); >> 5. handlers to create customized 'body-parser' if none of the above >> options fits. >> >> it seems the option 4 above is the one you are looking for. For example, >> supposing you need to send a JSON "{"foo":"bar"}" to the server, in the >> client: >> >> curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar"}' >> http://localhost:8080 >> >> and in a minimal snipped server example, you can receive it as: >> >> ... >> >> static void req_cb(void *cls, struct bk_httpreq *req, struct bk_httpres >> *res) { >> struct bk_str *payload = bk_httpreq_payload(req); >> printf("Payload string: %s; Size: %zd", bk_str_content(payload), >> bk_str_length(payload)); >> ... >> } >> >> int main(void) { >> struct bk_httpsrv *srv = bk_httpsrv_new(req_cb, NULL); >> bk_httpsrv_listen(srv, 8080, false); >> ... >> } >> >> ... >> >> then it prints "Payload string: {"foo":"bar"}; Size: 13" in the >> terminal. The "payload" variable above is an instance of the object >> "bk_str" which contains useful "methods" for string handling like >> "content", "length", "clear", "printf" and more. >> >> If you want to take a look how it was implemented, please use the branch " >> *new_api*" and specifically the line 78 from bk_httpuplds.c file: >> >> https://github.com/risoflora/libbrook/blob/new_api/src/bk_httpuplds.c#L78 >> >> hope this help you. >> >> (the library is under development, so there is parts undocumented yet but >> feel free to send any questions via Github issues) >> >> On Thu, May 17, 2018 at 1:52 AM, Austin Einter >> wrote: >> >>> Hello >>> I am using libmicrohttpd and facing issue with POST processing. >>> >>> The POST message received contains xml or json data. libmicrohttpd does >>> not support xml / json. So I can not use post process or iterate callbacks >>> for body data processing. In fact I tried and it does not work (iterate >>> callbacks not called even when I call post process). >>> >>> So I am left with two options. >>> >>> 1) Either write body parser, where I need to take care of content-length >>> case, chunked data case etc. >>> >>> 2) OR use http-parser kind of open source >>> >>> Is there any other option available? >>> >>> I am not very pro to write my own parser, instead I would like to use >>> http-parser. I just looked at http-parser. To use http-parser, we need to >>> provide the entire message to http-parser, either single time or one chunk >>> at a time. >>> >>> >>> Is there anyway in libmicrohttpd, I can collect the entire POST message >>> as it comes / received at socket level? >>> >>> If possible, kindly provide required steps / info. >>> >>> Best Regards >>> Austin >>> >> >> -- >> Silvio Clécio >> > >
Re: [libmicrohttpd] POST data processing
Could you show a smallest code example showing how you are trying (both server and client side)? (for client side I've used curl and it sends fine any payload kind) On Thu, May 17, 2018 at 11:33 AM, Austin Einterwrote: > Hello > When I tried, if content type is xml, post processor crate failed (it > returned null). > When content type is binary, it created post processor, however when I > call post process api, the iterate callback not called. > > That made me to search google. > Somewhere I read it does not support xml and json. > > Data I am getting in access content callback, however my iterate callback > is not called. > Please note that same code is working for text data (file upload or form > data). > > Thanks > Austin > > On Thu, May 17, 2018 at 6:56 PM, silvioprog wrote: > >> Hello Austin, >> >> As Christian explained, via "upload_data" you can receive any payload >> kind. I'm writting a library which maps MHD callbacks to "objects" and it >> needs to support the following features: >> >> 1. receive one or more files *on demand* (to receive large files about 5 >> GB+ without "frying" CPU/RAM) in same request (files via form-data); >> 2. receive form fields (HTML-form fields via x-www-form-urlencoded); >> 3. 1 and 2 (both files and fields via form-data); >> 4. payload contents (raw-data, json/xml and so on via application/); >> 5. handlers to create customized 'body-parser' if none of the above >> options fits. >> >> it seems the option 4 above is the one you are looking for. For example, >> supposing you need to send a JSON "{"foo":"bar"}" to the server, in the >> client: >> >> curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar"}' >> http://localhost:8080 >> >> and in a minimal snipped server example, you can receive it as: >> >> ... >> >> static void req_cb(void *cls, struct bk_httpreq *req, struct bk_httpres >> *res) { >> struct bk_str *payload = bk_httpreq_payload(req); >> printf("Payload string: %s; Size: %zd", bk_str_content(payload), >> bk_str_length(payload)); >> ... >> } >> >> int main(void) { >> struct bk_httpsrv *srv = bk_httpsrv_new(req_cb, NULL); >> bk_httpsrv_listen(srv, 8080, false); >> ... >> } >> >> ... >> >> then it prints "Payload string: {"foo":"bar"}; Size: 13" in the >> terminal. The "payload" variable above is an instance of the object >> "bk_str" which contains useful "methods" for string handling like >> "content", "length", "clear", "printf" and more. >> >> If you want to take a look how it was implemented, please use the branch " >> *new_api*" and specifically the line 78 from bk_httpuplds.c file: >> >> https://github.com/risoflora/libbrook/blob/new_api/src/bk_httpuplds.c#L78 >> >> hope this help you. >> >> (the library is under development, so there is parts undocumented yet but >> feel free to send any questions via Github issues) >> >> On Thu, May 17, 2018 at 1:52 AM, Austin Einter >> wrote: >> >>> Hello >>> I am using libmicrohttpd and facing issue with POST processing. >>> >>> The POST message received contains xml or json data. libmicrohttpd does >>> not support xml / json. So I can not use post process or iterate callbacks >>> for body data processing. In fact I tried and it does not work (iterate >>> callbacks not called even when I call post process). >>> >>> So I am left with two options. >>> >>> 1) Either write body parser, where I need to take care of content-length >>> case, chunked data case etc. >>> >>> 2) OR use http-parser kind of open source >>> >>> Is there any other option available? >>> >>> I am not very pro to write my own parser, instead I would like to use >>> http-parser. I just looked at http-parser. To use http-parser, we need to >>> provide the entire message to http-parser, either single time or one chunk >>> at a time. >>> >>> >>> Is there anyway in libmicrohttpd, I can collect the entire POST message >>> as it comes / received at socket level? >>> >>> If possible, kindly provide required steps / info. >>> >>> Best Regards >>> Austin >>> >> >> -- >> Silvio Clécio >> > -- Silvio Clécio
Re: [libmicrohttpd] POST data processing
Hello When I tried, if content type is xml, post processor crate failed (it returned null). When content type is binary, it created post processor, however when I call post process api, the iterate callback not called. That made me to search google. Somewhere I read it does not support xml and json. Data I am getting in access content callback, however my iterate callback is not called. Please note that same code is working for text data (file upload or form data). Thanks Austin On Thu, May 17, 2018 at 6:56 PM, silvioprogwrote: > Hello Austin, > > As Christian explained, via "upload_data" you can receive any payload > kind. I'm writting a library which maps MHD callbacks to "objects" and it > needs to support the following features: > > 1. receive one or more files *on demand* (to receive large files about 5 > GB+ without "frying" CPU/RAM) in same request (files via form-data); > 2. receive form fields (HTML-form fields via x-www-form-urlencoded); > 3. 1 and 2 (both files and fields via form-data); > 4. payload contents (raw-data, json/xml and so on via application/); > 5. handlers to create customized 'body-parser' if none of the above > options fits. > > it seems the option 4 above is the one you are looking for. For example, > supposing you need to send a JSON "{"foo":"bar"}" to the server, in the > client: > > curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar"}' > http://localhost:8080 > > and in a minimal snipped server example, you can receive it as: > > ... > > static void req_cb(void *cls, struct bk_httpreq *req, struct bk_httpres > *res) { > struct bk_str *payload = bk_httpreq_payload(req); > printf("Payload string: %s; Size: %zd", bk_str_content(payload), > bk_str_length(payload)); > ... > } > > int main(void) { > struct bk_httpsrv *srv = bk_httpsrv_new(req_cb, NULL); > bk_httpsrv_listen(srv, 8080, false); > ... > } > > ... > > then it prints "Payload string: {"foo":"bar"}; Size: 13" in the terminal. > The "payload" variable above is an instance of the object "bk_str" which > contains useful "methods" for string handling like "content", "length", > "clear", "printf" and more. > > If you want to take a look how it was implemented, please use the branch " > *new_api*" and specifically the line 78 from bk_httpuplds.c file: > > https://github.com/risoflora/libbrook/blob/new_api/src/bk_httpuplds.c#L78 > > hope this help you. > > (the library is under development, so there is parts undocumented yet but > feel free to send any questions via Github issues) > > On Thu, May 17, 2018 at 1:52 AM, Austin Einter > wrote: > >> Hello >> I am using libmicrohttpd and facing issue with POST processing. >> >> The POST message received contains xml or json data. libmicrohttpd does >> not support xml / json. So I can not use post process or iterate callbacks >> for body data processing. In fact I tried and it does not work (iterate >> callbacks not called even when I call post process). >> >> So I am left with two options. >> >> 1) Either write body parser, where I need to take care of content-length >> case, chunked data case etc. >> >> 2) OR use http-parser kind of open source >> >> Is there any other option available? >> >> I am not very pro to write my own parser, instead I would like to use >> http-parser. I just looked at http-parser. To use http-parser, we need to >> provide the entire message to http-parser, either single time or one chunk >> at a time. >> >> >> Is there anyway in libmicrohttpd, I can collect the entire POST message >> as it comes / received at socket level? >> >> If possible, kindly provide required steps / info. >> >> Best Regards >> Austin >> > > -- > Silvio Clécio >
Re: [libmicrohttpd] POST data processing
Hello Austin, As Christian explained, via "upload_data" you can receive any payload kind. I'm writting a library which maps MHD callbacks to "objects" and it needs to support the following features: 1. receive one or more files *on demand* (to receive large files about 5 GB+ without "frying" CPU/RAM) in same request (files via form-data); 2. receive form fields (HTML-form fields via x-www-form-urlencoded); 3. 1 and 2 (both files and fields via form-data); 4. payload contents (raw-data, json/xml and so on via application/); 5. handlers to create customized 'body-parser' if none of the above options fits. it seems the option 4 above is the one you are looking for. For example, supposing you need to send a JSON "{"foo":"bar"}" to the server, in the client: curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar"}' http://localhost:8080 and in a minimal snipped server example, you can receive it as: ... static void req_cb(void *cls, struct bk_httpreq *req, struct bk_httpres *res) { struct bk_str *payload = bk_httpreq_payload(req); printf("Payload string: %s; Size: %zd", bk_str_content(payload), bk_str_length(payload)); ... } int main(void) { struct bk_httpsrv *srv = bk_httpsrv_new(req_cb, NULL); bk_httpsrv_listen(srv, 8080, false); ... } ... then it prints "Payload string: {"foo":"bar"}; Size: 13" in the terminal. The "payload" variable above is an instance of the object "bk_str" which contains useful "methods" for string handling like "content", "length", "clear", "printf" and more. If you want to take a look how it was implemented, please use the branch " *new_api*" and specifically the line 78 from bk_httpuplds.c file: https://github.com/risoflora/libbrook/blob/new_api/src/bk_httpuplds.c#L78 hope this help you. (the library is under development, so there is parts undocumented yet but feel free to send any questions via Github issues) On Thu, May 17, 2018 at 1:52 AM, Austin Einterwrote: > Hello > I am using libmicrohttpd and facing issue with POST processing. > > The POST message received contains xml or json data. libmicrohttpd does > not support xml / json. So I can not use post process or iterate callbacks > for body data processing. In fact I tried and it does not work (iterate > callbacks not called even when I call post process). > > So I am left with two options. > > 1) Either write body parser, where I need to take care of content-length > case, chunked data case etc. > > 2) OR use http-parser kind of open source > > Is there any other option available? > > I am not very pro to write my own parser, instead I would like to use > http-parser. I just looked at http-parser. To use http-parser, we need to > provide the entire message to http-parser, either single time or one chunk > at a time. > > > Is there anyway in libmicrohttpd, I can collect the entire POST message as > it comes / received at socket level? > > If possible, kindly provide required steps / info. > > Best Regards > Austin > -- Silvio Clécio
Re: [libmicrohttpd] POST data processing
On 05/17/2018 06:52 AM, Austin Einter wrote: > Hello > I am using libmicrohttpd and facing issue with POST processing. > > The POST message received contains xml or json data. libmicrohttpd does > not support xml / json. So I can not use post process or iterate > callbacks for body data processing. In fact I tried and it does not work > (iterate callbacks not called even when I call post process). > > So I am left with two options. > > 1) Either write body parser, where I need to take care of content-length > case, chunked data case etc. Nope, that is _always_ take care of by MHD. You just need to feed the "upload_data" you get in your MHD_AccessHandlerCallback as you receive it to your XML or JSON parser, either incrementally or as a whole. > Is there anyway in libmicrohttpd, I can collect the entire POST message > as it comes / received at socket level? Sure. You should look at GNU Taler (https://git.taler.net/), the exchange and the merchant-backend both use MHD to receive JSON-formatted uploads. The core logic for this is actually from/in GNUnet (https://gnunet.org/git/, gnunet.git, src/json/json_mhd.c). So if you are happy with libjansson, you can link against libgnunetjson (like Taler does) and use that code to parse JSON. XML usage would be analogous (and this is ~200 LOC only). Happy hacking! Christian 0xE29FC3CC.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature