I did a web scraping in Racket a couple of months ago, and I agree that it is painful. On the bright side, Bogdan seems to start working on a practical requests-like library <https://github.com/Bogdanp/racket-http-easy> due to this thread, so we should have a good practical HTTP request library soon :)
On Sun, May 17, 2020 at 2:09 PM Brian Adkins <br...@lojic.com> wrote: > Sorry - missed the fact that you already found the cookie library :) > > On Sunday, May 17, 2020 at 5:08:32 PM UTC-4, Brian Adkins wrote: >> >> I spent many years developing in Ruby before switching to Racket, so I >> understand the appeal of a "batteries included" language. Python excels in >> this area. If you're weighting the "batteries included" aspect very high, >> then Racket may not be suitable for you at this time, but in that case, I >> expect you'd just stick with Python. >> >> If you're patient, I think you may find many compelling reasons to >> consider Racket. I won't list them here (other than the fact that the >> community alone is compelling enough to invest more time learning the >> language), or advocate for the language in general, but if you decide to >> check it out more thoroughly later, another library that may be of interest >> is: >> >> https://docs.racket-lang.org/cookies/index.html >> >> I've been developing web applications in Racket for the last year and a >> half, or so, but I've been focused on server-side functionality, and I've >> either found a Racket package that does what I need directly, or I've found >> a set of them that allow me to easily layer the extra code on top that I >> need. I was used to the Rails framework, so I thought it would be tough >> switching to Racket, but in hindsight, it was trivial. >> >> I will be doing more web scraping soon for some data science >> applications, and given what Racket provides currently, it will be easy to >> write something that fulfills the requirements you listed in your first >> post, although I have no need for proxy servers personally. >> >> Brian >> >> On Sunday, May 17, 2020 at 4:02:59 PM UTC-4, fixpoint wrote: >>> >>> Thanks, but that library does not support proxies. Oh well, my Racket >>> adventures have come to an end as I’ll be unable to convince others at work >>> to try it without a practical, batteries included http library. >>> >>> On Sunday, May 17, 2020 at 2:14:19 PM UTC-5, Jens Axel Søgaard wrote: >>>> >>>> Take a look at the `http` package. >>>> >>>> https://docs.racket-lang.org/http/index.html >>>> >>>> I believe it can be used to receive and send cookies (and other >>>> headers), but >>>> I don't know how much is automatic. >>>> >>>> /Jens Axel >>>> >>>> Racket Stories >>>> https://racket-stories.com >>>> >>>> >>>> >>>> Den søn. 17. maj 2020 kl. 17.51 skrev fixpoint <pe...@kazmier.com>: >>>> >>>>> I was under the incorrect impression that net/url did not support >>>>> HTTP 1.1. I suspect I stumbled upon some out of date information on >>>>> stackoverflow. Thanks for pointing me in the right direction. Using >>>>> net/url, net/cookies, and json, I hacked the following code (please >>>>> be gentle, it is literally the first lines of Racket I've ever written). >>>>> There are two key problems with this code: >>>>> >>>>> 1. Because cookies handling is not part of net/url, my use of >>>>> get-pure-port/headers is not able to see cookies, and thus save >>>>> them, in any of the 3xx redirect responses. >>>>> 2. I need a post-pure-port/headers equivalent, so I can post data >>>>> and reuse the connection. It looks like you can only reuse connections >>>>> in >>>>> net/url with only get-pure-port/headers. >>>>> >>>>> Am I missing any other libraries out there? >>>>> >>>>> #lang racket >>>>> >>>>> (require json >>>>> net/head >>>>> net/url >>>>> net/cookies/user-agent) >>>>> >>>>> >>>>> (define (get-json-request url #:connection conn) >>>>> (define req-headers >>>>> (let ([cookies (cookie-header url)]) >>>>> (cond [(false? cookies) '()] >>>>> [else (list (format "Cookie: ~a" cookies))]))) >>>>> (display (format "Headers sent were:\n ~a\n" req-headers)) >>>>> (define-values >>>>> (in-port resp-headers) >>>>> (get-pure-port/headers url req-headers #:connection conn >>>>> #:redirections 1)) >>>>> (extract-and-save-cookies! (map string->bytes/utf-8 (string-split >>>>> resp-headers "\r\n")) url) >>>>> (display (format "\nReceived headers were:\n ~a" resp-headers)) >>>>> (read-json in-port)) >>>>> >>>>> >>>>> ;; Start a session >>>>> (define conn (make-http-connection)) >>>>> >>>>> ;; Make a request that will force server to set cookies >>>>> (displayln (jsexpr->string >>>>> (get-json-request >>>>> (string->url " >>>>> https://postman-echo.com/cookies/set?foo1=bar1&foo2=bar2";) >>>>> #:connection conn))) >>>>> >>>>> ;; Make another request to see what cookies we sent server >>>>> (displayln "\n--------------------------------------\n") >>>>> (displayln (jsexpr->string >>>>> (get-json-request >>>>> (string->url "https://postman-echo.com/cookies";) >>>>> #:connection conn))) >>>>> >>>>> ;; Close session >>>>> (http-connection-close conn) >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sunday, May 17, 2020 at 4:44:32 AM UTC-5, evdubs wrote: >>>>>> >>>>>> Have you taken a look at net/url >>>>>> <https://docs.racket-lang.org/net/url.html?q=net%2Furl>? From there, >>>>>> I see: >>>>>> >>>>>> - make-http-connection >>>>>> >>>>>> <https://docs.racket-lang.org/net/url.html?q=net%2Furl#%28def._%28%28lib._net%2Furl..rkt%29._make-http-connection%29%29>, >>>>>> which can allow calls to get-pure-port/headers to stay connected >>>>>> if the server allows it. >>>>>> - current-proxy-servers >>>>>> >>>>>> <https://docs.racket-lang.org/net/url.html?q=net%2Furl#%28def._%28%28lib._net%2Furl..rkt%29._current-proxy-servers%29%29> >>>>>> and current-no-proxy-servers, which show they respect http_proxy, >>>>>> https_proxy, and no_proxy. >>>>>> >>>>>> There are also get-pure-port >>>>>> <https://docs.racket-lang.org/net/url.html?q=net%2Furl#%28def._%28%28lib._net%2Furl..rkt%29._get-pure-port%2Fheaders%29%29> >>>>>> related functions that allow setting headers, which I think should let >>>>>> you >>>>>> set your cookies? For JSON, you can take look at the json >>>>>> <https://docs.racket-lang.org/json/index.html?q=json> docs if you're >>>>>> interested in converting the HTTP response bodies into a JSON expression. >>>>>> These expressions can then be treated like any other expression where you >>>>>> can map/filter/fold over lists and access map elements with hash table >>>>>> functions. >>>>>> >>>>>> To find this stuff, I just searched for "http" and "json" using the >>>>>> ". . . search manuals . . ." box at the top right of the docs. The search >>>>>> looks through all of the Racket API as well as many user-contributed >>>>>> packages. It makes it very easy to find whatever you might be interested >>>>>> in. >>>>>> >>>>>> Evan >>>>>> >>>>>> On Saturday, May 16, 2020 at 7:45:08 AM UTC-10, fixpoint wrote: >>>>>>> >>>>>>> I'm on the search for a new programming language to learn, so I >>>>>>> thought I'd check out Racket, but I'm having a hard time trying to do a >>>>>>> very common task that would make Racket practical for my use at work: >>>>>>> >>>>>>> - Make a series of HTTP requests to an API that returns JSON >>>>>>> responses >>>>>>> - Reuse the HTTP connection to avoid creating new TCP >>>>>>> connections for each request >>>>>>> - Honors the `http_proxy`, `https_proxy`, and >>>>>>> `no_proxy` environment variables >>>>>>> - Maintains a "session" where cookies set by the server are sent >>>>>>> back in subsequent requests >>>>>>> >>>>>>> My daily driver is Python, so I'm used to its `requests` library. Go >>>>>>> and Rust have similar libraries. While I don't mind piecing some of this >>>>>>> together, I'm struggling as a new user to figure out what libraries are >>>>>>> recommended and, more importantly, how to put them together to >>>>>>> accomplish >>>>>>> the above points. >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> p.s. The Racket documentation is by far the best looking >>>>>>> documentation I've read in any language. It's quite amazing! >>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Racket Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to racket...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/racket-users/f71ad314-677f-4f42-ae7c-9e738d88eded%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/racket-users/f71ad314-677f-4f42-ae7c-9e738d88eded%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> -- >>>> Jens Axel Søgaard >>>> >>>> -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to racket-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/racket-users/21032250-5dc7-47eb-8e81-f4c2f6b9e52d%40googlegroups.com > <https://groups.google.com/d/msgid/racket-users/21032250-5dc7-47eb-8e81-f4c2f6b9e52d%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/CADcueguVa_zR3XcRBqKt7668cmNFG%2BX82Rh5UKuJE_qrAmknog%40mail.gmail.com.
