Also look at Mechanize, which I believe can do a headless JS scrape of a site. It's purely a scraper, so less likely to be so test-centric.
Walter > On Jun 19, 2017, at 10:18 AM, Jason Fleetwood-Boldt <[email protected]> > wrote: > > > > I think he's scraping someone else's site. > > You obviously can't do this with Ruby alone, as there is no headless web > browser written entirely in Ruby (that's just nonsense) > > If you can get phantomjs working on your production site, that's probably the > way to go. Look deep into the internals of Capybara to understand how it > drives phantomjs. With phantomjs, you basically have a headless web browser > and you can use Capybara's DSL to access parts of the page, including > evaluating scripts and parsing the DOM. > > Just keep in mind phantomjs is an actual executable so it needs to be > compiled and built for your production environment explicitly, which might be > a little tricky depending on where your site is. > > But a little birdie told me a few months ago that the phantomjs team has > decided that once Chrome has a headless mode, which I believe is forthcoming, > they plan to abandon phantomjs in favor of Chrome's headless mode. Not sure > if that's really true or when that will happen. > > -Jason > > > >> On Jun 18, 2017, at 8:36 AM, Colin Law <[email protected]> wrote: >> >> On 18 June 2017 at 12:21, Martin Luy <[email protected]> wrote: >>> I'm extracting content from some websites. Currently I evaluate HTML code >>> using Nokogiri. But the relevant content is not contained in the responded >>> body of the HTTP GET request. This is because there is some Javascript code >>> like $(window).load() or $(document).ready() that will send some Ajax >>> requests and fill the original HTML code. >>> >>> So I'm searching for some library that automatically executes Javascript >>> code and Ajax requests just like a normal browser. >> >> Understood. Don't think I can help I am afraid. Does the site not work >> with js disabled in the browser? >> >> Colin >> >>> >>> Martin >>> >>> Von meinem Samsung Gerät gesendet. >>> >>> >>> -------- Ursprüngliche Nachricht -------- >>> Von: Colin Law <[email protected]> >>> Datum: 18.06.17 09:42 (GMT+01:00) >>> An: "Ruby on Rails: Talk" <[email protected]> >>> Betreff: Re: [Rails] browser simulator independent of web framework >>> >>> On 17 June 2017 at 22:58, Martin L. <[email protected]> wrote: >>>> Hi all, >>>> >>>> Is there any browser simulator that fulfills these requirements: >>>> >>>> - gem written in Ruby >>>> - automatically performing Ajax and Javascript code (XSS is not an issue >>>> in >>>> my case) >>>> - independent of the frameworks used by the website (Rails, JavaEE, >>>> ASP.NET, >>>> ...) >>>> - only client-side >>>> - no testing >>>> - no browser dependency >>> >>> What do you mean 'no testing'? If not for testing then what is it for? >>> >>> Colin >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Ruby on Rails: Talk" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/rubyonrails-talk/H_YImOIzNNo/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/rubyonrails-talk/CAL%3D0gLvqHArcWpbZ5gsfCiPg0EF%3D4kD8QzZbC3KnicN58uAZ8A%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Ruby on Rails: Talk" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/rubyonrails-talk/smyckbsskql3om4h9odf01hm.1497784510166%40email.android.com. >>> >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby on Rails: Talk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/rubyonrails-talk/CAL%3D0gLsaS8DzC2o0%3DHkKt_%2BMvvzqutm0mSHUj5jf7CoKvnqJEg%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. >> > > ---- > > Jason Fleetwood-Boldt > [email protected] > http://www.jasonfleetwoodboldt.com/writing > > If you'd like to reply by encrypted email you can find my public key on > jasonfleetwoodboldt.com (more about setting GPG: https://gpgtools.org) > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Talk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/rubyonrails-talk/A0A534AE-3ECA-41F9-9170-432A98DD743D%40datatravels.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/7E161DC2-A4E6-4A12-8C13-1FE180EE57CB%40wdstudio.com. For more options, visit https://groups.google.com/d/optout.
