Re: How can I scrape text from a web page that is generated by javascript?

2017-07-31 Thread Roger Eller via use-livecode
Indeed. In fact, Thierry Douez solved this over the weekend with a mix of that and some regex. On Sat, Jul 29, 2017 at 3:56 PM, Sannyasin Brahmanathaswami via use-livecode wrote: > FYI depending on what you are trying to get out you can also > > get the htmltext of widget "browser"; set the ht

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-29 Thread Sannyasin Brahmanathaswami via use-livecode
FYI depending on what you are trying to get out you can also get the htmltext of widget "browser"; set the htmltext of templateField(); put the text of templateField() and see if that may serve in some use case. On 7/24/17, 12:41 PM, "use-livecode on behalf of Mike Bonner via use-livecode"

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Bonner via use-livecode
Something like autoit can probably be set up to automate things. Set up an autoit script that does the scrape, then use os scheduling tools to run the script as often as you like. I haven't done it in windows in a while, but I think "at" is deprecated. Can't recall whats replaced it. On Mon, Ju

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Kerner via use-livecode
I always start my scrapes manually, so I am no help there with this plugin. They all take many hours, so I'm not starting them constantly. On Mon, Jul 24, 2017 at 5:03 PM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > So... I can make it scrape, then save to a CSV, then

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Roger Eller via use-livecode
So... I can make it scrape, then save to a CSV, then read the CSV into LC and use the data. But how to automate the scraping and CSV saving at a given interval, I'm not there yet. ~Roger On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode < use-livecode@lists.runrev.com> wrote: > The

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Kerner via use-livecode
The developer was also helpful when I was running into difficulty writing some of the parameter sets. On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > Scratch that. It's amazing what watching the video or reading the docs can > do to enlight

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Roger Eller via use-livecode
Scratch that. It's amazing what watching the video or reading the docs can do to enlighten the unenlightened. On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller wrote: > I installed the plugin, but am unfamiliar with how to use it. I didn't > see an obvious way to filter the page down to the curren

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Roger Eller via use-livecode
I installed the plugin, but am unfamiliar with how to use it. I didn't see an obvious way to filter the page down to the current prices I want to grab. ~Roger On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < use-livecode@lists.runrev.com> wrote: > Don't know what to tell you. I

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Kerner via use-livecode
Don't know what to tell you. I scrape a few hundred thousand pages at a time from multiple sites with this plugin. On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < use-livecode@lists.runrev.com> wrote: > No workie. I create a new sitemap with a URL of http://192.168.1.87 and > it

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Bob Sneidar via use-livecode
No workie. I create a new sitemap with a URL of http://192.168.1.87 and it tells me it's not a valid URL!!! That is strike 3 for me when I encounter that kind of problem. Even if I load the full url default page for that device, then copy paste the URL it STILL tells me it's invalid. Bob S >

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Bob Sneidar via use-livecode
I have tried this before and it does not work without getting into Javascript. The contents of a java object are not revealed in the htmltext. I have a really great application for this if I could get it going. I would love to write an app that gathered all the pertinent data from a web managed

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Roger Eller via use-livecode
Mike, Mike, hh, So many good ideas! Thanks guys! ~Roger On Mon, Jul 24, 2017 at 9:23 AM, hh via use-livecode < use-livecode@lists.runrev.com> wrote: > > Roger E. wrote: > > Is there a browser widget command that will do it? > > (Yet another option): > > YES. > > Set the URL of a browser widge

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread hh via use-livecode
> Roger E. wrote: > Is there a browser widget command that will do it? (Yet another option): YES. Set the URL of a browser widget to https://www.coinbase.com/charts and get (via a timer) the htmltext of the browser widget. This is the _generated_ source. You can now search and find the needed o

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Kerner via use-livecode
Roger, I use a chrome plugin called "Web Scraper", and then I process the CSV in LC - then again, I'm usually scraping many many many pages, and this automates the process. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Bonner via use-livecode
Ah, nevermind. Only works for spot, but if you look at the api and get an authentication key, then you should be able to request all kinds of information through oath2 On Mon, Jul 24, 2017 at 7:05 AM, Mike Bonner wrote: > If you need buy, sell and spot prices you can use this.. > -- spot price >

Re: How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Mike Bonner via use-livecode
If you need buy, sell and spot prices you can use this.. -- spot price get url https://api.coinbase.com/v2/prices/spot?currency=USD -- buy price get url https://api.coinbase.com/v2/prices/buy?currency=USD -- sell price get url https://api.coinbase.com/v2/prices/sell?currency=USD Data is returned a

How can I scrape text from a web page that is generated by javascript?

2017-07-24 Thread Roger Eller via use-livecode
There are three values at the top of this page that are updated by javascript. The $ amounts are not visible in the source. https://www.coinbase.com/charts What is the best approach to grab the text using LiveCode. Is there a browser widget command that will do it? Thanks. ~Roger