Re: How can I scrape text from a web page that is generated by javascript?
Indeed. In fact, Thierry Douez solved this over the weekend with a mix of that and some regex. On Sat, Jul 29, 2017 at 3:56 PM, Sannyasin Brahmanathaswami via use-livecodewrote: > FYI depending on what you are trying to get out you can also > > get the htmltext of widget "browser"; set the htmltext of templateField(); > put the text of templateField() > > and see if that may serve in some use case. > > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
FYI depending on what you are trying to get out you can also get the htmltext of widget "browser"; set the htmltext of templateField(); put the text of templateField() and see if that may serve in some use case. On 7/24/17, 12:41 PM, "use-livecode on behalf of Mike Bonner via use-livecode"wrote: Something like autoit can probably be set up to automate things. Set up an autoit script that does the scrape, then use os scheduling tools to run the script as often as you like. I haven't done it in windows in a while, but I think "at" is deprecated. Can't recall whats replaced it. On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > So... I can make it scrape, then save to a CSV, then read the CSV into LC > and use the data. But how to automate the scraping and CSV saving at a > given interval, I'm not there yet. > > ~Roger > > > On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > The developer was also helpful when I was running into difficulty writing > > some of the parameter sets. > > > > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > > > Scratch that. It's amazing what watching the video or reading the docs > > can > > > do to enlighten the unenlightened. > > > > > > > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller < > > roger.e.el...@sealedair.com> > > > wrote: > > > > > > > I installed the plugin, but am unfamiliar with how to use it. I > didn't > > > > see an obvious way to filter the page down to the current prices I > want > > > to > > > > grab. > > > > > > > > ~Roger > > > > > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < > > > > use-livecode@lists.runrev.com> wrote: > > > > > > > >> Don't know what to tell you. I scrape a few hundred thousand pages > > at a > > > >> time from multiple sites with this plugin. > > > >> > > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < > > > >> use-livecode@lists.runrev.com> wrote: > > > >> > > > >> > No workie. I create a new sitemap with a URL of > http://192.168.1.87 > > > and > > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I > > > >> > encounter that kind of problem. Even if I load the full url > default > > > page > > > >> > for that device, then copy paste the URL it STILL tells me it's > > > invalid. > > > >> > > > > >> > Bob S > > > >> > > > > >> > > > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > > > >> > use-livecode@lists.runrev.com> wrote: > > > >> > > > > > >> > > Roger, > > > >> > > I use a chrome plugin called "Web Scraper", and then I process > the > > > >> CSV in > > > >> > > LC - then again, I'm usually scraping many many many pages, and > > this > > > >> > > automates the process. > > > >> > > ___ > > > >> > > use-livecode mailing list > > > >> > > use-livecode@lists.runrev.com > > > >> > > Please visit this url to subscribe, unsubscribe and manage your > > > >> > subscription preferences: > > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > >> > > > > >> > ___ > > > >> > use-livecode mailing list > > > >> > use-livecode@lists.runrev.com > > > >> > Please visit this url to subscribe, unsubscribe and manage your > > > >> > subscription preferences: > > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> On the first day, God created the heavens and the Earth > > > >> On the second day, God created the oceans. > > > >> On the third day, God put the animals on hold for a few hours, > > > >>and did a little diving. > > > >> And God said, "This is good." > > > >> ___ > > > >> use-livecode mailing list > > > >> use-livecode@lists.runrev.com > > > >> Please visit this url to subscribe, unsubscribe and manage your > > > >> subscription preferences: > > > >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > > > > > > > > ___ > > > use-livecode mailing list > > > use-livecode@lists.runrev.com > > > Please visit this url to subscribe, unsubscribe and manage your > > > subscription preferences: > > >
Re: How can I scrape text from a web page that is generated by javascript?
Something like autoit can probably be set up to automate things. Set up an autoit script that does the scrape, then use os scheduling tools to run the script as often as you like. I haven't done it in windows in a while, but I think "at" is deprecated. Can't recall whats replaced it. On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > So... I can make it scrape, then save to a CSV, then read the CSV into LC > and use the data. But how to automate the scraping and CSV saving at a > given interval, I'm not there yet. > > ~Roger > > > On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > The developer was also helpful when I was running into difficulty writing > > some of the parameter sets. > > > > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > > > Scratch that. It's amazing what watching the video or reading the docs > > can > > > do to enlighten the unenlightened. > > > > > > > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller < > > roger.e.el...@sealedair.com> > > > wrote: > > > > > > > I installed the plugin, but am unfamiliar with how to use it. I > didn't > > > > see an obvious way to filter the page down to the current prices I > want > > > to > > > > grab. > > > > > > > > ~Roger > > > > > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < > > > > use-livecode@lists.runrev.com> wrote: > > > > > > > >> Don't know what to tell you. I scrape a few hundred thousand pages > > at a > > > >> time from multiple sites with this plugin. > > > >> > > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < > > > >> use-livecode@lists.runrev.com> wrote: > > > >> > > > >> > No workie. I create a new sitemap with a URL of > http://192.168.1.87 > > > and > > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I > > > >> > encounter that kind of problem. Even if I load the full url > default > > > page > > > >> > for that device, then copy paste the URL it STILL tells me it's > > > invalid. > > > >> > > > > >> > Bob S > > > >> > > > > >> > > > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > > > >> > use-livecode@lists.runrev.com> wrote: > > > >> > > > > > >> > > Roger, > > > >> > > I use a chrome plugin called "Web Scraper", and then I process > the > > > >> CSV in > > > >> > > LC - then again, I'm usually scraping many many many pages, and > > this > > > >> > > automates the process. > > > >> > > ___ > > > >> > > use-livecode mailing list > > > >> > > use-livecode@lists.runrev.com > > > >> > > Please visit this url to subscribe, unsubscribe and manage your > > > >> > subscription preferences: > > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > >> > > > > >> > ___ > > > >> > use-livecode mailing list > > > >> > use-livecode@lists.runrev.com > > > >> > Please visit this url to subscribe, unsubscribe and manage your > > > >> > subscription preferences: > > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> On the first day, God created the heavens and the Earth > > > >> On the second day, God created the oceans. > > > >> On the third day, God put the animals on hold for a few hours, > > > >>and did a little diving. > > > >> And God said, "This is good." > > > >> ___ > > > >> use-livecode mailing list > > > >> use-livecode@lists.runrev.com > > > >> Please visit this url to subscribe, unsubscribe and manage your > > > >> subscription preferences: > > > >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > > > > > > > > ___ > > > use-livecode mailing list > > > use-livecode@lists.runrev.com > > > Please visit this url to subscribe, unsubscribe and manage your > > > subscription preferences: > > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > > > > > > > -- > > On the first day, God created the heavens and the Earth > > On the second day, God created the oceans. > > On the third day, God put the animals on hold for a few hours, > >and did a little diving. > > And God said, "This is good." > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode >
Re: How can I scrape text from a web page that is generated by javascript?
I always start my scrapes manually, so I am no help there with this plugin. They all take many hours, so I'm not starting them constantly. On Mon, Jul 24, 2017 at 5:03 PM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > So... I can make it scrape, then save to a CSV, then read the CSV into LC > and use the data. But how to automate the scraping and CSV saving at a > given interval, I'm not there yet. > > ~Roger > > > On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > The developer was also helpful when I was running into difficulty writing > > some of the parameter sets. > > > > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > > > Scratch that. It's amazing what watching the video or reading the docs > > can > > > do to enlighten the unenlightened. > > > > > > > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller < > > roger.e.el...@sealedair.com> > > > wrote: > > > > > > > I installed the plugin, but am unfamiliar with how to use it. I > didn't > > > > see an obvious way to filter the page down to the current prices I > want > > > to > > > > grab. > > > > > > > > ~Roger > > > > > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < > > > > use-livecode@lists.runrev.com> wrote: > > > > > > > >> Don't know what to tell you. I scrape a few hundred thousand pages > > at a > > > >> time from multiple sites with this plugin. > > > >> > > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < > > > >> use-livecode@lists.runrev.com> wrote: > > > >> > > > >> > No workie. I create a new sitemap with a URL of > http://192.168.1.87 > > > and > > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I > > > >> > encounter that kind of problem. Even if I load the full url > default > > > page > > > >> > for that device, then copy paste the URL it STILL tells me it's > > > invalid. > > > >> > > > > >> > Bob S > > > >> > > > > >> > > > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > > > >> > use-livecode@lists.runrev.com> wrote: > > > >> > > > > > >> > > Roger, > > > >> > > I use a chrome plugin called "Web Scraper", and then I process > the > > > >> CSV in > > > >> > > LC - then again, I'm usually scraping many many many pages, and > > this > > > >> > > automates the process. > > > >> > > ___ > > > >> > > use-livecode mailing list > > > >> > > use-livecode@lists.runrev.com > > > >> > > Please visit this url to subscribe, unsubscribe and manage your > > > >> > subscription preferences: > > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > >> > > > > >> > ___ > > > >> > use-livecode mailing list > > > >> > use-livecode@lists.runrev.com > > > >> > Please visit this url to subscribe, unsubscribe and manage your > > > >> > subscription preferences: > > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> On the first day, God created the heavens and the Earth > > > >> On the second day, God created the oceans. > > > >> On the third day, God put the animals on hold for a few hours, > > > >>and did a little diving. > > > >> And God said, "This is good." > > > >> ___ > > > >> use-livecode mailing list > > > >> use-livecode@lists.runrev.com > > > >> Please visit this url to subscribe, unsubscribe and manage your > > > >> subscription preferences: > > > >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > >> > > > > > > > > > > > ___ > > > use-livecode mailing list > > > use-livecode@lists.runrev.com > > > Please visit this url to subscribe, unsubscribe and manage your > > > subscription preferences: > > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > > > > > > > -- > > On the first day, God created the heavens and the Earth > > On the second day, God created the oceans. > > On the third day, God put the animals on hold for a few hours, > >and did a little diving. > > And God said, "This is good." > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > -- On the first day, God created the heavens and the Earth On the second day, God created the oceans. On the third day, God put the animals on hold for a few hours, and did a
Re: How can I scrape text from a web page that is generated by javascript?
So... I can make it scrape, then save to a CSV, then read the CSV into LC and use the data. But how to automate the scraping and CSV saving at a given interval, I'm not there yet. ~Roger On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode < use-livecode@lists.runrev.com> wrote: > The developer was also helpful when I was running into difficulty writing > some of the parameter sets. > > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > Scratch that. It's amazing what watching the video or reading the docs > can > > do to enlighten the unenlightened. > > > > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller < > roger.e.el...@sealedair.com> > > wrote: > > > > > I installed the plugin, but am unfamiliar with how to use it. I didn't > > > see an obvious way to filter the page down to the current prices I want > > to > > > grab. > > > > > > ~Roger > > > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < > > > use-livecode@lists.runrev.com> wrote: > > > > > >> Don't know what to tell you. I scrape a few hundred thousand pages > at a > > >> time from multiple sites with this plugin. > > >> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < > > >> use-livecode@lists.runrev.com> wrote: > > >> > > >> > No workie. I create a new sitemap with a URL of http://192.168.1.87 > > and > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I > > >> > encounter that kind of problem. Even if I load the full url default > > page > > >> > for that device, then copy paste the URL it STILL tells me it's > > invalid. > > >> > > > >> > Bob S > > >> > > > >> > > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > > >> > use-livecode@lists.runrev.com> wrote: > > >> > > > > >> > > Roger, > > >> > > I use a chrome plugin called "Web Scraper", and then I process the > > >> CSV in > > >> > > LC - then again, I'm usually scraping many many many pages, and > this > > >> > > automates the process. > > >> > > ___ > > >> > > use-livecode mailing list > > >> > > use-livecode@lists.runrev.com > > >> > > Please visit this url to subscribe, unsubscribe and manage your > > >> > subscription preferences: > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode > > >> > > > >> > > > >> > ___ > > >> > use-livecode mailing list > > >> > use-livecode@lists.runrev.com > > >> > Please visit this url to subscribe, unsubscribe and manage your > > >> > subscription preferences: > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode > > >> > > > >> > > >> > > >> > > >> -- > > >> On the first day, God created the heavens and the Earth > > >> On the second day, God created the oceans. > > >> On the third day, God put the animals on hold for a few hours, > > >>and did a little diving. > > >> And God said, "This is good." > > >> ___ > > >> use-livecode mailing list > > >> use-livecode@lists.runrev.com > > >> Please visit this url to subscribe, unsubscribe and manage your > > >> subscription preferences: > > >> http://lists.runrev.com/mailman/listinfo/use-livecode > > >> > > > > > > > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > > -- > On the first day, God created the heavens and the Earth > On the second day, God created the oceans. > On the third day, God put the animals on hold for a few hours, >and did a little diving. > And God said, "This is good." > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
The developer was also helpful when I was running into difficulty writing some of the parameter sets. On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > Scratch that. It's amazing what watching the video or reading the docs can > do to enlighten the unenlightened. > > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller> wrote: > > > I installed the plugin, but am unfamiliar with how to use it. I didn't > > see an obvious way to filter the page down to the current prices I want > to > > grab. > > > > ~Roger > > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > >> Don't know what to tell you. I scrape a few hundred thousand pages at a > >> time from multiple sites with this plugin. > >> > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < > >> use-livecode@lists.runrev.com> wrote: > >> > >> > No workie. I create a new sitemap with a URL of http://192.168.1.87 > and > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I > >> > encounter that kind of problem. Even if I load the full url default > page > >> > for that device, then copy paste the URL it STILL tells me it's > invalid. > >> > > >> > Bob S > >> > > >> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > >> > use-livecode@lists.runrev.com> wrote: > >> > > > >> > > Roger, > >> > > I use a chrome plugin called "Web Scraper", and then I process the > >> CSV in > >> > > LC - then again, I'm usually scraping many many many pages, and this > >> > > automates the process. > >> > > ___ > >> > > use-livecode mailing list > >> > > use-livecode@lists.runrev.com > >> > > Please visit this url to subscribe, unsubscribe and manage your > >> > subscription preferences: > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode > >> > > >> > > >> > ___ > >> > use-livecode mailing list > >> > use-livecode@lists.runrev.com > >> > Please visit this url to subscribe, unsubscribe and manage your > >> > subscription preferences: > >> > http://lists.runrev.com/mailman/listinfo/use-livecode > >> > > >> > >> > >> > >> -- > >> On the first day, God created the heavens and the Earth > >> On the second day, God created the oceans. > >> On the third day, God put the animals on hold for a few hours, > >>and did a little diving. > >> And God said, "This is good." > >> ___ > >> use-livecode mailing list > >> use-livecode@lists.runrev.com > >> Please visit this url to subscribe, unsubscribe and manage your > >> subscription preferences: > >> http://lists.runrev.com/mailman/listinfo/use-livecode > >> > > > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > -- On the first day, God created the heavens and the Earth On the second day, God created the oceans. On the third day, God put the animals on hold for a few hours, and did a little diving. And God said, "This is good." ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
Scratch that. It's amazing what watching the video or reading the docs can do to enlighten the unenlightened. On Mon, Jul 24, 2017 at 2:59 PM, Roger Ellerwrote: > I installed the plugin, but am unfamiliar with how to use it. I didn't > see an obvious way to filter the page down to the current prices I want to > grab. > > ~Roger > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < > use-livecode@lists.runrev.com> wrote: > >> Don't know what to tell you. I scrape a few hundred thousand pages at a >> time from multiple sites with this plugin. >> >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < >> use-livecode@lists.runrev.com> wrote: >> >> > No workie. I create a new sitemap with a URL of http://192.168.1.87 and >> > it tells me it's not a valid URL!!! That is strike 3 for me when I >> > encounter that kind of problem. Even if I load the full url default page >> > for that device, then copy paste the URL it STILL tells me it's invalid. >> > >> > Bob S >> > >> > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < >> > use-livecode@lists.runrev.com> wrote: >> > > >> > > Roger, >> > > I use a chrome plugin called "Web Scraper", and then I process the >> CSV in >> > > LC - then again, I'm usually scraping many many many pages, and this >> > > automates the process. >> > > ___ >> > > use-livecode mailing list >> > > use-livecode@lists.runrev.com >> > > Please visit this url to subscribe, unsubscribe and manage your >> > subscription preferences: >> > > http://lists.runrev.com/mailman/listinfo/use-livecode >> > >> > >> > ___ >> > use-livecode mailing list >> > use-livecode@lists.runrev.com >> > Please visit this url to subscribe, unsubscribe and manage your >> > subscription preferences: >> > http://lists.runrev.com/mailman/listinfo/use-livecode >> > >> >> >> >> -- >> On the first day, God created the heavens and the Earth >> On the second day, God created the oceans. >> On the third day, God put the animals on hold for a few hours, >>and did a little diving. >> And God said, "This is good." >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your >> subscription preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode >> > > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
I installed the plugin, but am unfamiliar with how to use it. I didn't see an obvious way to filter the page down to the current prices I want to grab. ~Roger On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode < use-livecode@lists.runrev.com> wrote: > Don't know what to tell you. I scrape a few hundred thousand pages at a > time from multiple sites with this plugin. > > On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > No workie. I create a new sitemap with a URL of http://192.168.1.87 and > > it tells me it's not a valid URL!!! That is strike 3 for me when I > > encounter that kind of problem. Even if I load the full url default page > > for that device, then copy paste the URL it STILL tells me it's invalid. > > > > Bob S > > > > > > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > > > > Roger, > > > I use a chrome plugin called "Web Scraper", and then I process the CSV > in > > > LC - then again, I'm usually scraping many many many pages, and this > > > automates the process. > > > ___ > > > use-livecode mailing list > > > use-livecode@lists.runrev.com > > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > > -- > On the first day, God created the heavens and the Earth > On the second day, God created the oceans. > On the third day, God put the animals on hold for a few hours, >and did a little diving. > And God said, "This is good." > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
Don't know what to tell you. I scrape a few hundred thousand pages at a time from multiple sites with this plugin. On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode < use-livecode@lists.runrev.com> wrote: > No workie. I create a new sitemap with a URL of http://192.168.1.87 and > it tells me it's not a valid URL!!! That is strike 3 for me when I > encounter that kind of problem. Even if I load the full url default page > for that device, then copy paste the URL it STILL tells me it's invalid. > > Bob S > > > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > > Roger, > > I use a chrome plugin called "Web Scraper", and then I process the CSV in > > LC - then again, I'm usually scraping many many many pages, and this > > automates the process. > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > -- On the first day, God created the heavens and the Earth On the second day, God created the oceans. On the third day, God put the animals on hold for a few hours, and did a little diving. And God said, "This is good." ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
No workie. I create a new sitemap with a URL of http://192.168.1.87 and it tells me it's not a valid URL!!! That is strike 3 for me when I encounter that kind of problem. Even if I load the full url default page for that device, then copy paste the URL it STILL tells me it's invalid. Bob S > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode >wrote: > > Roger, > I use a chrome plugin called "Web Scraper", and then I process the CSV in > LC - then again, I'm usually scraping many many many pages, and this > automates the process. > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
I have tried this before and it does not work without getting into Javascript. The contents of a java object are not revealed in the htmltext. I have a really great application for this if I could get it going. I would love to write an app that gathered all the pertinent data from a web managed device (like the copiers I work with day to day) and save it to some kind of file so it could be one-click restored at some later point (like when we have to replace the hard drives). Bob S > On Jul 24, 2017, at 06:23 , hh via use-livecode >wrote: > >> Roger E. wrote: >> Is there a browser widget command that will do it? > > (Yet another option): > > YES. > > Set the URL of a browser widget to https://www.coinbase.com/charts > and get (via a timer) the htmltext of the browser widget. This is the > _generated_ source. > > You can now search and find the needed offsets of the tags. > OR > Get the needed class-elements via javascript (more lengthy). > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
Mike, Mike, hh, So many good ideas! Thanks guys! ~Roger On Mon, Jul 24, 2017 at 9:23 AM, hh via use-livecode < use-livecode@lists.runrev.com> wrote: > > Roger E. wrote: > > Is there a browser widget command that will do it? > > (Yet another option): > > YES. > > Set the URL of a browser widget to https://www.coinbase.com/charts > and get (via a timer) the htmltext of the browser widget. This is the > _generated_ source. > > You can now search and find the needed offsets of the tags. > OR > Get the needed class-elements via javascript (more lengthy). > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
> Roger E. wrote: > Is there a browser widget command that will do it? (Yet another option): YES. Set the URL of a browser widget to https://www.coinbase.com/charts and get (via a timer) the htmltext of the browser widget. This is the _generated_ source. You can now search and find the needed offsets of the tags. OR Get the needed class-elements via javascript (more lengthy). ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
Roger, I use a chrome plugin called "Web Scraper", and then I process the CSV in LC - then again, I'm usually scraping many many many pages, and this automates the process. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
Ah, nevermind. Only works for spot, but if you look at the api and get an authentication key, then you should be able to request all kinds of information through oath2 On Mon, Jul 24, 2017 at 7:05 AM, Mike Bonnerwrote: > If you need buy, sell and spot prices you can use this.. > -- spot price > get url https://api.coinbase.com/v2/prices/spot?currency=USD > -- buy price > get url https://api.coinbase.com/v2/prices/buy?currency=USD > -- sell price > get url https://api.coinbase.com/v2/prices/sell?currency=USD > > Data is returned as json > > On Mon, Jul 24, 2017 at 6:48 AM, Roger Eller via use-livecode < > use-livecode@lists.runrev.com> wrote: > >> There are three values at the top of this page that are updated by >> javascript. The $ amounts are not visible in the source. >> >> https://www.coinbase.com/charts >> >> What is the best approach to grab the text using LiveCode. Is there a >> browser widget command that will do it? >> >> Thanks. >> ~Roger >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your >> subscription preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode >> > > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: How can I scrape text from a web page that is generated by javascript?
If you need buy, sell and spot prices you can use this.. -- spot price get url https://api.coinbase.com/v2/prices/spot?currency=USD -- buy price get url https://api.coinbase.com/v2/prices/buy?currency=USD -- sell price get url https://api.coinbase.com/v2/prices/sell?currency=USD Data is returned as json On Mon, Jul 24, 2017 at 6:48 AM, Roger Eller via use-livecode < use-livecode@lists.runrev.com> wrote: > There are three values at the top of this page that are updated by > javascript. The $ amounts are not visible in the source. > > https://www.coinbase.com/charts > > What is the best approach to grab the text using LiveCode. Is there a > browser widget command that will do it? > > Thanks. > ~Roger > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
How can I scrape text from a web page that is generated by javascript?
There are three values at the top of this page that are updated by javascript. The $ amounts are not visible in the source. https://www.coinbase.com/charts What is the best approach to grab the text using LiveCode. Is there a browser widget command that will do it? Thanks. ~Roger ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode