Re: How can I scrape text from a web page that is generated by javascript?

[Roger Eller via use-livecode]

Indeed.  In fact, Thierry Douez solved this over the weekend with a mix of
that and some regex.


On Sat, Jul 29, 2017 at 3:56 PM, Sannyasin Brahmanathaswami via
use-livecode  wrote:

> FYI depending on what you are trying to get out you can also
>
> get the htmltext of widget "browser"; set the htmltext of templateField();
> put the text of templateField()
>
> and see if that may serve in some use case.
>
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Sannyasin Brahmanathaswami via use-livecode]

FYI depending on what you are trying to get out you can also 

get the htmltext of widget "browser"; set the htmltext of templateField(); put 
the text of templateField()

and see if that may serve in some use case.


 

On 7/24/17, 12:41 PM, "use-livecode on behalf of Mike Bonner via use-livecode" 
 wrote:

Something like autoit can probably be set up to automate things.  Set up an
autoit script that does the scrape, then use os scheduling tools to run the
script as often as you like.  I haven't done it in windows in a while, but
I think "at" is deprecated.  Can't recall whats replaced it.

On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode <
use-livecode@lists.runrev.com> wrote:

> So... I can make it scrape, then save to a CSV, then read the CSV into LC
> and use the data.  But how to automate the scraping and CSV saving at a
> given interval, I'm not there yet.
>
> ~Roger
>
>
> On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
> > The developer was also helpful when I was running into difficulty 
writing
> > some of the parameter sets.
> >
> > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> > use-livecode@lists.runrev.com> wrote:
> >
> > > Scratch that.  It's amazing what watching the video or reading the 
docs
> > can
> > > do to enlighten the unenlightened.
> > >
> > >
> > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> > roger.e.el...@sealedair.com>
> > > wrote:
> > >
> > > > I installed the plugin, but am unfamiliar with how to use it.  I
> didn't
> > > > see an obvious way to filter the page down to the current prices I
> want
> > > to
> > > > grab.
> > > >
> > > > ~Roger
> > > >
> > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > > use-livecode@lists.runrev.com> wrote:
> > > >
> > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> > at a
> > > >> time from multiple sites with this plugin.
> > > >>
> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > > >> use-livecode@lists.runrev.com> wrote:
> > > >>
> > > >> > No workie. I create a new sitemap with a URL of
> http://192.168.1.87
> > > and
> > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when 
I
> > > >> > encounter that kind of problem. Even if I load the full url
> default
> > > page
> > > >> > for that device, then copy paste the URL it STILL tells me it's
> > > invalid.
> > > >> >
> > > >> > Bob S
> > > >> >
> > > >> >
> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > > >> > use-livecode@lists.runrev.com> wrote:
> > > >> > >
> > > >> > > Roger,
> > > >> > > I use a chrome plugin called "Web Scraper", and then I process
> the
> > > >> CSV in
> > > >> > > LC - then again, I'm usually scraping many many many pages, and
> > this
> > > >> > > automates the process.
> > > >> > > ___
> > > >> > > use-livecode mailing list
> > > >> > > use-livecode@lists.runrev.com
> > > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >> >
> > > >> > ___
> > > >> > use-livecode mailing list
> > > >> > use-livecode@lists.runrev.com
> > > >> > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> On the first day, God created the heavens and the Earth
> > > >> On the second day, God created the oceans.
> > > >> On the third day, God put the animals on hold for a few hours,
> > > >>and did a little diving.
> > > >> And God said, "This is good."
> > > >> ___
> > > >> use-livecode mailing list
> > > >> use-livecode@lists.runrev.com
> > > >> Please visit this url to subscribe, unsubscribe and manage your
> > > >> subscription preferences:
> > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >>
> > > >
> > > >
> > > ___
> > > use-livecode mailing list
> > > use-livecode@lists.runrev.com
> > > Please visit this url to subscribe, unsubscribe and manage your
> > > subscription preferences:
> > > 

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Bonner via use-livecode]

Something like autoit can probably be set up to automate things.  Set up an
autoit script that does the scrape, then use os scheduling tools to run the
script as often as you like.  I haven't done it in windows in a while, but
I think "at" is deprecated.  Can't recall whats replaced it.

On Mon, Jul 24, 2017 at 3:03 PM, Roger Eller via use-livecode <
use-livecode@lists.runrev.com> wrote:

> So... I can make it scrape, then save to a CSV, then read the CSV into LC
> and use the data.  But how to automate the scraping and CSV saving at a
> given interval, I'm not there yet.
>
> ~Roger
>
>
> On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
> > The developer was also helpful when I was running into difficulty writing
> > some of the parameter sets.
> >
> > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> > use-livecode@lists.runrev.com> wrote:
> >
> > > Scratch that.  It's amazing what watching the video or reading the docs
> > can
> > > do to enlighten the unenlightened.
> > >
> > >
> > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> > roger.e.el...@sealedair.com>
> > > wrote:
> > >
> > > > I installed the plugin, but am unfamiliar with how to use it.  I
> didn't
> > > > see an obvious way to filter the page down to the current prices I
> want
> > > to
> > > > grab.
> > > >
> > > > ~Roger
> > > >
> > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > > use-livecode@lists.runrev.com> wrote:
> > > >
> > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> > at a
> > > >> time from multiple sites with this plugin.
> > > >>
> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > > >> use-livecode@lists.runrev.com> wrote:
> > > >>
> > > >> > No workie. I create a new sitemap with a URL of
> http://192.168.1.87
> > > and
> > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > > >> > encounter that kind of problem. Even if I load the full url
> default
> > > page
> > > >> > for that device, then copy paste the URL it STILL tells me it's
> > > invalid.
> > > >> >
> > > >> > Bob S
> > > >> >
> > > >> >
> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > > >> > use-livecode@lists.runrev.com> wrote:
> > > >> > >
> > > >> > > Roger,
> > > >> > > I use a chrome plugin called "Web Scraper", and then I process
> the
> > > >> CSV in
> > > >> > > LC - then again, I'm usually scraping many many many pages, and
> > this
> > > >> > > automates the process.
> > > >> > > ___
> > > >> > > use-livecode mailing list
> > > >> > > use-livecode@lists.runrev.com
> > > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >> >
> > > >> > ___
> > > >> > use-livecode mailing list
> > > >> > use-livecode@lists.runrev.com
> > > >> > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> On the first day, God created the heavens and the Earth
> > > >> On the second day, God created the oceans.
> > > >> On the third day, God put the animals on hold for a few hours,
> > > >>and did a little diving.
> > > >> And God said, "This is good."
> > > >> ___
> > > >> use-livecode mailing list
> > > >> use-livecode@lists.runrev.com
> > > >> Please visit this url to subscribe, unsubscribe and manage your
> > > >> subscription preferences:
> > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >>
> > > >
> > > >
> > > ___
> > > use-livecode mailing list
> > > use-livecode@lists.runrev.com
> > > Please visit this url to subscribe, unsubscribe and manage your
> > > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >
> >
> >
> >
> > --
> > On the first day, God created the heavens and the Earth
> > On the second day, God created the oceans.
> > On the third day, God put the animals on hold for a few hours,
> >and did a little diving.
> > And God said, "This is good."
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Kerner via use-livecode]

I always start my scrapes manually, so I am no help there with this
plugin.  They all take many hours, so I'm not starting them constantly.

On Mon, Jul 24, 2017 at 5:03 PM, Roger Eller via use-livecode <
use-livecode@lists.runrev.com> wrote:

> So... I can make it scrape, then save to a CSV, then read the CSV into LC
> and use the data.  But how to automate the scraping and CSV saving at a
> given interval, I'm not there yet.
>
> ~Roger
>
>
> On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
> > The developer was also helpful when I was running into difficulty writing
> > some of the parameter sets.
> >
> > On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> > use-livecode@lists.runrev.com> wrote:
> >
> > > Scratch that.  It's amazing what watching the video or reading the docs
> > can
> > > do to enlighten the unenlightened.
> > >
> > >
> > > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> > roger.e.el...@sealedair.com>
> > > wrote:
> > >
> > > > I installed the plugin, but am unfamiliar with how to use it.  I
> didn't
> > > > see an obvious way to filter the page down to the current prices I
> want
> > > to
> > > > grab.
> > > >
> > > > ~Roger
> > > >
> > > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > > use-livecode@lists.runrev.com> wrote:
> > > >
> > > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> > at a
> > > >> time from multiple sites with this plugin.
> > > >>
> > > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > > >> use-livecode@lists.runrev.com> wrote:
> > > >>
> > > >> > No workie. I create a new sitemap with a URL of
> http://192.168.1.87
> > > and
> > > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > > >> > encounter that kind of problem. Even if I load the full url
> default
> > > page
> > > >> > for that device, then copy paste the URL it STILL tells me it's
> > > invalid.
> > > >> >
> > > >> > Bob S
> > > >> >
> > > >> >
> > > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > > >> > use-livecode@lists.runrev.com> wrote:
> > > >> > >
> > > >> > > Roger,
> > > >> > > I use a chrome plugin called "Web Scraper", and then I process
> the
> > > >> CSV in
> > > >> > > LC - then again, I'm usually scraping many many many pages, and
> > this
> > > >> > > automates the process.
> > > >> > > ___
> > > >> > > use-livecode mailing list
> > > >> > > use-livecode@lists.runrev.com
> > > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >> >
> > > >> > ___
> > > >> > use-livecode mailing list
> > > >> > use-livecode@lists.runrev.com
> > > >> > Please visit this url to subscribe, unsubscribe and manage your
> > > >> > subscription preferences:
> > > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> On the first day, God created the heavens and the Earth
> > > >> On the second day, God created the oceans.
> > > >> On the third day, God put the animals on hold for a few hours,
> > > >>and did a little diving.
> > > >> And God said, "This is good."
> > > >> ___
> > > >> use-livecode mailing list
> > > >> use-livecode@lists.runrev.com
> > > >> Please visit this url to subscribe, unsubscribe and manage your
> > > >> subscription preferences:
> > > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > > >>
> > > >
> > > >
> > > ___
> > > use-livecode mailing list
> > > use-livecode@lists.runrev.com
> > > Please visit this url to subscribe, unsubscribe and manage your
> > > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >
> >
> >
> >
> > --
> > On the first day, God created the heavens and the Earth
> > On the second day, God created the oceans.
> > On the third day, God put the animals on hold for a few hours,
> >and did a little diving.
> > And God said, "This is good."
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



-- 
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a 

Re: How can I scrape text from a web page that is generated by javascript?

[Roger Eller via use-livecode]

So... I can make it scrape, then save to a CSV, then read the CSV into LC
and use the data.  But how to automate the scraping and CSV saving at a
given interval, I'm not there yet.

~Roger


On Mon, Jul 24, 2017 at 4:53 PM, Mike Kerner via use-livecode <
use-livecode@lists.runrev.com> wrote:

> The developer was also helpful when I was running into difficulty writing
> some of the parameter sets.
>
> On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
> > Scratch that.  It's amazing what watching the video or reading the docs
> can
> > do to enlighten the unenlightened.
> >
> >
> > On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller <
> roger.e.el...@sealedair.com>
> > wrote:
> >
> > > I installed the plugin, but am unfamiliar with how to use it.  I didn't
> > > see an obvious way to filter the page down to the current prices I want
> > to
> > > grab.
> > >
> > > ~Roger
> > >
> > > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > > use-livecode@lists.runrev.com> wrote:
> > >
> > >> Don't know what to tell you.  I scrape a few hundred thousand pages
> at a
> > >> time from multiple sites with this plugin.
> > >>
> > >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> > >> use-livecode@lists.runrev.com> wrote:
> > >>
> > >> > No workie. I create a new sitemap with a URL of http://192.168.1.87
> > and
> > >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > >> > encounter that kind of problem. Even if I load the full url default
> > page
> > >> > for that device, then copy paste the URL it STILL tells me it's
> > invalid.
> > >> >
> > >> > Bob S
> > >> >
> > >> >
> > >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > >> > use-livecode@lists.runrev.com> wrote:
> > >> > >
> > >> > > Roger,
> > >> > > I use a chrome plugin called "Web Scraper", and then I process the
> > >> CSV in
> > >> > > LC - then again, I'm usually scraping many many many pages, and
> this
> > >> > > automates the process.
> > >> > > ___
> > >> > > use-livecode mailing list
> > >> > > use-livecode@lists.runrev.com
> > >> > > Please visit this url to subscribe, unsubscribe and manage your
> > >> > subscription preferences:
> > >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >> >
> > >> >
> > >> > ___
> > >> > use-livecode mailing list
> > >> > use-livecode@lists.runrev.com
> > >> > Please visit this url to subscribe, unsubscribe and manage your
> > >> > subscription preferences:
> > >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> On the first day, God created the heavens and the Earth
> > >> On the second day, God created the oceans.
> > >> On the third day, God put the animals on hold for a few hours,
> > >>and did a little diving.
> > >> And God said, "This is good."
> > >> ___
> > >> use-livecode mailing list
> > >> use-livecode@lists.runrev.com
> > >> Please visit this url to subscribe, unsubscribe and manage your
> > >> subscription preferences:
> > >> http://lists.runrev.com/mailman/listinfo/use-livecode
> > >>
> > >
> > >
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
>
>
>
> --
> On the first day, God created the heavens and the Earth
> On the second day, God created the oceans.
> On the third day, God put the animals on hold for a few hours,
>and did a little diving.
> And God said, "This is good."
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Kerner via use-livecode]

The developer was also helpful when I was running into difficulty writing
some of the parameter sets.

On Mon, Jul 24, 2017 at 4:27 PM, Roger Eller via use-livecode <
use-livecode@lists.runrev.com> wrote:

> Scratch that.  It's amazing what watching the video or reading the docs can
> do to enlighten the unenlightened.
>
>
> On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller 
> wrote:
>
> > I installed the plugin, but am unfamiliar with how to use it.  I didn't
> > see an obvious way to filter the page down to the current prices I want
> to
> > grab.
> >
> > ~Roger
> >
> > On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> > use-livecode@lists.runrev.com> wrote:
> >
> >> Don't know what to tell you.  I scrape a few hundred thousand pages at a
> >> time from multiple sites with this plugin.
> >>
> >> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> >> use-livecode@lists.runrev.com> wrote:
> >>
> >> > No workie. I create a new sitemap with a URL of http://192.168.1.87
> and
> >> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> >> > encounter that kind of problem. Even if I load the full url default
> page
> >> > for that device, then copy paste the URL it STILL tells me it's
> invalid.
> >> >
> >> > Bob S
> >> >
> >> >
> >> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> >> > use-livecode@lists.runrev.com> wrote:
> >> > >
> >> > > Roger,
> >> > > I use a chrome plugin called "Web Scraper", and then I process the
> >> CSV in
> >> > > LC - then again, I'm usually scraping many many many pages, and this
> >> > > automates the process.
> >> > > ___
> >> > > use-livecode mailing list
> >> > > use-livecode@lists.runrev.com
> >> > > Please visit this url to subscribe, unsubscribe and manage your
> >> > subscription preferences:
> >> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> >> >
> >> >
> >> > ___
> >> > use-livecode mailing list
> >> > use-livecode@lists.runrev.com
> >> > Please visit this url to subscribe, unsubscribe and manage your
> >> > subscription preferences:
> >> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >> >
> >>
> >>
> >>
> >> --
> >> On the first day, God created the heavens and the Earth
> >> On the second day, God created the oceans.
> >> On the third day, God put the animals on hold for a few hours,
> >>and did a little diving.
> >> And God said, "This is good."
> >> ___
> >> use-livecode mailing list
> >> use-livecode@lists.runrev.com
> >> Please visit this url to subscribe, unsubscribe and manage your
> >> subscription preferences:
> >> http://lists.runrev.com/mailman/listinfo/use-livecode
> >>
> >
> >
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



-- 
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Roger Eller via use-livecode]

Scratch that.  It's amazing what watching the video or reading the docs can
do to enlighten the unenlightened.


On Mon, Jul 24, 2017 at 2:59 PM, Roger Eller 
wrote:

> I installed the plugin, but am unfamiliar with how to use it.  I didn't
> see an obvious way to filter the page down to the current prices I want to
> grab.
>
> ~Roger
>
> On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
>> Don't know what to tell you.  I scrape a few hundred thousand pages at a
>> time from multiple sites with this plugin.
>>
>> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
>> use-livecode@lists.runrev.com> wrote:
>>
>> > No workie. I create a new sitemap with a URL of http://192.168.1.87 and
>> > it tells me it's not a valid URL!!! That is strike 3 for me when I
>> > encounter that kind of problem. Even if I load the full url default page
>> > for that device, then copy paste the URL it STILL tells me it's invalid.
>> >
>> > Bob S
>> >
>> >
>> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
>> > use-livecode@lists.runrev.com> wrote:
>> > >
>> > > Roger,
>> > > I use a chrome plugin called "Web Scraper", and then I process the
>> CSV in
>> > > LC - then again, I'm usually scraping many many many pages, and this
>> > > automates the process.
>> > > ___
>> > > use-livecode mailing list
>> > > use-livecode@lists.runrev.com
>> > > Please visit this url to subscribe, unsubscribe and manage your
>> > subscription preferences:
>> > > http://lists.runrev.com/mailman/listinfo/use-livecode
>> >
>> >
>> > ___
>> > use-livecode mailing list
>> > use-livecode@lists.runrev.com
>> > Please visit this url to subscribe, unsubscribe and manage your
>> > subscription preferences:
>> > http://lists.runrev.com/mailman/listinfo/use-livecode
>> >
>>
>>
>>
>> --
>> On the first day, God created the heavens and the Earth
>> On the second day, God created the oceans.
>> On the third day, God put the animals on hold for a few hours,
>>and did a little diving.
>> And God said, "This is good."
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Roger Eller via use-livecode]

I installed the plugin, but am unfamiliar with how to use it.  I didn't see
an obvious way to filter the page down to the current prices I want to grab.

~Roger

On Mon, Jul 24, 2017 at 1:34 PM, Mike Kerner via use-livecode <
use-livecode@lists.runrev.com> wrote:

> Don't know what to tell you.  I scrape a few hundred thousand pages at a
> time from multiple sites with this plugin.
>
> On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
> > No workie. I create a new sitemap with a URL of http://192.168.1.87 and
> > it tells me it's not a valid URL!!! That is strike 3 for me when I
> > encounter that kind of problem. Even if I load the full url default page
> > for that device, then copy paste the URL it STILL tells me it's invalid.
> >
> > Bob S
> >
> >
> > > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> > use-livecode@lists.runrev.com> wrote:
> > >
> > > Roger,
> > > I use a chrome plugin called "Web Scraper", and then I process the CSV
> in
> > > LC - then again, I'm usually scraping many many many pages, and this
> > > automates the process.
> > > ___
> > > use-livecode mailing list
> > > use-livecode@lists.runrev.com
> > > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
> >
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
> >
>
>
>
> --
> On the first day, God created the heavens and the Earth
> On the second day, God created the oceans.
> On the third day, God put the animals on hold for a few hours,
>and did a little diving.
> And God said, "This is good."
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Kerner via use-livecode]

Don't know what to tell you.  I scrape a few hundred thousand pages at a
time from multiple sites with this plugin.

On Mon, Jul 24, 2017 at 12:12 PM, Bob Sneidar via use-livecode <
use-livecode@lists.runrev.com> wrote:

> No workie. I create a new sitemap with a URL of http://192.168.1.87 and
> it tells me it's not a valid URL!!! That is strike 3 for me when I
> encounter that kind of problem. Even if I load the full url default page
> for that device, then copy paste the URL it STILL tells me it's invalid.
>
> Bob S
>
>
> > On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode <
> use-livecode@lists.runrev.com> wrote:
> >
> > Roger,
> > I use a chrome plugin called "Web Scraper", and then I process the CSV in
> > LC - then again, I'm usually scraping many many many pages, and this
> > automates the process.
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>



-- 
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Bob Sneidar via use-livecode]

No workie. I create a new sitemap with a URL of http://192.168.1.87 and it 
tells me it's not a valid URL!!! That is strike 3 for me when I encounter that 
kind of problem. Even if I load the full url default page for that device, then 
copy paste the URL it STILL tells me it's invalid. 

Bob S


> On Jul 24, 2017, at 06:13 , Mike Kerner via use-livecode 
>  wrote:
> 
> Roger,
> I use a chrome plugin called "Web Scraper", and then I process the CSV in
> LC - then again, I'm usually scraping many many many pages, and this
> automates the process.
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Bob Sneidar via use-livecode]

I have tried this before and it does not work without getting into Javascript. 
The contents of a java object are not revealed in the htmltext. I have a really 
great application for this if I could get it going. I would love to write an 
app  that gathered all the pertinent data from a web managed device (like the 
copiers I work with day to day) and save it to some kind of file so it could be 
one-click restored at some later point (like when we have to replace the hard 
drives). 

Bob S


> On Jul 24, 2017, at 06:23 , hh via use-livecode 
>  wrote:
> 
>> Roger E. wrote:
>> Is there a browser widget command that will do it?
> 
> (Yet another option):
> 
> YES.
> 
> Set the URL of a browser widget to https://www.coinbase.com/charts
> and get (via a timer) the htmltext of the browser widget. This is the
> _generated_ source.
> 
> You can now search and find the needed offsets of the tags.
> OR
> Get the needed class-elements via javascript (more lengthy).
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Roger Eller via use-livecode]

Mike, Mike, hh,

So many good ideas!  Thanks guys!

~Roger


On Mon, Jul 24, 2017 at 9:23 AM, hh via use-livecode <
use-livecode@lists.runrev.com> wrote:

> > Roger E. wrote:
> > Is there a browser widget command that will do it?
>
> (Yet another option):
>
> YES.
>
> Set the URL of a browser widget to https://www.coinbase.com/charts
> and get (via a timer) the htmltext of the browser widget. This is the
> _generated_ source.
>
> You can now search and find the needed offsets of the tags.
> OR
> Get the needed class-elements via javascript (more lengthy).
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[hh via use-livecode]

> Roger E. wrote:
> Is there a browser widget command that will do it?

(Yet another option):

YES.

Set the URL of a browser widget to https://www.coinbase.com/charts
and get (via a timer) the htmltext of the browser widget. This is the
_generated_ source.

You can now search and find the needed offsets of the tags.
OR
Get the needed class-elements via javascript (more lengthy).

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Kerner via use-livecode]

Roger,
I use a chrome plugin called "Web Scraper", and then I process the CSV in
LC - then again, I'm usually scraping many many many pages, and this
automates the process.
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Bonner via use-livecode]

Ah, nevermind. Only works for spot, but if you look at the api and get an
authentication key, then you should be able to request all kinds of
information through oath2

On Mon, Jul 24, 2017 at 7:05 AM, Mike Bonner  wrote:

> If you need buy, sell and spot prices you can use this..
> -- spot price
> get url https://api.coinbase.com/v2/prices/spot?currency=USD
> -- buy price
> get url https://api.coinbase.com/v2/prices/buy?currency=USD
> -- sell price
> get url https://api.coinbase.com/v2/prices/sell?currency=USD
>
> Data is returned as json
>
> On Mon, Jul 24, 2017 at 6:48 AM, Roger Eller via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
>>  There are three values at the top of this page that are updated by
>> javascript.  The $ amounts are not visible in the source.
>>
>> https://www.coinbase.com/charts
>>
>> What is the best approach to grab the text using LiveCode.  Is there a
>> browser widget command that will do it?
>>
>> Thanks.
>> ~Roger
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: How can I scrape text from a web page that is generated by javascript?

[Mike Bonner via use-livecode]

If you need buy, sell and spot prices you can use this..
-- spot price
get url https://api.coinbase.com/v2/prices/spot?currency=USD
-- buy price
get url https://api.coinbase.com/v2/prices/buy?currency=USD
-- sell price
get url https://api.coinbase.com/v2/prices/sell?currency=USD

Data is returned as json

On Mon, Jul 24, 2017 at 6:48 AM, Roger Eller via use-livecode <
use-livecode@lists.runrev.com> wrote:

>  There are three values at the top of this page that are updated by
> javascript.  The $ amounts are not visible in the source.
>
> https://www.coinbase.com/charts
>
> What is the best approach to grab the text using LiveCode.  Is there a
> browser widget command that will do it?
>
> Thanks.
> ~Roger
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

How can I scrape text from a web page that is generated by javascript?

[Roger Eller via use-livecode]

 There are three values at the top of this page that are updated by
javascript.  The $ amounts are not visible in the source.

https://www.coinbase.com/charts

What is the best approach to grab the text using LiveCode.  Is there a
browser widget command that will do it?

Thanks.
~Roger
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode