Re: [CentOS] script to make webpage snapshot

[Anthony K]

On 12/08/16 19:55, Anthony K wrote:

On 12/08/16 06:46, Valeri Galtsev wrote:

Dear Experts,

Could someone recommend a script or utility one can run from command 
line

on Linux or UNIX machine to make a snapshot of webpage?

Looks like this *[0]* is what you are after:

CmdShots is a FireFox add-on that takes full-page screenshots through 
the Command-line.



*[0]* https://github.com/omarabid/CmdShots 


For completeness sake, the author first sought a solution on Stack 
Overflow [1].  When one was not forthcoming, he created his own solution.


[1] 
http://stackoverflow.com/questions/13158083/take-a-full-page-screenshot-with-firefox



___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Anthony K]

On 12/08/16 06:46, Valeri Galtsev wrote:

Dear Experts,

Could someone recommend a script or utility one can run from command line
on Linux or UNIX machine to make a snapshot of webpage?

Looks like this *[0]* is what you are after:

CmdShots is a FireFox add-on that takes full-page screenshots through 
the Command-line.



*[0]* https://github.com/omarabid/CmdShots


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Mark Parsons]

First time on this list, sorry if I mess up on something.

I use http://wkhtmltopdf.org/ there is a to image version as well. I
haven't had any trouble with it. I've mostly used on Windows but there are
Linux versions.

On Aug 11, 2016 7:09 PM, "Kahlil Hodgson" 
wrote:

> I have some some angularjs sites that I test with protractor and a
> chrome webdriver. I read in the docs at some point that I could take
> and save screenshots if I wanted. You may be able to write a simple
> nodejs script to kick of the webdriver and take the screenhsot.  Or
> someone may have already writen one :-)
>
> https://www.seleniumeasy.com/selenium-tutorials/take-
> screenshot-with-selenium-webdriver
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Kahlil Hodgson]

I have some some angularjs sites that I test with protractor and a
chrome webdriver. I read in the docs at some point that I could take
and save screenshots if I wanted. You may be able to write a simple
nodejs script to kick of the webdriver and take the screenhsot.  Or
someone may have already writen one :-)

https://www.seleniumeasy.com/selenium-tutorials/take-screenshot-with-selenium-webdriver
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Frank Cox]

On Thu, 11 Aug 2016 17:19:15 -0500 (CDT)
Valeri Galtsev wrote:

> This involves a person sitting with open web browser and clicking mouse. I
> need to do it for a few webpages once a day, as they change daily. I
> prefer to once spend time on making cron job, and forget about it forever.
> Hence I need it done from command line.

html2ps followed by pstoimg and/or imagemagick to create a graphic from the 
postscript.

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Valeri Galtsev]

On Thu, August 11, 2016 5:27 pm, John R Pierce wrote:
> On 8/11/2016 3:10 PM, Valeri Galtsev wrote:
>> I usually am not good at explaining what I need. I really only need an
>> image of what one would see in web browser if one point to that URL. I
>> do
>> not care it to be interactive. I also don't want to get the content
>> ("mirror") of stuff that URL points to on variety of "depths" - I don't
>> want to use wget or curl for this reason. That is what I tried first and
>> it breaks with at lest one of the web sites - they do seem protect
>> themselves from "robots" or similar. And we don't need it. We just need
>> to
>> show what they page shows today, that's all.
>
> then screen capture is about it too many sites, ALL the content is
> dynamic, for instance,
> https://www.google.com/maps/@36.9460899,-122.0268105,664a,20y,41.31t/data=!3m1!1e3
>
> that page is composed of tiles of image data superimposed on the fly
> with ajax code running in the browser to fetch the layers displayed.
>
> you simply can't fetch the html and make any sense out of it, the
> browser is running a complex application to display that.
>

Yes, I understand as much, thanks. I'm still sure it is not hopeless task.

Valeri


Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[John R Pierce]

On 8/11/2016 3:10 PM, Valeri Galtsev wrote:

I usually am not good at explaining what I need. I really only need an
image of what one would see in web browser if one point to that URL. I do
not care it to be interactive. I also don't want to get the content
("mirror") of stuff that URL points to on variety of "depths" - I don't
want to use wget or curl for this reason. That is what I tried first and
it breaks with at lest one of the web sites - they do seem protect
themselves from "robots" or similar. And we don't need it. We just need to
show what they page shows today, that's all.


then screen capture is about it too many sites, ALL the content is 
dynamic, for instance, 
https://www.google.com/maps/@36.9460899,-122.0268105,664a,20y,41.31t/data=!3m1!1e3


that page is composed of tiles of image data superimposed on the fly 
with ajax code running in the browser to fetch the layers displayed.


you simply can't fetch the html and make any sense out of it, the 
browser is running a complex application to display that.



--
john r pierce, recycling bits in santa cruz

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Michael Abbondanza]

Could try https://screenshotlayer.com/

I have had success automating their API.

On Thu, Aug 11, 2016 at 3:19 PM, Valeri Galtsev 
wrote:

>
>
> On Thu, August 11, 2016 5:13 pm, Dave Stevens wrote:
> > Quoting Valeri Galtsev :
> >
> >>
> >> On Thu, August 11, 2016 5:02 pm, John R Pierce wrote:
> >>> On 8/11/2016 1:46 PM, Valeri Galtsev wrote:
>  Could someone recommend a script or utility one can run from command
> line
>  on Linux or UNIX machine to make a snapshot of webpage?
> 
>  We have a signage (xibo) and whoever creates/changes content, likes
> to add
>  URLs of some webpages there. All works well if these are webpages on
> our
>  servers (which are pretty fast), but some external servers often take
> time
>  to respond and take time to assemble the page, in addition these
> servers
>  sometimes get really busy, and when response is longer than time
> devoted
>  for that content in signage window, this window hangs forever with
> blank
>  white field until you restart client. Trivial workaround: just to get
> snapshot (as, say daily cron job), and point signage client to that
> snapshot definitely will solve it, and simultaneously we will stop
> bugging
>  other people servers often without much need for it.
> 
>  But when I tried to search for some utility or script that makes
> webpage
>  snapshot, I discovered that my ability to search degraded somehow...
> >>>
> >>> many/most webpages these days are heavily dynamic content, a static
> snapshot would likely break.  plus any site-relative links on that
> snapshot would be pointing to your server, not the original, any ajax
> code on that webpage would try to interact with your server which
> won't be running the right back end stuff, etcetc.
> >>
> >> I usually am not good at explaining what I need. I really only need an
> image of what one would see in web browser if one point to that URL. I
> do
> >> not care it to be interactive. I also don't want to get the content
> ("mirror") of stuff that URL points to on variety of "depths" - I don't
> want to use wget or curl for this reason. That is what I tried first
> and it breaks with at lest one of the web sites - they do seem protect
> themselves from "robots" or similar. And we don't need it. We just need
> to
> >> show what they page shows today, that's all.
> >>
> >> Valeri
> >
> > why not File -> Print -> .pdf?
>
> This involves a person sitting with open web browser and clicking mouse. I
> need to do it for a few webpages once a day, as they change daily. I
> prefer to once spend time on making cron job, and forget about it forever.
> Hence I need it done from command line.
>
> Valeri
>
> >
> > D
> >
> >>
> >>>
> >>> --
> >>> john r pierce, recycling bits in santa cruz
> >>>
> >>
> >>
> >> 
> >> Valeri Galtsev
> >> Sr System Administrator
> >> Department of Astronomy and Astrophysics
> >> Kavli Institute for Cosmological Physics
> >> University of Chicago
> >> Phone: 773-702-4247
> >> 
> >> ___
> >> CentOS mailing list
> >> CentOS@centos.org
> >> https://lists.centos.org/mailman/listinfo/centos
> >>
> >
> >
> >
> > --
> > "As long as politics is the shadow cast on society by big business, the
> attenuation of the shadow will not change the substance."
> >
> > -- John Dewey
> >
> >
> >
> >
> >
>
>
> 
> Valeri Galtsev
> Sr System Administrator
> Department of Astronomy and Astrophysics
> Kavli Institute for Cosmological Physics
> University of Chicago
> Phone: 773-702-4247
> 
>
> ___
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos
>



-- 

Michael Abbondanza

Technical Support Engineer

408-479-2207


Now Available - Free Hadoop On-Demand Training

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Valeri Galtsev]

On Thu, August 11, 2016 5:13 pm, Dave Stevens wrote:
> Quoting Valeri Galtsev :
>
>>
>> On Thu, August 11, 2016 5:02 pm, John R Pierce wrote:
>>> On 8/11/2016 1:46 PM, Valeri Galtsev wrote:
 Could someone recommend a script or utility one can run from command
line
 on Linux or UNIX machine to make a snapshot of webpage?

 We have a signage (xibo) and whoever creates/changes content, likes
to add
 URLs of some webpages there. All works well if these are webpages on our
 servers (which are pretty fast), but some external servers often take
time
 to respond and take time to assemble the page, in addition these servers
 sometimes get really busy, and when response is longer than time devoted
 for that content in signage window, this window hangs forever with blank
 white field until you restart client. Trivial workaround: just to get
snapshot (as, say daily cron job), and point signage client to that
snapshot definitely will solve it, and simultaneously we will stop
bugging
 other people servers often without much need for it.

 But when I tried to search for some utility or script that makes webpage
 snapshot, I discovered that my ability to search degraded somehow...
>>>
>>> many/most webpages these days are heavily dynamic content, a static
snapshot would likely break.  plus any site-relative links on that
snapshot would be pointing to your server, not the original, any ajax
code on that webpage would try to interact with your server which
won't be running the right back end stuff, etcetc.
>>
>> I usually am not good at explaining what I need. I really only need an
image of what one would see in web browser if one point to that URL. I
do
>> not care it to be interactive. I also don't want to get the content
("mirror") of stuff that URL points to on variety of "depths" - I don't
want to use wget or curl for this reason. That is what I tried first
and it breaks with at lest one of the web sites - they do seem protect
themselves from "robots" or similar. And we don't need it. We just need
to
>> show what they page shows today, that's all.
>>
>> Valeri
>
> why not File -> Print -> .pdf?

This involves a person sitting with open web browser and clicking mouse. I
need to do it for a few webpages once a day, as they change daily. I
prefer to once spend time on making cron job, and forget about it forever.
Hence I need it done from command line.

Valeri

>
> D
>
>>
>>>
>>> --
>>> john r pierce, recycling bits in santa cruz
>>>
>>
>>
>> 
>> Valeri Galtsev
>> Sr System Administrator
>> Department of Astronomy and Astrophysics
>> Kavli Institute for Cosmological Physics
>> University of Chicago
>> Phone: 773-702-4247
>> 
>> ___
>> CentOS mailing list
>> CentOS@centos.org
>> https://lists.centos.org/mailman/listinfo/centos
>>
>
>
>
> --
> "As long as politics is the shadow cast on society by big business, the
attenuation of the shadow will not change the substance."
>
> -- John Dewey
>
>
>
>
>



Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247


___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Valeri Galtsev]

On Thu, August 11, 2016 4:13 pm, Liam O'Toole wrote:
> On 2016-08-11, Valeri Galtsev
>  wrote:
>> Dear Experts,
>>
>> Could someone recommend a script or utility one can run from command
>> line on Linux or UNIX machine to make a snapshot of webpage?
>>
>> We have a signage (xibo) and whoever creates/changes content, likes to
>> add URLs of some webpages there. All works well if these are webpages
>> on our servers (which are pretty fast), but some external servers
>> often take time to respond and take time to assemble the page, in
>> addition these servers sometimes get really busy, and when response is
>> longer than time devoted for that content in signage window, this
>> window hangs forever with blank white field until you restart client.
>> Trivial workaround: just to get snapshot (as, say daily cron job), and
>> point signage client to that snapshot definitely will solve it, and
>> simultaneously we will stop bugging other people servers often without
>> much need for it.
>>
>> But when I tried to search for some utility or script that makes
>> webpage snapshot, I discovered that my ability to search degraded
>> somehow...
>>
>> Thanks for all your pointers!
>>
>> Valeri
>
> Not an answer to the question you asked, but maybe this is a job for a
> caching proxy server like squid?
>

Thanks! It didn't occur to me. It will be much more sophisticated than
just an image "snapshot" of the webpage, but should solve our problem. If
I don't find anything doing "snapshot" successfully, this is what I will
do.

Valeri


Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Dave Stevens]

Quoting Valeri Galtsev :



On Thu, August 11, 2016 5:02 pm, John R Pierce wrote:

On 8/11/2016 1:46 PM, Valeri Galtsev wrote:

Could someone recommend a script or utility one can run from command
line
on Linux or UNIX machine to make a snapshot of webpage?

We have a signage (xibo) and whoever creates/changes content, likes to
add
URLs of some webpages there. All works well if these are webpages on our
servers (which are pretty fast), but some external servers often take
time
to respond and take time to assemble the page, in addition these servers
sometimes get really busy, and when response is longer than time devoted
for that content in signage window, this window hangs forever with blank
white field until you restart client. Trivial workaround: just to get
snapshot (as, say daily cron job), and point signage client to that
snapshot definitely will solve it, and simultaneously we will stop
bugging
other people servers often without much need for it.

But when I tried to search for some utility or script that makes webpage
snapshot, I discovered that my ability to search degraded somehow...


many/most webpages these days are heavily dynamic content, a static
snapshot would likely break.  plus any site-relative links on that
snapshot would be pointing to your server, not the original, any ajax
code on that webpage would try to interact with your server which won't
be running the right back end stuff, etcetc.


I usually am not good at explaining what I need. I really only need an
image of what one would see in web browser if one point to that URL. I do
not care it to be interactive. I also don't want to get the content
("mirror") of stuff that URL points to on variety of "depths" - I don't
want to use wget or curl for this reason. That is what I tried first and
it breaks with at lest one of the web sites - they do seem protect
themselves from "robots" or similar. And we don't need it. We just need to
show what they page shows today, that's all.

Valeri


why not File -> Print -> .pdf?

D





--
john r pierce, recycling bits in santa cruz





Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos





--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."

-- John Dewey





___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Valeri Galtsev]

On Thu, August 11, 2016 5:02 pm, John R Pierce wrote:
> On 8/11/2016 1:46 PM, Valeri Galtsev wrote:
>> Could someone recommend a script or utility one can run from command
>> line
>> on Linux or UNIX machine to make a snapshot of webpage?
>>
>> We have a signage (xibo) and whoever creates/changes content, likes to
>> add
>> URLs of some webpages there. All works well if these are webpages on our
>> servers (which are pretty fast), but some external servers often take
>> time
>> to respond and take time to assemble the page, in addition these servers
>> sometimes get really busy, and when response is longer than time devoted
>> for that content in signage window, this window hangs forever with blank
>> white field until you restart client. Trivial workaround: just to get
>> snapshot (as, say daily cron job), and point signage client to that
>> snapshot definitely will solve it, and simultaneously we will stop
>> bugging
>> other people servers often without much need for it.
>>
>> But when I tried to search for some utility or script that makes webpage
>> snapshot, I discovered that my ability to search degraded somehow...
>
> many/most webpages these days are heavily dynamic content, a static
> snapshot would likely break.  plus any site-relative links on that
> snapshot would be pointing to your server, not the original, any ajax
> code on that webpage would try to interact with your server which won't
> be running the right back end stuff, etcetc.

I usually am not good at explaining what I need. I really only need an
image of what one would see in web browser if one point to that URL. I do
not care it to be interactive. I also don't want to get the content
("mirror") of stuff that URL points to on variety of "depths" - I don't
want to use wget or curl for this reason. That is what I tried first and
it breaks with at lest one of the web sites - they do seem protect
themselves from "robots" or similar. And we don't need it. We just need to
show what they page shows today, that's all.

Valeri

>
> --
> john r pierce, recycling bits in santa cruz
>



Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[John R Pierce]

On 8/11/2016 1:46 PM, Valeri Galtsev wrote:

Could someone recommend a script or utility one can run from command line
on Linux or UNIX machine to make a snapshot of webpage?

We have a signage (xibo) and whoever creates/changes content, likes to add
URLs of some webpages there. All works well if these are webpages on our
servers (which are pretty fast), but some external servers often take time
to respond and take time to assemble the page, in addition these servers
sometimes get really busy, and when response is longer than time devoted
for that content in signage window, this window hangs forever with blank
white field until you restart client. Trivial workaround: just to get
snapshot (as, say daily cron job), and point signage client to that
snapshot definitely will solve it, and simultaneously we will stop bugging
other people servers often without much need for it.

But when I tried to search for some utility or script that makes webpage
snapshot, I discovered that my ability to search degraded somehow...


many/most webpages these days are heavily dynamic content, a static 
snapshot would likely break.  plus any site-relative links on that 
snapshot would be pointing to your server, not the original, any ajax 
code on that webpage would try to interact with your server which won't 
be running the right back end stuff, etcetc.


--
john r pierce, recycling bits in santa cruz

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Liam O'Toole]

On 2016-08-11, Valeri Galtsev
 wrote:
> Dear Experts,
>
> Could someone recommend a script or utility one can run from command
> line on Linux or UNIX machine to make a snapshot of webpage?
>
> We have a signage (xibo) and whoever creates/changes content, likes to
> add URLs of some webpages there. All works well if these are webpages
> on our servers (which are pretty fast), but some external servers
> often take time to respond and take time to assemble the page, in
> addition these servers sometimes get really busy, and when response is
> longer than time devoted for that content in signage window, this
> window hangs forever with blank white field until you restart client.
> Trivial workaround: just to get snapshot (as, say daily cron job), and
> point signage client to that snapshot definitely will solve it, and
> simultaneously we will stop bugging other people servers often without
> much need for it.
>
> But when I tried to search for some utility or script that makes
> webpage snapshot, I discovered that my ability to search degraded
> somehow...
>
> Thanks for all your pointers!
>
> Valeri

Not an answer to the question you asked, but maybe this is a job for a
caching proxy server like squid?

-- 

Liam

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] script to make webpage snapshot

[Frank Cox]

On Thu, 11 Aug 2016 15:46:42 -0500 (CDT)
Valeri Galtsev wrote:

> Could someone recommend a script or utility one can run from command line
> on Linux or UNIX machine to make a snapshot of webpage?

wget?
httrack?

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

[CentOS] script to make webpage snapshot

[Valeri Galtsev]

Dear Experts,

Could someone recommend a script or utility one can run from command line
on Linux or UNIX machine to make a snapshot of webpage?

We have a signage (xibo) and whoever creates/changes content, likes to add
URLs of some webpages there. All works well if these are webpages on our
servers (which are pretty fast), but some external servers often take time
to respond and take time to assemble the page, in addition these servers
sometimes get really busy, and when response is longer than time devoted
for that content in signage window, this window hangs forever with blank
white field until you restart client. Trivial workaround: just to get
snapshot (as, say daily cron job), and point signage client to that
snapshot definitely will solve it, and simultaneously we will stop bugging
other people servers often without much need for it.

But when I tried to search for some utility or script that makes webpage
snapshot, I discovered that my ability to search degraded somehow...

Thanks for all your pointers!

Valeri

Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247

___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos