Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

from Jason Harris:

> Some rationale for why gzip encoding is enabled:

> https://davidwalsh.name/check-gzip

> A look at the headers and how wget behaves:

>   wget --no-check-certificate -S --header="Accept-Encoding: gzip" 
> "https://svnweb.freebsd.org/ports/head/x11/xcb-proto/files/patch-xcbgen_align.py?revision=425597=co;

> What to add to your /usr/local/etc/lynx.cfg:

> PREFERRED_ENCODING:NONE

> HTH.

Yes, that worked, files in 
https://svnweb.freebsd.org/ports/head/x11/xcb-proto/files
downloaded true, file sizes were correct, no gzip.

My thanks.

But this really needs to be better documented.

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Anatoly]

> 
> URL where I was stung was
> https://svnweb.freebsd.org/ports/head/x11/xcb-proto/files
> with two patch files in that directory.
> 
Tested it with links 2.13. It works ok.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

> I have never seen lynx compress an uncompressed file.  However, if lynx sends 
> a header that it can _accept_ gzip encoding, which I believe it might, the 
> webserver can easily gzip the contents to save bandwidth.  lynx could
possibly be saving that compressed content to disk, with a .gz extension...
  
> Personally, I use elinks (and used to be its FreeBSD maintainer) way more 
> often than lynx.
  
> Can you send an URL to recreate the problem?

URL where I was stung was
https://svnweb.freebsd.org/ports/head/x11/xcb-proto/files
with two patch files in that directory.

I was thinking about using links or elinks instead of lynx, but 
textproc/docbook-tools uses www/lynx as a dependency.

If I don't want links' crude graphics implementation, elinks might be smaller 
and good enough.

Building links with directfb option can take a long time; one is better off 
with Firefox or Seamonkey.

I also emailed the upstream maintainer, Thomas Dickey 
(dic...@invisible-island.net).

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

> Thomas Mueller skrev:

> > Documentation on Lynx and w3m are awful hard to find!

> > I couldn't find anything on auto_uncompress or anything else that might be 
> > put in ~/.w3m/config.

> > If the file on the server is already compressed, for instance a tarball, 
> > then I want to download it that way.

> > But a browser/downloader has no proper business compressing a file to be 
> > downloaded.

> For w3m you can set accept_encoding. Default is

> accept_encoding gzip, compress, bzip, bzip2, deflate

> Try to remove gz, bzip, bzip2 or set it to "".

> If you think this is a bug you should report it upstream:

> https://github.com/tats/w3m

> Herbert

I looked through w3m documentation and found nothing on accept_encoding.

Their website suggests that w3m is more of a pager than a web browser.

Elvis (enhanced vi clone) is/was also a rudimentary text-mode web browser.

I don't install w3m explicitly, only if it is pulled in by the ports system as 
a dependency.

Maybe I will take the same approach regarding lynx?

I intend to check a NetBSD installation with lynx to see if that misbehaves the 
same way regarding gratuitous, unwanted gzip compression of downloads.

I haven't used links in some time; remember a build on FreeBSD took six hours, 
but that was with DirectFB on a computer with 256 MB RAM.

I think I need to email dic...@invisible-island.net .

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Herbert J. Skuhra]

Thomas Mueller skrev:
> 
> Documentation on Lynx and w3m are awful hard to find!
> 
> I couldn't find anything on auto_uncompress or anything else that might be 
> put in ~/.w3m/config.
> 
> If the file on the server is already compressed, for instance a tarball, then 
> I want to download it that way.
> 
> But a browser/downloader has no proper business compressing a file to be 
> downloaded.

For w3m you can set accept_encoding. Default is

accept_encoding gzip, compress, bzip, bzip2, deflate

Try to remove gz, bzip, bzip2 or set it to "".

If you think this is a bug you should report it upstream:

https://github.com/tats/w3m

--
Herbert
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Matthias Apitz]

El día lunes, abril 24, 2017 a las 11:48:32p. m. +, Thomas Mueller escribió:

> Curl is tricky to get right from the command line, I tried.
> 
> I have no recent experience with links.  Should I build with graphics, 
> without graphics, or build links1?
> 
> Lynx and w3m people hide the documentation, but I looked through sample 
> lynx.cfg.
> 
> Do I need to set
> GZIP_PATH=/dev/null ?

What about:

lynx -dump http://www.unixarea.de/OtroLugar.txt > x.txt

matthias

-- 
Matthias Apitz, ✉ g...@unixarea.de, ⌂ http://www.unixarea.de/  ☎ 
+49-176-38902045
Aus "Nie wieder Krieg!" wurde "Nie wieder Krieg ohne Deutschlands Truppen"
The "No wars anymore!" changed now to "No wars anymore without German battle 
groups!"
El "¡Nunca jamás guerra!" ha cambiado a "¡Nunca jamás guerra sin tropas 
alemanas!" 
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Dewayne Geraghty]

Yes, I'm with Anatoly.  Some browsers accept encrypted or compressed
content and then try to display it. I've tried lynx (GPL2),wget (GPL3) but
curl (MIT) with minimum options does the trick nicely.
I use curl in place of fetch, so in a make.conf there's
FETCH_CMD= /usr/local/bin/curl --create-dirs --connect-timeout 10 -m 240
--retry 1 -C - --ipv4 -O -o -

curl is harder to master than wget ;)

Hopefully this helps :)
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Anatoly]

On Mon, 24 Apr 2017 23:48:32 +
"Thomas Mueller"  wrote:

> I have no recent experience with links.  Should I build with
> graphics, without graphics, or build links1?
If you wish text console only, no graphics needed. Under x11 it's nice
fast simple graphical browser when started as 'links -g' otherwise. If
you wish graphics without x11, may experiment with direct fb and
svgalib. Links1 is an old version (contemporary is 2.x).

I think w3m doesn't gzip anything itself, but When talking to server it
says that gzip supported. Then it receives compressed stream from
server and fails (or just lazy) to decompress it.

-w3m must not say to server "Accept-Encoding: gzip, deflate" if it can't
gunzip
-server must not send gzipped stream if above not said. And
unfortunately, I think, some servers may not respect this.
-w3m must gunzip it otherwise.

I never used w3m, have not it istalled yet, but quick look at manual on
the web gives a idea to try
w3m -header "accept-encoding: identity"
to explicitly say to server send everything in plain.

Can you display content you trying to save on w3m screen online? If
yes, may try to fight for gzipping off, but if no, you must investigate
why gunzipping is not working within w3m (is it compiled against zlib
 or something at all?). Btw, what is the url you trying to save?

> Lynx was nice before it got intimately mixed with gzip.
But every browser nowadays can do "content gunzipping" due to existing
web standard
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

from Anatoly:

> If you wish console browser, try links
> if you wish http get tool, try curl

Curl is tricky to get right from the command line, I tried.

I have no recent experience with links.  Should I build with graphics, without 
graphics, or build links1?

Lynx and w3m people hide the documentation, but I looked through sample 
lynx.cfg.

Do I need to set
GZIP_PATH=/dev/null ?

Lynx was nice before it got intimately mixed with gzip.

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Anatoly]

If you wish console browser, try links
if you wish http get tool, try curl

On Mon, 24 Apr 2017 04:54:27 +
"Thomas Mueller"  wrote:

> > If you're just trying to grab a file, fetch(1) may prove adequate.
> > (It's in base.)  
>   
> > Peace,
> > david
> >-
> > David H. Wolfskill
> 
> I tried fetch, but got something entirely different, the stuff on the
> web page, but not the desired file.
> 
> File compression, such as PKZIP, Infozip, gzip, bzip2, 7-zip, RAR, xz
> are useful in places but can be overbearing in other places.
> 
> Compressing man pages is more pain in the ass than benefit.
> 
> Tom
> 
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to
> "freebsd-ports-unsubscr...@freebsd.org"

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

> > Thomas Mueller skrev:

> > I don't use lynx (text-mode web browser) much, but have run into a problem 
> > that I never had before.
> > Lynx, and also w3m, download what are supposed to be text files and then I 
> > see the gzip'ed version on the hard drive.

> > Lynx used to download files as is!

> > I looked through "man lynx", also /usr/local/etc/lynx.cfg, and couldn't 
> > figure how to disable the annoying, gratuitous gzip.

> > Mozilla Seamonkey and Firefox can download straight without altering.

> > My previous experience was that Lynx was trustworthy and would download 
> > files as is.

> For w3m you can try to set 'auto_uncompress 1' in ~/.w3m/config.

> Herbert

Documentation on Lynx and w3m are awful hard to find!

I couldn't find anything on auto_uncompress or anything else that might be put 
in ~/.w3m/config.

If the file on the server is already compressed, for instance a tarball, then I 
want to download it that way.

But a browser/downloader has no proper business compressing a file to be 
downloaded.

I might try on a NetBSD installation where I believe I installed lynx to 
compare the download automatic gzip behavior.

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Herbert J. Skuhra]

Thomas Mueller skrev:
> 
> I don't use lynx (text-mode web browser) much, but have run into a problem 
> that I never had before.
> Lynx, and also w3m, download what are supposed to be text files and then I 
> see the gzip'ed version on the hard drive.
> 
> Lynx used to download files as is!
> 
> I looked through "man lynx", also /usr/local/etc/lynx.cfg, and couldn't 
> figure how to disable the annoying, gratuitous gzip.
> 
> Mozilla Seamonkey and Firefox can download straight without altering.
> 
> My previous experience was that Lynx was trustworthy and would download files 
> as is.

For w3m you can try to set 'auto_uncompress 1' in ~/.w3m/config.

--
Herbert
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

> If you're just trying to grab a file, fetch(1) may prove adequate.
> (It's in base.)
  
> Peace,
> david
>-
> David H. Wolfskill  

I tried fetch, but got something entirely different, the stuff on the web page, 
but not the desired file.

File compression, such as PKZIP, Infozip, gzip, bzip2, 7-zip, RAR, xz are 
useful in places but can be overbearing in other places.

Compressing man pages is more pain in the ass than benefit.

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Downloading with lynx or w3m, how to download as is, without gratuitous gzip

[Thomas Mueller]

I don't use lynx (text-mode web browser) much, but have run into a problem that 
I never had before.

Lynx, and also w3m, download what are supposed to be text files and then I see 
the gzip'ed version on the hard drive.

Lynx used to download files as is!

I looked through "man lynx", also /usr/local/etc/lynx.cfg, and couldn't figure 
how to disable the annoying, gratuitous gzip.

Mozilla Seamonkey and Firefox can download straight without altering.

My previous experience was that Lynx was trustworthy and would download files 
as is.

Otherwise, what other text-mode web browsers are there in case my graphic 
interface (X) is not installed or is not operational?

In this case, the problem arose due to a conflict reported by svn in 
/usr/ports/x11/xcb-proto/files
https://svnweb.freebsd.org/ports/head/x11/xcb-proto/files

I solved that problem with "svn revert -R" but still would like to know how to 
make lynx properly functional again.

Lynx seems much easier to use than w3m, if I could find a way around its fetish 
with gzip; w3m also showed this fetish.

Tom

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"