Re: [arch-general] pacman doesn't show download progress

[nmset]

Le vendredi 18 août 2017 11:55:41 CEST mpan a écrit :
> Don’t write them to fix that, as nothing is broken: Content-Length is
> not required.

Of course I won't, ;-)

Re: [arch-general] pacman doesn't show download progress

[mpan]

>> Notice how the Content-Length: header is missing.
> Thanks for the clear explanations, at least I understand what's happening, 
> it's up to remote to fix.
  Don’t write them to fix that, as nothing is broken: Content-Length is
not required.



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] pacman doesn't show download progress

[nmset]

Le jeudi 17 août 2017 20:43:33 CEST Eli Schwartz a écrit :
> As for your log, that's just the HTTP headers, something you can also
> see with curl -I http://...
> 
> Notice how the Content-Length: header is missing.

Thanks for the clear explanations, at least I understand what's happening, 
it's up to remote to fix.

Re: [arch-general] pacman doesn't show download progress

[Guus Snijders via arch-general]

Op 17 aug. 2017 21:49 schreef "Eli Schwartz" :

On 08/17/2017 02:46 PM, Eli Schwartz wrote:
> ... and it's probably worth mentioning that sometime in the last 10
> minutes they seem to have fixed this.

Or not. It's erratically switching between working and not-working.


So it's probably one faulty server behind a load balancer? Seems strange
behaviour for a single machine.


Mvg, Guus Snijders

Re: [arch-general] pacman doesn't show download progress

[Eli Schwartz]

On 08/17/2017 02:46 PM, Eli Schwartz wrote:
> ... and it's probably worth mentioning that sometime in the last 10
> minutes they seem to have fixed this.

Or not. It's erratically switching between working and not-working.

-- 
Eli Schwartz



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] pacman doesn't show download progress

[Eli Schwartz]

On 08/17/2017 02:43 PM, Eli Schwartz wrote:
> Well, look at this:
> 
> ```
> [eschwartz@arch ~]$ pacman -Sp linux
> http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
> [eschwartz@arch ~]$ wget -P /tmp/
> http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
> --2017-08-17 14:31:17--
> http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
> Resolving archlinux.mirrors.ovh.net... 213.32.5.7,
> 2001:41d0:202:100:213:32:5:7
> Connecting to archlinux.mirrors.ovh.net|213.32.5.7|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: unspecified [application/octet-stream]
> Saving to: ‘/tmp/linux-4.12.7-1-i686.pkg.tar.xz’
> 
> linux-4.12.7-1-i686.pkg.tar.xz  [   <=>
>   ]   1.37M   912KB/s
> 
> ```
> 
> Length is unspecified, and wget does its <=> slidy-back-and-forth thing.
> And it seems pacman just gives up on showing progress information and
> instead shows nothing (which isn't unreasonable).
> 
> As for your log, that's just the HTTP headers, something you can also
> see with curl -I http://...
> 
> Notice how the Content-Length: header is missing.

... and it's probably worth mentioning that sometime in the last 10
minutes they seem to have fixed this.

-- 
Eli Schwartz



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] pacman doesn't show download progress

[Eli Schwartz]

On 08/17/2017 02:23 PM, SET wrote:
> Le jeudi 17 août 2017 06:46:37 CEST Darshit Shah via arch-general a écrit :
> 
>> No, right next to the packages, in the package cache
>> (/var/cache/pacman/pkg by default)
> 
> Though the problem is resolved by choosing another mirror, here is a sample 
> log file : oxygen-icons-1:5.37.0-1-any.pkg.tar.xz.part.log 
> 
>   HTTP/1.1 200 OK
>   Server: nginx/1.6.2
>   Date: Thu, 17 Aug 2017 18:14:56 GMT
>   Content-Type: application/octet-stream
>   X-Varnish: 537330300
>   Age: 0
>   Via: 1.1 varnish-v4
>   Transfer-Encoding: chunked
>   Accept-Ranges: bytes
>   X-IPLB-Instance: 5865
> 
> I don't know how to understand it.

Well, look at this:

```
[eschwartz@arch ~]$ pacman -Sp linux
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
[eschwartz@arch ~]$ wget -P /tmp/
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
--2017-08-17 14:31:17--
http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i686.pkg.tar.xz
Resolving archlinux.mirrors.ovh.net... 213.32.5.7,
2001:41d0:202:100:213:32:5:7
Connecting to archlinux.mirrors.ovh.net|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/linux-4.12.7-1-i686.pkg.tar.xz’

linux-4.12.7-1-i686.pkg.tar.xz  [   <=>
  ]   1.37M   912KB/s

```

Length is unspecified, and wget does its <=> slidy-back-and-forth thing.
And it seems pacman just gives up on showing progress information and
instead shows nothing (which isn't unreasonable).

As for your log, that's just the HTTP headers, something you can also
see with curl -I http://...

Notice how the Content-Length: header is missing.

-- 
Eli Schwartz



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] pacman doesn't show download progress

[SET]

Le jeudi 17 août 2017 06:46:37 CEST Darshit Shah via arch-general a écrit :

> No, right next to the packages, in the package cache
> (/var/cache/pacman/pkg by default)

Though the problem is resolved by choosing another mirror, here is a sample 
log file : oxygen-icons-1:5.37.0-1-any.pkg.tar.xz.part.log 

  HTTP/1.1 200 OK
  Server: nginx/1.6.2
  Date: Thu, 17 Aug 2017 18:14:56 GMT
  Content-Type: application/octet-stream
  X-Varnish: 537330300
  Age: 0
  Via: 1.1 varnish-v4
  Transfer-Encoding: chunked
  Accept-Ranges: bytes
  X-IPLB-Instance: 5865

I don't know how to understand it.

Re: [arch-general] pacman doesn't show download progress

[Florian Pritz via arch-general]

On 16.08.2017 21:47, SET wrote:
> Le mercredi 16 août 2017 20:48:59 CEST Bartłomiej Piotrowski a écrit :
> 
>> Which mirror was it? We will try to contact them about the issue.
>> 
>> B
> 
> It is :
>  http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch

I've notified the mirror admin. I'll post here again if I know more.

Florian



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] pacman doesn't show download progress

[Darshit Shah via arch-general]

* nm...@netcourrier.com  [170816 17:58]:

Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :

XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S
-o %o.log -O %o %u


Where will be the log file ? current working directory ?


No, right next to the packages, in the package cache 
(/var/cache/pacman/pkg by default)


--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6


signature.asc
Description: PGP signature

Re: [arch-general] pacman doesn't show download progress

[Bartłomiej Piotrowski]

On 2017-08-16 20:41, SET wrote:
> Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
>> My guess is that there exist some mirror(s) that return a bad HTTP
>> Response, that is a response without a file-size.
> 
> You're probably right. pacman has expected output if I change remote mirror. 
> It had to download 150 MB, I interrupted the download and restarted many 
> times. No output while downloading with one mirror, and normal output with 
> another one. At least up to now.
> 
> Thanks.
> 

Which mirror was it? We will try to contact them about the issue.

B

Re: [arch-general] pacman doesn't show download progress

[SET]

Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
> My guess is that there exist some mirror(s) that return a bad HTTP
> Response, that is a response without a file-size.

You're probably right. pacman has expected output if I change remote mirror. 
It had to download 150 MB, I interrupted the download and restarted many 
times. No output while downloading with one mirror, and normal output with 
another one. At least up to now.

Thanks.

Re: [arch-general] pacman doesn't show download progress

[SET]

Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
> Could you also share the first few lines of your mirrorlist file?

The first few lines verbatim :

Server = http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch
#Server = http://mirror.archlinux.ikoula.com/archlinux/$repo/os/$arch
Server = http://archlinux.polymorf.fr/$repo/os/$arch
##
## Arch Linux repository mirrorlist
## Generated on 2016-03-10
##

## Worldwide
#Server = http://mirror.rackspace.com/archlinux/$repo/os/$arch

## Australia
#Server = http://mirror.aarnet.edu.au/pub/archlinux/$repo/os/$arch

Re: [arch-general] pacman doesn't show download progress

[nmset]

Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
> XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S
> -o %o.log -O %o %u

Where will be the log file ? current working directory ?

Re: [arch-general] pacman doesn't show download progress

[Darshit Shah via arch-general]

Could you also share the first few lines of your mirrorlist file?

I don't have much idea about how `pacman` is implemented, so what I'm 
saying next is pure guesswork and speculation.


My guess is that there exist some mirror(s) that return a bad HTTP 
Response, that is a response without a file-size. A HTTP/1.1 response, 
almost always includes a "Content-Length" header which tells the client 
program what the total size of the file is. This data is also obviously 
important for drawing an accurate progress bar. Now, if the 
"Content-Length" header was missing, pacman cannot know the final size 
of the file before hand and hence it is not drawing the progress bars.


As a test, I would request that the next time you see something like 
this, go through these steps:


1. Delete the just downloaded package from the cache, so that pacman 
  will download it again

2. Replace the XferCommand line with this:
	`XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S 
	-o %o.log -O %o %u`

3. Run pacman again

This will create a log file containing the Server's response next to the 
package in the pacman cache. Take a look at the log file and also please 
share the contents here with us. If my guess is correct, we should be 
able to see this in the log file, else, we find another way to debug 
this


* Ismael Bouya  [170816 11:13]:

(Tue, Aug 15, 2017 at 02:05:44PM -0700) Moses Miller via arch-general :

Can you explain in more detail what the problem is, and show actual and
expected output?


Hello,
I have the same "problem" for a few weeks too.
Here is the output of pacman -Syu I got just now on two machines:
https://paste.ee/p/hqCTu
https://paste.ee/p/eLu01

In the first one, you see download progress on the packages.
In the second one you don't see them. Here you don't see them at all (probably
because there are only a few packages), but it happens that I only see a few of
them. The behavior is completely random, and on both machines: it's just luck
that I could get the two different examples today

The only line that differs between both machines in pacman.conf is IgnorePkg:
--
[options]
HoldPkg = pacman glibc
IgnorePkg   = (...)
Architecture = auto
Color
CheckSpace
SigLevel= Required DatabaseOptional
LocalFileSigLevel = Optional
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
[multilib]
Include = /etc/pacman.d/mirrorlist


[NB: I "think" it's related to curl (didn't check if pacman uses it though)
since I have a similar experience once with that command in a non-related use.]

Hope it can help.
--
Ismael




--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6


signature.asc
Description: PGP signature

Re: [arch-general] pacman doesn't show download progress

[Ismael Bouya]

(Tue, Aug 15, 2017 at 02:05:44PM -0700) Moses Miller via arch-general :
> Can you explain in more detail what the problem is, and show actual and
> expected output?

Hello,
I have the same "problem" for a few weeks too.
Here is the output of pacman -Syu I got just now on two machines:
https://paste.ee/p/hqCTu
https://paste.ee/p/eLu01

In the first one, you see download progress on the packages.
In the second one you don't see them. Here you don't see them at all (probably
because there are only a few packages), but it happens that I only see a few of
them. The behavior is completely random, and on both machines: it's just luck
that I could get the two different examples today

The only line that differs between both machines in pacman.conf is IgnorePkg:
--
[options]
HoldPkg = pacman glibc
IgnorePkg   = (...)
Architecture = auto
Color
CheckSpace
SigLevel= Required DatabaseOptional
LocalFileSigLevel = Optional
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
[multilib]
Include = /etc/pacman.d/mirrorlist


[NB: I "think" it's related to curl (didn't check if pacman uses it though)
since I have a similar experience once with that command in a non-related use.]

Hope it can help.
-- 
Ismael


signature.asc
Description: PGP signature

Re: [arch-general] pacman doesn't show download progress

[Ralf Mardorf]

On Tue, 15 Aug 2017 23:04:35 +0200, SET wrote:
># Misc options
>#UseSyslog
>Color
>#TotalDownload
>CheckSpace

Here a progress bar is displayed, but I'm using the "ILoveCandy" Easter
egg.

[root@archlinux ~]# grep Misc\ options -A8 /etc/pacman.conf 
# Misc options
#UseSyslog
#UseDelta
#TotalDownload
CheckSpace
#VerbosePkgLists
Color
ILoveCandy

[root@archlinux ~]# pacman -Q pacman
pacman 5.0.2-2

Re: [arch-general] pacman doesn't show download progress

[Moses Miller via arch-general]

Can you explain in more detail what the problem is, and show actual and
expected output?

On 08/15/2017 02:04 PM, SET wrote:
> Le mardi 15 août 2017 21:43:46 CEST Eli Schwartz a écrit :
>> Have you set an external downloader?
> 
> No, I'm using the default downloader.
> 
> [options]
> #RootDir = /
> #DBPath  = /var/lib/pacman/
> #CacheDir= /var/cache/pacman/pkg/
> #LogFile = /var/log/pacman.log
> #GPGDir  = /etc/pacman.d/gnupg/
> HoldPkg = pacman glibc
> #XferCommand = /usr/bin/curl -C - -f %u > %o
> #XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
> #CleanMethod = KeepInstalled
> #UseDelta= 0.7
> Architecture = auto
> IgnoreGroup = xorg
> IgnorePkg   = xf86-input-keyboard xf86-input-mouse xf86-video-vmware
> IgnorePkg   = qt5-doc minuet ocaml
> #NoUpgrade   =
> #NoExtract   =
> # Misc options
> #UseSyslog
> Color
> #TotalDownload
> CheckSpace
> 



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] pacman doesn't show download progress

[SET]

Le mardi 15 août 2017 21:43:46 CEST Eli Schwartz a écrit :
> Have you set an external downloader?

No, I'm using the default downloader.

[options]
#RootDir = /
#DBPath  = /var/lib/pacman/
#CacheDir= /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
#GPGDir  = /etc/pacman.d/gnupg/
HoldPkg = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta= 0.7
Architecture = auto
IgnoreGroup = xorg
IgnorePkg   = xf86-input-keyboard xf86-input-mouse xf86-video-vmware
IgnorePkg   = qt5-doc minuet ocaml
#NoUpgrade   =
#NoExtract   =
# Misc options
#UseSyslog
Color
#TotalDownload
CheckSpace

Re: [arch-general] pacman doesn't show download progress

[Eli Schwartz]

On 08/15/2017 03:37 PM, SET wrote:
> pacman -Syu show database download progress but not packages being 
> downloaded. 
> They do get downloaded and installation progress is rightly displayed. This 
> can be annoying when connection speed is poor. This has been observed on 
> three 
> machines.
> 
> I am obviously requesting a solution and am ready to provide any additional 
> required information.

I'm not sure what to say, other than that I see progress every time I
download packages, unless they are already cached and therefore do not
need to be downloaded.

Have you set an external downloader?

-- 
Eli Schwartz



signature.asc
Description: OpenPGP digital signature

[arch-general] pacman doesn't show download progress

[SET]

pacman -Syu show database download progress but not packages being downloaded. 
They do get downloaded and installation progress is rightly displayed. This 
can be annoying when connection speed is poor. This has been observed on three 
machines.

I am obviously requesting a solution and am ready to provide any additional 
required information.

Thanks.