Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Shanker Wang]

Hi, the sync is still in progress, which started on Nov 11 09:39:26

Nov 11 09:39:26 4b36bab6e1bc ftpsync-debian[7]: Mirrorsync start
Nov 11 09:39:26 4b36bab6e1bc ftpsync-debian[7]: Running mirrorsync, update is 
required, 
/data/mirrors/debian//Archive-Update-Required-neomirrors.tuna.tsinghua.edu.cn 
exists
Nov 11 09:39:26 4b36bab6e1bc ftpsync-debian[7]: Running stage1: rsync -e 
/ftpsync/bin/rsync-ssl-tunnel -6 --safe-links --bwlimit=0 -prltvHSB8192 
--timeout 3600 --stats 
--filter=protect_Archive-Update-in-Progress-neomirrors.tuna.tsinghua.edu.cn 
--filter=protect_project/trace/neomirrors.tuna.tsinghua.edu.cn 
--filter=protect_project/trace/neomirrors.tuna.tsinghua.edu.cn-stage1 
--exclude=project/trace/_hierarchy --filter=protect_project/trace/_hierarchy 
--exclude=project/trace/_traces --filter=protect_project/trace/_traces 
--filter=protect_Archive-Update-Required-neomirrors.tuna.tsinghua.edu.cn 
--exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease 
--include=i18n/by-hash/** --exclude=i18n/* --exclude=ls-lR*  --exclude=.~tmp~/  
 mirrors.tuna.tsinghua.edu...@syncproxy.cna.debian.org::debian 
/data/mirrors/debian/

I'm trying to figure out why the sync is taking so much time.

Cheers,

Miao Wang

> 在 2017年11月14日，08:10，Bastian Blank  写道：
> 
> Hi
> 
> It happened again, this time on neomirror, and for 12(!) days.
> 
> https://mirror-master.debian.org/status/mirror-info/neomirrors.tuna.tsinghua.edu.cn.html
> 
> Please fix your setup.  In the meantime I will remove it as candicate
> for any *.d.o.
> 
> Bastian
> 
> -- 
> Men will always be men -- no matter where they are.
>   -- Harry Mudd, "Mudd's Women", stardate 1329.8

Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Bastian Blank]

Hi

It happened again, this time on neomirror, and for 12(!) days.

https://mirror-master.debian.org/status/mirror-info/neomirrors.tuna.tsinghua.edu.cn.html

Please fix your setup.  In the meantime I will remove it as candicate
for any *.d.o.

Bastian

-- 
Men will always be men -- no matter where they are.
-- Harry Mudd, "Mudd's Women", stardate 1329.8

Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Shanker Wang]

> 在 2017年09月16日，16:47，Peter Palfrader  写道：
> 
> On Fri, 15 Sep 2017, Shanker Wang wrote:
> 
>>> On Thu, Sep 07, 2017 at 11:02:40PM +0800, Shanker Wang wrote:
> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
> out of date.
>>> 
>>> Currently both mirrors are updated almost three hours apart, even if
>>> triggered at the same time.  Please investigate why this happens.
>> 
>> That's because the trigger cannot assure that sync is started immediately.
>> When the limit of concurrent syncing jobs is reached, other jobs will be 
>> pending in queue.
> 
> Then these services probably should not be available behind a common
> name.  Apt gets very confused if the data on "one server" changes
> between requests, therefore the window where mirrors have a different
> state should be kept as small as possible.
> 

You are right. The main reason why there are two servers balancing the 
traffic is that the throughput of the storage is becoming the bottleneck.
The most ideal solution should be utilizing a load balancer to "pin" a 
user onto a specific server. However, we do not have an additional server
as a load balancer currently. 

I admit that the inconsistence between the two servers should be a problem.
To partly work around this, the following measures were taken:

* Load balancing is achieved by randomly return one DNS record of two on 
  the name server, whose ttl is about 24 hours, so that for a period of 
  time, the name resolution result of our common name will be stable.

* For requests for a file which does not exist on one server, the server 
  will internally forward the request to the other and will response with
  the file on the other server if exists.


Those are all we have done to work around this problem. Though we have not 
received issues from users about the inconsistence, we will keep a close
eye on that. Any suggestions are welcomed.

> To which of the two mirrors should ftp2.cn.debian.org point, and should
> we replace mirrors.tuna.tsinghua.edu.cn in our list with nano and neo?
> 
> -- 
>|  .''`.   ** Debian **
>  Peter Palfrader   | : :' :  The  universal
> https://www.palfrader.org/ | `. `'  Operating System
>|   `-https://www.debian.org/

Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Peter Palfrader]

On Fri, 15 Sep 2017, Shanker Wang wrote:

> > On Thu, Sep 07, 2017 at 11:02:40PM +0800, Shanker Wang wrote:
> >>> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
> >>> out of date.
> > 
> > Currently both mirrors are updated almost three hours apart, even if
> > triggered at the same time.  Please investigate why this happens.
> 
> That's because the trigger cannot assure that sync is started immediately.
> When the limit of concurrent syncing jobs is reached, other jobs will be 
> pending in queue.

Then these services probably should not be available behind a common
name.  Apt gets very confused if the data on "one server" changes
between requests, therefore the window where mirrors have a different
state should be kept as small as possible.

To which of the two mirrors should ftp2.cn.debian.org point, and should
we replace mirrors.tuna.tsinghua.edu.cn in our list with nano and neo?

-- 
|  .''`.   ** Debian **
  Peter Palfrader   | : :' :  The  universal
 https://www.palfrader.org/ | `. `'  Operating System
|   `-https://www.debian.org/

Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Shanker Wang]

> 在 2017年09月15日，18:05，Bastian Blank  写道：
> 
> Hi
> 
> On Thu, Sep 07, 2017 at 11:02:40PM +0800, Shanker Wang wrote:
>>> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
>>> out of date.
> 
> Currently both mirrors are updated almost three hours apart, even if
> triggered at the same time.  Please investigate why this happens.

That's because the trigger cannot assure that sync is started immediately.
When the limit of concurrent syncing jobs is reached, other jobs will be 
pending in queue.


> 
>>> | weasel@melartin:~$ curl -s 
>>> http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn
>>>  | grep serial 
>>> | Archive serial: 2017090602
>>> | weasel@melartin:~$ curl -s 
>>> http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn
>>>  | grep serial 
>>> | Archive serial: 2017090702
> 
> Regards,
> Bastian
> 
> -- 
> Extreme feminine beauty is always disturbing.
>   -- Spock, "The Cloud Minders", stardate 5818.4

Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Bastian Blank]

Hi

On Thu, Sep 07, 2017 at 11:02:40PM +0800, Shanker Wang wrote:
> > One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
> > out of date.

Currently both mirrors are updated almost three hours apart, even if
triggered at the same time.  Please investigate why this happens.

> > | weasel@melartin:~$ curl -s 
> > http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn
> >  | grep serial 
> > | Archive serial: 2017090602
> > | weasel@melartin:~$ curl -s 
> > http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn
> >  | grep serial 
> > | Archive serial: 2017090702

Regards,
Bastian

-- 
Extreme feminine beauty is always disturbing.
-- Spock, "The Cloud Minders", stardate 5818.4

Bug#874574: [THU Mirror:2987] Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Shanker Wang]

Thanks for notification! 

> 在 2017年09月07日，20:29，Peter Palfrader  写道：
> 
> Package: mirrors
> User: mirr...@packages.debian.org
> Usertags: mirror-problem
> 
> Hi!
> 
> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
> out of date.
> 
> | weasel@melartin:~$ curl -s 
> http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn
>  | grep serial 
> | Archive serial: 2017090602
> | weasel@melartin:~$ curl -s 
> http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn
>  | grep serial 
> | Archive serial: 2017090702
> 
> o Please investigate why nanomirrors does not update

Because of an unexpected termination of the sync process, the lock file was not 
removed and blocked further retries. The lock file has been removed and the 
sync is now in progress. 

> 
> o Please set MIRRORNAME on both hosts to their respective hostname, such that
> the tracefiles are
>   
> http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/nanomirrors.tuna.tsinghua.edu.cn
>  
> 
> and
>   
> http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/neomirrors.tuna.tsinghua.edu.cn
>  
> 
> respectively.

We have changed this setting. Thanks for your advice.

> 
> o The tracefiles suggest that the ftpsync version you are using is not the
> current one.  Please update

We will soon perform that upgrade.

> 
> Cheers,
> -- 
>   |  .''`.   ** Debian **
> Peter Palfrader   | : :' :  The  universal
> https://www.palfrader.org/ | `. `'  Operating System
>   |   `-https://www.debian.org/
> 
> -- 
> 您收到此邮件是因为您订阅了 Google 网上论坛的“THU Opensource Mirror Admin”群组。
> 要退订此群组并停止接收此群组的电子邮件，请发送电子邮件到thu-opensource-mirror-admin+unsubscr...@googlegroups.com。
> 要查看更多选项，请访问 https://groups.google.com/d/optout。


smime.p7s
Description: S/MIME cryptographic signature