Hi Kotresh,

thanks for your response...

i have made further tests based on ubuntu 16.04.3 (latest upgrades) and gfs 3.12.5 with following rsync version :

1. ii  rsync                              3.1.1-3ubuntu1
2. ii  rsync                              3.1.1-3ubuntu1.2
3. ii  rsync                              3.1.2-2ubuntu0.1

in each test all nodes had the same rsync version installed. all tests failed with the below shown error. then i started a test with the same setup. this time upgrades for ubuntu 16.04.3 were disabled. again i have tested the above shown rsync versions on all nodes. the geo replication works fine in any case. then i have installed  3.1.1-3ubuntu1.2 on each node as it is installed with latest ubuntu upgrade. afterwards i have just upgraded all slave nodes with the latest ubuntu upgrades, reboot...the geo-replication still works fine. after upgrade and reboot of the master nodes with the latest ubuntu upgrades the below shown error appeared again... geo rep is broken.

i believe one of the below shown packages caused the error because directly after upgrade of these packages the error appeared
on the corresponding master node....
anyone else who has experienced this issue...?
any help would be appreciated.

best regards
Dietmar




Am 25.01.2018 um 04:14 schrieb Kotresh Hiremath Ravishankar:
It is clear that rsync is failing. Are the rsync versions on all masters and slave nodes same?
I have seen that has caused problems sometimes.

-Kotresh HR

On Wed, Jan 24, 2018 at 10:29 PM, Dietmar Putz <dietmar.p...@3qsdn.com <mailto:dietmar.p...@3qsdn.com>> wrote:

    Hi all,

    i have made some tests on the latest Ubuntu 16.04.3 server image.
    Upgrades were disabled...
    the configuration was always the same...a distributed replicated
    volume on 4 VM's with geo-replication to a dist. repl .volume on 4
    VM's.
    i started with 3.7.20, upgrade to 3.8.15, to 3.10.9 to 3.12.5.
    After each upgrade i have tested the geo-replication which worked
    well anytime.
    then i have made an update / upgrade on the first master node.
    directly after upgrade the below shown error appeared on that node.
    after upgrade on the second master node the error appeared there
    also... geo replication is faulty.

    this error affects gfs 3.7.20, 3.8.15, 3.10.9 and 3.12.5 on Ubuntu
    16.04.3
    in one test i have updated rsync from 3.1.1 to 3.1.2 but with no
    effect.

    does anyone else experienced this behavior...any idea ?

    best regards
    Dietmar


    gfs 3.12.5 geo-rep log on master :

    [2018-01-24 15:50:35.347959] I [master(/brick1/mvol1):1385:crawl]
    _GMaster: slave's time    stime=(1516808792, 0)
    [2018-01-24 15:50:35.604094] I
    [master(/brick1/mvol1):1863:syncjob] Syncer: Sync Time Taken   
    duration=0.0294    num_files=1 job=2    return_code=3
    [2018-01-24 15:50:35.605490] E
    [resource(/brick1/mvol1):210:errlog] Popen: command returned
    error    cmd=rsync -aR0 --inplace --files-from=- --super --stats
    --numeric-ids --no-implied-dirs --existing --xattrs --acls
    --ignore-missing-args . -e ssh -oPasswordAuthentication=no
    -oStrictHostKeyChecking=no -i
    /var/lib/glusterd/geo-replication/secret.pem -p 22
    -oControlMaster=auto -S
    /tmp/gsyncd-aux-ssh-MZwEp2/cbad1c5f88978ecd713bdb1478fbabbe.sock
    --compress root@gl-node5-int:/proc/2013/cwd error=3
    [2018-01-24 15:50:35.628978] I
    [syncdutils(/brick1/mvol1):271:finalize] <top>: exiting.



    after this upgrade one server fails :
    Start-Date: 2018-01-18  04:33:52
    Commandline: /usr/bin/unattended-upgrade
    Upgrade:
    libdns-export162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    libisccfg140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    bind9-host:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    dnsutils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    libc6:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
    libisc160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    locales:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
    libisc-export160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    libc-bin:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
    liblwres141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    libdns162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    multiarch-support:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
    libisccc140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10),
    libbind9-140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
    1:9.10.3.dfsg.P4-8ubuntu1.10)
    End-Date: 2018-01-18  04:34:32



    strace rsync :

    30743 23:34:47 newfstatat(3, "6737", {st_mode=S_IFDIR|0755,
    st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
    30743 23:34:47 newfstatat(3, "6741", {st_mode=S_IFDIR|0755,
    st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
    30743 23:34:47 getdents(3, /* 0 entries */, 131072) = 0
    30743 23:34:47 munmap(0x7fa4feae7000, 135168) = 0
    30743 23:34:47 close(3)                 = 0
    30743 23:34:47 write(2, "rsync: getcwd(): No such file or
    directory (2)", 46) = 46
    30743 23:34:47 write(2, "\n", 1)        = 1
    30743 23:34:47 rt_sigaction(SIGUSR1, {SIG_IGN, [], SA_RESTORER,
    0x7fa4fdf404b0}, NULL, 8) = 0
    30743 23:34:47 rt_sigaction(SIGUSR2, {SIG_IGN, [], SA_RESTORER,
    0x7fa4fdf404b0}, NULL, 8) = 0
    30743 23:34:47 write(2, "rsync error: errors selecting
    input/output files, dirs (code 3) at util.c(1056)
    [Receiver=3.1.1]", 96) = 96
    30743 23:34:47 write(2, "\n", 1)        = 1
    30743 23:34:47 exit_group(3)            = ?
    30743 23:34:47 +++ exited with 3 +++




    Am 19.01.2018 um 17:27 schrieb Joe Julian:
    ubuntu 16.04

    _______________________________________________
    Gluster-users mailing list
    Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
    http://lists.gluster.org/mailman/listinfo/gluster-users
    <http://lists.gluster.org/mailman/listinfo/gluster-users>




--
Thanks and Regards,
Kotresh H R

_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Reply via email to