Re: [Gluster-users] geo-replication appears to be working, but "failures" are logged for each file transferred.

Tue, 02 Feb 2016 21:48:48 -0800 

Hi Kris,
These failures are mostly while transferring data from Master to Slave. Geo-replication is not intelligent to find exact number and reason for failures. I sent patch to effectively handle the rsync/tar failures. This patch is expected to be available with 3.7.8 release. 
http://review.gluster.org/#/c/12856/
Above patch identifies the reason for failure and logs in /var/log/glusterfs/geo-replication/<MASTERVOL>/*.log. 

Answers inline.

regards
Aravinda

On 02/03/2016 07:16 AM, Kris Laib wrote:

I recently setup geo-replication between two sites, and everything appears to be working, however, 
if I do a "gluster volume geo-replication SourceVol DestVol status detail", I see that 
the "failures" column seems to be incrementing for each file it transfers.   I've looked 
at the logs in /var/log/glusterfs/geo-replication/SourceVol/* , but nothing stood out to me.

I've verified that the MD5 sum matches on the files at the destination after 
it's been replicated, so I'm unsure what is triggering the failure.  Aside from 
the incrementing failure column, it all looks good to me.
Looks like these issues can be ignored. Errors may be due to xattrs syncing. With the above patch, logging is improved to log the failure reasons. 

All boxes in both clusters are CentOS 6.7, running glusterfs 3.7.5


My questions:

1)  What's causing the failures to be logged (please let me know what 
additional logs to provide)?

Possible reasons for Sync failures,
- May be failure to sync xattrs/acls to Slave.
- I/O error on Slave
- Files unlinked in Slave
- GFID Conflict

2)  How do I reset the failures counter?   It stays at the current value even 
once the files are deleted from the source

Will fix this issue in upcoming releases.

3)  Is there documentation somewhere I missed that explains what Entry, Data, 
Meta, and Failures are in the status detail view for geo-replication?
Entry - Pending Entry operations - Entry operations like CREATE, RENAME, UNLINK etc, which are executed as RPC calls. Data - Pending Data operations - Rsync/Tar is used to transfer data from Master to Slave. Meta - Pending Meta operations - Changing ownership, permissions are executed as RPC calls. 

Failures - Going to be obsolete since Errors will be logged in log files.

4)  Should "gluster volume geo-replication status" show me anything on the destination?  
Mine simply outputs "No active geo-replication sessions", though the command on the 
source works fine.

Geo-replication status only available from Master.



Thanks,
Kris

-----------

Here's an example of what I'm seeing:


[[email protected] temp]$ mount | grep "gfs-prd.mgmt:/prd_items"
gfs-prd.mgmt:/prd_items on /mnt/temp type fuse.glusterfs 
(rw,default_permissions,allow_other,max_read=131072)
[[email protected] temp]$ pwd
/mnt/temp
[[email protected] temp]$ gluster volume geo-replication prd_items 
[email protected]::dr_items status detail
MASTER NODE                    MASTER VOL    MASTER BRICK                       
        SLAVE USER    SLAVE                                           SLAVE 
NODE                     STATUS     CRAWL STATUS       LAST_SYNCED            
ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    
CHECKPOINT COMPLETION TIME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gfs-prd02.mgmt    prd_items     /data/glusterfs/prd_items_brick1/brick1    
georep        [email protected]::dr_items    gfs-slc03.mgmt    Active     
Changelog Crawl    2016-02-03 01:28:26    0        0       0       2           
N/A                N/A                     N/A
gfs-prd03.mgmt    prd_items     /data/glusterfs/prd_items_brick1/brick1    
georep        [email protected]::dr_items    gfs-slc02.mgmt    Passive    
N/A                N/A                    N/A      N/A     N/A     N/A         
N/A                N/A                     N/A
[[email protected] temp]$ dd if=/dev/urandom of=test01 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.179957 s, 5.8 MB/s
[[email protected] temp]$ dd if=/dev/urandom of=test02 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.181704 s, 5.8 MB/s
[[email protected] temp]$ dd if=/dev/urandom of=test03 bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.181575 s, 5.8 MB/s
[[email protected] temp]$ gluster volume geo-replication prd_items 
[email protected]::dr_items status detail
MASTER NODE                    MASTER VOL    MASTER BRICK                       
        SLAVE USER    SLAVE                                           SLAVE 
NODE                     STATUS     CRAWL STATUS       LAST_SYNCED            
ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    
CHECKPOINT COMPLETION TIME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gfs-prd02.mgmt    prd_items     /data/glusterfs/prd_items_brick1/brick1    
georep        [email protected]::dr_items    gfs-slc03.mgmt    Active     
Changelog Crawl    2016-02-03 01:29:11    0        0       0       5           
N/A                N/A                     N/A
gfs-prd03.mgmt    prd_items     /data/glusterfs/prd_items_brick1/brick1    
georep        [email protected]::dr_items    gfs-slc02.mgmt    Passive    
N/A                N/A                    N/A      N/A     N/A     N/A         
N/A                N/A                     N/A




Snippet from 
/var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.log
==============

[2016-02-03 01:26:14.702147] I 
[master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 
crawls, 1 turns
[2016-02-03 01:26:14.718751] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462711, 0)
[2016-02-03 01:27:15.371404] I 
[master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 
crawls, 1 turns
[2016-02-03 01:27:15.379936] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462771, 0)
[2016-02-03 01:27:58.15177] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462831, 0)
[2016-02-03 01:27:59.54204] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:00.101433] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:01.104391] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:02.312911] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:03.356967] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:04.364518] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:05.565407] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:06.610732] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:07.620142] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462877
[2016-02-03 01:28:08.818785] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: 
changelogs CHANGELOG.1454462877 could not be processed - moving on...
[2016-02-03 01:28:08.826875] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: 
SKIPPED GFID = f5a44b23-cdd8-4278-92d1-1b2a16e8c9d5
[2016-02-03 01:28:15.94074] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462876, 0)
[2016-02-03 01:28:18.669955] I 
[master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 17 
crawls, 3 turns
[2016-02-03 01:28:27.720731] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462891, 0)
[2016-02-03 01:28:58.387859] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462906, 0)
[2016-02-03 01:28:59.360420] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:00.430566] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:01.610398] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:02.667408] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:03.692369] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:04.947613] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:05.932200] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:06.947702] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:08.186101] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462937
[2016-02-03 01:29:09.203042] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: 
changelogs CHANGELOG.1454462937 could not be processed - moving on...
[2016-02-03 01:29:09.205854] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: 
SKIPPED GFID = c99a05c9-9e7d-41cf-b148-9bec91f527c8
[2016-02-03 01:29:15.432744] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462936, 0)
[2016-02-03 01:29:16.743381] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:17.944102] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:18.970493] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:20.77699] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:21.197900] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:22.259825] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:23.349780] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:24.469357] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:25.526793] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: 
incomplete sync, retrying changelogs: CHANGELOG.1454462952
[2016-02-03 01:29:26.612977] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: 
changelogs CHANGELOG.1454462952 could not be processed - moving on...
[2016-02-03 01:29:26.615757] W 
[master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: 
SKIPPED GFID = 
a7dc6c28-9239-44af-909f-a2938bb387b6,550ffdd3-4636-45b8-b278-fbfe120b9d85
[2016-02-03 01:29:29.828999] I 
[master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 16 
crawls, 3 turns
[2016-02-03 01:30:15.98556] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454462951, 0)
[2016-02-03 01:30:30.709733] I 
[master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 
crawls, 1 turns
[2016-02-03 01:31:12.918898] I 
[master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's 
time: (1454463011, 0)
[2016-02-03 01:31:31.423204] I 
[master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 
crawls, 1 turns


Snippet from 
/var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1-changes.log
===============
[2016-02-03 01:26:12.472345] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:26:12.472396] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1721, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:26:12.472415] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:26:12.472426] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:26:12.472440] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2478 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462772.472206), (vec: 1, 
len: 4104)
[2016-02-03 01:26:12.472467] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:26:12.472481] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:26:12.472502] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6b9, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:26:14.952267] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462772
 to processed directory
[2016-02-03 01:27:12.531138] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:27:12.531177] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1722, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:27:12.531195] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:27:12.531207] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:27:12.531220] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2479 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462832.531035), (vec: 1, 
len: 4104)
[2016-02-03 01:27:12.531241] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:27:12.531256] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:27:12.531274] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6ba, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:27:15.660249] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462832
 to processed directory
[2016-02-03 01:27:57.576147] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:27:57.576198] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1723, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:27:57.576217] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:27:57.576230] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:27:57.576246] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2480 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462877.576035), (vec: 1, 
len: 4104)
[2016-02-03 01:27:57.576282] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:27:57.576297] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:27:57.576327] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6bb, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:28:08.833040] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462877
 to processed directory
[2016-02-03 01:28:12.589377] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:28:12.589420] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1724, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:12.589439] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:28:12.589451] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:28:12.589466] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2481 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462892.589270), (vec: 1, 
len: 4104)
[2016-02-03 01:28:12.589487] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:28:12.589500] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:28:12.589521] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6bc, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:28:15.398144] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462892
 to processed directory
[2016-02-03 01:28:27.604453] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:28:27.604510] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1725, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:27.604530] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:28:27.604542] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:28:27.604557] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2482 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462907.604293), (vec: 1, 
len: 4104)
[2016-02-03 01:28:27.604580] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:28:27.604594] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:28:27.604615] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6bd, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:28:28.066455] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462907
 to processed directory
[2016-02-03 01:28:57.634864] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:28:57.634938] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1726, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:28:57.635006] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:28:57.635028] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:28:57.635049] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2483 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462937.634652), (vec: 1, 
len: 4104)
[2016-02-03 01:28:57.635079] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:28:57.635098] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:28:57.635127] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6be, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:29:09.211888] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462937
 to processed directory
[2016-02-03 01:29:12.650651] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:29:12.650705] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1727, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:29:12.650726] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:29:12.650739] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:29:12.650755] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2484 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454462952.650534), (vec: 1, 
len: 4104)
[2016-02-03 01:29:12.650787] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:29:12.650802] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:29:12.650824] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6bf, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:29:26.622432] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462952
 to processed directory
[2016-02-03 01:30:12.693458] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:30:12.693499] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1728, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:30:12.693528] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:30:12.693544] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:30:12.693562] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2485 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454463012.693353), (vec: 1, 
len: 4104)
[2016-02-03 01:30:12.693587] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:30:12.693610] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:30:12.693638] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6c0, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:30:15.432599] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463012
 to processed directory
[2016-02-03 01:31:12.754211] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:31:12.754243] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1729, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:31:12.754260] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:31:12.754273] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:31:12.754294] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2486 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454463072.754125), (vec: 1, 
len: 4104)
[2016-02-03 01:31:12.754323] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:31:12.754338] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:31:12.754357] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6c1, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:31:13.156812] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463072
 to processed directory
[2016-02-03 01:32:12.816173] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:32:12.816219] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1730, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:32:12.816236] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:32:12.816248] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:32:12.816261] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2487 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454463132.816061), (vec: 1, 
len: 4104)
[2016-02-03 01:32:12.816298] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:32:12.816319] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:32:12.816361] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6c2, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:32:13.864741] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463132
 to processed directory
[2016-02-03 01:33:12.877209] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2
[2016-02-03 01:33:12.877247] T [rpcsvc.c:495:rpcsvc_request_create] 
0-rpc-service: received rpc-message (XID: 0x1731, Ver: 2, Program: 1886350951, 
ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog)
[2016-02-03 01:33:12.877276] T 
[auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: 
pid: 0, uid: 0, gid: 0, owner: 00000000
[2016-02-03 01:33:12.877308] T [rpcsvc.c:316:rpcsvc_program_actor] 
0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for
[2016-02-03 01:33:12.877332] D 
[gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2488 
[/data/glusterfs/prd_items_brick1/brick1] (time: 1454463192.877131), (vec: 1, 
len: 4104)
[2016-02-03 01:33:12.877360] T [rpcsvc.c:1273:rpcsvc_submit_generic] 
0-rpc-service: Tx message: 8
[2016-02-03 01:33:12.877380] T [rpcsvc.c:863:rpcsvc_record_build_header] 
0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24
[2016-02-03 01:33:12.877408] T [rpcsvc.c:1321:rpcsvc_submit_generic] 
0-rpc-service: submitted reply for rpc-message (XID: 0x6c3, Program: 
LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport 
(socket.gfchangelog)
[2016-02-03 01:33:14.577963] D [gf-changelog-api.c:53:gf_changelog_done] 
0-gfchangelog: moving 
/var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463192
 to processed directory


Snippet from 
/var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1.gluster.log
=============
[2016-02-01 20:45:23.591390] I [MSGID: 108031] 
[afr-common.c:1783:afr_local_discovery_cbk] 0-prd_items-replicate-0: selecting 
local read_child prd_items-client-0
[2016-02-03 01:27:58.464278] E [fuse-bridge.c:3336:fuse_xattr_cbk] 
0-glusterfs-fuse: extended attribute not supported by the backend storage
[2016-02-03 01:29:03.558047] E [fuse-bridge.c:3336:fuse_xattr_cbk] 
0-glusterfs-fuse: extended attribute not supported by the backend storage
[2016-02-03 01:29:25.386117] E [fuse-bridge.c:3336:fuse_xattr_cbk] 
0-glusterfs-fuse: extended attribute not supported by the backend storage






_______________________________________________
Gluster-users mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-users



_______________________________________________
Gluster-users mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-users

Reply via email to