Re: [ceph-users] Multisite Ceph setup sync issue
Hi Amit,
Still same. Please see the below output. Anything else I can try? Please update
[cephuser@zabbix-client ~]$ radosgw-admin period update --commit 2>/dev/null
Sending period to new master zone 71931e0e-1be6-449f-af34-edb4166c4e4a
[cephuser@zabbix-client ~]$ sudo systemctl start ceph-radosgw@rgw.`hostname -s`
[cephuser@zabbix-client ~]$ sudo systemctl restart ceph-radosgw@rgw.`hostname
-s`
[cephuser@zabbix-client ~]$ sudo systemctl status ceph-radosgw@rgw.`hostname -s`
● ceph-radosgw@rgw.zabbix-client.service - Ceph rados gateway
Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled;
vendor preset: disabled)
Active: active (running) since Wed 2019-01-30 22:38:15 IST; 7s ago
Main PID: 8234 (radosgw)
CGroup:
/system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.zabbix-client.service
└─8234 /usr/bin/radosgw -f --cluster ceph --name
client.rgw.zabbix-client --setuser ceph --setgroup ceph
Jan 30 22:38:15 zabbix-client systemd[1]: Started Ceph rados gateway.
[cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1
2>/dev/null
realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose)
metadata sync failed to read sync status: (2) No such file or directory
data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
failed to retrieve sync info: (5) Input/output error
[cephuser@zabbix-client ~]$
/Krishna
From: Amit Ghadge
Sent: Wednesday, January 30, 2019 9:17 PM
To: Krishna Verma
Cc: Casey Bodley ; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Multisite Ceph setup sync issue
EXTERNAL MAIL
Have you commit your changes on slave gateway?
First, run commit command on slave gateway and then try.
-AmitG
On Wed, 30 Jan 2019, 21:06 Krishna Verma,
mailto:kve...@cadence.com>> wrote:
Hi Casey,
Thanks for your reply, however I tried with "--source-zone" option with sync
command but getting below error:
Sync status From slave gateway to master zone "noida1"
[cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1
2>/dev/null
realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose)
metadata sync failed to read sync status: (2) No such file or directory
data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
failed to retrieve sync info: (5) Input/output error
[cephuser@zabbix-client ~]$
Sync status From Master Gateway to slave zone " san-jose":
[cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone san-jose
2>/dev/null
realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
metadata sync no sync (zone is master)
[cephuser@zabbix-server ~]$
Zone detail from master gateway :
[cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null
{
"id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
"name": "noida",
"api_name": "noida",
"is_master": "true",
"endpoints": [
"http:\/\/zabbix-server:7480"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
"zones": [
{
"id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
"name": "noida1",
"endpoints": [
"http:\/\/vlno-ceph01:7480"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "1102c891-d81c-480e-9487-c9f874287d13"
}
[cephuser@zabbix-server ~]$
Zone detail from slave gateway:
[cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null
{
"id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
"name": "noida",
"api_name": "noida",
"is_master": "true",
"endpoints": [
"http:\/\/zabbix-server:7480"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "71931e0e-1b
Re: [ceph-users] Multisite Ceph setup sync issue
Have you commit your changes on slave gateway?
First, run commit command on slave gateway and then try.
-AmitG
On Wed, 30 Jan 2019, 21:06 Krishna Verma, wrote:
> Hi Casey,
>
> Thanks for your reply, however I tried with "--source-zone" option with
> sync command but getting below error:
>
> Sync status From slave gateway to master zone "noida1"
>
> [cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone
> noida1 2>/dev/null
> realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
> zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
>zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose)
> metadata sync failed to read sync status: (2) No such file or directory
> data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
> failed to retrieve sync info: (5) Input/output
> error
> [cephuser@zabbix-client ~]$
>
> Sync status From Master Gateway to slave zone " san-jose":
>
> [cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone
> san-jose 2>/dev/null
> realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
> zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
>zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
> metadata sync no sync (zone is master)
> [cephuser@zabbix-server ~]$
>
> Zone detail from master gateway :
>
> [cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null
> {
> "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
> "name": "noida",
> "api_name": "noida",
> "is_master": "true",
> "endpoints": [
> "http:\/\/zabbix-server:7480"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
> "zones": [
> {
> "id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
> "name": "noida1",
> "endpoints": [
> "http:\/\/vlno-ceph01:7480"
> ],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "1102c891-d81c-480e-9487-c9f874287d13"
> }
>
> [cephuser@zabbix-server ~]$
>
>
> Zone detail from slave gateway:
>
> [cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null
> {
> "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
> "name": "noida",
> "api_name": "noida",
> "is_master": "true",
> "endpoints": [
> "http:\/\/zabbix-server:7480"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
> "zones": [
> {
> "id": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac",
> "name": "san-jose",
> "endpoints": [
> "http:\/\/zabbix-client:7480"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> },
> {
> "id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
> "name": "noida1",
> "endpoints": [
> "http:\/\/vlno-ceph01:7480"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id"
Re: [ceph-users] Multisite Ceph setup sync issue
Hi Casey,
Thanks for your reply, however I tried with "--source-zone" option with sync
command but getting below error:
Sync status From slave gateway to master zone "noida1"
[cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone noida1
2>/dev/null
realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose)
metadata sync failed to read sync status: (2) No such file or directory
data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
failed to retrieve sync info: (5) Input/output error
[cephuser@zabbix-client ~]$
Sync status From Master Gateway to slave zone " san-jose":
[cephuser@zabbix-server ~]$ radosgw-admin sync status --source-zone san-jose
2>/dev/null
realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
metadata sync no sync (zone is master)
[cephuser@zabbix-server ~]$
Zone detail from master gateway :
[cephuser@zabbix-server ~]$ radosgw-admin zonegroup get 2>/dev/null
{
"id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
"name": "noida",
"api_name": "noida",
"is_master": "true",
"endpoints": [
"http:\/\/zabbix-server:7480"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
"zones": [
{
"id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
"name": "noida1",
"endpoints": [
"http:\/\/vlno-ceph01:7480"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "1102c891-d81c-480e-9487-c9f874287d13"
}
[cephuser@zabbix-server ~]$
Zone detail from slave gateway:
[cephuser@zabbix-client ~]$ radosgw-admin zonegroup get 2>/dev/null
{
"id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
"name": "noida",
"api_name": "noida",
"is_master": "true",
"endpoints": [
"http:\/\/zabbix-server:7480"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
"zones": [
{
"id": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac",
"name": "san-jose",
"endpoints": [
"http:\/\/zabbix-client:7480"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
},
{
"id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
"name": "noida1",
"endpoints": [
"http:\/\/vlno-ceph01:7480"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "1102c891-d81c-480e-9487-c9f874287d13"
}
[cephuser@zabbix-client ~]
I need your expert advice.
/Krishna
-Original Message-
From: Casey Bodley
Sent: Wednesday, January 30, 2019 1:54 AM
To: Krishna Verma
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Multisite Ceph setup sync issue
EXTERNAL MAIL
On Tue, Jan 29, 2019 at 12:24 PM Krishna Verma wrote:
>
> Hi Ceph Users,
>
>
>
> I need your to fix sync issue in multisite setup.
>
>
>
> I have 2 cluster in different datacenter that we want to use for
> bidirectional data replication. By followed the documentation
> https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ceph.com_docs_
Re: [ceph-users] Multisite Ceph setup sync issue
On Tue, Jan 29, 2019 at 12:24 PM Krishna Verma wrote:
>
> Hi Ceph Users,
>
>
>
> I need your to fix sync issue in multisite setup.
>
>
>
> I have 2 cluster in different datacenter that we want to use for
> bidirectional data replication. By followed the documentation
> http://docs.ceph.com/docs/master/radosgw/multisite/ I have setup the gateway
> on each site but when I am checking the sync status its getting failed as
> below:
>
>
>
> Admin node at master :
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin data sync status
>
> ERROR: source zone not specified
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin realm list
>
> {
>
> "default_info": "1102c891-d81c-480e-9487-c9f874287d13",
>
> "realms": [
>
> "georep",
>
> "geodata"
>
> ]
>
> }
>
>
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin zonegroup list
>
> read_default_id : 0
>
> {
>
> "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
>
> "zonegroups": [
>
> "noida"
>
> ]
>
> }
>
>
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin zone list
>
> {
>
> "default_info": "71931e0e-1be6-449f-af34-edb4166c4e4a",
>
> "zones": [
>
> "noida1"
>
> ]
>
> }
>
>
>
> [cephuser@vlno-ceph01 cluster]$
>
>
>
> [cephuser@vlno-ceph01 cluster]$ cat ceph.conf
>
> [global]
>
> fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55
>
> mon_initial_members = vlno-ceph01
>
> mon_host = 172.23.16.67
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.23.16.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
> [cephuser@vlno-ceph01 cluster]$
>
>
>
> On Master Gateway :
>
>
>
> [cephuser@zabbix-server ~]$ cat /etc/ceph/ceph.conf
>
> [global]
>
> fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55
>
> mon_initial_members = vlno-ceph01
>
> mon_host = 172.23.16.67
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.23.16.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
> [client.rgw.zabbix-server]
>
> host = zabbix-server
>
> rgw frontends = "civetweb port=7480"
>
> rgw_zone=noida1
>
> [cephuser@zabbix-server ~]$
>
>
>
>
>
> On Secondary site admin node.
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin realm list
>
> {
>
> "default_info": "1102c891-d81c-480e-9487-c9f874287d13",
>
> "realms": [
>
> "georep"
>
> ]
>
> }
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zonegroup list
>
> read_default_id : 0
>
> {
>
> "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
>
> "zonegroups": [
>
> "noida",
>
> "default"
>
> ]
>
> }
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zone list
>
> {
>
> "default_info": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac",
>
> "zones": [
>
> "san-jose"
>
> ]
>
> }
>
>
>
> [cephuser@vlsj-kverma1 cluster]$
>
>
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ cat ceph.conf
>
> [global]
>
> fsid = c626be3a-4536-48b9-8db8-470437052313
>
> mon_initial_members = vlsj-kverma1
>
> mon_host = 172.18.84.131
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.18.84.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
>
>
>
>
> [cephuser@vlsj-kverma1 cluster]$
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin data sync status
>
> 2019-01-28 10:33:12.163298 7f11c24c79c0 1 Cannot find zone
> id=45c690a8-f39c-4b1d-9faf-e0e991ceaaac (name=san-jose), switching to local
> zonegroup configuration
>
> ERROR: source zone not specified
>
> [cephuser@vlsj-kverma1 cluster]$
>
>
>
> On Secondary site Gateway host:
>
>
>
> [cephuser@zabbix-client ceph]$ cat /etc/ceph/ceph.conf
>
> [global]
>
> fsid = c626be3a-4536-48b9-8db8-470437052313
>
> mon_initial_members = vlsj-kverma1
>
> mon_host = 172.18.84.131
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.18.84.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
> [client.rgw.zabbix-client]
>
> host = zabbix-client
>
> rgw frontends = "civetweb port=7480"
>
> rgw_zone=san-jose
>
>
>
> [cephuser@zabbix-client ceph]$
>
>
>
>
>
>
>
> Appreciate any help in the setup.
>
>
>
> /Krishna
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
The 'radosgw-admin data sync status' command requires a --source-zone
argument, which is generally the zone name on the opposite cluster.
But you're probably just looking for the
