Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ingo Reimann]

Hi Ben!



Thanks for your advice. I included the names of our gateways, but did omit 
the external name of the service itself. Now, everything is working again.



And yes, this change is worth a note J



Best regards,



Ingo



Von: Ben Hines [mailto:bhi...@gmail.com]
Gesendet: Dienstag, 23. Mai 2017 02:16
An: Ingo Reimann
Cc: Radoslaw Zarzynski; ceph-users
Betreff: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?



We used this workaround when upgrading to Kraken (which had a similar issue)



>modify the zonegroup and populate the 'hostnames' array with all backend 
>server hostnames as well as the hostname terminated by haproxy



Which i'm fine with. It's definitely a change that should be noted in a more 
prominent release note. Without the hostname in there, ceph interpreted the 
hostname as a bucket name if the hostname rgw was being hit with differed 
from the hostname of the actual server. Pre Kraken, i didn't need that 
setting at all and it just worked.



-Ben



On Mon, May 22, 2017 at 1:11 AM, Ingo Reimann <ireim...@dunkel.de> wrote:

Hi Radek,

are there any news about this issue? We are also stuck with 10.2.5 and can`t
update to 10.2.7.
We use a couple of radosgws that are loadbalanced behind a Keepalived/LVS.
Removal of rgw_dns_name does only help, if I address the gateway directly,
but not in general.

Best regards,

Ingo

-Ursprüngliche Nachricht-
Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von
Radoslaw Zarzynski
Gesendet: Mittwoch, 3. Mai 2017 11:59
An: Łukasz Jagiełło
Cc: ceph-users@lists.ceph.com
Betreff: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?


Hello Łukasz,

Thanks for your testing and sorry for my mistake. It looks that two commits
need to be reverted to get the previous behaviour:

The already mentioned one:
  https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0
Its dependency:
  https://github.com/ceph/ceph/commit/b72fc1b820ede3cd186d887d9d30f7f91fe3764b

They have been merged in the same pull request:
  https://github.com/ceph/ceph/pull/11760
and form the difference visible between v10.2.5 and v10.2.6 in the matter of
"in_hosted_domain" handling:
  https://github.com/ceph/ceph/blame/v10.2.5/src/rgw/rgw_rest.cc#L1773
  https://github.com/ceph/ceph/blame/v10.2.6/src/rgw/rgw_rest.cc#L1781-L1782

I'm really not sure we want to revert them. Still, it can be that they just
unhide a misconfiguration issue while fixing the problems we had with
handling of virtual hosted buckets.

Regards,
Radek

On Wed, May 3, 2017 at 3:12 AM, Łukasz Jagiełło <jagiello.luk...@gmail.com>
wrote:
> Hi,
>
> I tried today revert [1] from 10.2.7 but the problem is still there
> even without the change. Revert to 10.2.5 fix the issue instantly.
>
> https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7
> f4c6d0
>
> On Thu, Apr 27, 2017 at 4:53 AM, Radoslaw Zarzynski
> <rzarzyn...@mirantis.com> wrote:
>>




Ingo Reimann

Teamleiter Technik
Dunkel GmbH <https://www.dunkel.de/>
Dunkel GmbH
Philipp-Reis-Straße 2
65795 Hattersheim
Fon: +49 6190 889-100 <tel:%2B49%206190%20889-100>
Fax: +49 6190 889-399 <tel:%2B49%206190%20889-399>
eMail: supp...@dunkel.de
http://www.Dunkel.de/   Amtsgericht Frankfurt/Main
HRB: 37971
Geschäftsführer: Axel Dunkel
Ust-ID: DE 811622001

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Hines]

We used this workaround when upgrading to Kraken (which had a similar issue)

>modify the zonegroup and populate the 'hostnames' array with all backend
server hostnames as well as the hostname terminated by haproxy

Which i'm fine with. It's definitely a change that should be noted in a
more prominent release note. Without the hostname in there, ceph
interpreted the hostname as a bucket name if the hostname rgw was being hit
with differed from the hostname of the actual server. Pre Kraken, i didn't
need that setting at all and it just worked.

-Ben

On Mon, May 22, 2017 at 1:11 AM, Ingo Reimann <ireim...@dunkel.de> wrote:

> Hi Radek,
>
> are there any news about this issue? We are also stuck with 10.2.5 and
> can`t
> update to 10.2.7.
> We use a couple of radosgws that are loadbalanced behind a Keepalived/LVS.
> Removal of rgw_dns_name does only help, if I address the gateway directly,
> but not in general.
>
> Best regards,
>
> Ingo
>
> -Ursprüngliche Nachricht-
> Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von
> Radoslaw Zarzynski
> Gesendet: Mittwoch, 3. Mai 2017 11:59
> An: Łukasz Jagiełło
> Cc: ceph-users@lists.ceph.com
> Betreff: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
>
> Hello Łukasz,
>
> Thanks for your testing and sorry for my mistake. It looks that two commits
> need to be reverted to get the previous behaviour:
>
> The already mentioned one:
>   https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca
> 16d7f4c6d0
> Its dependency:
>   https://github.com/ceph/ceph/commit/b72fc1b820ede3cd186d887d9d30f7
> f91fe3764b
>
> They have been merged in the same pull request:
>   https://github.com/ceph/ceph/pull/11760
> and form the difference visible between v10.2.5 and v10.2.6 in the matter
> of
> "in_hosted_domain" handling:
>   https://github.com/ceph/ceph/blame/v10.2.5/src/rgw/rgw_rest.cc#L1773
>   https://github.com/ceph/ceph/blame/v10.2.6/src/rgw/rgw_
> rest.cc#L1781-L1782
>
> I'm really not sure we want to revert them. Still, it can be that they just
> unhide a misconfiguration issue while fixing the problems we had with
> handling of virtual hosted buckets.
>
> Regards,
> Radek
>
> On Wed, May 3, 2017 at 3:12 AM, Łukasz Jagiełło <jagiello.luk...@gmail.com
> >
> wrote:
> > Hi,
> >
> > I tried today revert [1] from 10.2.7 but the problem is still there
> > even without the change. Revert to 10.2.5 fix the issue instantly.
> >
> > https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7
> > f4c6d0
> >
> > On Thu, Apr 27, 2017 at 4:53 AM, Radoslaw Zarzynski
> > <rzarzyn...@mirantis.com> wrote:
> >>
>
>
>
> Ingo Reimann
>
> Teamleiter Technik
> Dunkel GmbH <https://www.dunkel.de/>
> Dunkel GmbH
> Philipp-Reis-Straße 2
> 65795 Hattersheim
> Fon: +49 6190 889-100
> Fax: +49 6190 889-399
> eMail: supp...@dunkel.de
> http://www.Dunkel.de/   Amtsgericht Frankfurt/Main
> HRB: 37971
> Geschäftsführer: Axel Dunkel
> Ust-ID: DE 811622001
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ingo Reimann]

Hi Radek,

are there any news about this issue? We are also stuck with 10.2.5 and can`t 
update to 10.2.7.
We use a couple of radosgws that are loadbalanced behind a Keepalived/LVS. 
Removal of rgw_dns_name does only help, if I address the gateway directly, 
but not in general.

Best regards,

Ingo

-Ursprüngliche Nachricht-
Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von 
Radoslaw Zarzynski
Gesendet: Mittwoch, 3. Mai 2017 11:59
An: Łukasz Jagiełło
Cc: ceph-users@lists.ceph.com
Betreff: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

Hello Łukasz,

Thanks for your testing and sorry for my mistake. It looks that two commits 
need to be reverted to get the previous behaviour:

The already mentioned one:
  https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0
Its dependency:
  https://github.com/ceph/ceph/commit/b72fc1b820ede3cd186d887d9d30f7f91fe3764b

They have been merged in the same pull request:
  https://github.com/ceph/ceph/pull/11760
and form the difference visible between v10.2.5 and v10.2.6 in the matter of 
"in_hosted_domain" handling:
  https://github.com/ceph/ceph/blame/v10.2.5/src/rgw/rgw_rest.cc#L1773
  https://github.com/ceph/ceph/blame/v10.2.6/src/rgw/rgw_rest.cc#L1781-L1782

I'm really not sure we want to revert them. Still, it can be that they just 
unhide a misconfiguration issue while fixing the problems we had with 
handling of virtual hosted buckets.

Regards,
Radek

On Wed, May 3, 2017 at 3:12 AM, Łukasz Jagiełło <jagiello.luk...@gmail.com> 
wrote:
> Hi,
>
> I tried today revert [1] from 10.2.7 but the problem is still there
> even without the change. Revert to 10.2.5 fix the issue instantly.
>
> https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7
> f4c6d0
>
> On Thu, Apr 27, 2017 at 4:53 AM, Radoslaw Zarzynski
> <rzarzyn...@mirantis.com> wrote:
>>



Ingo Reimann

Teamleiter Technik
Dunkel GmbH <https://www.dunkel.de/>
Dunkel GmbH
Philipp-Reis-Straße 2
65795 Hattersheim
Fon: +49 6190 889-100
Fax: +49 6190 889-399
eMail: supp...@dunkel.de
http://www.Dunkel.de/   Amtsgericht Frankfurt/Main
HRB: 37971
Geschäftsführer: Axel Dunkel
Ust-ID: DE 811622001
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Łukasz Jagiełło]

chance as the credentials were stored remotely. This explains
> >> >> the presence of "error reading user info" in the user-keystone.txt.
> >> >>
> >> >> What is common for both scenarios are the low-level things related
> >> >> to StringToSign crafting/signature generation at RadosGW's side.
> >> >> Following one has been composed for the request from admin.txt:
> >> >>
> >> >>GET
> >> >>
> >> >>
> >> >>Wed, 26 Apr 2017 09:18:42 GMT
> >> >>/bbpsrvc15.cscs.ch/
> >> >>
> >> >> If you could provide a similar log from v10.2.5, I would be really
> >> >> grateful.
> >> >>
> >> >> Regards,
> >> >> Radek
> >> >>
> >> >> [1]
> >> >>
> >> >> https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_
> s3.cc#L3269-L3272
> >> >> [2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_
> common.h#L170
> >> >>
> >> >> On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch>
> >> >> wrote:
> >> >>>
> >> >>> Hello Radek,
> >> >>>
> >> >>> Please find attached the failed request for both the admin user and
> a
> >> >>> standard user (backed by keystone).
> >> >>>
> >> >>> Kind regards,
> >> >>>
> >> >>> Ben Morrice
> >> >>>
> >> >>> 
> __
> >> >>> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> >> >>> EPFL BBP
> >> >>> Biotech Campus
> >> >>> Chemin des Mines 9
> >> >>> 1202 Geneva
> >> >>> Switzerland
> >> >>>
> >> >>> 
> >> >>> From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
> >> >>> Sent: Tuesday, April 25, 2017 7:38 PM
> >> >>> To: Morrice Ben
> >> >>> Cc: ceph-users@lists.ceph.com
> >> >>> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
> >> >>>
> >> >>> Hello Ben,
> >> >>>
> >> >>> Could you provide full RadosGW's log for the failed request?
> >> >>> I mean the lines starting from header listing, through the start
> >> >>> marker ("== starting new request...") till the end marker?
> >> >>>
> >> >>> At the moment we can't see any details related to the signature
> >> >>> calculation.
> >> >>>
> >> >>> Regards,
> >> >>> Radek
> >> >>>
> >> >>> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi all,
> >> >>>>
> >> >>>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7
> >> >>>> (RHEL7)
> >> >>>> and authentication is in a very bad state. This installation is
> part
> >> >>>> of
> >> >>>> a
> >> >>>> multigw configuration, and I have just updated one host in the
> >> >>>> secondary
> >> >>>> zone (all other hosts/zones are running 10.2.5).
> >> >>>>
> >> >>>> On the 10.2.7 server I cannot authenticate as a user (normally
> backed
> >> >>>> by
> >> >>>> OpenStack Keystone), but even worse I can also not authenticate
> with
> >> >>>> an
> >> >>>> admin user.
> >> >>>>
> >> >>>> Please see [1] for the results of performing a list bucket
> operation
> >> >>>> with
> >> >>>> python boto (script works against rgw 10.2.5)
> >> >>>>
> >> >>>> Also, if I try to authenticate from the 'master' rgw zone with a
> >> >>>> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
> >> >>>>
> >> >>>> "ERROR: failed to fetch datalog info"
> >> >>>>
> >> >>>> "failed to retrie

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Radoslaw Zarzynski]

> Regards,
>> >> Radek
>> >>
>> >> [1]
>> >>
>> >> https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
>> >> [2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170
>> >>
>> >> On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch>
>> >> wrote:
>> >>>
>> >>> Hello Radek,
>> >>>
>> >>> Please find attached the failed request for both the admin user and a
>> >>> standard user (backed by keystone).
>> >>>
>> >>> Kind regards,
>> >>>
>> >>> Ben Morrice
>> >>>
>> >>> __
>> >>> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
>> >>> EPFL BBP
>> >>> Biotech Campus
>> >>> Chemin des Mines 9
>> >>> 1202 Geneva
>> >>> Switzerland
>> >>>
>> >>> 
>> >>> From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
>> >>> Sent: Tuesday, April 25, 2017 7:38 PM
>> >>> To: Morrice Ben
>> >>> Cc: ceph-users@lists.ceph.com
>> >>> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
>> >>>
>> >>> Hello Ben,
>> >>>
>> >>> Could you provide full RadosGW's log for the failed request?
>> >>> I mean the lines starting from header listing, through the start
>> >>> marker ("== starting new request...") till the end marker?
>> >>>
>> >>> At the moment we can't see any details related to the signature
>> >>> calculation.
>> >>>
>> >>> Regards,
>> >>> Radek
>> >>>
>> >>> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch>
>> >>> wrote:
>> >>>>
>> >>>> Hi all,
>> >>>>
>> >>>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7
>> >>>> (RHEL7)
>> >>>> and authentication is in a very bad state. This installation is part
>> >>>> of
>> >>>> a
>> >>>> multigw configuration, and I have just updated one host in the
>> >>>> secondary
>> >>>> zone (all other hosts/zones are running 10.2.5).
>> >>>>
>> >>>> On the 10.2.7 server I cannot authenticate as a user (normally backed
>> >>>> by
>> >>>> OpenStack Keystone), but even worse I can also not authenticate with
>> >>>> an
>> >>>> admin user.
>> >>>>
>> >>>> Please see [1] for the results of performing a list bucket operation
>> >>>> with
>> >>>> python boto (script works against rgw 10.2.5)
>> >>>>
>> >>>> Also, if I try to authenticate from the 'master' rgw zone with a
>> >>>> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>> >>>>
>> >>>> "ERROR: failed to fetch datalog info"
>> >>>>
>> >>>> "failed to retrieve sync info: (13) Permission denied"
>> >>>>
>> >>>> The above errors correlates to the errors in the log on the server
>> >>>> running
>> >>>> 10.2.7 (debug level 20) at [2]
>> >>>>
>> >>>> I'm not sure what I have done wrong or can try next?
>> >>>>
>> >>>> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
>> >>>> authentication functionality
>> >>>>
>> >>>> [1]
>> >>>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>> >>>> > >>>>
>> >>>>
>> >>>> encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>> >>>>
>> >>>> [2]
>> >>>> /bbpsrvc15.cscs.ch/admin/log
>> >>>> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
>> >>>> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
>> >>>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
>> >>>

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Morrice]

Hello again,

I can work around this issue. If the host header is an IP address, the 
request is treated as a virtual:


So if I auth to to my backends via IP, things work as expected.

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 28/04/17 09:26, Ben Morrice wrote:

Hello Radek,

Thanks again for your anaylsis.

I can confirm on 10.2.7, if I remove the conf "rgw dns name" I can 
auth to directly to the radosgw host.


In our environment we terminate SSL and route connections via haproxy, 
but it's still sometimes useful to be able to communicate directly to 
the backend radosgw server.


It seems that it's not possible to set multiple "rgw dns name" entries 
in ceph.conf


Is the only solution to modify the zonegroup and populate the 
'hostnames' array with all backend server hostnames as well as the 
hostname terminated by haproxy?


Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 13:53, Radoslaw Zarzynski wrote:

Bingo! From the 10.2.5-admin:

   GET

   Thu, 27 Apr 2017 07:49:59 GMT
   /

And also:

   2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0
   2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
s->info.domain= s->info.request_uri=/

The most interesting part is the "final ... in_hosted_domain=0".
It looks we need to dig around RGWREST::preprocess(),
rgw_find_host_in_domains() & company.

There is a commit introduced in v10.2.6 that touches this area [1].
I'm definitely not saying it's the root cause. It might be that a change
in the code just unhidden a configuration issue [2].

I will talk about the problem on the today's sync-up.

Thanks for the logs!
Regards,
Radek

[1] 
https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0

[2] http://tracker.ceph.com/issues/17440

On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice <ben.morr...@epfl.ch> 
wrote:

Hello Radek,

Thank-you for your analysis so far! Please find attached logs for 
both the
admin user and a keystone backed user from 10.2.5 (same host as 
before, I
have simply downgraded the packages). Both users can authenticate 
and list

buckets on 10.2.5.

Also - I tried version 10.2.6 and see the same behavior as 10.2.7, 
so the

bug i'm hitting looks like it was introduced in 10.2.6

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 04:45, Radoslaw Zarzynski wrote:

Thanks for the logs, Ben.

It looks that two completely different authenticators have failed:
the local, RADOS-backed auth (admin.txt) and Keystone-based
one as well. In the second case I'm pretty sure that Keystone has
rejected [1][2] to authenticate provided signature/StringToSign.
RGW tried to fallback to the local auth which obviously didn't have
any chance as the credentials were stored remotely. This explains
the presence of "error reading user info" in the user-keystone.txt.

What is common for both scenarios are the low-level things related
to StringToSign crafting/signature generation at RadosGW's side.
Following one has been composed for the request from admin.txt:

GET


Wed, 26 Apr 2017 09:18:42 GMT
/bbpsrvc15.cscs.ch/

If you could provide a similar log from v10.2.5, I would be really
grateful.

Regards,
Radek

[1]
https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272 

[2] 
https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170


On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch> 
wrote:

Hello Radek,

Please find attached the failed request for both the admin user and a
standard user (backed by keystone).

Kind regards,

Ben Morrice

__ 


Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland


From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
Sent: Tuesday, April 25, 2017 7:38 PM
To: Morrice Ben
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

Hello Ben,

Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("== starting new request...") till the end marker?

At the moment we can't see any details relat

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Morrice]

Hello Radek,

Thanks again for your anaylsis.

I can confirm on 10.2.7, if I remove the conf "rgw dns name" I can auth 
to directly to the radosgw host.


In our environment we terminate SSL and route connections via haproxy, 
but it's still sometimes useful to be able to communicate directly to 
the backend radosgw server.


It seems that it's not possible to set multiple "rgw dns name" entries 
in ceph.conf


Is the only solution to modify the zonegroup and populate the 
'hostnames' array with all backend server hostnames as well as the 
hostname terminated by haproxy?


Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 13:53, Radoslaw Zarzynski wrote:

Bingo! From the 10.2.5-admin:

   GET

   Thu, 27 Apr 2017 07:49:59 GMT
   /

And also:

   2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0
   2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
s->info.domain= s->info.request_uri=/

The most interesting part is the "final ... in_hosted_domain=0".
It looks we need to dig around RGWREST::preprocess(),
rgw_find_host_in_domains() & company.

There is a commit introduced in v10.2.6 that touches this area [1].
I'm definitely not saying it's the root cause. It might be that a change
in the code just unhidden a configuration issue [2].

I will talk about the problem on the today's sync-up.

Thanks for the logs!
Regards,
Radek

[1] https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0
[2] http://tracker.ceph.com/issues/17440

On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice <ben.morr...@epfl.ch> wrote:

Hello Radek,

Thank-you for your analysis so far! Please find attached logs for both the
admin user and a keystone backed user from 10.2.5 (same host as before, I
have simply downgraded the packages). Both users can authenticate and list
buckets on 10.2.5.

Also - I tried version 10.2.6 and see the same behavior as 10.2.7, so the
bug i'm hitting looks like it was introduced in 10.2.6

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 04:45, Radoslaw Zarzynski wrote:

Thanks for the logs, Ben.

It looks that two completely different authenticators have failed:
the local, RADOS-backed auth (admin.txt) and Keystone-based
one as well. In the second case I'm pretty sure that Keystone has
rejected [1][2] to authenticate provided signature/StringToSign.
RGW tried to fallback to the local auth which obviously didn't have
any chance as the credentials were stored remotely. This explains
the presence of "error reading user info" in the user-keystone.txt.

What is common for both scenarios are the low-level things related
to StringToSign crafting/signature generation at RadosGW's side.
Following one has been composed for the request from admin.txt:

GET


Wed, 26 Apr 2017 09:18:42 GMT
/bbpsrvc15.cscs.ch/

If you could provide a similar log from v10.2.5, I would be really
grateful.

Regards,
Radek

[1]
https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
[2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170

On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch> wrote:

Hello Radek,

Please find attached the failed request for both the admin user and a
standard user (backed by keystone).

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland


From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
Sent: Tuesday, April 25, 2017 7:38 PM
To: Morrice Ben
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

Hello Ben,

Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("== starting new request...") till the end marker?

At the moment we can't see any details related to the signature
calculation.

Regards,
Radek

On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch> wrote:

Hi all,

I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7
(RHEL7)
and authentication is in a very bad state. This installation is part of
a
multigw configuration, and I have just updated one host in the secondary
zone (all other hosts/zones are running 10.2.5).

On the 10.2.7 server I cannot authenticate as a u

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Radoslaw Zarzynski]

Bingo! From the 10.2.5-admin:

  GET

  Thu, 27 Apr 2017 07:49:59 GMT
  /

And also:

  2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0
  2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
s->info.domain= s->info.request_uri=/

The most interesting part is the "final ... in_hosted_domain=0".
It looks we need to dig around RGWREST::preprocess(),
rgw_find_host_in_domains() & company.

There is a commit introduced in v10.2.6 that touches this area [1].
I'm definitely not saying it's the root cause. It might be that a change
in the code just unhidden a configuration issue [2].

I will talk about the problem on the today's sync-up.

Thanks for the logs!
Regards,
Radek

[1] https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0
[2] http://tracker.ceph.com/issues/17440

On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice <ben.morr...@epfl.ch> wrote:
> Hello Radek,
>
> Thank-you for your analysis so far! Please find attached logs for both the
> admin user and a keystone backed user from 10.2.5 (same host as before, I
> have simply downgraded the packages). Both users can authenticate and list
> buckets on 10.2.5.
>
> Also - I tried version 10.2.6 and see the same behavior as 10.2.7, so the
> bug i'm hitting looks like it was introduced in 10.2.6
>
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> On 27/04/17 04:45, Radoslaw Zarzynski wrote:
>>
>> Thanks for the logs, Ben.
>>
>> It looks that two completely different authenticators have failed:
>> the local, RADOS-backed auth (admin.txt) and Keystone-based
>> one as well. In the second case I'm pretty sure that Keystone has
>> rejected [1][2] to authenticate provided signature/StringToSign.
>> RGW tried to fallback to the local auth which obviously didn't have
>> any chance as the credentials were stored remotely. This explains
>> the presence of "error reading user info" in the user-keystone.txt.
>>
>> What is common for both scenarios are the low-level things related
>> to StringToSign crafting/signature generation at RadosGW's side.
>> Following one has been composed for the request from admin.txt:
>>
>>GET
>>
>>
>>Wed, 26 Apr 2017 09:18:42 GMT
>>/bbpsrvc15.cscs.ch/
>>
>> If you could provide a similar log from v10.2.5, I would be really
>> grateful.
>>
>> Regards,
>> Radek
>>
>> [1]
>> https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
>> [2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170
>>
>> On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch> wrote:
>>>
>>> Hello Radek,
>>>
>>> Please find attached the failed request for both the admin user and a
>>> standard user (backed by keystone).
>>>
>>> Kind regards,
>>>
>>> Ben Morrice
>>>
>>> __________________
>>> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
>>> EPFL BBP
>>> Biotech Campus
>>> Chemin des Mines 9
>>> 1202 Geneva
>>> Switzerland
>>>
>>> 
>>> From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
>>> Sent: Tuesday, April 25, 2017 7:38 PM
>>> To: Morrice Ben
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
>>>
>>> Hello Ben,
>>>
>>> Could you provide full RadosGW's log for the failed request?
>>> I mean the lines starting from header listing, through the start
>>> marker ("== starting new request...") till the end marker?
>>>
>>> At the moment we can't see any details related to the signature
>>> calculation.
>>>
>>> Regards,
>>> Radek
>>>
>>> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7
>>>> (RHEL7)
>>>> and authentication is in a very bad state. This installation is part of
>>>> a
>>>> multigw configuration, and I have just updated one host 

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Morrice]

Hello Radek,

Thank-you for your analysis so far! Please find attached logs for both 
the admin user and a keystone backed user from 10.2.5 (same host as 
before, I have simply downgraded the packages). Both users can 
authenticate and list buckets on 10.2.5.


Also - I tried version 10.2.6 and see the same behavior as 10.2.7, so 
the bug i'm hitting looks like it was introduced in 10.2.6


Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 04:45, Radoslaw Zarzynski wrote:

Thanks for the logs, Ben.

It looks that two completely different authenticators have failed:
the local, RADOS-backed auth (admin.txt) and Keystone-based
one as well. In the second case I'm pretty sure that Keystone has
rejected [1][2] to authenticate provided signature/StringToSign.
RGW tried to fallback to the local auth which obviously didn't have
any chance as the credentials were stored remotely. This explains
the presence of "error reading user info" in the user-keystone.txt.

What is common for both scenarios are the low-level things related
to StringToSign crafting/signature generation at RadosGW's side.
Following one has been composed for the request from admin.txt:

   GET


   Wed, 26 Apr 2017 09:18:42 GMT
   /bbpsrvc15.cscs.ch/

If you could provide a similar log from v10.2.5, I would be really grateful.

Regards,
Radek

[1] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
[2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170

On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch> wrote:

Hello Radek,

Please find attached the failed request for both the admin user and a standard 
user (backed by keystone).

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland


From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
Sent: Tuesday, April 25, 2017 7:38 PM
To: Morrice Ben
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

Hello Ben,

Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("== starting new request...") till the end marker?

At the moment we can't see any details related to the signature
calculation.

Regards,
Radek

On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch> wrote:

Hi all,

I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
and authentication is in a very bad state. This installation is part of a
multigw configuration, and I have just updated one host in the secondary
zone (all other hosts/zones are running 10.2.5).

On the 10.2.7 server I cannot authenticate as a user (normally backed by
OpenStack Keystone), but even worse I can also not authenticate with an
admin user.

Please see [1] for the results of performing a list bucket operation with
python boto (script works against rgw 10.2.5)

Also, if I try to authenticate from the 'master' rgw zone with a
"radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:

"ERROR: failed to fetch datalog info"

"failed to retrieve sync info: (13) Permission denied"

The above errors correlates to the errors in the log on the server running
10.2.7 (debug level 20) at [2]

I'm not sure what I have done wrong or can try next?

By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
authentication functionality

[1]
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva

[2]
/bbpsrvc15.cscs.ch/admin/log
2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
2017-04-20 16:43:04.916255 7ff87c6c0700 15
auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
err_no=-2027 new_err_no=-2027
2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
/admin/log:get_obj:op status=0
2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
/admin/log:get_obj:http status=403
2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
req=0x7ff87c6ba710 op status=0 http_status=403 ==
2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
- -
2017-04-20 16:43:

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Radoslaw Zarzynski]

Thanks for the logs, Ben.

It looks that two completely different authenticators have failed:
the local, RADOS-backed auth (admin.txt) and Keystone-based
one as well. In the second case I'm pretty sure that Keystone has
rejected [1][2] to authenticate provided signature/StringToSign.
RGW tried to fallback to the local auth which obviously didn't have
any chance as the credentials were stored remotely. This explains
the presence of "error reading user info" in the user-keystone.txt.

What is common for both scenarios are the low-level things related
to StringToSign crafting/signature generation at RadosGW's side.
Following one has been composed for the request from admin.txt:

  GET


  Wed, 26 Apr 2017 09:18:42 GMT
  /bbpsrvc15.cscs.ch/

If you could provide a similar log from v10.2.5, I would be really grateful.

Regards,
Radek

[1] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
[2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170

On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch> wrote:
> Hello Radek,
>
> Please find attached the failed request for both the admin user and a 
> standard user (backed by keystone).
>
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> 
> From: Radoslaw Zarzynski <rzarzyn...@mirantis.com>
> Sent: Tuesday, April 25, 2017 7:38 PM
> To: Morrice Ben
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
>
> Hello Ben,
>
> Could you provide full RadosGW's log for the failed request?
> I mean the lines starting from header listing, through the start
> marker ("== starting new request...") till the end marker?
>
> At the moment we can't see any details related to the signature
> calculation.
>
> Regards,
> Radek
>
> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch> wrote:
>> Hi all,
>>
>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
>> and authentication is in a very bad state. This installation is part of a
>> multigw configuration, and I have just updated one host in the secondary
>> zone (all other hosts/zones are running 10.2.5).
>>
>> On the 10.2.7 server I cannot authenticate as a user (normally backed by
>> OpenStack Keystone), but even worse I can also not authenticate with an
>> admin user.
>>
>> Please see [1] for the results of performing a list bucket operation with
>> python boto (script works against rgw 10.2.5)
>>
>> Also, if I try to authenticate from the 'master' rgw zone with a
>> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>>
>> "ERROR: failed to fetch datalog info"
>>
>> "failed to retrieve sync info: (13) Permission denied"
>>
>> The above errors correlates to the errors in the log on the server running
>> 10.2.7 (debug level 20) at [2]
>>
>> I'm not sure what I have done wrong or can try next?
>>
>> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
>> authentication functionality
>>
>> [1]
>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>> > encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>>
>> [2]
>> /bbpsrvc15.cscs.ch/admin/log
>> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
>> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
>> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
>> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
>> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
>> err_no=-2027 new_err_no=-2027
>> 2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
>> /admin/log:get_obj:op status=0
>> 2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
>> /admin/log:get_obj:http status=403
>> 2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
>> req=0x7ff87c6ba710 op status=0 http_status=403 ==
>> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
>> 2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
>> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
>> - -
>> 2017-04-

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Radoslaw Zarzynski]

Hello Ben,

Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("== starting new request...") till the end marker?

At the moment we can't see any details related to the signature
calculation.

Regards,
Radek

On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice  wrote:
> Hi all,
>
> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
> and authentication is in a very bad state. This installation is part of a
> multigw configuration, and I have just updated one host in the secondary
> zone (all other hosts/zones are running 10.2.5).
>
> On the 10.2.7 server I cannot authenticate as a user (normally backed by
> OpenStack Keystone), but even worse I can also not authenticate with an
> admin user.
>
> Please see [1] for the results of performing a list bucket operation with
> python boto (script works against rgw 10.2.5)
>
> Also, if I try to authenticate from the 'master' rgw zone with a
> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>
> "ERROR: failed to fetch datalog info"
>
> "failed to retrieve sync info: (13) Permission denied"
>
> The above errors correlates to the errors in the log on the server running
> 10.2.7 (debug level 20) at [2]
>
> I'm not sure what I have done wrong or can try next?
>
> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
> authentication functionality
>
> [1]
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>  encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>
> [2]
> /bbpsrvc15.cscs.ch/admin/log
> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
> err_no=-2027 new_err_no=-2027
> 2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
> /admin/log:get_obj:op status=0
> 2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
> /admin/log:get_obj:http status=403
> 2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
> req=0x7ff87c6ba710 op status=0 http_status=403 ==
> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
> 2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
> - -
> 2017-04-20 16:43:04.917212 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1544: shard_id=20
> mdlog_marker=1_1492686039.901886_5551978.1
> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
> 2017-04-20 16:43:04.917236 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: init request
> 2017-04-20 16:43:04.917240 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status
> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
> blocked
> 2017-04-20 16:43:04.918285 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status complete
> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
> 13:00:39.0.901886s
> 2017-04-20 16:43:04.918316 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: sending rest request
> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
> 20 14:43:04 2017
> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
> 14:43:04 2017
> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
> dest=/admin/log
> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
>
> --
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> 

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Morrice]

Hello Orit,

Could it be that something has changed in 10.2.5+ which is related to 
reading the endpoints from the zone/period config?


In my master zone I have specified the endpoint with a trailing 
backslash (which is also escaped), however I do not define the secondary 
endpoint this way. Am I hitting a bug here?


Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 21/04/17 09:36, Ben Morrice wrote:

Hello Orit,

Please find attached the output from the radosgw commands and the 
relevant section from ceph.conf (radosgw)


bbp-gva-master is running 10.2.5

bbp-gva-secondary is running 10.2.7

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 21/04/17 07:55, Orit Wasserman wrote:

Hi Ben,

On Thu, Apr 20, 2017 at 6:08 PM, Ben Morrice  
wrote:

Hi all,

I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 
(RHEL7)
and authentication is in a very bad state. This installation is part 
of a
multigw configuration, and I have just updated one host in the 
secondary

zone (all other hosts/zones are running 10.2.5).

On the 10.2.7 server I cannot authenticate as a user (normally 
backed by

OpenStack Keystone), but even worse I can also not authenticate with an
admin user.

Please see [1] for the results of performing a list bucket operation 
with

python boto (script works against rgw 10.2.5)

Also, if I try to authenticate from the 'master' rgw zone with a
"radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:

"ERROR: failed to fetch datalog info"

"failed to retrieve sync info: (13) Permission denied"

The above errors correlates to the errors in the log on the server 
running

10.2.7 (debug level 20) at [2]

I'm not sure what I have done wrong or can try next?

By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
authentication functionality

Can you provide the following info:
radosgw-admin period get
radsogw-admin zonegroup get
radsogw-admin zone get

Can you provide your ceph.conf?

Thanks,
Orit


[1]
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva 



[2]
/bbpsrvc15.cscs.ch/admin/log
2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
2017-04-20 16:43:04.916255 7ff87c6c0700 15
auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
err_no=-2027 new_err_no=-2027
2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
/admin/log:get_obj:op status=0
2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
/admin/log:get_obj:http status=403
2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
req=0x7ff87c6ba710 op status=0 http_status=403 ==
2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() 
returned -2027

2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log 
HTTP/1.1" 403 0

- -
2017-04-20 16:43:04.917212 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
incremental_sync:1544: shard_id=20
mdlog_marker=1_1492686039.901886_5551978.1
sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
2017-04-20 16:43:04.917236 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: 
operate()

2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: init request
2017-04-20 16:43:04.917240 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: 
operate()

2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status
2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 
is io

blocked
2017-04-20 16:43:04.918285 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: 
operate()

2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status complete
2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
13:00:39.0.901886s
2017-04-20 16:43:04.918316 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: 
operate()

2017-04-20 

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Morrice]

Hello Orit,

Please find attached the output from the radosgw commands and the 
relevant section from ceph.conf (radosgw)


bbp-gva-master is running 10.2.5

bbp-gva-secondary is running 10.2.7

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 21/04/17 07:55, Orit Wasserman wrote:

Hi Ben,

On Thu, Apr 20, 2017 at 6:08 PM, Ben Morrice  wrote:

Hi all,

I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
and authentication is in a very bad state. This installation is part of a
multigw configuration, and I have just updated one host in the secondary
zone (all other hosts/zones are running 10.2.5).

On the 10.2.7 server I cannot authenticate as a user (normally backed by
OpenStack Keystone), but even worse I can also not authenticate with an
admin user.

Please see [1] for the results of performing a list bucket operation with
python boto (script works against rgw 10.2.5)

Also, if I try to authenticate from the 'master' rgw zone with a
"radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:

"ERROR: failed to fetch datalog info"

"failed to retrieve sync info: (13) Permission denied"

The above errors correlates to the errors in the log on the server running
10.2.7 (debug level 20) at [2]

I'm not sure what I have done wrong or can try next?

By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
authentication functionality

Can you provide the following info:
radosgw-admin period get
radsogw-admin zonegroup get
radsogw-admin zone get

Can you provide your ceph.conf?

Thanks,
Orit


[1]
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva

[2]
/bbpsrvc15.cscs.ch/admin/log
2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
2017-04-20 16:43:04.916255 7ff87c6c0700 15
auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
err_no=-2027 new_err_no=-2027
2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
/admin/log:get_obj:op status=0
2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
/admin/log:get_obj:http status=403
2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
req=0x7ff87c6ba710 op status=0 http_status=403 ==
2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
- -
2017-04-20 16:43:04.917212 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
incremental_sync:1544: shard_id=20
mdlog_marker=1_1492686039.901886_5551978.1
sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
2017-04-20 16:43:04.917236 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: init request
2017-04-20 16:43:04.917240 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status
2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
blocked
2017-04-20 16:43:04.918285 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status complete
2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
13:00:39.0.901886s
2017-04-20 16:43:04.918316 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: sending rest request
2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
20 14:43:04 2017
2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
14:43:04 2017
2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
dest=/admin/log
2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET

--
Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva

Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Orit Wasserman]

Hi Ben,

On Thu, Apr 20, 2017 at 6:08 PM, Ben Morrice  wrote:
> Hi all,
>
> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
> and authentication is in a very bad state. This installation is part of a
> multigw configuration, and I have just updated one host in the secondary
> zone (all other hosts/zones are running 10.2.5).
>
> On the 10.2.7 server I cannot authenticate as a user (normally backed by
> OpenStack Keystone), but even worse I can also not authenticate with an
> admin user.
>
> Please see [1] for the results of performing a list bucket operation with
> python boto (script works against rgw 10.2.5)
>
> Also, if I try to authenticate from the 'master' rgw zone with a
> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>
> "ERROR: failed to fetch datalog info"
>
> "failed to retrieve sync info: (13) Permission denied"
>
> The above errors correlates to the errors in the log on the server running
> 10.2.7 (debug level 20) at [2]
>
> I'm not sure what I have done wrong or can try next?
>
> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
> authentication functionality

Can you provide the following info:
radosgw-admin period get
radsogw-admin zonegroup get
radsogw-admin zone get

Can you provide your ceph.conf?

Thanks,
Orit

>
> [1]
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>  encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>
> [2]
> /bbpsrvc15.cscs.ch/admin/log
> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
> err_no=-2027 new_err_no=-2027
> 2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
> /admin/log:get_obj:op status=0
> 2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
> /admin/log:get_obj:http status=403
> 2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done
> req=0x7ff87c6ba710 op status=0 http_status=403 ==
> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
> 2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
> - -
> 2017-04-20 16:43:04.917212 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1544: shard_id=20
> mdlog_marker=1_1492686039.901886_5551978.1
> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
> 2017-04-20 16:43:04.917236 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: init request
> 2017-04-20 16:43:04.917240 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status
> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
> blocked
> 2017-04-20 16:43:04.918285 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status complete
> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
> 13:00:39.0.901886s
> 2017-04-20 16:43:04.918316 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: sending rest request
> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
> 20 14:43:04 2017
> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
> 14:43:04 2017
> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
> dest=/admin/log
> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
>
> --
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list

[ceph-users] RGW 10.2.5->10.2.7 authentication fail?

[Ben Morrice]

Hi all,

I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 
(RHEL7) and authentication is in a very bad state. This installation is 
part of a multigw configuration, and I have just updated one host in the 
secondary zone (all other hosts/zones are running 10.2.5).


On the 10.2.7 server I cannot authenticate as a user (normally backed by 
OpenStack Keystone), but even worse I can also not authenticate with an 
admin user.


Please see [1] for the results of performing a list bucket operation 
with python boto (script works against rgw 10.2.5)


Also, if I try to authenticate from the 'master' rgw zone with a 
"radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:


"ERROR: failed to fetch datalog info"

"failed to retrieve sync info: (13) Permission denied"

The above errors correlates to the errors in the log on the server 
running 10.2.7 (debug level 20) at [2]


I'm not sure what I have done wrong or can try next?

By the way, downgrading the packages from 10.2.7 to 10.2.5 returns 
authentication functionality


[1]
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva


[2]
/bbpsrvc15.cscs.ch/admin/log
2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated 
digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
2017-04-20 16:43:04.916255 7ff87c6c0700 15 
auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=

2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER: 
err_no=-2027 new_err_no=-2027
2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET 
/admin/log:get_obj:op status=0
2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET 
/admin/log:get_obj:http status=403
2017-04-20 16:43:04.916343 7ff87c6c0700  1 == req done 
req=0x7ff87c6ba710 op status=0 http_status=403 ==

2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610: 
10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 
403 0 - -
2017-04-20 16:43:04.917212 7ff9777e6700 20 
cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync: 
incremental_sync:1544: shard_id=20 
mdlog_marker=1_1492686039.901886_5551978.1 
sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync: 
incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
2017-04-20 16:43:04.917236 7ff9777e6700 20 
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate: 
shard_id=20: init request
2017-04-20 16:43:04.917240 7ff9777e6700 20 
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate: 
shard_id=20: reading shard status
2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is 
io blocked
2017-04-20 16:43:04.918285 7ff9777e6700 20 
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate: 
shard_id=20: reading shard status complete
2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20 
marker=1_1492686039.901886_5551978.1 last_update=2017-04-20 
13:00:39.0.901886s
2017-04-20 16:43:04.918316 7ff9777e6700 20 
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate: 
shard_id=20: sending rest request
2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu 
Apr 20 14:43:04 2017
2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20 
14:43:04 2017
2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource(): 
dest=/admin/log

2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET

--
Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com