Re: [ceph-users] Possible way to clean up leaked multipart objects?

[William Schroeder]

David,

We would love some testing of the tool. Are you set up to compile and deploy 
Ceph changes? If your situation is not related to the leaked multipart objects 
due to retries, it will let you know nothing was fixed.  That is still a useful 
test.

Another variation of multipart leak comes from multipart sessions not being 
aborted or completed. That is something that Ceph tools can already assist 
with. My fix is for objects that do not show up in standard bucket listings but 
are accounted for in bucket stats.

On Aug 31, 2017, at 4:26 PM, David Turner 
mailto:drakonst...@gmail.com>> wrote:

Jewel 10.2.7.  I found a discrepancy in object counts for a multisite 
configuration and it's looking like it might be orphaned multipart files 
causing it.  It doesn't look like this PR has received much attention.  Is 
there anything I can do to help you with testing/confirming a use case for this 
tool?

On Tue, Aug 29, 2017 at 5:28 PM William Schroeder 
mailto:william.schroe...@ctl.io>> wrote:
Hello!

Our team finally had a chance to take another look at the problem identified by 
Brian Felton in http://tracker.ceph.com/issues/16767.  Basically, if any 
multipart objects are retried before an Abort or Complete, they remain on the 
system, taking up space and leaving their accounting in “radosgw-admin bucket 
stats”.  The problem is confirmed in Hammer and Jewel.

This past week, we succeeded in some experimental code to remove those parts.  
I am not sure if this code has any unintended consequences, so **I would 
greatly appreciate reviews of the new tool**!  I have tested it successfully 
against objects created and leaked in the ceph-demo Docker image for Jewel.  
Here is a pull request with the patch:

https://github.com/ceph/ceph/pull/17349

Basically, we added a new subcommand for “bucket” called “fixmpleak”.  This 
lists objects in the “multipart” namespace, and it identifies objects that are 
not associated with current .meta files in that list.  It then deletes those 
objects with a delete op, which results in the accounting being corrected and 
the space being reclaimed on the OSDs.

This is not a preventative measure, which would be a lot more complicated, but 
we figure to run this tool hourly against all our buckets to keep things clean.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] luminous ceph-osd crash

[Marcin Dulak]

Hi,

/var/log/ceph/ceph-osd.0.log is attached.
My sdb is 128MB and sdc (journal) is 16MB:

[root@server0 ~]# ceph-disk list
/dev/dm-0 other, xfs, mounted on /
/dev/dm-1 swap, swap
/dev/sda :
 /dev/sda1 other, 0x83
 /dev/sda2 other, xfs, mounted on /boot
 /dev/sda3 other, LVM2_member
/dev/sdb :
 /dev/sdb1 ceph data, active, cluster ceph, osd.0, journal /dev/sdc1
/dev/sdc :
 /dev/sdc1 ceph journal, for /dev/sdb1

Marcin

On Thu, Aug 31, 2017 at 3:05 PM, Sage Weil  wrote:

> Hi Marcin,
>
> Can you reproduce the crash with 'debug bluestore = 20' set, and then
> ceph-post-file /var/log/ceph/ceph-osd.0.log?
>
> My guess is that we're not handling a very small device properly?
>
> sage
>
>
> On Thu, 31 Aug 2017, Marcin Dulak wrote:
>
> > Hi,
> >
> > I have a virtual CentOS 7.3 test setup at:
> > https://github.com/marcindulak/github-test-local/
> blob/a339ff7505267545f593f
> > d949a6453a56cdfd7fe/vagrant-ceph-rbd-tutorial-centos7.sh
> >
> > It seems to crash reproducibly with luminous, and works with kraken.
> > Is this a known issue?
> >
> > [ceph_deploy.conf][DEBUG ] found configuration file at:
> > /home/ceph/.cephdeploy.conf
> > [ceph_deploy.cli][INFO  ] Invoked (1.5.37): /bin/ceph-deploy osd activate
> > server0:/dev/sdb1:/dev/sdc server1:/dev/sdb1:/dev/sdc
> > server2:/dev/sdb1:/dev/sdc
> > [ceph_deploy.cli][INFO  ] ceph-deploy options:
> > [ceph_deploy.cli][INFO  ]  username  : None
> > [ceph_deploy.cli][INFO  ]  verbose   : False
> > [ceph_deploy.cli][INFO  ]  overwrite_conf: False
> > [ceph_deploy.cli][INFO  ]  subcommand: activate
> > [ceph_deploy.cli][INFO  ]  quiet : False
> > [ceph_deploy.cli][INFO  ]  cd_conf   :
> > 
> > [ceph_deploy.cli][INFO  ]  cluster   : ceph
> > [ceph_deploy.cli][INFO  ]  func  :  at
> > 0x109fb90>
> > [ceph_deploy.cli][INFO  ]  ceph_conf : None
> > [ceph_deploy.cli][INFO  ]  default_release   : False
> > [ceph_deploy.cli][INFO  ]  disk  : [('server0',
> > '/dev/sdb1', '/dev/sdc'), ('server1', '/dev/sdb1', '/dev/sdc'),
> ('server2',
> > '/dev/sdb1', '/dev/sdc')]
> > [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
> > server0:/dev/sdb1:/dev/sdc server1:/dev/sdb1:/dev/sdc
> > server2:/dev/sdb1:/dev/sdc
> > [server0][DEBUG ] connection detected need for sudo
> > [server0][DEBUG ] connected to host: server0
> > [server0][DEBUG ] detect platform information from remote host
> > [server0][DEBUG ] detect machine type
> > [server0][DEBUG ] find the location of an executable
> > [ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
> > [ceph_deploy.osd][DEBUG ] activating host server0 disk /dev/sdb1
> > [ceph_deploy.osd][DEBUG ] will use init type: systemd
> > [server0][DEBUG ] find the location of an executable
> > [server0][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate
> > --mark-init systemd --mount /dev/sdb1
> > [server0][WARNIN] main_activate: path = /dev/sdb1
> > [server0][WARNIN] get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is
> > /sys/dev/block/8:17/dm/uuid
> > [server0][WARNIN] command: Running command: /sbin/blkid -o udev -p
> /dev/sdb1
> > [server0][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o
> value
> > -- /dev/sdb1
> > [server0][WARNIN] command: Running command: /usr/bin/ceph-conf
> > --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
> > [server0][WARNIN] command: Running command: /usr/bin/ceph-conf
> > --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
> > [server0][WARNIN] mount: Mounting /dev/sdb1 on
> /var/lib/ceph/tmp/mnt.wfKzzb
> > with options noatime,inode64
> > [server0][WARNIN] command_check_call: Running command: /usr/bin/mount -t
> xfs
> > -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.wfKzzb
> > [server0][WARNIN] command: Running command: /sbin/restorecon
> > /var/lib/ceph/tmp/mnt.wfKzzb
> > [server0][WARNIN] activate: Cluster uuid is
> > 04e79ca9-308c-41a5-b40d-a2737c34238d
> > [server0][WARNIN] command: Running command: /usr/bin/ceph-osd
> --cluster=ceph
> > --show-config-value=fsid
> > [server0][WARNIN] activate: Cluster name is ceph
> > [server0][WARNIN] activate: OSD uuid is 46d7cc0b-a087-4c8c-b00c-
> ff584c941cf9
> > [server0][WARNIN] activate: OSD id is 0
> > [server0][WARNIN] activate: Initializing OSD...
> > [server0][WARNIN] command_check_call: Running command: /usr/bin/ceph
> > --cluster ceph --name client.bootstrap-osd --keyring
> > /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
> > /var/lib/ceph/tmp/mnt.wfKzzb/activate.monmap
> > [server0][WARNIN] got monmap epoch 1
> > [server0][WARNIN] command_check_call: Running command: /usr/bin/ceph-osd
> > --cluster ceph --mkfs -i 0 --monmap
> > /var/lib/ceph/tmp/mnt.wfKzzb/activate.monmap --osd-data
> > /var/lib/ceph/tmp/mnt.wfKzzb --osd-uuid 46d7cc0b-a087-4c8c-b00c-
> ff584c941cf9
> > --setuser ceph --setg

Re: [ceph-users] Possible way to clean up leaked multipart objects?

[David Turner]

Jewel 10.2.7.  I found a discrepancy in object counts for a multisite
configuration and it's looking like it might be orphaned multipart files
causing it.  It doesn't look like this PR has received much attention.  Is
there anything I can do to help you with testing/confirming a use case for
this tool?

On Tue, Aug 29, 2017 at 5:28 PM William Schroeder 
wrote:

> Hello!
>
>
>
> Our team finally had a chance to take another look at the problem
> identified by Brian Felton in http://tracker.ceph.com/issues/16767.
> Basically, if any multipart objects are retried before an Abort or
> Complete, they remain on the system, taking up space and leaving their
> accounting in “radosgw-admin bucket stats”.  The problem is confirmed in
> Hammer and Jewel.
>
>
>
> This past week, we succeeded in some experimental code to remove those
> parts.  I am not sure if this code has any unintended consequences, so **I
> would greatly appreciate reviews of the new tool**!  I have tested it
> successfully against objects created and leaked in the ceph-demo Docker
> image for Jewel.  Here is a pull request with the patch:
>
>
>
> https://github.com/ceph/ceph/pull/17349
>
>
>
> Basically, we added a new subcommand for “bucket” called “fixmpleak”.
> This lists objects in the “multipart” namespace, and it identifies objects
> that are not associated with current .meta files in that list.  It then
> deletes those objects with a delete op, which results in the accounting
> being corrected and the space being reclaimed on the OSDs.
>
>
>
> This is not a preventative measure, which would be a lot more complicated,
> but we figure to run this tool hourly against all our buckets to keep
> things clean.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [rgw][s3] Object not in objects list

[Stanley Zhang]

Your bucket index got corrupted. I believe there is no easy way to 
restore the index other than downloading existing objects and re-upload 
them, correct me if anybody else know a better way.


You can check out all your objects in that bucket with:

rados -p .rgw.buckets ls | grep default.32785769.2

Plus what's your region map? where is the your bucket index stored? from 
the naming of data pool, you seems using same pool for bucket index?


Regards

Stanley


On 31/08/17 9:01 PM, Rudenko Aleksandr wrote:

Hi,

Maybe someone have thoughts?

---
Best regards,

Alexander Rudenko



On 30 Aug 2017, at 12:28, Rudenko Aleksandr > wrote:


Hi,

I use ceph 0.94.10(hammer) with radosgw as S3-compatible object store.

I have few objects in some bucket with strange problem.

I use awscli as s3 client.

GET/HEAD objects work fine but list object doesn’t.
In objects list I don’t see these objects.

Object metadata:

radosgw-admin bi list --bucket={my-bucket} --object={my-object}

Return [].

But:

rados -p .rgw.buckets stat default.32785769.2_{my-object}

.rgw.buckets/default.32785769.2_{my-object} mtime 2017-08-15 
18:07:29.00, size 97430



Bucket versioning not enabled.
Bucket has more 13M objects.

Where can I find the problem?

---
Best regards,

Alexander Rudenko







___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_

This email has been filtered by SMX. For more info visit http://smxemail.com
_

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Object gateway and LDAP Auth

[Josh]

Hello!

I've setup LDAP authentication on an object gateway and am attempting to
create a bucket via s3 using python's boto3. It works fine using the access
and secret key for a radosgw user, but access is denied using a token
generated via radosgw-token with the LDAP user's credentials. The user does
exist in the directory (I'm using Active Directory), and I am able to query
for that user using the creds specified in rgw_ldap_binddn and
rgw_ldap_secret.

I've bumped the rgw logging to 20 and can see the request come in, but it
ultimately gets denied:
2017-08-30 15:44:55.754721 7f4878ff9700  2 req 1:0.76:s3:PUT
/foobar:create_bucket:authorizing
2017-08-30 15:44:55.754738 7f4878ff9700 10 v4 signature format = 
2017-08-30 15:44:55.754746 7f4878ff9700 10 v4 credential format =
/20170830/us-east-1/s3/aws4_request
2017-08-30 15:44:55.754750 7f4878ff9700 10 access key id = 
2017-08-30 15:44:55.754755 7f4878ff9700 10 credential scope =
20170830/us-east-1/s3/aws4_request
2017-08-30 15:44:55.754769 7f4878ff9700 20 get_system_obj_state:
rctx=0x7f4878ff2060 obj=default.rgw.users.keys: state=0x7f48f40131a8
s->prefetch_data=0
2017-08-30 15:44:55.754778 7f4878ff9700 10 cache get:
name=default.rgw.users.keys+ : miss
2017-08-30 15:44:55.755312 7f4878ff9700 10 cache put:
name=default.rgw.users.keys+ info.flags=0
2017-08-30 15:44:55.755321 7f4878ff9700 10 adding
default.rgw.users.keys+ to cache LRU end
2017-08-30 15:44:55.755328 7f4878ff9700 10 error reading user info,
uid= can't authenticate
2017-08-30 15:44:55.755330 7f4878ff9700 10 failed to authorize request
2017-08-30 15:44:55.755331 7f4878ff9700 20 handler->ERRORHANDLER:
err_no=-2028 new_err_no=-2028
2017-08-30 15:44:55.755393 7f4878ff9700  2 req 1:0.000747:s3:PUT
/foobar:create_bucket:op status=0
2017-08-30 15:44:55.755398 7f4878ff9700  2 req 1:0.000752:s3:PUT
/foobar:create_bucket:http status=403
2017-08-30 15:44:55.755402 7f4878ff9700  1 == req done
req=0x7f4878ff3710 op status=0 http_status=403 ==
2017-08-30 15:44:55.755409 7f4878ff9700 20 process_request() returned -2028

I am also running a tcpdump on the machine while I see these log messages,
but strangely I see no traffic destined for my configured LDAP server.
Here's some info on my setup. It seems like I'm missing something very
obvious; any help would be appreciated!

# rpm -q ceph-radosgw
ceph-radosgw-10.2.9-0.el7.x86_64

# grep rgw /etc/ceph/ceph.conf
[client.rgw.hostname]
rgw_frontends = civetweb port=8081s ssl_certificate=/path/to/private/key.pem
debug rgw = 20
rgw_s3_auth_use_ldap = true
rgw_ldap_secret = "/path/to/creds/file"
rgw_ldap_uri = "ldaps://hostname.domain.com:636"
rgw_ldap_binddn = "CN=valid_user,OU=Accounts,DC=domain,DC=com"
rgw_ldap_searchdn = "ou=Accounts,dc=domain,dc=com"
rgw_ldap_dnattr = "uid"
rgw_ldap_searchfilter = "objectclass=user"


Thanks,
Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] (no subject)

[Marc Roos]

Should these messages not be gone in 12.2.0?

2017-08-31 20:49:33.500773 7f5aa1756d40 -1 WARNING: the following 
dangerous and experimental features are enabled: bluestore
2017-08-31 20:49:33.501026 7f5aa1756d40 -1 WARNING: the following 
dangerous and experimental features are enabled: bluestore
2017-08-31 20:49:33.540667 7f5aa1756d40 -1 WARNING: the following 
dangerous and experimental features are enabled: bluestore

ceph-selinux-12.2.0-0.el7.x86_64
ceph-mon-12.2.0-0.el7.x86_64
collectd-ceph-5.7.1-2.el7.x86_64
ceph-base-12.2.0-0.el7.x86_64
ceph-osd-12.2.0-0.el7.x86_64
ceph-mgr-12.2.0-0.el7.x86_64
ceph-12.2.0-0.el7.x86_64
ceph-common-12.2.0-0.el7.x86_64
ceph-mds-12.2.0-0.el7.x86_64



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW Multisite metadata sync init

[David Turner]

All of the messages from sync error list are listed below.  The number on
the left is how many times the error message is found.

   1811 "message": "failed to sync bucket instance:
(16) Device or resource busy"
  7 "message": "failed to sync bucket instance: (5)
Input\/output error"
 65 "message": "failed to sync object"

On Tue, Aug 29, 2017 at 10:00 AM Orit Wasserman  wrote:

>
> Hi David,
>
> On Mon, Aug 28, 2017 at 8:33 PM, David Turner 
> wrote:
>
>> The vast majority of the sync error list is "failed to sync bucket
>> instance: (16) Device or resource busy".  I can't find anything on Google
>> about this error message in relation to Ceph.  Does anyone have any idea
>> what this means? and/or how to fix it?
>>
>
> Those are intermediate errors resulting from several radosgw trying to
> acquire the same sync log shard lease. It doesn't effect the sync progress.
> Are there any other errors?
>
> Orit
>
>>
>> On Fri, Aug 25, 2017 at 2:48 PM Casey Bodley  wrote:
>>
>>> Hi David,
>>>
>>> The 'data sync init' command won't touch any actual object data, no.
>>> Resetting the data sync status will just cause a zone to restart a full
>>> sync of the --source-zone's data changes log. This log only lists which
>>> buckets/shards have changes in them, which causes radosgw to consider them
>>> for bucket sync. So while the command may silence the warnings about data
>>> shards being behind, it's unlikely to resolve the issue with missing
>>> objects in those buckets.
>>>
>>> When data sync is behind for an extended period of time, it's usually
>>> because it's stuck retrying previous bucket sync failures. The 'sync error
>>> list' may help narrow down where those failures are.
>>>
>>> There is also a 'bucket sync init' command to clear the bucket sync
>>> status. Following that with a 'bucket sync run' should restart a full sync
>>> on the bucket, pulling in any new objects that are present on the
>>> source-zone. I'm afraid that those commands haven't seen a lot of polish or
>>> testing, however.
>>>
>>> Casey
>>>
>>> On 08/24/2017 04:15 PM, David Turner wrote:
>>>
>>> Apparently the data shards that are behind go in both directions, but
>>> only one zone is aware of the problem.  Each cluster has objects in their
>>> data pool that the other doesn't have.  I'm thinking about initiating a
>>> `data sync init` on both sides (one at a time) to get them back on the same
>>> page.  Does anyone know if that command will overwrite any local data that
>>> the zone has that the other doesn't if you run `data sync init` on it?
>>>
>>> On Thu, Aug 24, 2017 at 1:51 PM David Turner 
>>> wrote:
>>>
 After restarting the 2 RGW daemons on the second site again, everything
 caught up on the metadata sync.  Is there something about having 2 RGW
 daemons on each side of the multisite that might be causing an issue with
 the sync getting stale?  I have another realm set up the same way that is
 having a hard time with its data shards being behind.  I haven't told them
 to resync, but yesterday I noticed 90 shards were behind.  It's caught back
 up to only 17 shards behind, but the oldest change not applied is 2 months
 old and no order of restarting RGW daemons is helping to resolve this.

 On Thu, Aug 24, 2017 at 10:59 AM David Turner 
 wrote:

> I have a RGW Multisite 10.2.7 set up for bi-directional syncing.  This
> has been operational for 5 months and working fine.  I recently created a
> new user on the master zone, used that user to create a bucket, and put in
> a public-acl object in there.  The Bucket created on the second site, but
> the user did not and the object errors out complaining about the 
> access_key
> not existing.
>
> That led me to think that the metadata isn't syncing, while bucket and
> data both are.  I've also confirmed that data is syncing for other buckets
> as well in both directions. The sync status from the second site was this.
>
>
>1.
>
>  metadata sync syncing
>
>2.
>
>full sync: 0/64 shards
>
>3.
>
>incremental sync: 64/64 shards
>
>4.
>
>metadata is caught up with master
>
>5.
>
>  data sync source: f4c12327-4721-47c9-a365-86332d84c227 
> (public-atl01)
>
>6.
>
>syncing
>
>7.
>
>full sync: 0/128 shards
>
>8.
>
>incremental sync: 128/128 shards
>
>9.
>
>data is caught up with source
>
>
>
> Sync status leads me to think that the second site believes it is up
> to date, even though it is missing a freshly created user.  I restarted 
> all
> of th

Re: [ceph-users] Very slow start of osds after reboot

[Hervé Ballans]

Hi Piotr,

Just to verify one point, how are connected your disks (physically), in 
a NON-RAID or RAID0 mode ?


rv

Le 31/08/2017 à 16:24, Piotr Dzionek a écrit :
For a last 3 weeks I have been running latest LTS Luminous Ceph 
release on CentOS7. It started with 4th RC and now I have Stable Release.
Cluster runs fine, however I noticed that if I do a reboot of one the 
nodes, it takes a really long time for cluster to be in ok status.
Osds are starting up, but not as soon as the server is up. They are up 
one by one during a period of 5 minutes. I checked the logs and all 
osds have following errors.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] a metadata lost problem when mds breaks down

[Sage Weil]

On Thu, 31 Aug 2017, Mark Meyers wrote:
> Hi:
> 
> I encountered a metadata lost problem of files when testing the 
> outcome of ceph mds's unpredictable breakdown. 
> The test shell script is like(not real code):
> 
> while i < 10
> do 
> touch $i
> i++
> done
> echo c > /proc/sysrq-trigger
> 
> The script is run in the same machine which mds is running on, and 
> the directory is mount point of cephfs. 
> I created 10,000 files using touch command and forced the system to 
> break down.
> When the machine rebooted again, the number of new files created is 
> smaller than 1. So the metadata of those files are lost.
> My question: Are those losts of metadata allowed by ceph? Or is this 
> a bug of ceph mds which could be solved in the future? My ceph version is 
> 12.1.0, ubuntu kernel version 4.4.0.
> 
> Thanks for your reading, I really appreciate your rely. Thanks!

You have to call fsync(2) on the containing directory if you want metadata 
changes (link, unlink, rename, create, mknod, open(...O_CREAT)) to be 
stable... just like you have to call fsync(2) after write(2) if you want 
the data to be durable.  In the case of creating a file, calling fsync() 
on the new file is also sufficient to make the new file exist (on most 
file systems, including CephFS).

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Very slow start of osds after reboot

[Dan van der Ster]

Random theory... I just noticed that the ceph-osd's are listed twice
[1] in the output of systemctl list-dependencies.

Is that correct?!!!

-- dan

[1] > systemctl list-dependencies
...
● ├─ceph-mds.target
● ├─ceph-mon.target
● ├─ceph-osd.target
● │ ├─ceph-osd@48.service
● │ ├─ceph-osd@49.service
● │ ├─ceph-osd@50.service
● │ ├─ceph-osd@51.service
● │ ├─ceph-osd@53.service
● │ ├─ceph-osd@54.service
● │ ├─ceph-osd@55.service
● │ ├─ceph-osd@56.service
● │ ├─ceph-osd@59.service
● │ ├─ceph-osd@61.service
● │ ├─ceph-osd@63.service
● │ ├─ceph-osd@65.service
● │ ├─ceph-osd@68.service
● │ ├─ceph-osd@70.service
● │ ├─ceph-osd@74.service
● │ ├─ceph-osd@80.service
● │ ├─ceph-osd@81.service
● │ ├─ceph-osd@82.service
● │ ├─ceph-osd@83.service
● │ ├─ceph-osd@84.service
● │ ├─ceph-osd@89.service
● │ ├─ceph-osd@90.service
● │ ├─ceph-osd@91.service
● │ └─ceph-osd@92.service
● ├─ceph.target
● │ ├─ceph-mds.target
● │ ├─ceph-mon.target
● │ └─ceph-osd.target
● │   ├─ceph-osd@48.service
● │   ├─ceph-osd@49.service
● │   ├─ceph-osd@50.service
● │   ├─ceph-osd@51.service
● │   ├─ceph-osd@53.service
● │   ├─ceph-osd@54.service
● │   ├─ceph-osd@55.service
● │   ├─ceph-osd@56.service
● │   ├─ceph-osd@59.service
● │   ├─ceph-osd@61.service
● │   ├─ceph-osd@63.service
● │   ├─ceph-osd@65.service
● │   ├─ceph-osd@68.service
● │   ├─ceph-osd@70.service
● │   ├─ceph-osd@74.service
● │   ├─ceph-osd@80.service
● │   ├─ceph-osd@81.service
● │   ├─ceph-osd@82.service
● │   ├─ceph-osd@83.service
● │   ├─ceph-osd@84.service
● │   ├─ceph-osd@89.service
● │   ├─ceph-osd@90.service
● │   ├─ceph-osd@91.service
● │   └─ceph-osd@92.service
● ├─getty.target
...



On Thu, Aug 31, 2017 at 4:57 PM, Dan van der Ster  wrote:
> Hi,
>
> I see the same with jewel on el7 -- it started one of the recent point
> releases around ~10.2.5, IIRC.
>
> Problem seems to be the same -- daemon is started before the osd is
> mounted... then the service waits several seconds before trying again.
>
> Aug 31 15:41:47 ceph-osd: 2017-08-31 15:41:47.267661 7f2e49731800 -1
> #033[0;31m ** ERROR: unable to open OSD superblock on
> /var/lib/ceph/osd/ceph-89: (2) No such file or directory#033[0m
> Aug 31 15:41:47 ceph-osd: starting osd.55 at :/0 osd_data
> /var/lib/ceph/osd/ceph-55 /var/lib/ceph/osd/ceph-55/journal
> Aug 31 15:41:47 systemd: ceph-osd@89.service: main process exited,
> code=exited, status=1/FAILURE
> Aug 31 15:41:47 systemd: Unit ceph-osd@89.service entered failed state.
> Aug 31 15:41:47 systemd: ceph-osd@89.service failed.
> Aug 31 15:41:47 kernel: XFS (sdi1): Ending clean mount
> Aug 31 15:41:47 rc.local: Removed symlink
> /etc/systemd/system/ceph-osd.target.wants/ceph-osd@54.service.
> Aug 31 15:41:47 systemd: Reloading.
> Aug 31 15:41:47 systemd: Reloading.
> Aug 31 15:41:47 rc.local: Created symlink from
> /etc/systemd/system/ceph-osd.target.wants/ceph-osd@54.service to
> /usr/lib/systemd/system/ceph-osd@.service.
> Aug 31 15:41:47 systemd: Reloading.
> Aug 31 15:41:55 ceph-osd: 2017-08-31 15:41:55.425566 7f74b92e1800 -1
> osd.55 123659 log_to_monitors {default=true}
> Aug 31 15:42:07 systemd: ceph-osd@84.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@61.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@83.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@80.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@70.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@65.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@82.service holdoff time over,
> scheduling restart.
> Aug 31 15:42:07 systemd: ceph-osd@89.service holdoff time over,
> scheduling restart.
>
>
> -- Dan
>
>
>
> On Thu, Aug 31, 2017 at 4:24 PM, Piotr Dzionek  wrote:
>> Hi,
>>
>> For a last 3 weeks I have been running latest LTS Luminous Ceph release on
>> CentOS7. It started with 4th RC and now I have Stable Release.
>> Cluster runs fine, however I noticed that if I do a reboot of one the nodes,
>> it takes a really long time for cluster to be in ok status.
>> Osds are starting up, but not as soon as the server is up. They are up one
>> by one during a period of 5 minutes. I checked the logs and all osds have
>> following errors.
>>
>> 2017-08-30 15:27:52.541366 7f7dabd0d700 30 Event(0x7f7dbc9f4a80 nevent=5000
>> time_id=62).process_events event_wq process is 11 mask is 1
>> 2017-08-30 15:51:03.639222 7faf11c3ed00  0 set uid:gid to 167:167
>> (ceph:ceph)
>> 2017-08-30 15:51:03.639342 7faf11c3ed00  0 ceph version 12.2.0
>> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
>> pid 3037
>> 2017-08-30 15:51:03.672898 7faf11c3ed00 -1 ESC[0;31m ** ERROR: unable to
>> open OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or
>> directoryESC[0m
>> 2017-08-30 15:51:42.453334 7f9f55f11d00  0 set uid:gid to 167:167
>> (ceph:ceph)
>> 2017-08-30 15:51:42.453352 7f9f5

[ceph-users] a metadata lost problem when mds breaks down

[Mark Meyers]

Hi:

I encountered a metadata lost problem of files when testing the outcome 
of ceph mds's unpredictable breakdown. 
The test shell script is like(not real code):

while i < 10
do 
touch $i
i++
done
echo c > /proc/sysrq-trigger

The script is run in the same machine which mds is running on, and the 
directory is mount point of cephfs. 
I created 10,000 files using touch command and forced the system to 
break down.
When the machine rebooted again, the number of new files created is 
smaller than 1. So the metadata of those files are lost.
My question: Are those losts of metadata allowed by ceph? Or is this a 
bug of ceph mds which could be solved in the future? My ceph version is 12.1.0, 
ubuntu kernel version 4.4.0.

Thanks for your reading, I really appreciate your rely. Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Very slow start of osds after reboot

[Dan van der Ster]

Hi,

I see the same with jewel on el7 -- it started one of the recent point
releases around ~10.2.5, IIRC.

Problem seems to be the same -- daemon is started before the osd is
mounted... then the service waits several seconds before trying again.

Aug 31 15:41:47 ceph-osd: 2017-08-31 15:41:47.267661 7f2e49731800 -1
#033[0;31m ** ERROR: unable to open OSD superblock on
/var/lib/ceph/osd/ceph-89: (2) No such file or directory#033[0m
Aug 31 15:41:47 ceph-osd: starting osd.55 at :/0 osd_data
/var/lib/ceph/osd/ceph-55 /var/lib/ceph/osd/ceph-55/journal
Aug 31 15:41:47 systemd: ceph-osd@89.service: main process exited,
code=exited, status=1/FAILURE
Aug 31 15:41:47 systemd: Unit ceph-osd@89.service entered failed state.
Aug 31 15:41:47 systemd: ceph-osd@89.service failed.
Aug 31 15:41:47 kernel: XFS (sdi1): Ending clean mount
Aug 31 15:41:47 rc.local: Removed symlink
/etc/systemd/system/ceph-osd.target.wants/ceph-osd@54.service.
Aug 31 15:41:47 systemd: Reloading.
Aug 31 15:41:47 systemd: Reloading.
Aug 31 15:41:47 rc.local: Created symlink from
/etc/systemd/system/ceph-osd.target.wants/ceph-osd@54.service to
/usr/lib/systemd/system/ceph-osd@.service.
Aug 31 15:41:47 systemd: Reloading.
Aug 31 15:41:55 ceph-osd: 2017-08-31 15:41:55.425566 7f74b92e1800 -1
osd.55 123659 log_to_monitors {default=true}
Aug 31 15:42:07 systemd: ceph-osd@84.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@61.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@83.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@80.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@70.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@65.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@82.service holdoff time over,
scheduling restart.
Aug 31 15:42:07 systemd: ceph-osd@89.service holdoff time over,
scheduling restart.


-- Dan



On Thu, Aug 31, 2017 at 4:24 PM, Piotr Dzionek  wrote:
> Hi,
>
> For a last 3 weeks I have been running latest LTS Luminous Ceph release on
> CentOS7. It started with 4th RC and now I have Stable Release.
> Cluster runs fine, however I noticed that if I do a reboot of one the nodes,
> it takes a really long time for cluster to be in ok status.
> Osds are starting up, but not as soon as the server is up. They are up one
> by one during a period of 5 minutes. I checked the logs and all osds have
> following errors.
>
> 2017-08-30 15:27:52.541366 7f7dabd0d700 30 Event(0x7f7dbc9f4a80 nevent=5000
> time_id=62).process_events event_wq process is 11 mask is 1
> 2017-08-30 15:51:03.639222 7faf11c3ed00  0 set uid:gid to 167:167
> (ceph:ceph)
> 2017-08-30 15:51:03.639342 7faf11c3ed00  0 ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
> pid 3037
> 2017-08-30 15:51:03.672898 7faf11c3ed00 -1 ESC[0;31m ** ERROR: unable to
> open OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or
> directoryESC[0m
> 2017-08-30 15:51:42.453334 7f9f55f11d00  0 set uid:gid to 167:167
> (ceph:ceph)
> 2017-08-30 15:51:42.453352 7f9f55f11d00  0 ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
> pid 7366
> 2017-08-30 15:51:42.453590 7f9f55f11d00 -1 ESC[0;31m ** ERROR: unable to
> open OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or
> directoryESC[0m
> 2017-08-30 15:52:03.199062 7effa00cad00  0 set uid:gid to 167:167
> (ceph:ceph)
> 2017-08-30 15:52:03.199081 7effa00cad00  0 ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
> pid 7747
> 2017-08-30 15:52:03.199323 7effa00cad00 -1 ESC[0;31m ** ERROR: unable to
> open OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or
> directoryESC[0m
> 2017-08-30 15:52:23.967466 7ff008c2cd00  0 set uid:gid to 167:167
> (ceph:ceph)
> 2017-08-30 15:52:23.967483 7ff008c2cd00  0 ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
> pid 8016
> 2017-08-30 15:52:23.967714 7ff008c2cd00 -1 ESC[0;31m ** ERROR: unable to
> open OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or
> directoryESC[0m
> 2017-08-30 15:52:44.716646 7fc2bd322d00  0 set uid:gid to 167:167
> (ceph:ceph)
> 2017-08-30 15:52:44.716664 7fc2bd322d00  0 ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
> pid 8808
> 2017-08-30 15:52:44.716892 7fc2bd322d00 -1 ESC[0;31m ** ERROR: unable to
> open OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or
> directoryESC[0m
> 2017-08-30 15:53:06.214611 7f4583e70d00  0 set uid:gid to 167:167
> (ceph:ceph)
> 2017-08-30 15:53:06.214629 7f4583e70d00  0 ceph version 12.2.0
> (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown),
> pid 9184
> 2017-08-30 15:53:06.214855 7f4583e70d00 -1 ESC[0;31m ** ERROR: unable to
> open OSD superbloc

Re: [ceph-users] Changing the failure domain

[David Turner]

How long are you seeing these blocked requests for?  Initially or
perpetually?  Changing the failure domain causes all PGs to peer at the
same time.  This would be the cause if it happens really quickly.  There is
no way to avoid all of them peering while making a change like this.  After
that, It could easily be caused because a fair majority of your data is
probably set to move around.  I would check what might be causing the
blocked requests during this time.  See if there is an OSD that might be
dying (large backfills have the tendancy to find a couple failing drives)
which could easily cause things to block.  Also checking if your disks or
journals are maxed out with iostat could shine some light on any mitigating
factor.

On Thu, Aug 31, 2017 at 9:01 AM Laszlo Budai 
wrote:

> Dear all!
>
> In our Hammer cluster we are planning to switch our failure domain from
> host to chassis. We have performed some simulations, and regardless of the
> settings we have used some slow requests have appeared all the time.
>
> we had the the following settings:
>
>   "osd_max_backfills": "1",
>  "osd_backfill_full_ratio": "0.85",
>  "osd_backfill_retry_interval": "10",
>  "osd_backfill_scan_min": "1",
>  "osd_backfill_scan_max": "4",
>  "osd_kill_backfill_at": "0",
>  "osd_debug_skip_full_check_in_backfill_reservation": "false",
>  "osd_debug_reject_backfill_probability": "0",
>
> "osd_min_recovery_priority": "0",
>  "osd_allow_recovery_below_min_size": "true",
>  "osd_recovery_threads": "1",
>  "osd_recovery_thread_timeout": "60",
>  "osd_recovery_thread_suicide_timeout": "300",
>  "osd_recovery_delay_start": "0",
>  "osd_recovery_max_active": "1",
>  "osd_recovery_max_single_start": "1",
>  "osd_recovery_max_chunk": "8388608",
>  "osd_recovery_forget_lost_objects": "false",
>  "osd_recovery_op_priority": "1",
>  "osd_recovery_op_warn_multiple": "16",
>
>
> we have also tested it with the CFQ IO scheduler on the OSDs and the
> following params:
>  "osd_disk_thread_ioprio_priority": "7"
>  "osd_disk_thread_ioprio_class": "idle"
>
> and the nodeep-scrub set.
>
> Is there anything else to try? Is there a good way to switch from one kind
> of failure domain to an other without slow requests?
>
> Thank you in advance for any suggestions.
>
> Kind regards,
> Laszlo
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Very slow start of osds after reboot

[Sean Purdy]

Datapoint: I have the same issue on 12.1.1, three nodes, 6 disks per node.

On Thu, 31 Aug 2017, Piotr Dzionek said:
> For a last 3 weeks I have been running latest LTS Luminous Ceph release on
> CentOS7. It started with 4th RC and now I have Stable Release.
> Cluster runs fine, however I noticed that if I do a reboot of one the nodes,
> it takes a really long time for cluster to be in ok status.
> Osds are starting up, but not as soon as the server is up. They are up one
> by one during a period of 5 minutes. I checked the logs and all osds have
> following errors.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] where is a RBD in use

[Maxime Guyot]

Hi Götz,

Something like "rbd status image-spec" usually works for me. Man page says:
"Show the status of the image, including which clients have it open."
I'll tell you which IPs have it open which should help you to track it down.

Cheers,
Maxime

On Thu, 31 Aug 2017 at 16:26 Götz Reinicke 
wrote:

> Hi,
>
> Is it possible to see which clients are using an RBD? … I found an RBD in
> one of my pools but cant remember if I ever use / mounted it to a client.
>
> Thx for feedback ! Regards . Götz
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] where is a RBD in use

[Götz Reinicke]

Hi,

Is it possible to see which clients are using an RBD? … I found an RBD in one 
of my pools but cant remember if I ever use / mounted it to a client.

Thx for feedback ! Regards . Götz


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Very slow start of osds after reboot

[Piotr Dzionek]

Hi,

For a last 3 weeks I have been running latest LTS Luminous Ceph release 
on CentOS7. It started with 4th RC and now I have Stable Release.
Cluster runs fine, however I noticed that if I do a reboot of one the 
nodes, it takes a really long time for cluster to be in ok status.
Osds are starting up, but not as soon as the server is up. They are up 
one by one during a period of 5 minutes. I checked the logs and all osds 
have following errors.


2017-08-30 15:27:52.541366 7f7dabd0d700 30 Event(0x7f7dbc9f4a80 nevent=5000 
time_id=62).process_events event_wq process is 11 mask is 1
2017-08-30 15:51:03.639222 7faf11c3ed00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:51:03.639342 7faf11c3ed00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 3037
2017-08-30 15:51:03.672898 7faf11c3ed00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:51:42.453334 7f9f55f11d00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:51:42.453352 7f9f55f11d00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 7366
2017-08-30 15:51:42.453590 7f9f55f11d00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:52:03.199062 7effa00cad00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:52:03.199081 7effa00cad00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 7747
2017-08-30 15:52:03.199323 7effa00cad00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:52:23.967466 7ff008c2cd00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:52:23.967483 7ff008c2cd00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 8016
2017-08-30 15:52:23.967714 7ff008c2cd00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:52:44.716646 7fc2bd322d00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:52:44.716664 7fc2bd322d00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 8808
2017-08-30 15:52:44.716892 7fc2bd322d00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:53:06.214611 7f4583e70d00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:53:06.214629 7f4583e70d00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 9184
2017-08-30 15:53:06.214855 7f4583e70d00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:53:26.955944 7f1dfea39d00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:53:26.955962 7f1dfea39d00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 9417
2017-08-30 15:53:26.956191 7f1dfea39d00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:53:47.714131 7fabbc5cfd00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:53:47.714149 7fabbc5cfd00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 10469
2017-08-30 15:53:47.714383 7fabbc5cfd00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:54:08.708602 7f32fae4fd00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:54:08.708616 7f32fae4fd00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 11152
2017-08-30 15:54:08.708898 7f32fae4fd00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:54:29.454000 7faea1004d00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:54:29.454016 7faea1004d00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 11751
2017-08-30 15:54:29.454243 7faea1004d00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:54:50.207237 7f0c1701ed00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:54:50.207253 7f0c1701ed00  0 ceph version 12.2.0 
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc), process (unknown), 
pid 12878
2017-08-30 15:54:50.207431 7f0c1701ed00 -1 ESC[0;31m ** ERROR: unable to open 
OSD superblock on /var/lib/ceph/osd/ceph-27: (2) No such file or directoryESC[0m
2017-08-30 15:55:11.237106 7f571b1ffd00  0 set uid:gid to 167:167 (ceph:ceph)
2017-08-30 15:55:11.237122 7f571b1ffd00  0 ceph

Re: [ceph-users] luminous ceph-osd crash

[Sage Weil]

Hi Marcin,

Can you reproduce the crash with 'debug bluestore = 20' set, and then 
ceph-post-file /var/log/ceph/ceph-osd.0.log?

My guess is that we're not handling a very small device properly?

sage


On Thu, 31 Aug 2017, Marcin Dulak wrote:

> Hi,
> 
> I have a virtual CentOS 7.3 test setup at:
> https://github.com/marcindulak/github-test-local/blob/a339ff7505267545f593f
> d949a6453a56cdfd7fe/vagrant-ceph-rbd-tutorial-centos7.sh
> 
> It seems to crash reproducibly with luminous, and works with kraken.
> Is this a known issue?
> 
> [ceph_deploy.conf][DEBUG ] found configuration file at:
> /home/ceph/.cephdeploy.conf
> [ceph_deploy.cli][INFO  ] Invoked (1.5.37): /bin/ceph-deploy osd activate
> server0:/dev/sdb1:/dev/sdc server1:/dev/sdb1:/dev/sdc
> server2:/dev/sdb1:/dev/sdc
> [ceph_deploy.cli][INFO  ] ceph-deploy options:
> [ceph_deploy.cli][INFO  ]  username                      : None
> [ceph_deploy.cli][INFO  ]  verbose                       : False
> [ceph_deploy.cli][INFO  ]  overwrite_conf                : False
> [ceph_deploy.cli][INFO  ]  subcommand                    : activate
> [ceph_deploy.cli][INFO  ]  quiet                         : False
> [ceph_deploy.cli][INFO  ]  cd_conf                       :
> 
> [ceph_deploy.cli][INFO  ]  cluster                       : ceph
> [ceph_deploy.cli][INFO  ]  func                          :  0x109fb90>
> [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
> [ceph_deploy.cli][INFO  ]  default_release               : False
> [ceph_deploy.cli][INFO  ]  disk                          : [('server0',
> '/dev/sdb1', '/dev/sdc'), ('server1', '/dev/sdb1', '/dev/sdc'), ('server2',
> '/dev/sdb1', '/dev/sdc')]
> [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
> server0:/dev/sdb1:/dev/sdc server1:/dev/sdb1:/dev/sdc
> server2:/dev/sdb1:/dev/sdc
> [server0][DEBUG ] connection detected need for sudo
> [server0][DEBUG ] connected to host: server0 
> [server0][DEBUG ] detect platform information from remote host
> [server0][DEBUG ] detect machine type
> [server0][DEBUG ] find the location of an executable
> [ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
> [ceph_deploy.osd][DEBUG ] activating host server0 disk /dev/sdb1
> [ceph_deploy.osd][DEBUG ] will use init type: systemd
> [server0][DEBUG ] find the location of an executable
> [server0][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate
> --mark-init systemd --mount /dev/sdb1
> [server0][WARNIN] main_activate: path = /dev/sdb1
> [server0][WARNIN] get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is
> /sys/dev/block/8:17/dm/uuid
> [server0][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/sdb1
> [server0][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value
> -- /dev/sdb1
> [server0][WARNIN] command: Running command: /usr/bin/ceph-conf
> --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
> [server0][WARNIN] command: Running command: /usr/bin/ceph-conf
> --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
> [server0][WARNIN] mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.wfKzzb
> with options noatime,inode64
> [server0][WARNIN] command_check_call: Running command: /usr/bin/mount -t xfs
> -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.wfKzzb
> [server0][WARNIN] command: Running command: /sbin/restorecon
> /var/lib/ceph/tmp/mnt.wfKzzb
> [server0][WARNIN] activate: Cluster uuid is
> 04e79ca9-308c-41a5-b40d-a2737c34238d
> [server0][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph
> --show-config-value=fsid
> [server0][WARNIN] activate: Cluster name is ceph
> [server0][WARNIN] activate: OSD uuid is 46d7cc0b-a087-4c8c-b00c-ff584c941cf9
> [server0][WARNIN] activate: OSD id is 0
> [server0][WARNIN] activate: Initializing OSD...
> [server0][WARNIN] command_check_call: Running command: /usr/bin/ceph
> --cluster ceph --name client.bootstrap-osd --keyring
> /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
> /var/lib/ceph/tmp/mnt.wfKzzb/activate.monmap
> [server0][WARNIN] got monmap epoch 1
> [server0][WARNIN] command_check_call: Running command: /usr/bin/ceph-osd
> --cluster ceph --mkfs -i 0 --monmap
> /var/lib/ceph/tmp/mnt.wfKzzb/activate.monmap --osd-data
> /var/lib/ceph/tmp/mnt.wfKzzb --osd-uuid 46d7cc0b-a087-4c8c-b00c-ff584c941cf9
> --setuser ceph --setgroup ceph
> [server0][WARNIN]/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x
> 86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/
> rpm/el7/BUILD/ceph-12.2.0/src/os/bluestore/BlueFS.cc: In function 'void
> BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread
> 7fef4f0cfd00 time 2017-08-31 10:05:31.892519
> [server0][WARNIN]/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x
> 86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/
> rpm/el7/BUILD/ceph-12.2.0/src/os/bluestore/BlueFS.cc: 172: FAILED
> assert(bdev[id]->get_size() >= offset + length)
> [ser

[ceph-users] Changing the failure domain

[Laszlo Budai]

Dear all!

In our Hammer cluster we are planning to switch our failure domain from host to 
chassis. We have performed some simulations, and regardless of the settings we 
have used some slow requests have appeared all the time.

we had the the following settings:

 "osd_max_backfills": "1",
"osd_backfill_full_ratio": "0.85",
"osd_backfill_retry_interval": "10",
"osd_backfill_scan_min": "1",
"osd_backfill_scan_max": "4",
"osd_kill_backfill_at": "0",
"osd_debug_skip_full_check_in_backfill_reservation": "false",
"osd_debug_reject_backfill_probability": "0",

   "osd_min_recovery_priority": "0",
"osd_allow_recovery_below_min_size": "true",
"osd_recovery_threads": "1",
"osd_recovery_thread_timeout": "60",
"osd_recovery_thread_suicide_timeout": "300",
"osd_recovery_delay_start": "0",
"osd_recovery_max_active": "1",
"osd_recovery_max_single_start": "1",
"osd_recovery_max_chunk": "8388608",
"osd_recovery_forget_lost_objects": "false",
"osd_recovery_op_priority": "1",
"osd_recovery_op_warn_multiple": "16",


we have also tested it with the CFQ IO scheduler on the OSDs and the following 
params:
"osd_disk_thread_ioprio_priority": "7"
"osd_disk_thread_ioprio_class": "idle"

and the nodeep-scrub set.

Is there anything else to try? Is there a good way to switch from one kind of 
failure domain to an other without slow requests?

Thank you in advance for any suggestions.

Kind regards,
Laszlo


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Day Netherlands: 20-09-2017

[Etienne Menguy]

Hi,


Do you know if some of the talks will be in english?


Étienne



From: ceph-users  on behalf of Wido den 
Hollander 
Sent: Thursday, August 24, 2017 17:09
To: ceph-us...@ceph.com
Subject: [ceph-users] Ceph Day Netherlands: 20-09-2017

Hi,

In less then a month the Ceph Day in NL is coming up. It will be hosted by BIT 
[0] at their great venue in Ede, NL.

The schedule hasn't been posted yet, we are still working on that. There will 
be a great talk from the people of BIT showing off their SSD-only cluster 
spread out over multiple DCs.

There is a direct train from Schiphol Airport (Amsterdam) to Ede where a 
taxi-service will be arranged to bring you back and from the venue without any 
charge.

Registration is free! :-)

More information: http://ceph.com/cephdays/netherlands2017/

Wido

[0]: https://www.bit.nl/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] luminous ceph-osd crash

[Marcin Dulak]

Hi,

I have a virtual CentOS 7.3 test setup at:
https://github.com/marcindulak/github-test-local/blob/a339ff
7505267545f593fd949a6453a56cdfd7fe/vagrant-ceph-rbd-tutorial-centos7.sh

It seems to crash reproducibly with luminous, and works with kraken.
Is this a known issue?

[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.37): /bin/ceph-deploy osd activate
server0:/dev/sdb1:/dev/sdc server1:/dev/sdb1:/dev/sdc
server2:/dev/sdb1:/dev/sdc
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username  : None
[ceph_deploy.cli][INFO  ]  verbose   : False
[ceph_deploy.cli][INFO  ]  overwrite_conf: False
[ceph_deploy.cli][INFO  ]  subcommand: activate
[ceph_deploy.cli][INFO  ]  quiet : False
[ceph_deploy.cli][INFO  ]  cd_conf   :

[ceph_deploy.cli][INFO  ]  cluster   : ceph
[ceph_deploy.cli][INFO  ]  func  : 
[ceph_deploy.cli][INFO  ]  ceph_conf : None
[ceph_deploy.cli][INFO  ]  default_release   : False
[ceph_deploy.cli][INFO  ]  disk  : [('server0',
'/dev/sdb1', '/dev/sdc'), ('server1', '/dev/sdb1', '/dev/sdc'), ('server2',
'/dev/sdb1', '/dev/sdc')]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks
server0:/dev/sdb1:/dev/sdc server1:/dev/sdb1:/dev/sdc
server2:/dev/sdb1:/dev/sdc
[server0][DEBUG ] connection detected need for sudo
[server0][DEBUG ] connected to host: server0
[server0][DEBUG ] detect platform information from remote host
[server0][DEBUG ] detect machine type
[server0][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host server0 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: systemd
[server0][DEBUG ] find the location of an executable
[server0][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate
--mark-init systemd --mount /dev/sdb1
[server0][WARNIN] main_activate: path = /dev/sdb1
[server0][WARNIN] get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is
/sys/dev/block/8:17/dm/uuid
[server0][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/sdb1
[server0][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value
-- /dev/sdb1
[server0][WARNIN] command: Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[server0][WARNIN] command: Running command: /usr/bin/ceph-conf
--cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[server0][WARNIN] mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.wfKzzb
with options noatime,inode64
[server0][WARNIN] command_check_call: Running command: /usr/bin/mount -t
xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.wfKzzb
[server0][WARNIN] command: Running command: /sbin/restorecon
/var/lib/ceph/tmp/mnt.wfKzzb
[server0][WARNIN] activate: Cluster uuid is
04e79ca9-308c-41a5-b40d-a2737c34238d
[server0][WARNIN] command: Running command: /usr/bin/ceph-osd
--cluster=ceph --show-config-value=fsid
[server0][WARNIN] activate: Cluster name is ceph
[server0][WARNIN] activate: OSD uuid is 46d7cc0b-a087-4c8c-b00c-ff584c941cf9
[server0][WARNIN] activate: OSD id is 0
[server0][WARNIN] activate: Initializing OSD...
[server0][WARNIN] command_check_call: Running command: /usr/bin/ceph
--cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/tmp/mnt.wfKzzb/activate.monmap
[server0][WARNIN] got monmap epoch 1
[server0][WARNIN] command_check_call: Running command: /usr/bin/ceph-osd
--cluster ceph --mkfs -i 0 --monmap
/var/lib/ceph/tmp/mnt.wfKzzb/activate.monmap --osd-data
/var/lib/ceph/tmp/mnt.wfKzzb --osd-uuid
46d7cc0b-a087-4c8c-b00c-ff584c941cf9 --setuser ceph --setgroup ceph
[server0][WARNIN]
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/os/bluestore/BlueFS.cc:
In function 'void BlueFS::add_block_extent(unsigned int, uint64_t,
uint64_t)' thread 7fef4f0cfd00 time 2017-08-31 10:05:31.892519
[server0][WARNIN]
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.0/rpm/el7/BUILD/ceph-12.2.0/src/os/bluestore/BlueFS.cc:
172: FAILED assert(bdev[id]->get_size() >= offset + length)
[server0][WARNIN]  ceph version 12.2.0
(32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
[server0][WARNIN]  1: (ceph::__ceph_assert_fail(char const*, char const*,
int, char const*)+0x110) [0x7fef4fb4c510]
[server0][WARNIN]  2: (BlueFS::add_block_extent(unsigned int, unsigned
long, unsigned long)+0x4d8) [0x7fef4fad1f88]
[server0][WARNIN]  3: (BlueStore::_open_db(bool)+0xc4f) [0x7fef4f9f597f]
[server0][WARNIN]  4: (BlueStore::mk

Re: [ceph-users] [rgw][s3] Object not in objects list

[Rudenko Aleksandr]

Hi,

Maybe someone have thoughts?

---
Best regards,

Alexander Rudenko



On 30 Aug 2017, at 12:28, Rudenko Aleksandr 
mailto:arude...@croc.ru>> wrote:

Hi,

I use ceph 0.94.10(hammer) with radosgw as S3-compatible object store.

I have few objects in some bucket with strange problem.

I use awscli as s3 client.

GET/HEAD objects work fine but list object doesn’t.
In objects list I don’t see these objects.

Object metadata:

radosgw-admin bi list --bucket={my-bucket} --object={my-object}

Return [].

But:

rados -p .rgw.buckets stat default.32785769.2_{my-object}

.rgw.buckets/default.32785769.2_{my-object} mtime 2017-08-15 18:07:29.00, 
size 97430


Bucket versioning not enabled.
Bucket has more 13M objects.

Where can I find the problem?

---
Best regards,

Alexander Rudenko




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com