Re: [ceph-users] RGW Multisite Sync Memory Usage

2017-07-26 Thread Casey Bodley 

Hi Ryan,

Sorry to hear about the crashes. Based on the fact that it's happening 
on the source zone, I'm guessing that you're hitting this infinite loop 
that leads to OOM: http://tracker.ceph.com/issues/20386. The jewel 
backport for this one is still pending, so I raised its priority to 
Urgent. I'm afraid there isn't a workaround here - the infinite loop 
reproduces once the 'data changes log' grows above 1000 entries.


Casey


On 07/26/2017 11:05 AM, Ryan Leimenstoll wrote:

Hi all,

We are currently trying to migrate our RGW Object Storage service from one zone 
to another (in the same zonegroup) in part to make use of erasure coded data 
pools. That being said, the rgw daemon is reliably getting OOM killed on the 
rgw origin host serving the original zone (and thus the current production 
data) as a result of high rgw memory usage. We are willing to consider more 
memory for the rgw daemon’s hosts to solve this problem, but was wondering what 
would be expected memory wise (at least as a rule of thumb). I noticed there 
were a few memory related rgw sync fixes in 10.2.9, but so far upgrading hasn’t 
seemed to prevent crashing.


Some details about our cluster:
Ceph Version: 10.2.9
OS: RHEL 7.3

584 OSDs
Serving RBD, CephFS, and RGW

RGW Origin Hosts:
Virtualized via KVM/QEMU, RHEL 7.3
Memory: 32GB
CPU: 12 virtual cores (Hypervisor processors: Intel E5-2630)

First zone data and index pools:
pool name KB  objects   clones degraded  
unfound   rdrd KB   wrwr KB
.rgw.buckets112190858231 3423974600
0   2713542251 265848150719475841837 153970795085
.rgw.buckets.index0 497200  
  0   3721485483   5926323574 360300980


Thanks,
Ryan Leimenstoll
University of Maryland Institute for Advanced Computer Studies

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

2017-07-26 Thread Gregory Farnum 
On Wed, Jul 26, 2017 at 1:03 PM Patrick Donnelly 
wrote:

> On Wed, Jul 26, 2017 at 2:26 AM,   wrote:
> > Hello!
> >
> > Based on the documentation for defining quotas in CephFS for any
> directory (http://docs.ceph.com/docs/master/cephfs/quota/), I defined a
> quota for attribute max_bytes:
> > ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> > getfattr: Removing leading '/' from absolute path names
> > # file: mnt/ceph-fuse/MTY/
> > ceph.quota.max_bytes="1"
> >
> > To validate if the quota is working, I write a 128MB file in
> /mnt/ceph-fuse/MTY:
> > ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> > 2+0 records in
> > 2+0 records out
> > 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
> >
> > This file is created correctly, and the utilization statistcs confirm it:
> > ld4257:~ # rados df
> > pool name KB  objects   clones degraded
> unfound   rdrd KB   wrwr KB
> > hdb-backup131072   3200
>   08843251 88572586
> > hdb-backup_metadata27920   2700
>   0  301   168115 645955386
> > rbd0000
>   00000
> > templates  0000
>   00000
> >   total used 9528188   59
> >   total avail   811829446772
> >   total space   811838974960
> >
> >
> > Question:
> > Why can I create a file with size 128MB after defining a quota of 100MB?
>
> I don't have a cluster to check this on now but perhaps because a
> sparse file (you wrote all zeros) does not consume its entire file
> size in the quota (only what it uses).  Retry with /dev/urandom.
>
> (And the usual disclaimer: quotas only work with libcephfs/ceph-fuse.
> The kernel client does not support quotas.)



Also, with older clients you needed to explicitly turn it on with the
config option. I think that includes Jewel.

>
>
> --
> Patrick Donnelly
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

2017-07-26 Thread Patrick Donnelly 
On Wed, Jul 26, 2017 at 2:26 AM,   wrote:
> Hello!
>
> Based on the documentation for defining quotas in CephFS for any directory 
> (http://docs.ceph.com/docs/master/cephfs/quota/), I defined a quota for 
> attribute max_bytes:
> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> getfattr: Removing leading '/' from absolute path names
> # file: mnt/ceph-fuse/MTY/
> ceph.quota.max_bytes="1"
>
> To validate if the quota is working, I write a 128MB file in 
> /mnt/ceph-fuse/MTY:
> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> 2+0 records in
> 2+0 records out
> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
>
> This file is created correctly, and the utilization statistcs confirm it:
> ld4257:~ # rados df
> pool name KB  objects   clones degraded  
> unfound   rdrd KB   wrwr KB
> hdb-backup131072   3200   
>  08843251 88572586
> hdb-backup_metadata27920   2700   
>  0  301   168115 645955386
> rbd0000   
>  00000
> templates  0000   
>  00000
>   total used 9528188   59
>   total avail   811829446772
>   total space   811838974960
>
>
> Question:
> Why can I create a file with size 128MB after defining a quota of 100MB?

I don't have a cluster to check this on now but perhaps because a
sparse file (you wrote all zeros) does not consume its entire file
size in the quota (only what it uses).  Retry with /dev/urandom.

(And the usual disclaimer: quotas only work with libcephfs/ceph-fuse.
The kernel client does not support quotas.)

-- 
Patrick Donnelly
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD Snapsot space accounting ...

2017-07-26 Thread David Turner 
Because the snapshot is the full status of the rbd at the time it was
created, which was 100MB. Any modifications made to the rbd after the
snapshot will show up in the rbds size until about snapshot is taken. If
you delete the snapshot, it rolls up where the size was reported.

Basically the oldest point in time an object was modified in an
rbd/snapshots will own the size of the object. The end itself is the
current state, so any object not modified since the last snapshot will not
count towards the size of the current rbd.

On Wed, Jul 26, 2017, 3:00 PM Laszlo Budai  wrote:

> Dear all,
>
> Where can I read more about how the space used by a snapshot of an RBD
> image is calculated? Or can someone explain it here?
> I can see that before the snapshot is created, the size of the image is
> let's say 100M as reported by  the rbd du command, while after taking the
> snapshot, I can see in the rbd du output that the 100M appears to belong to
> the snapshot, and the image is only consuming a small amount. How comes?
>
> Thank you,
> Laszlo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD Snapsot space accounting ...

2017-07-26 Thread Laszlo Budai 

Dear all,

Where can I read more about how the space used by a snapshot of an RBD image is 
calculated? Or can someone explain it here?
I can see that before the snapshot is created, the size of the image is let's 
say 100M as reported by  the rbd du command, while after taking the snapshot, I 
can see in the rbd du output that the 100M appears to belong to the snapshot, 
and the image is only consuming a small amount. How comes?

Thank you,
Laszlo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] bluestore-osd and block.dbs of other osds on ssd

2017-07-26 Thread Kenneth Waegeman 

Hi all,

Using filestore, we have some clusters were we put some journals of 
regular osds(hdd) together with eg. cache or metadata osd on one SSD. 
Even with the OS too on the OSD, this gave us better performance than 
with journals on disk.


Now using bluestore, i was thinking if it is possible to have a 
bluestore OSD on an SSD, together with the block.db/block.wal of HDD 
osds ? Something like this for the SSD:


Number  Start   EndSizeFile system  NameFlags
 1  1049kB  106MB  105MB   xfs  ceph data
 2  106MB   150GB  150GBceph block
 3  150GB   151GB  1074MB   ceph block.db
 4  151GB   152GB  604MBceph block.wal

Using ceph-deploy/ceph-disk , this does not seem possible at the moment. 
Adding the db/wal partitions is not a problem, but having the OSD to 
share the disk is:


- ceph-disk does not accept partitions, it needs full disks to make the 
xfs and block partitions


- it always needs to have the first two partitions

- it will take all the space left of the disk for the OSD block partition.


I probably could hack something in by resizing the partitions, like 
above, but I'd rather not :)


Will such kind of feature be possible, or is this just a bad idea with 
bluestore?



Thank you very much!

Kenneth

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

2017-07-26 Thread Wido den Hollander 

> Op 26 juli 2017 om 13:43 schreef c.mo...@web.de:
> 
> 
> 26. Juli 2017 11:29, "Wido den Hollander"  schrieb:
> 
> >> Op 26 juli 2017 om 11:26 schreef c.mo...@web.de:
> >> 
> >> Hello!
> >> 
> >> Based on the documentation for defining quotas in CephFS for any directory
> >> (http://docs.ceph.com/docs/master/cephfs/quota), I defined a quota for 
> >> attribute max_bytes:
> >> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> >> getfattr: Removing leading '/' from absolute path names
> >> # file: mnt/ceph-fuse/MTY/
> >> ceph.quota.max_bytes="1"
> >> 
> >> To validate if the quota is working, I write a 128MB file in 
> >> /mnt/ceph-fuse/MTY:
> >> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> >> 2+0 records in
> >> 2+0 records out
> >> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
> >> 
> >> This file is created correctly, and the utilization statistcs confirm it:
> >> ld4257:~ # rados df
> >> pool name KB objects clones degraded unfound rd rd KB wr wr KB
> >> hdb-backup 131072 32 0 0 0 8 8 43251 88572586
> >> hdb-backup_metadata 27920 27 0 0 0 301 168115 6459 55386
> >> rbd 0 0 0 0 0 0 0 0 0
> >> templates 0 0 0 0 0 0 0 0 0
> >> total used 9528188 59
> >> total avail 811829446772
> >> total space 811838974960
> >> 
> >> Question:
> >> Why can I create a file with size 128MB after defining a quota of 100MB?
> > 
> > What kernel version does the client use? Quotas rely on client support.
> > 
> > Also, quotas are lazy and can take a bit of time before they start to block 
> > writes.
> > 
> > Wido
> > 
> >> THX
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> ld4257:~ # uname -r
> 4.4.59-92.24-default
> 

Hmm, thinking about it, I don't think the kernel supports CephFS quotas yet.

Somebody else might be able to confirm, but afaik it's not supported.

Wido

> In the meantime I have created 4 files of 210MB space allocation in total:
> ld4257:~ # ll -h /mnt/ceph-fuse/MTY/
> total 210M
> -rw-r--r-- 1 root root 100M Jul 26 13:40 100MBfile_from_ld4257
> -rw-r--r-- 1 root root  10M Jul 26 11:54 10MBfile_from_ld4257
> -rw-r--r-- 1 root root  50M Jul 26 11:55 50MBfile_from_ld2398
> -rw-r--r-- 1 root root  50M Jul 26 11:56 50MBfile_from_ld4257
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RGW Multisite Sync Memory Usage

2017-07-26 Thread Ryan Leimenstoll 
Hi all, 

We are currently trying to migrate our RGW Object Storage service from one zone 
to another (in the same zonegroup) in part to make use of erasure coded data 
pools. That being said, the rgw daemon is reliably getting OOM killed on the 
rgw origin host serving the original zone (and thus the current production 
data) as a result of high rgw memory usage. We are willing to consider more 
memory for the rgw daemon’s hosts to solve this problem, but was wondering what 
would be expected memory wise (at least as a rule of thumb). I noticed there 
were a few memory related rgw sync fixes in 10.2.9, but so far upgrading hasn’t 
seemed to prevent crashing. 


Some details about our cluster:
Ceph Version: 10.2.9
OS: RHEL 7.3

584 OSDs
Serving RBD, CephFS, and RGW

RGW Origin Hosts:
Virtualized via KVM/QEMU, RHEL 7.3
Memory: 32GB
CPU: 12 virtual cores (Hypervisor processors: Intel E5-2630)

First zone data and index pools:
pool name KB  objects   clones degraded  
unfound   rdrd KB   wrwr KB
.rgw.buckets112190858231 3423974600
0   2713542251 265848150719475841837 153970795085
.rgw.buckets.index0 497200  
  0   3721485483   5926323574 360300980


Thanks,
Ryan Leimenstoll
University of Maryland Institute for Advanced Computer Studies

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Broken Ceph Cluster when adding new one - Proxmox 5.0 & Ceph Luminous

2017-07-26 Thread Alexandre DERUMIER 
Hi Phil,


It's possible that rocksdb have a bug with some old cpus currently (old xeon 
and some opteron)
I have the same behaviour with new cluster when creating mons
http://tracker.ceph.com/issues/20529

What is your cpu model ?

in your log: 

sh[1869]:  in thread 7f6d85db3c80 thread_name:ceph-osd
sh[1869]:  ceph version 12.1.0 (330b5d17d66c6c05b08ebc129d3e6e8f92f73c60) 
luminous (dev)
sh[1869]:  1: (()+0x9bc562) [0x558561169562]
sh[1869]:  2: (()+0x110c0) [0x7f6d835cb0c0]
sh[1869]:  3: 
(rocksdb::VersionBuilder::SaveTo(rocksdb::VersionStorageInfo*)+0x871) 
[0x5585615788b1]
sh[1869]:  4: 
(rocksdb::VersionSet::Recover(std::vector const&, bool)+0x26bc) 
[0x55856145ca4c]
sh[1869]:  5: 
(rocksdb::DBImpl::Recover(std::vector const&, bool, bool, 
bool)+0x11f) [0x558561423e6f]
sh[1869]:  6: (rocksdb::DB::Open(rocksdb::DBOptions const&, 
std::__cxx11::basic_string 
const&, std:
sh[1869]:  7: (rocksdb::DB::Open(rocksdb::Options const&, 
std::__cxx11::basic_string 
const&, rocksdb:
sh[1869]:  8: (RocksDBStore::do_open(std::ostream&, bool)+0x68e) 
[0x5585610af76e]
sh[1869]:  9: (RocksDBStore::create_and_open(std::ostream&)+0xd7) 
[0x5585610b0d27]
sh[1869]:  10: (BlueStore::_open_db(bool)+0x326) [0x55856103c6d6]
sh[1869]:  11: (BlueStore::mkfs()+0x856) [0x55856106d406]
sh[1869]:  12: (OSD::mkfs(CephContext*, ObjectStore*, 
std::__cxx11::basic_string 
const&, uuid_d, int)+0x348) [0x558560bc98f8]
sh[1869]:  13: (main()+0xe58) [0x558560b1da78]
sh[1869]:  14: (__libc_start_main()+0xf1) [0x7f6d825802b1]
sh[1869]:  15: (_start()+0x2a) [0x558560ba4dfa]
sh[1869]: 2017-07-16 14:46:00.763521 7f6d85db3c80 -1 *** Caught signal (Illegal 
instruction) **
sh[1869]:  in thread 7f6d85db3c80 thread_name:ceph-osd
sh[1869]:  ceph version 12.1.0 (330b5d17d66c6c05b08ebc129d3e6e8f92f73c60) 
luminous (dev)
sh[1869]:  1: (()+0x9bc562) [0x558561169562]

- Mail original -
De: "Phil Schwarz" 
À: "Udo Lembke" , "ceph-users" 
Envoyé: Dimanche 16 Juillet 2017 15:04:16
Objet: Re: [ceph-users] Broken Ceph Cluster when adding new one - Proxmox 5.0 & 
Ceph Luminous

Le 15/07/2017 à 23:09, Udo Lembke a écrit : 
> Hi, 
> 
> On 15.07.2017 16:01, Phil Schwarz wrote: 
>> Hi, 
>> ... 
>> 
>> While investigating, i wondered about my config : 
>> Question relative to /etc/hosts file : 
>> Should i use private_replication_LAN Ip or public ones ? 
> private_replication_LAN!! And the pve-cluster should use another network 
> (nics) if possible. 
> 
> Udo 
> 
OK, thanks Udo. 

After investigation, i did : 
- set Noout OSDs 
- Stopped CPU-pegging LXC 
- Check the cabling 
- Restart the whole cluster 

Everything went fine ! 

But, when i tried to add a new OSD : 

fdisk /dev/sdc --> Deleted the partition table 
parted /dev/sdc --> mklabel msdos (Disk came from a ZFS FreeBSD system) 
dd if=/dev/null of=/dev/sdc 
ceph-disk zap /dev/sdc 
dd if=/dev/zero of=/dev/sdc bs=10M count=1000 

And recreated the OSD via Web GUI. 
Same result, the OSD is known by the node, but not by the cluster. 

Logs seem to show an issue with this bluestore OSD, have a look at the file. 

I'm gonna give a try to OSD recreating using Filestore. 

Thanks 


___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph v10.2.9 - rbd cli deadlock ?

2017-07-26 Thread Jason Dillaman 
While I cannot reproduce what you are seeing, I can see how it could
theoretically be possible for this to deadlock on a thread shutdown if
the process was being shutdown before the service thread had a chance
to actually start executing. I've opened a tracker ticket for the
issue [1].

[1] http://tracker.ceph.com/issues/20776

On Tue, Jul 25, 2017 at 6:18 PM, Kjetil Jørgensen  wrote:
> Hi,
>
> I'm not sure yet whether or not this is made worse by config, however - if I
> do something along the lines of:
>>
>> seq 100 | xargs -P100 -n1 bash -c 'exec rbd.original showmapped'
>
>
> I'll end up with at least one of the invocations deadlocked like below.
> Doing the same on our v10.2.7 clusters seems to work fine.
>
> The stacktraces according to GDB looks something like this for all the ones
> I've looked at at least:
>>
>> warning: the debug information found in "/usr/bin/rbd" does not match
>> "/usr/bin/rbd.original" (CRC mismatch).
>> # Yes - we've diverted rbd to rbd.original with a shell-wrapper around it
>
>
>> [New LWP 285438]
>> [New LWP 285439]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> 0x7fbbea58798d in pthread_join (threadid=140444952844032,
>> thread_return=thread_return@entry=0x0) at pthread_join.c:90
>> 90  pthread_join.c: No such file or directory.
>> Thread 3 (Thread 0x7fbbe3865700 (LWP 285439)):
>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1  0x55a852fcf896 in Cond::Wait (mutex=..., this=0x55a85cdeb258) at
>> ./common/Cond.h:56
>> #2  CephContextServiceThread::entry (this=0x55a85cdeb1c0) at
>> common/ceph_context.cc:101
>> #3  0x7fbbea5866ba in start_thread (arg=0x7fbbe3865700) at
>> pthread_create.c:333
>> #4  0x7fbbe80743dd in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>> Thread 2 (Thread 0x7fbbe4804700 (LWP 285438)):
>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1  0x55a852fb297b in ceph::log::Log::entry (this=0x55a85cd98830) at
>> log/Log.cc:457
>> #2  0x7fbbea5866ba in start_thread (arg=0x7fbbe4804700) at
>> pthread_create.c:333
>> #3  0x7fbbe80743dd in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>> Thread 1 (Thread 0x7fbbfda1e100 (LWP 285436)):
>> #0  0x7fbbea58798d in pthread_join (threadid=140444952844032,
>> thread_return=thread_return@entry=0x0) at pthread_join.c:90
>> #1  0x55a852fb6270 in Thread::join (this=this@entry=0x55a85cdeb1c0,
>> prval=prval@entry=0x0) at common/Thread.cc:171
>> #2  0x55a852fca060 in CephContext::join_service_thread
>> (this=this@entry=0x55a85cd95780) at common/ceph_context.cc:637
>> #3  0x55a852fcc2c7 in CephContext::~CephContext (this=0x55a85cd95780,
>> __in_chrg=) at common/ceph_context.cc:507
>> #4  0x55a852fcc9bc in CephContext::put (this=0x55a85cd95780) at
>> common/ceph_context.cc:578
>> #5  0x55a852eac2b1 in
>> boost::intrusive_ptr::~intrusive_ptr (this=0x7ffef7ef5060,
>> __in_chrg=) at
>> /usr/include/boost/smart_ptr/intrusive_ptr.hpp:97
>> #6  main (argc=, argv=) at
>> tools/rbd/rbd.cc:17
>
>
> Cheers,
> --
> Kjetil Joergensen 
> Staff Curmudgeon, Medallia Inc
> Phone: +1 (650) 739-6580
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

2017-07-26 Thread c . monty 
26. Juli 2017 11:29, "Wido den Hollander"  schrieb:

>> Op 26 juli 2017 om 11:26 schreef c.mo...@web.de:
>> 
>> Hello!
>> 
>> Based on the documentation for defining quotas in CephFS for any directory
>> (http://docs.ceph.com/docs/master/cephfs/quota), I defined a quota for 
>> attribute max_bytes:
>> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
>> getfattr: Removing leading '/' from absolute path names
>> # file: mnt/ceph-fuse/MTY/
>> ceph.quota.max_bytes="1"
>> 
>> To validate if the quota is working, I write a 128MB file in 
>> /mnt/ceph-fuse/MTY:
>> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
>> 2+0 records in
>> 2+0 records out
>> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
>> 
>> This file is created correctly, and the utilization statistcs confirm it:
>> ld4257:~ # rados df
>> pool name KB objects clones degraded unfound rd rd KB wr wr KB
>> hdb-backup 131072 32 0 0 0 8 8 43251 88572586
>> hdb-backup_metadata 27920 27 0 0 0 301 168115 6459 55386
>> rbd 0 0 0 0 0 0 0 0 0
>> templates 0 0 0 0 0 0 0 0 0
>> total used 9528188 59
>> total avail 811829446772
>> total space 811838974960
>> 
>> Question:
>> Why can I create a file with size 128MB after defining a quota of 100MB?
> 
> What kernel version does the client use? Quotas rely on client support.
> 
> Also, quotas are lazy and can take a bit of time before they start to block 
> writes.
> 
> Wido
> 
>> THX
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


ld4257:~ # uname -r
4.4.59-92.24-default

In the meantime I have created 4 files of 210MB space allocation in total:
ld4257:~ # ll -h /mnt/ceph-fuse/MTY/
total 210M
-rw-r--r-- 1 root root 100M Jul 26 13:40 100MBfile_from_ld4257
-rw-r--r-- 1 root root  10M Jul 26 11:54 10MBfile_from_ld4257
-rw-r--r-- 1 root root  50M Jul 26 11:55 50MBfile_from_ld2398
-rw-r--r-- 1 root root  50M Jul 26 11:56 50MBfile_from_ld4257
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

2017-07-26 Thread Wido den Hollander 

> Op 26 juli 2017 om 11:26 schreef c.mo...@web.de:
> 
> 
> Hello!
> 
> Based on the documentation for defining quotas in CephFS for any directory 
> (http://docs.ceph.com/docs/master/cephfs/quota/), I defined a quota for 
> attribute max_bytes:
> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> getfattr: Removing leading '/' from absolute path names
> # file: mnt/ceph-fuse/MTY/
> ceph.quota.max_bytes="1"
> 
> To validate if the quota is working, I write a 128MB file in 
> /mnt/ceph-fuse/MTY:
> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> 2+0 records in
> 2+0 records out
> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
> 
> This file is created correctly, and the utilization statistcs confirm it:
> ld4257:~ # rados df
> pool name KB  objects   clones degraded  
> unfound   rdrd KB   wrwr KB
> hdb-backup131072   3200   
>  08843251 88572586
> hdb-backup_metadata27920   2700   
>  0  301   168115 645955386
> rbd0000   
>  00000
> templates  0000   
>  00000
>   total used 9528188   59
>   total avail   811829446772
>   total space   811838974960
> 
> 
> Question:
> Why can I create a file with size 128MB after defining a quota of 100MB?
> 

What kernel version does the client use? Quotas rely on client support.

Also, quotas are lazy and can take a bit of time before they start to block 
writes.

Wido

> 
> THX
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Defining quota in CephFS - quota is ignored

2017-07-26 Thread c . monty 
Hello!

Based on the documentation for defining quotas in CephFS for any directory 
(http://docs.ceph.com/docs/master/cephfs/quota/), I defined a quota for 
attribute max_bytes:
ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
getfattr: Removing leading '/' from absolute path names
# file: mnt/ceph-fuse/MTY/
ceph.quota.max_bytes="1"

To validate if the quota is working, I write a 128MB file in /mnt/ceph-fuse/MTY:
ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
2+0 records in
2+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s

This file is created correctly, and the utilization statistcs confirm it:
ld4257:~ # rados df
pool name KB  objects   clones degraded  
unfound   rdrd KB   wrwr KB
hdb-backup131072   3200
08843251 88572586
hdb-backup_metadata27920   2700 
   0  301   168115 645955386
rbd0000
00000
templates  0000
00000
  total used 9528188   59
  total avail   811829446772
  total space   811838974960


Question:
Why can I create a file with size 128MB after defining a quota of 100MB?


THX
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Linear space complexity or memory leak in `Radosgw-admin bucket check --fix`

2017-07-26 Thread Orit Wasserman 
Hi Hans,

On Wed, Jul 26, 2017 at 10:24 AM, Ben Hines  wrote:
> Which version of Ceph?
>
> On Tue, Jul 25, 2017 at 4:19 AM, Hans van den Bogert 
> wrote:
>>
>> Hi All,
>>
>> I don't seem to be able to fix a bucket, a bucket which has become
>> inconsistent due to the use of the `inconsistent-index` flag 8).
>>
>> My ceph-admin VM has 4GB of RAM, but that doesn't seem to be enough to do
>> a `radosgw-admin bucket check --fix` which holds 6M items, as the
>> radosgw-admin process is killed eventually by the Out-Of-Memory-Manager. Is
>> this high RAM usage to be expected, or should I file a bug?
>>

At the current implementation we send all entries needing fixing to
the OSD in one go.
In case of a large bucket this it can cause high memory usage or even
http://tracker.ceph.com/issues/20772.
I am working on fix it.

Regards,
Orit

>> Regards,
>>
>> Hans
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] upgrading to newer jewel release, no cluster uuid assigned

2017-07-26 Thread Jasper Spaans 
That value is in ceph.conf, but I wouldn't expect that to have helped,
looking at the ceph-disk code (in the module level function `activate`)::

ceph_fsid = read_one_line(path, 'ceph_fsid')
if ceph_fsid is None:
raise Error('No cluster uuid assigned.')

Maybe there is a thinko there, as ceph_fsid is only used to find the
cluster name by scanning config files (which does succeed if there is
only a ceph.conf that does not contain an fsid - meaning the ceph_fsid
value is not used at all.)


Cheers,
Jasper


On 25/07/2017 19:22, David Turner wrote:
> Does your ceph.conf file have your cluster uuid lasted in it? You should
> be able to see what it is from ceph status and add it to your config if
> it's missing.
> 
> 
> On Tue, Jul 25, 2017, 7:38 AM Jasper Spaans
> > wrote:
> 
> Hi list,
> 
> We had some troubles activating our OSDs after upgrading from Ceph
> 10.2.7 to 10.2.9. The error we got was 'No cluster uuid assigned' after
> calling ceph-disk trigger --sync /dev/sda3 .
> 
> Our cluster runs on Ubuntu 16.04, has been deployed using the
> Ceph-ansible roles, and we're using the collocated dmcrypt mode (so, 3
> partitions per drive for data, journal and lockbox, with the first two
> encrypted using dmcrypt).
> 
> After some probing (read: diffing the source code) it turned out our
> lockbox directories did not contain a 'ceph_fsid' file, so I just
> bluntly put them in using something along the lines of:
> 
> for fs in $(mount|grep lockbox|cut -d' ' -f3) ; do \
>   mount $fs -o rw,remount
>   echo $our_fs_uuid > $fs/ceph_fsid
>   mount $fs -o ro,remount
> done
> 
> After doing this on all of our nodes, I was able to upgrade and activate
> the OSDs again, and it even survives a reboot.
> 
> Looking at the release notes, I couldn't find any mention of this - so
> I'll post it here in the hopes someone may find it useful.
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Linear space complexity or memory leak in `Radosgw-admin bucket check --fix`

2017-07-26 Thread Ben Hines 
Which version of Ceph?

On Tue, Jul 25, 2017 at 4:19 AM, Hans van den Bogert 
wrote:

> Hi All,
>
> I don't seem to be able to fix a bucket, a bucket which has become
> inconsistent due to the use of the `inconsistent-index` flag 8).
>
> My ceph-admin VM has 4GB of RAM, but that doesn't seem to be enough to do
> a `radosgw-admin bucket check --fix` which holds 6M items, as the
> radosgw-admin process is killed eventually by the Out-Of-Memory-Manager. Is
> this high RAM usage to be expected, or should I file a bug?
>
> Regards,
>
> Hans
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com