Hi All,
I just would to be sure about keystone configuration for Rados Gateway.
I read the documentation http://ceph.com/docs/master/radosgw/keystone/ and
http://ceph.com/docs/master/radosgw/config-ref/?highlight=keystone
but I didn't catch if after having configured the rados gateway
Hi,
I'm still trying to find why there is much more write operations on
filestore since Emperor/Firefly than from Dumpling.
So, I add monitoring of all perf counters values from OSD.
From what I see : «filestore.ops» reports an average of 78 operations
per seconds. But, block device monitoring
Le 22/03/2015 22:44, hp cre a écrit :
Hello Yann,
Thanks for your reply. Unfortunately, I found it by chance during a
search, since you didn't include me in the reply, I never got it on my
email.
Well that wasn't intended, but that's because I replied to the list,
which is usually the
Hi all,
I have a ceph cluster(0.80.7) in production.
Now I encounter a bottleneck of iosp, so I want to add a cache
tier with SSDs to provide better I/O performance. Here is the procedure:
1. Create a cache pool
2. Set up a cahce tire
ceph osd tier add cold-storage hot-storage
3. Set
Erg... I sent to fast. Bad title, please read «More writes on
blockdevice than on filestore)
Le lundi 23 mars 2015 à 14:21 +0100, Olivier Bonvalet a écrit :
Hi,
I'm still trying to find why there is much more write operations on
filestore since Emperor/Firefly than from Dumpling.
So, I
Yes, I understand that.
The initial purpose of first email was just an advise for new comers. My
fault was in that I was selected ext4 for SSD disks as backend.
But I did not foresee that inode number can reach its limit before the
free space :)
And maybe there must be some sort of warning
Hi Somnath,
Thank you, please find my answers below
Somnath Roy somnath@sandisk.com a écrit le 22/03/15 18:16 :
Hi Frederick,
Need some information here.
1. Just to clarify, you are saying it is happening g in 0.87.1 and not
in Firefly ?
That's a possibility, others running
Hi,
consider following values for a pool:
Size = 3
OSDs = 400
%Data = 100
Target PGs per OSD = 200 (This is default)
The PG calculator generates number of PGs for this pool as : 32768.
Questions:
1. The Ceph documentation recommends around 100 PGs/OSD, whereas the
calculator takes 200 as
On 22/03/2015 08:29, Bogdan SOLGA wrote:
Hello, everyone!
I have a few questions related to the CephFS part of Ceph:
* is it production ready?
Like it says at http://ceph.com/docs/master/cephfs/: CephFS currently
lacks a robust ‘fsck’ check and repair function. Please use caution when
You could fix this by changing your block size when formatting the
mount-point with the mkfs -b command. I had this same issue when dealing
with the filesystem using glusterfs and the solution is to either use a
filesystem that allocates inodes automatically or change the block size
when you
Hi Steffen
We have recently encountered the errors described below. Initially one must set
check_obsoletes=1 in the yum priorities.conf file.
However subsequent yum updates cause problems.
The solution we use is to disable the epel repo by default:
yum-config-manager --disable epel
and
On Mon, Mar 23, 2015 at 6:21 AM, Olivier Bonvalet ceph.l...@daevel.fr wrote:
Hi,
I'm still trying to find why there is much more write operations on
filestore since Emperor/Firefly than from Dumpling.
Do you have any history around this? It doesn't sound familiar,
although I bet it's because
Hi Greg,
the low-/high-CPU comportement is absolutely persistent while a host is
UP, no oscillation.
But rebooting a node can make its comportment switch low-/high-CPU, as
seen this morning after checking the BIOS settings (especially numa)
were the same on 2 hosts.
Hosts are identical,
Hi,
Le lundi 23 mars 2015 à 07:29 -0700, Gregory Farnum a écrit :
On Mon, Mar 23, 2015 at 6:21 AM, Olivier Bonvalet ceph.l...@daevel.fr wrote:
Hi,
I'm still trying to find why there is much more write operations on
filestore since Emperor/Firefly than from Dumpling.
Do you have any
On Sun, Mar 22, 2015 at 2:55 AM, Saverio Proto ziopr...@gmail.com wrote:
Hello,
I started to work with CEPH few weeks ago, I might ask a very newbie
question, but I could not find an answer in the docs or in the ml
archive for this.
Quick description of my setup:
I have a ceph cluster with
On Mon, Mar 23, 2015 at 4:31 AM, f...@univ-lr.fr f...@univ-lr.fr wrote:
Hi Somnath,
Thank you, please find my answers below
Somnath Roy somnath@sandisk.com a écrit le 22/03/15 18:16 :
Hi Frederick,
Need some information here.
1. Just to clarify, you are saying it is happening g in
On Sun, Mar 22, 2015 at 11:22 AM, Somnath Roy somnath@sandisk.com wrote:
You should be having replicated copies on other OSDs (disks), so, no need to
worry about the data loss. You add a new drive and follow the steps in the
following link (either 1 or 2)
Except that's not the case if
On Mon, 23 Mar 2015 11:51:56 +0100 (CET) Alexandre DERUMIER wrote:
the combination of all the
things mentioned before in the Ceph/FS stack caused a 12x amplification
(instead of 2x) _before_ hitting the SSD.
oh, ok, pretty strange.
BTW, is it through ceph-fs ? or rbd/rados ?
See the
Hello,
thanks for the answers.
This was exacly what I was looking for:
mon_osd_down_out_interval = 900
I was not waiting long enoght to see my cluster recovering by itself.
That's why I tried to increase min_size, because I did not understand
what min_size was for.
Now that I know what is
On Mon, 23 Mar 2015, Xinze Chi wrote:
hi ceph:
Currently, there is not a command which can stop scrubbing when
the pg is doing scrub or deep
scrub. What about add a command to support it ? I think this s every
used for system administrator.
I have add a issue to track it.
On Mon, 23 Mar 2015 15:26:07 +0300 Kamil Kuramshin wrote:
Yes, I understand that.
The initial purpose of first email was just an advise for new comers. My
fault was in that I was selected ext4 for SSD disks as backend.
But I did not foresee that inode number can reach its limit before the
On Sat, Mar 21, 2015 at 10:46 AM, shylesh kumar shylesh.mo...@gmail.com wrote:
Hi ,
I was going through this simplified crush algorithm given in ceph website.
def crush(pg):
all_osds = ['osd.0', 'osd.1', 'osd.2', ...]
result = []
# size is the number of copies; primary+replicas
On Mon, Mar 23, 2015 at 7:17 AM, Saverio Proto ziopr...@gmail.com wrote:
Hello,
thanks for the answers.
This was exacly what I was looking for:
mon_osd_down_out_interval = 900
I was not waiting long enoght to see my cluster recovering by itself.
That's why I tried to increase min_size,
Does anybody know how to succesful install Calamari in rhel7 ? I have tried the
vagrant thug without sucesss and it seems like a nightmare there is a Kind of
Sidur when you do vagrant up where it seems not to find the vm path...
Regards
Jesus Chavez
SYSTEMS ENGINEER-C.SALES
Hello,
My name is Amy Wilson. I am the Director of eBrand Business (a 3D logo design and professional CGI animation studio).
Ive just visited the Ceph website and, I have to say, you have a really nice business over there. My congratulations!
hi :
there is a pool , data is not empty (by using 'rados df' or rados stats -p)
,but can not list the objs in that pool (using 'rados ls -p 'or python api)
. how do it happened ? The pool was created by a normal cmd . By the way
using the c code read and write works find.
My vag understanding is that this is mapped through the zone associated with
the specific user. So define your desiree pools and zones mapping to the pools
and assign users to desired regions+zones and thus to different pools per user.
Den 13/03/2015 kl. 07.48 skrev Sreenath BH
We are looking for someone to develop a course on a Ceph implementation in a
large computer manufacturer hybrid or public cloud. The project would involve
meeting with internal engineers to discover latest developments and applying
their style and standards to the courseware. Project timeframe
Hi David,
I also see only the RBD pool getting created by default in 0.93.
With regards to resizing placement groups, I believe you can use:
ceph osd pool set [pool name] pg_num
ceph osd pool set [pool name] pgp_num
Be forewarned, this will trigger data migration.
Cheers,
Hi,
We have a huge write IO Problem in our preproductive Ceph Cluster. First our
Hardware:
4 OSD Nodes with:
Supermicro X10 Board
32GB DDR4 RAM
2x Intel Xeon E5-2620
LSI SAS 9300-8i Host Bus Adapter
Intel Corporation 82599EB 10-Gigabit
2x Intel SSDSA2CT040G3 in software raid 1 for system
I have a doubt . In a scenario (3nodes x 4osd each x 2replica) I tested
with a node down and as long as you have space available all objects were
there.
Is it possible all replicas of an object to be saved in the same node?
Is it possible to lose any?
Is there a mechanism that prevents
hi harry,
your question is more related to the ceph-user, so i am replying to ceph-users.
On Wed, Mar 18, 2015 at 12:02 AM, harryxiyou harryxi...@gmail.com wrote:
Hi all,
I wanna deploy Ceph and I see the doc here
(http://docs.ceph.com/docs/dumpling/start/quick-start-preflight/). I
wonder
Hi Dennis,
I am interested in your project.
I wrote a Win32 cephfs client https://github.com/ceph/ceph-dokan.
But ceph-dokan runs in user-mode. I see you port code from
kernel cephfs, are you planning to write a kernel mode AIX-cephfs?
Thanks!
2015-03-04 17:59 GMT+08:00 Dennis
Sage,
we use apache as a filter for security and additional functionality
reasons. I do like the idea, but we'd need some kind of interface to
filter/modify/process requests.
Best regards
Axel Dunkel
-Ursprüngliche Nachricht-
Von: ceph-devel-ow...@vger.kernel.org
This would be a good thing to bring up in the meeting on Wednesday.
yes !
Yes, we can discuss details on Wed's call.
I wonder how much effect flow-control and header/data crc had.
yes. I known that sommath also disable crc for his bench
I disabled ceph's header/data crc for both
-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Brendan Moloney
Sent: 23 March 2015 21:02
To: Noah Mehl
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD + Flashcache + udev + Partition uuid
This would be in addition to
Yes, we are also facing similar issue on load (and running after some time).
This is a tcmalloc behavior.
You can try setting the following env variable to a bigger value say 128MB or
so.
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
This env variable is supposed to alleviate the issue but what we
Hi!
I'm using ceph cluster, packed to a number of docker containers.
There are two things, which you need to know:
1. Ceph OSDs are using FS attributes, which may not be supported by filesystem
inside docker container, so you need to mount external directory inside a
container to store OSD
Ok, so the decompile error is because I didn't download the CRUSH map
(found that out using hexdump), but I still can't compile an
unmodified CRUSH map.
[root@nodezz ~]# crushtool -d map.crush -o map
[root@nodezz ~]# crushtool -c map -o map.crush
map:105 error: parse error at ''
For some reason
On Mon, 23 Mar 2015, Robert LeBlanc wrote:
Ok, so the decompile error is because I didn't download the CRUSH map
(found that out using hexdump), but I still can't compile an
unmodified CRUSH map.
[root@nodezz ~]# crushtool -d map.crush -o map
[root@nodezz ~]# crushtool -c map -o map.crush
OK, sorry for all the quick e-mails, but I got it to compile. For some
reason there are a few errors from decompiling the CRUSH map.
1. The decompiled map has alg straw2 which is not vaild, removing
the 2 lets it compile
2. The hosts have weight 0.000, which I don't think prevents the map
from
Hello,
Again refer to my original, old mail:
http://lists.opennebula.org/pipermail/ceph-users-ceph.com/2014-October/043949.html
I was strictly looking at the SMART values, in the case of these
Intel DC S3700 SSDs the Host_Writes_32MiB values.
Which, according to what name implies and what all
hi ceph:
Currently, there is not a command which can stop scrubbing when
the pg is doing scrub or deep
scrub. What about add a command to support it ? I think this s every
used for system administrator.
I have add a issue to track it. http://tracker.ceph.com/issues/11202.
--
Regards,
On 23 Mar 2015, at 03:58, Haomai Wang haomaiw...@gmail.com wrote:
On Mon, Mar 23, 2015 at 2:53 AM, Josef Johansson jose...@gmail.com
mailto:jose...@gmail.com wrote:
Hi all!
Trying to figure out how much my journals are used, using SSDs as journals
and SATA-drives as storage, I dive
In my case there was cache pool for ec-pool serving RBD-images, and
object size is 4Mb, and client was an /kernel-rbd /client
each SSD disk is 60G disk, 2 disk per node, 6 nodes in total = 12 OSDs
in total
23.03.2015 12:00, Christian Balzer пишет:
Hello,
This is rather confusing, as
On 02/17/2015 11:13 AM, Mohamed Pakkeer wrote:
Hi Joao,
We followed your instruction to create the store dump
ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db list store.dump'
for above store's location, let's call it $STORE:
for m in osdmap pgmap; do
for k in first_committed
Hi,
Isn't it in the nature of ssd to have write amplication ?
Generaly, they have a erase block size of 128k,
so the worst case could be 128/4 = 32x write amplification.
(of course ssd algorithms and optimisations reduce this write amplification).
Now, it could be great to see if it's coming
Recently got a problem with OSDs based on SSD disks used in cache tier
for EC-pool
superuser@node02:~$ df -i
FilesystemInodes IUsed *IFree* IUse% Mounted on
...
/dev/sdb13335808 3335808 *0* 100%
/var/lib/ceph/osd/ceph-45
/dev/sda1
Hello,
This is rather confusing, as cache-tiers are just normal OSDs/pools and
thus should have Ceph objects of around 4MB in size by default.
This is matched on what I see with Ext4 here (normal OSD, not a cache
tier):
---
size:
/dev/sde1 2.7T 204G 2.4T 8% /var/lib/ceph/osd/ceph-0
(not tested, but I think with journal and O_DSYNC writes, it can give use ssd
write amplification)
also, I think that enterprise ssd with supercapacitor, should be able to cache
theses o_dsync writes in
the ssd buffer, and do bigger writes to reduce amplification.
Don't known how ssd internal
- Mail original -
Thanks stephane the thing is that those steps needs to be run in the node where
the osd lives, I dont have that node any more since the operating Systems got
corrupted so I Couldnt make it work :(
Jesus,
On my test cluster, what i have done is
Hi all, I did HA failover test shutting down 1 node and I see that only 1 OSD
came up after reboot:
[root@geminis ceph]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 50G 4.5G 46G 9% /
devtmpfs 126G 0 126G 0% /dev
tmpfs
I have been looking at the options for SSD caching for a bit now. Here is my
take on the current options:
1) bcache - Seems to have lots of reliability issues mentioned on mailing list
with little sign of improvement.
2) flashcache - Seems to be no longer (or minimally?) developed/maintained,
Just to add, the main reason it seems to make a difference is the metadata
updates which lie on the actual OSD. When you are doing small block writes,
these metadata updates seem to take almost as long as the actual data, so
although the writes are getting coalesced, the actual performance isn't
Hi all!
I had a CEPH Cluster with 10x OSDs all of them in one node.
Since the cluster was built from the beginning with just one OSDs node
the crushmap had as a default
the replication to be on OSDs.
Here is the relevant part from my crushmap:
# rules
rule replicated_ruleset {
We deployed with just putting the journal on an SSD directly, why would this
not work for you? Just wondering really :)
Thanks!
~Noah
On Mar 23, 2015, at 4:36 PM, Brendan Moloney molo...@ohsu.edu wrote:
I have been looking at the options for SSD caching for a bit now. Here is my
take on
This would be in addition to having the journal on SSD. The journal doesn't
help at all with small random reads and has a fairly limited ability to
coalesce writes.
In my case, the SSDs we are using for journals should have plenty of
bandwidth/IOPs/space to spare, so I want to see if I can
I was trying to decompile and edit the CRUSH map to adjust the CRUSH
rules. My first attempt created a map that would decompile, but I
could not recompile the CRUSH even if didn't modify it. When trying to
download the CRUSH fresh, now the decompile fails.
[root@nodezz ~]# ceph osd getmap -o
Hi Chris and Craig,
Thank you for sharing your experience with me about S3 API RADOS gateway!
Jerry Lam
Senior Software Developer, Big Data
Ontario Institute for Cancer Research
MaRS Centre
661 University Avenue
Suite 510
Toronto, Ontario
Canada M5G 0A3
Email: jerry@oicr.on.ca
Toll-free:
Hello, John!
Thank you very much for your reply and for the provided information! As a
follow-up to your email, a few other questions have arisen:
- is the http://ceph.com/docs/master/cephfs/ page referring to the
current release version (Giant) or to the HEAD (Hammer) version? if it's
You just need to change your rule from
step chooseleaf firstn 0 type osd
to
step chooseleaf firstn 0 type host
There will be data movement as it will want to move about half the
objects to the new host. There will be data generation as you move
from size 1 to size 2. As far as I know a deep
Georgios,
it really depends on how busy and powerful your cluster is, as Robert
wrote.
If in doubt, lower the backfill value as pointed out by Robert.
Look at the osd_scrub_load_threshold and with new enough
versions of Ceph at the osd_scrub_sleep setting, this is very helpful in keeping
deep
On 23/03/2015 19:00, Bogdan SOLGA wrote:
* is the http://ceph.com/docs/master/cephfs/ page referring to the
current release version (Giant) or to the HEAD (Hammer) version?
if it's referring to Giant -- are there any major improvements and
fixes for CephFS included in the
I don't believe that you can set the schedule of the deep scrubs.
People that want that kind of control disable deep scrubs and run a
script to scrub all PGs. For the other options, you should look
through http://ceph.com/docs/master/rados/configuration/osd-config-ref/
and find what you feel might
I'm trying to create a CRUSH ruleset and I'm using crushtool to test
the rules, but it doesn't seem to mapping things correctly. I have two
roots, on for spindles and another for SSD. I have two rules, one for
each root. The output of crushtool on rule 0 shows objects being
mapped to SSD OSDs when
A. I see now. Has anyone used
cachecadehttp://www.lsi.com/products/raid-controllers/pages/megaraid-cachecade-pro-software.aspx
from LSI for both the read and write cache to SSD? I don’t know if you can
attach a cachecade device to a JBOD, but if you could it would probably perform
really
Hello,
If you had used performance or slow in your subject future generations
would be able find this thread and what it is about more easily. ^_-
Also, check the various SSD + performance threads in the ML archives.
On Fri, 20 Mar 2015 14:13:19 + Rottmann Jonas wrote:
Hi,
We have a
67 matches
Mail list logo
