Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

> > Warren Wang > > > > On 6/24/16, 10:23 AM, "Wade Holler" <wade.hol...@gmail.com> wrote: > >>On the vm.vfs_cace_pressure = 1 : We had this initially and I still >>think it is the best choice for most configs. However with our large >>memory

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

ph.com > Cc: Somnath Roy; Warren Wang - ISD; Wade Holler; Blair Bethwaite; Ceph > Development > Subject: Re: [ceph-users] Dramatic performance drop at certain number of > objects in pool > > > Hello, > > On Thu, 23 Jun 2016 22:24:59 + Somnath Roy wrote: > >> Or

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

No. Our application writes very small objects. On Wed, Jun 22, 2016 at 10:01 PM, Blair Bethwaite <blair.bethwa...@gmail.com> wrote: > On 23 June 2016 at 11:41, Wade Holler <wade.hol...@gmail.com> wrote: >> Workload is native librados with python. ALL 4k objects. > &g

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

rkload? I wonder whether for certain workloads (e.g. > RBD) it's better to increase default object size somewhat before > pushing the split/merge up a lot... > > Cheers, > > On 23 June 2016 at 11:26, Wade Holler <wade.hol...@gmail.com> wrote: >> Based on everyones suggest

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

ectory is under the >> calculated threshold and a write occurs (maybe a read, I forget). >> > If it's a read a plain scrub might do the trick. > > Christian >> Warren >> >> >> From: ceph-users >> <ceph-users-boun...@lists.ceph.com<mailto:ceph-users-bou

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

Thanks everyone for your replies. I sincerely appreciate it. We are testing with different pg_num and filestore_split_multiple settings. Early indications are well not great. Regardless it is nice to understand the symptoms better so we try to design around it. Best Regards, Wade On Mon,

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

e look like (including slab)? > > Cheers, > > On 16 June 2016 at 22:14, Wade Holler <wade.hol...@gmail.com> wrote: >> Hi All, >> >> I have a repeatable condition when the object count in a pool gets to >> 320-330 million the object write time dramatically and

Re: [ceph-users] Dramatic performance drop at certain number of objects in pool

short periods of high read IOPS where (almost) no > writes occur? What does your memory usage look like (including slab)? > > Cheers, > > On 16 June 2016 at 22:14, Wade Holler <wade.hol...@gmail.com> wrote: > > Hi All, > > > > I have a repeatable conditi

[ceph-users] Performance drop when object count in a pool hits a threshold

Hi All, I have a repeatable condition when the object count in a pool gets to 320-330 million the object write time dramatically and almost instantly increases as much as 10X, exhibited by fs_apply_latency going from 10ms to 100s of ms. Can someone point me in a direction / have an explanation ?

Re: [ceph-users] RadosGW performance s3 many objects

We (my customer ) are trying to test at Jewell now but I can say that the above behavior was also observed by my customer at Infernalis. After 300 million or so objects in a single bucket the cluster basically fell down as described above. Few hundred osds in this cluster. We are concerned that

[ceph-users] Unsubscribe

___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Incorrect crush map

Hi Ben, What OS+Version ? Best Regards, Wade On Tue, May 3, 2016 at 2:44 PM Ben Hines wrote: > My crush map keeps putting some OSDs on the wrong node. Restarting them > fixes it temporarily, but they eventually hop back to the other node that > they aren't really on. > > Is

[ceph-users] pg to RadosGW object list

Hi All, I searched google and what not but haven't found this yet. Does anyone know how to do PG -> applicable RadosGW Object mapping? Best Regards, Wade ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Increasing time to save RGW objects

Hi there, What is the best way to "look at the rgw admin socket " to see what operations are taking a long time ? Best Regards Wade On Mon, Feb 8, 2016 at 12:16 PM Gregory Farnum wrote: > On Mon, Feb 8, 2016 at 8:49 AM, Kris Jurka wrote: > > > > I've been

Re: [ceph-users] Optimal OSD count for SSDs / NVMe disks

a.v...@gmail.com> wrote: > Am 03.02.2016 um 17:24 schrieb Wade Holler: > > AFAIK when using XFS, parallel write as you described is not enabled. > Not sure I'm getting this. If I have multiple OSDs on the same NVMe > (separated by different data-partitions) I have multiple parallel write

Re: [ceph-users] Optimal OSD count for SSDs / NVMe disks

Cheers Wade On Thu, Feb 4, 2016 at 7:24 AM Sascha Vogt <sascha.v...@gmail.com> wrote: > Hi, > > Am 04.02.2016 um 12:59 schrieb Wade Holler: > > You referenced parallel writes for journal and data. Which is default > > for btrfs but but XFS. Now you are mentioning multiple

Re: [ceph-users] Optimal OSD count for SSDs / NVMe disks

Hi Sascha, What is your file system type, XFS or Btrfs ? Thanks Wade On Wed, Feb 3, 2016 at 7:01 AM Sascha Vogt wrote: > Hi all, > > we recently tried adding a cache tier to our ceph cluster. We had 5 > spinning disks per hosts with a single journal NVMe disk, hosting

Re: [ceph-users] Optimal OSD count for SSDs / NVMe disks

but if your KVM instances are really that short lived, could you get away with size=2 on the cache pool from and availability perspective ? On Wed, Feb 3, 2016 at 7:44 AM Sascha Vogt <sascha.v...@gmail.com> wrote: > Hi Wade, > > Am 03.02.2016 um 13:26 schrieb Wade Holler: > > Wha

Re: [ceph-users] ceph random read performance is better than sequential read?

Could you share the fio command and your read_ahead_kb setting for the OSD devices ? "performance is better" is a little too general. I understand that we usually mean higher IOPS or higher aggregate throughput when we say performance is better. However, application random read performance

Re: [ceph-users] attempt to access beyond end of device on osd prepare

327.3.1 as I recall. Thank for the additional information / confirmation. On Mon, Feb 1, 2016 at 6:05 PM Simon Ironside <sirons...@caffetine.org> wrote: > On 01/02/16 17:47, Wade Holler wrote: > > I can at least say that I've seen this. (a lot) > > > > Running Infern

Re: [ceph-users] attempt to access beyond end of device on osd prepare

I can at least say that I've seen this. (a lot) Running Infernalis with Btrfs on Cent 7.2. I haven't seen any other issue in the cluster that I would say is related. Take it for what you will. Best Regards, Wade On Mon, Feb 1, 2016 at 12:39 PM Simon Ironside wrote:

Re: [ceph-users] SSD OSDs - more Cores or more GHz

Great commentary. While it is fundamentally true that higher clock speed equals lower latency, I'm my practical experience we are more often interested in latency at the concurrency profile of the applications. So in this regard I favor more cores when I have to choose, such that we can support

Re: [ceph-users] CentOS 7.2, Infernalis, preparing osd's and partprobe issues.

> /dev/sdb[2,3,4] and I was able to start all osd just running systemctl > start ceph.target > > Cheers > Goncalo > > > -- > *From:* Wade Holler [wade.hol...@gmail.com] > *Sent:* 08 January 2016 01:15 > *To:* Goncalo Borges; Loic Dacha

Re: [ceph-users] ceph osd tree output

appropriate ? Thank you ahead of time for your help! Best Regards, Wade On Mon, Jan 11, 2016 at 5:43 PM John Spray <jsp...@redhat.com> wrote: > On Mon, Jan 11, 2016 at 10:32 PM, Wade Holler <wade.hol...@gmail.com> > wrote: > > Does anyone else have any suggestions here? I a

Re: [ceph-users] ceph osd tree output

:12 AM Wade Holler <wade.hol...@gmail.com> wrote: > It is not set in the conf file. So why do I still have this behavior ? > > On Fri, Jan 8, 2016 at 11:08 AM hnuzhoulin <hnuzhoul...@gmail.com> wrote: > >> Yeah,this setting can not see in asok config. >> You just

Re: [ceph-users] ceph osd tree output

t in the crushmap, even if the crushmap does not reflect > reality. > > Regards, > > Mart > > > > > > On 01/08/2016 02:16 AM, Wade Holler wrote: > > Sure. Apologies for all the text: We have 12 Nodes for OSDs, 15 OSDs per > node, but I will only include a

Re: [ceph-users] using cache-tier with writeback mode, raods bench result degrade

My experience is performance degrades dramatically when dirty objects are flushed. Best Regards, Wade On Fri, Jan 8, 2016 at 11:08 AM hnuzhoulin wrote: > Hi,guyes > Recentlly,I am testing cache-tier using writeback mode.but I found a > strange things. > the performance

Re: [ceph-users] ceph osd tree output

f these restart is necessary) > > what I use this config is when I changed crushmap manually,and I do not > want the service init script to rebuild crushmap as default way. > > maybe this is not siut for your problem.just have a try. > > 在 Fri, 08 Jan 2016 21:51:32 +0800，Wa

Re: [ceph-users] ceph osd tree output

} host cpn4 { Thank you for your review ! Wade On Thu, Jan 7, 2016 at 6:03 PM Shinobu Kinjo <ski...@redhat.com> wrote: > Can you share the output with us? > > Rgds, > Shinobu > > - Original Message ----- > From: "Wade Holler" <

Re: [ceph-users] CentOS 7.2, Infernalis, preparing osd's and partprobe issues.

I commented out partprobe and everything seems to work just fine. *If someone has experience with why this is very bad please advise. Make sure you know about http://tracker.ceph.com/issues/13833 also. *ps we are running btrfs in the test jig and had to add the "-f" to the btrfs_args for

[ceph-users] ceph osd tree output

Sometimes my ceph osd tree output is wrong. Ie. Wrong osds on the wrong hosts ? Anyone else have this issue? I have seen this at Infernalis and Jewell. Thanks Wade ___ ceph-users mailing list ceph-users@lists.ceph.com

[ceph-users] Combo for Reliable SSD testing

All, I am testing an all SSD and NVMe (journal) config for a customers first endeavor investigating Ceph for performance oriented workloads. Can someone recommend a good performance and reliable ( under high load ) combination? Terrible high level question I know but we have had a number of

Re: [ceph-users] more performance issues :(

Have a look at the iostsat -x 1 1000 output to see what the drives are doing On Wed, Dec 23, 2015 at 4:35 PM Florian Rommel < florian.rom...@datalounges.com> wrote: > Ah, totally forgot the additional details :) > > OS is SUSE Enterprise Linux 12.0 with all patches, > Ceph version 0.94.3 > 4

Re: [ceph-users] requests are blocked

I had major host stability problems under load with -327 . Repeatable test cases under high load with XFS or BTRFS would result in hung kernel tasks and of course the sympathetic behavior you mention. requests are blocked mean that the op tracker in ceph hasn't received a timely response from

Re: [ceph-users] requests are blocked

ge, maybe there is a sysctl option to tweak on OSDs ? this > will be nasty if it goes into our production! > > > > -- > > Dan > > > > *From:* Wade Holler [mailto:wade.hol...@gmail.com] > *Sent:* Tuesday, December 22, 2015 4:36 PM > *To:* Dan Nica <dan.n...@staff.bluemat

Re: [ceph-users] rbd image mount on multiple clients

Hi Dan , When we say "mount" we are usually referring to a file system. Mounting a non shared filesystem on multiple hosts concurrently will certainly break things since each non shared filesystem host thinks it has exclusive access. Of course this is not true if a shared / clustered filesystem

Re: [ceph-users] radosgw bucket index sharding tips?

I'm interested in this too. Should start testing next week at 1B+ objects and I sure would like a recommendation of what config to start with. We learned the hard way that not sharding is very bad at scales like this. On Wed, Dec 16, 2015 at 2:06 PM Florian Haas wrote: > Hi

Re: [ceph-users] recommendations for file sharing

Keep it simple is my approach. #1 If needed Add rudimentary HA with pacemaker. http://linux-ha.org/wiki/Samba Cheers Wade On Tue, Dec 15, 2015 at 5:45 AM Alex Leake wrote: > Good Morning, > > > I have a production Ceph cluster at the University I work at, which runs >