Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread James Page
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi All The Ubuntu Kernel team have spent the last few weeks investigating the apparent performance disparity between RHEL 7 and Ubuntu 14.04; we've focussed efforts in a few ways (see below). All testing has been done using the latest Firefly

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Blair Bethwaite
Hi James, Interesting results, but did you do any tests with a NUMA system? IIUC the original report was from a dual socket setup, and that'd presumably be the standard setup for most folks (both OSD server and client side). Cheers, On 20 February 2015 at 20:07, James Page james.p...@ubuntu.com

Re: K/V interface buffer transaction

2015-02-20 Thread Haomai Wang
Yeah, I remind of that we should has a pending work for ObjectStore refactor bp[1]. We need to change KeyValueDB interface to adopt new improvment [1]: http://pad.ceph.com/p/hammer-osd_transaction_encoding On Fri, Feb 20, 2015 at 7:00 AM, Somnath Roy somnath@sandisk.com wrote: Thanks Sage

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Haomai Wang
Actually, I'm concerned about the correctness of benchmark using MemStore. AFAR it may cause lots of memory frag and cause performance degraded hugely. Maybe set filestore_blackhole=true is more precious? On Fri, Feb 20, 2015 at 5:49 PM, Blair Bethwaite blair.bethwa...@gmail.com wrote: Hi

Re: NewStore update

2015-02-20 Thread Haomai Wang
So cool! A little notes: 1. What about sync thread in NewStore? 2. Could we consider skipping WAL for large overwrite(backfill, RGW)? 3. Sorry, what means [aio_]fsync? On Fri, Feb 20, 2015 at 7:50 AM, Sage Weil sw...@redhat.com wrote: Hi everyone, We talked a bit about the proposed KeyFile

Re: giant integration branch for v0.87.1 ready for QE

2015-02-20 Thread Yuri Weinstein
Team leads, Please review QE validation results summary in http://tracker.ceph.com/issues/10501 Loic - this RC looks ready for release (in my opinion) ! Thx YuriW - Original Message - From: Yuri Weinstein ywein...@redhat.com To: Loic Dachary l...@dachary.org Cc: Ceph Development

giant branch for v0.87.1 is ready for release

2015-02-20 Thread Loic Dachary
Hi, QE approved https://github.com/ceph/ceph/commit/4178e32dd085adeead84fb168ab8a8a121256259 for release. The release notes should now be written based on the content of the issues and pull requests mentioned here http://workbench.dachary.org/ceph/ceph-backports/wikis/giant#already-merged

Re: Clocks out of sync

2015-02-20 Thread Gregory Farnum
Clocks in the labs have seemed a lot less well-synced lately than they had been previously. :( I think there was some issue and then a change to the NTP configuration, but I'm not clear on the details. -Greg On Fri, Feb 20, 2015 at 3:08 PM, David Zafman dzaf...@redhat.com wrote: On 2 of my

Re: giant branch for v0.87.1 is ready for release

2015-02-20 Thread Sage Weil
On Fri, 20 Feb 2015, Loic Dachary wrote: Hi, QE approved https://github.com/ceph/ceph/commit/4178e32dd085adeead84fb168ab8a8a121256259 for release. The release notes should now be written based on the content of the issues and pull requests mentioned here

Clocks out of sync

2015-02-20 Thread David Zafman
On 2 of my rados thrash runs clocks out of sync. Is this an occasional issue or did we have an infrastructure problem? On burnupi19 and burnupi25: 2015-02-20 12:52:52.636017 mon.1 10.214.134.14:6789/0 177 : cluster [WRN] message from mon.0 was stamped 0.501458s in the future, clocks not

Re: Clocks out of sync

2015-02-20 Thread Yuri Weinstein
David, We had ntp server issue tot too long ago, could be the same or new http://tracker.ceph.com/issues/10675 Thx YuriW - Original Message - From: David Zafman dzaf...@redhat.com To: ceph-devel@vger.kernel.org Sent: Friday, February 20, 2015 3:08:29 PM Subject: Clocks out of sync On

Disk failing plana74

2015-02-20 Thread David Zafman
A recent test run had an EIO on the following disk: plana74 /dev/sdb The machine is locked right now. David Zafman Senior Developer -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: disable pg_num warning?

2015-02-20 Thread Sage Weil
On Fri, 20 Feb 2015, Wyllys Ingersoll wrote: Is there a way to disable the warning (shown when doing ceph -s, for example) about the number of pgs (too few pgs per osd) ? All of the PG-related warnings are tuanble: OPTION(mon_pg_warn_min_per_osd, OPT_INT, 30) // min # pgs per (in) osd before

disable pg_num warning?

2015-02-20 Thread Wyllys Ingersoll
Is there a way to disable the warning (shown when doing ceph -s, for example) about the number of pgs (too few pgs per osd) ? Would it be considered risky to disable it? Consider when a cluster is initially created with just a single pool, and you don't anticipate that this pool will ever be

Re: NewStore update

2015-02-20 Thread Sage Weil
On Fri, 20 Feb 2015, Haomai Wang wrote: So cool! A little notes: 1. What about sync thread in NewStore? My thought right now is that there will be a WAL thread and (maybe) a transaction commit completion thread. What do you mean by sync thread? One thing I want to avoid is the current

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Alexandre DERUMIER
http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/ It's possible that this could be having an effect on the results. Isn't auto numa balancing enabled by default since kernel 3.8 ? it can be checked with cat /proc/sys/kernel/numa_balancing - Mail

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread James Page
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Mark On 20/02/15 15:51, Mark Nelson wrote: I've CC'ed both Colin and Jay on this mail - so if anyone has any specific questions about the testing they can chime in with responses. Good testing! Other than the NUMA questions in the other

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Mark Nelson
I think paying attention to NUMA is good advice. One of the things that apparently changed in RHEL7 is that they are now doing automatic NUMA tuning: http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/ It's possible that this could be having an effect on the

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Mark Nelson
On 02/20/2015 03:07 AM, James Page wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi All The Ubuntu Kernel team have spent the last few weeks investigating the apparent performance disparity between RHEL 7 and Ubuntu 14.04; we've focussed efforts in a few ways (see below). All

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Mark Nelson
On 02/20/2015 10:03 AM, Alexandre DERUMIER wrote: http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/ It's possible that this could be having an effect on the results. Isn't auto numa balancing enabled by default since kernel 3.8 ? No idea, I'm behind the

Re: NewStore update

2015-02-20 Thread Haomai Wang
OK, I just viewed part of codes and realized it. It looks like we want to sync metadata each time when WAL and we ahead do_transaction jobs before WAL things. It may cause larger latency than before? Because the latency of do_transactions couldn't be simply ignore under some latency sensitive

Re: disable pg_num warning?

2015-02-20 Thread Wyllys Ingersoll
Great, thanks! On Fri, Feb 20, 2015 at 11:15 AM, Sage Weil s...@newdream.net wrote: On Fri, 20 Feb 2015, Wyllys Ingersoll wrote: Is there a way to disable the warning (shown when doing ceph -s, for example) about the number of pgs (too few pgs per osd) ? All of the PG-related warnings are

Re: NewStore update

2015-02-20 Thread Mark Nelson
On 02/20/2015 09:00 AM, Sage Weil wrote: On Fri, 20 Feb 2015, Haomai Wang wrote: So cool! A little notes: 1. What about sync thread in NewStore? My thought right now is that there will be a WAL thread and (maybe) a transaction commit completion thread. What do you mean by sync thread?

Re: Disk failing plana74

2015-02-20 Thread Yuri Weinstein
I ran smart and it came back good, hmm ubuntu@plana74:~$ /usr/libexec/smart.pl All 4 drives happy as clams Thx YuriW - Original Message - From: David Zafman dzaf...@redhat.com To: Sandon Van Ness svann...@redhat.com Cc: ceph-devel@vger.kernel.org Sent: Friday, February 20, 2015 1:10:48

Re: Memstore performance improvements v0.90 vs v0.87

2015-02-20 Thread Stefan Priebe
Am 20.02.2015 um 17:03 schrieb Alexandre DERUMIER: http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/ It's possible that this could be having an effect on the results. Isn't auto numa balancing enabled by default since kernel 3.8 ? it can be checked with cat