By the way, I'm getting closer to getting wip-rocksdb in a state where it
can be merged, but it is failing to build due to this line:
$(shell (./build_tools/build_detect_version))
in Makefile.am which results in
automake: warnings are treated as errors
warning: Makefile.am:59: shell (./build_tools/build_detect_version:
non-POSIX variable name
Makefile.am:59: (probably a GNU make extension)
Makefile.am: installing './depcomp'
autoreconf: automake failed with exit status: 1
Any suggestions? You can see these build results at
http://ceph.com/gitbuilder.cgi
http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-deb-trusty-amd64-basic/log.cgi?log=92212c722100065922468e4185759be0435877ff
sage
On Thu, 31 Jul 2014, Shu, Xinxin wrote:
> Does your report base on wip-rocksdb-mark branch?
>
> Cheers,
> xinxin
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Mark Nelson
> Sent: Tuesday, July 29, 2014 12:56 AM
> To: Shu, Xinxin; [email protected]
> Subject: Re: First attempt at rocksdb monitor store stress testing
>
> Hi Xinxin,
>
> Thanks, I'll give it a try. I want to figure out what's going on in Rocksdb
> when the test stalls with leveled compaction. In the mean time, here are the
> test results with spinning disks and SSDs:
>
> http://nhm.ceph.com/mon-store-stress/Monitor_Store_Stress_Short_Tests.pdf
>
> Mark
>
> On 07/27/2014 11:45 PM, Shu, Xinxin wrote:
> > Hi mark,
> >
> > I tested this option on my setup , same issue happened , I will dig into it
> > , if you want to get info log , there is a workaround, set this option to
> > none:
> >
> > Rocksdb_log = ""
> >
> > Cheers,
> > xinxin
> >
> > -----Original Message-----
> > From: Mark Nelson [mailto:[email protected]]
> > Sent: Saturday, July 26, 2014 12:10 AM
> > To: Shu, Xinxin; [email protected]
> > Subject: Re: First attempt at rocksdb monitor store stress testing
> >
> > Hi Xinxin,
> >
> > I'm trying to enable the rocksdb log file as described in config_opts using:
> >
> > rocksdb_log = <path to log file>
> >
> > The file gets created but is empty. Any ideas?
> >
> > Mark
> >
> > On 07/24/2014 08:28 PM, Shu, Xinxin wrote:
> >> Hi mark,
> >>
> >> I am looking forward to your results on SSDs .
> >> rocksdb generates a crc of data to be written , this cannot be switch off
> >> (but can be subsititued with xxhash), there are two options , Option.
> >> verify_checksums_in_compaction and ReadOptions. verify_checksums, If we
> >> disable these two options , i think cpu usage will goes down . If we use
> >> universal compaction , this is not friendly with read operation.
> >>
> >> Btw , can you list your rocksdb configuration?
> >>
> >> Cheers,
> >> xinxin
> >>
> >> -----Original Message-----
> >> From: Mark Nelson [mailto:[email protected]]
> >> Sent: Friday, July 25, 2014 7:45 AM
> >> To: Shu, Xinxin; [email protected]
> >> Subject: Re: First attempt at rocksdb monitor store stress testing
> >>
> >> Earlier today I modified the rocksdb options so I could enable universal
> >> compaction. Over all performance is lower but I don't see the hang/stall
> >> in the middle of the test either. Instead the disk is basically pegged
> >> with 100% writes. I suspect average latency is higher than leveldb, but
> >> the highest latency is about 5-6s while we were seeing 30s spikes for
> >> leveldb with levelled (heh) compaction.
> >>
> >> I haven't done much tuning either way yet. It may be that if we keep
> >> level 0 and level 1 roughly the same size we can reduce stalls in the
> >> levelled setups. It will also be interesting to see what happens in these
> >> tests on SSDs.
> >>
> >> Mark
> >>
> >> On 07/24/2014 06:13 AM, Mark Nelson wrote:
> >>> Hi Xinxin,
> >>>
> >>> Thanks! I wonder as well if it might be interesting to expose the
> >>> options related to universal compaction? It looks like rocksdb
> >>> provides a lot of interesting knobs you can adjust!
> >>>
> >>> Mark
> >>>
> >>> On 07/24/2014 12:08 AM, Shu, Xinxin wrote:
> >>>> Hi mark,
> >>>>
> >>>> I think this maybe related to 'verify_checksums' config option
> >>>> ,when ReadOptions is initialized, default this option is true ,
> >>>> all data read from underlying storage will be verified against
> >>>> corresponding checksums, however, this option cannot be
> >>>> configured in wip-rocksdb branch. I will modify code to make this option
> >>>> configurable .
> >>>>
> >>>> Cheers,
> >>>> xinxin
> >>>>
> >>>> -----Original Message-----
> >>>> From: [email protected]
> >>>> [mailto:[email protected]] On Behalf Of Mark Nelson
> >>>> Sent: Thursday, July 24, 2014 7:14 AM
> >>>> To: [email protected]
> >>>> Subject: First attempt at rocksdb monitor store stress testing
> >>>>
> >>>> Hi Guys,
> >>>>
> >>>> So I've been interested lately in leveldb 99th percentile latency
> >>>> (and the amount of write amplification we are seeing) with leveldb.
> >>>> Joao mentioned he has written a tool called mon-store-stress in
> >>>> wip-leveldb-misc to try to provide a means to roughly guess at
> >>>> what's happening on the mons under heavy load. I cherry-picked it
> >>>> over to wip-rocksdb and after a couple of hacks was able to get
> >>>> everything built and running with some basic tests. There was
> >>>> little tuning done and I don't know how realistic this workload is,
> >>>> so don't assume this means anything yet, but some initial results are
> >>>> here:
> >>>>
> >>>> http://nhm.ceph.com/mon-store-stress/First%20Attempt.pdf
> >>>>
> >>>> Command that was used to run the tests:
> >>>>
> >>>> ./ceph-test-mon-store-stress --mon-keyvaluedb <leveldb|rocksdb>
> >>>> --write-min-size 50K --write-max-size 2M --percent-write 70
> >>>> --percent-read 30 --keep-state --test-seed 1406137270 --stop-at
> >>>> 5000 foo
> >>>>
> >>>> The most interesting bit right now is that rocksdb seems to be
> >>>> hanging in the middle of the test (left it running for several
> >>>> hours). CPU usage on one core was quite high during the hang.
> >>>> Profiling using perf with dwarf symbols I see:
> >>>>
> >>>> - 49.14% ceph-test-mon-s ceph-test-mon-store-stress [.]
> >>>> unsigned int
> >>>> rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned
> >>>> int, char const*, unsigned long)
> >>>> - unsigned int
> >>>> rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned
> >>>> int, char const*, unsigned long)
> >>>> 51.70%
> >>>> rocksdb::ReadBlockContents(rocksdb::RandomAccessFile*,
> >>>> rocksdb::Footer const&, rocksdb::ReadOptions const&,
> >>>> rocksdb::BlockHandle const&, rocksdb::BlockContents*,
> >>>> rocksdb::Env*,
> >>>> bool)
> >>>> 48.30%
> >>>> rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice
> >>>> const&, rocksdb::CompressionType, rocksdb::BlockHandle*)
> >>>>
> >>>> Not sure what's going on yet, may need to try to enable
> >>>> logging/debugging in rocksdb. Thoughts/Suggestions welcome. :)
> >>>>
> >>>> Mark
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >>>> in the body of a message to [email protected] More
> >>>> majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>
> >>
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to [email protected] More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
