...@inktank.com]
Sent: 11 December 2013 14:00
To: Loic Dachary
Cc: Andreas Joachim Peters; ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
On 12/11/2013 06:28 AM, Loic Dachary wrote:
On 11/12/2013 10:49, Andreas Joachim Peters wrote: Hi Loic,
I am a little bit
%.
Cheers
Cheers Andreas.
From: Mark Nelson [mark.nel...@inktank.com]
Sent: 11 December 2013 14:00
To: Loic Dachary
Cc: Andreas Joachim Peters; ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
On 12/11/2013 06:28 AM
Hi Loic,
I am a little bit confused which kind of tool you actually want. You want a
simple benchmark to check for degradation or you want a full profiler tool?
Most of the external tools have the problem that you measure the whole thing
including buffer allocation and initialization. We
On 11/12/2013 10:49, Andreas Joachim Peters wrote: Hi Loic,
I am a little bit confused which kind of tool you actually want. You want a
simple benchmark to check for degradation or you want a full profiler tool?
I was not sure, hence the confusion.
Most of the external tools have the
On 12/11/2013 06:28 AM, Loic Dachary wrote:
On 11/12/2013 10:49, Andreas Joachim Peters wrote: Hi Loic,
I am a little bit confused which kind of tool you actually want. You want a
simple benchmark to check for degradation or you want a full profiler tool?
I was not sure, hence the
Maybe using
http://google-perftools.googlecode.com/svn/trunk/doc/cpuprofile.html
is enough. fsbench looks overkill indeed.
/me exploring options ;-)
On 09/12/2013 17:45, Loic Dachary wrote:
Hi,
Mark Nelson suggested we use perf ( linux-tools ) for benchmarking. It looks
like something
Hi,
Mark Nelson suggested we use perf ( linux-tools ) for benchmarking. It looks
like something that would help indeed : the benchmark program would only
concern itself with doing some work according to the options and let
performances be collected from the outside, using tools that are
I will mention that this is a good tool if you want really detailed
profiling or cpu counter data about what's going on. Other tools that
are more generic (ie ones that just read data from proc, ie collectl,
sar, etc) may also be options.
Mark
On 12/09/2013 10:45 AM, Loic Dachary wrote:
ssse3 22 112
rec6 ssse3 16 86
From: Loic Dachary [l...@dachary.org]
Sent: 12 November 2013 19:06
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas,
On 12/11
Hi Andreas,
On 12/11/2013 02:11, Andreas Joachim Peters wrote:
Hi Loic,
I am finally doing the benchmark tool and I found a bunch of wrong parameter
checks which can make the whole thing SEGV.
All the RAID-6 codes have restrictions on the parameters but they are not
correctly enforced
Hi Loic,
I am finally doing the benchmark tool and I found a bunch of wrong parameter
checks which can make the whole thing SEGV.
All the RAID-6 codes have restrictions on the parameters but they are not
correctly enforced for Liberation Blaum-Roth codes in the CEPH wrapper class
... see
.
From: Loic Dachary [l...@dachary.org]
Sent: 27 September 2013 11:40
To: Andreas Joachim Peters
Cc: Ceph Development
Subject: Re: CEPH Erasure Encoding + OSD Scalability
On 26/09/2013 23:49, Andreas Joachim Peters wrote: Sure,
this text is clear
]
latency=27.737 ms
Cheers Andreas.
From: Loic Dachary [l...@dachary.org]
Sent: 27 September 2013 11:40
To: Andreas Joachim Peters
Cc: Ceph Development
Subject: Re: CEPH Erasure Encoding + OSD Scalability
On 26/09/2013 23:49, Andreas Joachim Peters wrote: Sure
will change an
may require a different strategy.
Cheers
Thanks Andreas.
From: Loic Dachary [l...@dachary.org]
Sent: 26 September 2013 21:18
To: Andreas Joachim Peters
Cc: Ceph Development
Subject: Re: CEPH Erasure Encoding + OSD Scalability
[re-adding ceph-devel to the cc]
On 26/09/2013 20:36, Andreas-Joachim Peters wrote: Hi Loic,
today I forked he CEPH repository and will commit my changes to my GitHub
fork asap ... (I am not familiar with GitHub in particular).
I was finalizing the minimim_to_decode function today with test
Hi Andreas,
It looks like this code would be useful as a standalone program instead of
being integrated within unit tests. There are a few support program of that
kind. The unit tests are probably not the best place for benchmarks. What do
you think ?
Cheers
Note: got the updated file ;-)
Yes, sure. I actually thought the same in the meanwhile ... I have some
questions:
Q: Can/should it stay in the framework of google test's or you would prefer
just a plain executable ?
I have added local parity support to your erasure class adding a new argument:
erasure-code-lp and
two new
On 25/09/2013 20:33, Andreas Joachim Peters wrote: Yes, sure. I actually
thought the same in the meanwhile ... I have some questions:
Q: Can/should it stay in the framework of google test's or you would prefer
just a plain executable ?
A plain executable would make sense. An simple
On Wed, 25 Sep 2013, Loic Dachary wrote:
On 25/09/2013 20:33, Andreas Joachim Peters wrote: Yes, sure. I actually
thought the same in the meanwhile ... I have some questions:
Q: Can/should it stay in the framework of google test's or you would prefer
just a plain executable ?
Hi Andreas,
Very inefficient implementation indeed :-) I've integrated your change in
https://github.com/ceph/ceph/pull/619
Cheers
On 23/09/2013 01:00, Andreas Joachim Peters wrote:
alignment = k*w*packetsize*sizeof(int);
in_length += (alignment - (in_length%alignment);
--
Loïc
()%alignment;
+ unsigned in_length = in.length() + tail?(alignment - tail):0;
Cheers Andreas.
__
From: Loic Dachary [l...@dachary.org]
Sent: 23 September 2013 09:27
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding
Hi Loic,
I have modified the Jerasure unit test, to record the encoding reconstruction
performance and to store this value in the optional google-test xml outputfile.
I have put (4,2) with a 4MB random object as default and one can pass a
different object size via '--object-size=1000' (for
encoding all blocks, then CRC32C on all blocks.
Cheers Andreas.
From: Loic Dachary [l...@dachary.org]
Sent: 21 September 2013 17:11
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Loic,
I was applying the changes and the
situation improves, however there is still one important thing which
actually dominated all the measurements which were needing larger packet
sizes (everything besides Raid6):
pad_in_length(unsigned in_length)
The
Hi Andreas,
It's probably too soon to be smart about reducing the number of copies, but
you're right : this copy is not necessary. The following pull request gets rid
of it:
https://github.com/ceph/ceph/pull/615
Cheers
On 20/09/2013 18:49, Loic Dachary wrote:
Hi,
This is a first attempt
Hi Loic,
I have now some benchmarks on a Xeon 2.27 GHz 4-core with gcc 4.4 (-O2) for
ENCODING based on the CEPH Jerasure port.
I measured for objects from 128k to 512 MB with random contents (if you encode
1 GB objects you see slow downs due to caching inefficiencies ...), otherwise
results
Hi Andreas,
Great work on these benchmarks ! It's definitely an incentive to improve as
much as possible. Could you push / send the scripts and sequence of operations
you've used ? I'll reproduce this locally while getting rid of the extra copy.
It would be useful to capture that into a script
Very exciting work guys!
I suspect there will definitely be people who want lower CPU
consumption, especially as ARM and Atom processors become more prolific. :)
Mark
On 09/20/2013 07:33 AM, Loic Dachary wrote:
Hi Andreas,
Great work on these benchmarks ! It's definitely an incentive to
On Fri, 20 Sep 2013, Loic Dachary wrote:
Hi Andreas,
Great work on these benchmarks ! It's definitely an incentive to improve as
much as possible. Could you push / send the scripts and sequence of
operations you've used ? I'll reproduce this locally while getting rid of the
extra copy.
Hi,
This is a first attempt at avoiding unnecessary copy:
https://github.com/dachary/ceph/blob/03445a5926cd073c11cd8693fb110729e40f35fa/src/osd/ErasureCodePluginJerasure/ErasureCodeJerasure.cc#L66
I'm not sure how it could be made more readable / terse with bufferlist
iterators. Any kind of
.
From: Loic Dachary [l...@dachary.org]
Sent: 25 August 2013 13:49
To: Andreas Joachim Peters
Cc: Ceph Development
Subject: Re: CEPH Erasure Encoding + OSD Scalability
On 24/08/2013 21:41, Loic Dachary wrote:
On 24/08/2013 15:30, Andreas-Joachim Peters wrote:
Hi Loic,
I will start to review
Cc: Ceph Development
Subject: Re: CEPH Erasure Encoding + OSD Scalability
On 24/08/2013 21:41, Loic Dachary wrote:
On 24/08/2013 15:30, Andreas-Joachim Peters wrote:
Hi Loic,
I will start to review
Cool :-)
...maybe you can briefly explain few things beforehand:
1) the buffer
Erasure Encoding + OSD Scalability
Hi Andreas,
Since it looks like we're going to use jerasure-1.2, we will be
able to try (C)RS using
https://github.com/tsuraan/Jerasure/blob/master/src/cauchy.c
https://github.com/tsuraan
: 06 July 2013 22:47
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
mailto:ceph-devel@vger.kernel.org mailto:ceph-devel@vger.kernel.org
mailto:ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi
in a virtual machine
you need this emulation ...
Cheers Andreas.
From: Loic Dachary [l...@dachary.org]
Sent: 06 July 2013 22:47
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas
:37
To: Andreas Joachim Peters
Cc: Loic Dachary; ceph-devel@vger.kernel.org
Subject: RE: CEPH Erasure Encoding + OSD Scalability
On Sun, 7 Jul 2013, Andreas Joachim Peters wrote:
Considering the crc32c-intel code you added ... I would provide a
function which provides a crc32c checksum
-abstract-api
Cheers
Cheers Andreas.
From: Sage Weil [s...@inktank.com]
Sent: 08 July 2013 05:37
To: Andreas Joachim Peters
Cc: Loic Dachary; ceph-devel@vger.kernel.org
Subject: RE: CEPH Erasure Encoding + OSD Scalability
On Sun, 7 Jul 2013, Andreas
.
From: Loic Dachary [l...@dachary.org]
Sent: 05 July 2013 23:23
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas,
On 04/07/2013 23:01, Andreas Joachim Peters wrote: Hi Loic,
thanks for the responses!
Maybe
; ceph-devel@vger.kernel.org
Subject: RE: CEPH Erasure Encoding + OSD Scalability
On Sun, 7 Jul 2013, Andreas Joachim Peters wrote:
Considering the crc32c-intel code you added ... I would provide a
function which provides a crc32c checksum and detects if it can do it
using SSE4.2 or implements
this emulation ...
Cheers Andreas.
From: Loic Dachary [l...@dachary.org]
Sent: 06 July 2013 22:47
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas,
Since it looks like we're going to use
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas,
Since it looks like we're going to use jerasure-1.2, we will be able to try
(C)RS using
https://github.com/tsuraan/Jerasure/blob/master/src/cauchy.c
https://github.com/tsuraan/Jerasure/blob/master/src/cauchy.h
Do you know
checksum run's at ~2GByte/s.
Cheers Andreas.
From: Loic Dachary [l...@dachary.org]
Sent: 05 July 2013 23:23
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas,
On 04/07/2013 23:01
.
From: Loic Dachary [l...@dachary.org]
Sent: 05 July 2013 23:23
To: Andreas Joachim Peters
Cc: ceph-devel@vger.kernel.org
Subject: Re: CEPH Erasure Encoding + OSD Scalability
Hi Andreas,
On 04/07/2013 23:01, Andreas Joachim Peters wrote: Hi Loic,
thanks
Hi Andreas,
Since it looks like we're going to use jerasure-1.2, we will be able to try
(C)RS using
https://github.com/tsuraan/Jerasure/blob/master/src/cauchy.c
https://github.com/tsuraan/Jerasure/blob/master/src/cauchy.h
Do you know of a better / faster implementation ? Is there a tradeoff
44 matches
Mail list logo
