Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Brad Diggs
Reducing the record size would negatively impact performance. For rational why, see thesection titled "Match Average I/O Block Sizes" in my blog post on filesystem caching:http://www.thezonemanager.com/2009/03/filesystem-cache-optimization.htmlBrad Brad Diggs | Principal Sales Consultant

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Brad Diggs
Jim,You are spot on. I was hoping that the writes would be close enough to identical thatthere would be a high ratio of duplicate data since I use the same record size, page size,compression algorithm, … etc. However, that was not the case. The main thing that Iwanted to prove though was that if

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Robert Milkowski
Citing yourself: The average block size for a given data block should be used as the metric to map all other datablock sizes to. For example, the ZFS recordsize is 128kb by default. If the average block (or page) size of a directory server is 2k, then the mismatch in size will result in

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Jim Klimov
Thanks for running and publishing the tests :) A comment on your testing technique follows, though. 2011-12-29 1:14, Brad Diggs wrote: As promised, here are the findings from my testing. I created 6 directory server instances ... However, once I started modifying the data of the replicated

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Brad Diggs
S11 FCSBrad Brad Diggs | Principal Sales Consultant |972.814.3698eMail:brad.di...@oracle.comTech Blog:http://TheZoneManager.comLinkedIn:http://www.linkedin.com/in/braddiggs On Dec 29, 2011, at 8:11 AM, Robert Milkowski wrote:And these results are from S11 FCS I assume.On older builds or Illumos

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Nico Williams
On Thu, Dec 29, 2011 at 9:53 AM, Brad Diggs brad.di...@oracle.com wrote: Jim, You are spot on.  I was hoping that the writes would be close enough to identical that there would be a high ratio of duplicate data since I use the same record size, page size, compression algorithm, … etc.  

Re: [zfs-discuss] S11 vs illumos zfs compatiblity

2011-12-29 Thread sol
Richard Elling wrote:  many of the former Sun ZFS team  regularly contribute to ZFS through the illumos developer community.   Does this mean that if they provide a bug fix via illumos then the fix won't make it into the Oracle code? ___

Re: [zfs-discuss] S11 vs illumos zfs compatiblity

2011-12-29 Thread Nico Williams
On Thu, Dec 29, 2011 at 2:06 PM, sol a...@yahoo.com wrote: Richard Elling wrote:  many of the former Sun ZFS team regularly contribute to ZFS through the illumos developer community. Does this mean that if they provide a bug fix via illumos then the fix won't make it into the Oracle code? If

Re: [zfs-discuss] S11 vs illumos zfs compatiblity

2011-12-29 Thread Richard Elling
On Dec 29, 2011, at 1:29 PM, Nico Williams wrote: On Thu, Dec 29, 2011 at 2:06 PM, sol a...@yahoo.com wrote: Richard Elling wrote: many of the former Sun ZFS team regularly contribute to ZFS through the illumos developer community. Does this mean that if they provide a bug fix via illumos

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Matthew Ahrens
On Mon, Dec 12, 2011 at 11:04 PM, Erik Trimble tr...@netdemons.com wrote: On 12/12/2011 12:23 PM, Richard Elling wrote: On Dec 11, 2011, at 2:59 PM, Mertol Ozyoney wrote: Not exactly. What is dedup'ed is the stream only, which is infect not very efficient. Real dedup aware replication is

Re: [zfs-discuss] Improving L1ARC cache efficiency with dedup

2011-12-29 Thread Nico Williams
On Thu, Dec 29, 2011 at 6:44 PM, Matthew Ahrens mahr...@delphix.com wrote: On Mon, Dec 12, 2011 at 11:04 PM, Erik Trimble tr...@netdemons.com wrote: (1) when constructing the stream, every time a block is read from a fileset (or volume), its checksum is sent to the receiving machine. The

[zfs-discuss] Resolving performance issue w/ deduplication (NexentaStor)

2011-12-29 Thread Ray Van Dolson
Hi all; We have a dev box running NexentaStor Community Edition 3.1.1 w/ 24GB (we don't run dedupe on production boxes -- and we do pay for Nexenta licenses on prd as well) RAM and an 8.5TB pool with deduplication enabled (1.9TB or so in use). Dedupe ratio is only 1.26x. The box has an

Re: [zfs-discuss] Resolving performance issue w/ deduplication (NexentaStor)

2011-12-29 Thread Fajar A. Nugraha
On Fri, Dec 30, 2011 at 1:31 PM, Ray Van Dolson rvandol...@esri.com wrote: Is there a non-disruptive way to undeduplicate everything and expunge the DDT? AFAIK, no  zfs send/recv and then back perhaps (we have the extra space)? That should work, but it's disruptive :D Others might provide

Re: [zfs-discuss] Resolving performance issue w/ deduplication (NexentaStor)

2011-12-29 Thread Ray Van Dolson
On Thu, Dec 29, 2011 at 10:59:04PM -0800, Fajar A. Nugraha wrote: On Fri, Dec 30, 2011 at 1:31 PM, Ray Van Dolson rvandol...@esri.com wrote: Is there a non-disruptive way to undeduplicate everything and expunge the DDT? AFAIK, no  zfs send/recv and then back perhaps (we have the extra