[zfs-discuss] zpool on raw disk. Do I need to format?

2010-07-01 Thread Peter Taps
Folks,

I am learning more about zfs storage. It appears, zfs pool can be created on a 
raw disk. There is no need to create any partitions, etc. on the disk. Does 
this mean there is no need to run format on a raw disk?

I have added a new disk to my system. It shows up as /dev/rdsk/c8t1d0s0. Do I 
need to format it before I convert it to zfs storage? Or, can I simply use it 
as:

# zfs create MyData /dev/rdsk/c8t1d0s0

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] zpool on raw disk. Do I need to format?

2010-07-01 Thread Peter Taps
Awesome. Thank you, CIndy.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] zfs - filesystem versus directory

2010-07-02 Thread Peter Taps
Thank you all, especially Edward, for the enlightenment.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Legality and the future of zfs...

2010-07-07 Thread Peter Taps
Folks,

As you may have heard, NetApp has a lawsuit against Sun in 2007 (and now 
carried over to Oracle) for patent infringement with the zfs file system. Now, 
NetApp is taking a stronger stance and threatening zfs storage suppliers to 
stop selling zfs-based storage.

http://www.theregister.co.uk/2010/07/06/netapp_coraid/?utm_source=feedburnerutm_medium=twitterutm_campaign=Feed%3A+shovelarts+%28Shovel+Arts%29

Given this, I am wondering what you think is the future of zfs as an open 
source project.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Legality and the future of zfs...

2010-07-09 Thread Peter Taps
Folks,

I would appreciate it if you can create a separate thread for Mac Mini.

Back to the original subject.

NetApp has deep pockets. A few companies have already backed out of zfs as they 
cannot afford to go through a lawsuit. I am in a stealth startup company and we 
rely on zfs for our application. The future of our company, and many other 
businesses, depends on what happens to zfs. If you are in a similar boat, what 
actions are you planning?

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Is there any support for bi-directional synchronization in zfs?

2010-07-14 Thread Peter Taps
Folks,

This is probably a very naive question.

Is it possible to set zfs for bi-directional synchronization of data across two 
locations? I am thinking this is almost impossible. Consider two files A and B 
at two different sites. There are three possible cases that require 
synchronization:

   1. A is changed. B is unchanged.
   2. B is changed. A is unchanged.
   3. A is changed. B is changed.

While it is possible to achieve synchronization for the first two cases, case 3 
requires special merging and is almost impossible.

I am thinking it is the same problem even at the block level.

Even to achieve 1 and 2 is a bit tricky given the latency between the two 
sites. Is there anything in zfs that makes it easier? 

Thank you in advance for sharing your thoughts.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] ZFS + WebDAV + AD integration

2010-07-14 Thread Peter Taps
Folks,

I now know that ZFS is capable of preserving AD account sids. I have verified 
the scenario with CIFS integration.

I am now wondering if it is possible to achieve a similar AD integration over 
WebDAV. Is it possible to retain security permissions on files and folders over 
WebDAV?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Encryption?

2010-07-14 Thread Peter Taps
 Btw, if you want a commercially supported and maintained product, have
 you looked at NexentaStor? Regardless of what happens with OpenSolaris,
 we aren't going anywhere. (Full disclosure: I'm a Nexenta Systems
 employee. :-)
 
 -- Garrett

Hi Garrett,

I would like to know why you think Nexenta would continue to stay if 
OpenSolaris goes away.

I feel the fate of Nexenta is no different than the fate of my startup company. 
Both of us are heavily dependent on zfs. And we know OpenSolaris version of zfs 
is the most stable version. 

Any business that is dependent on zfs must plan for two things as a contingency:

1. Look for an alternative for zfs
2. Look for an alternative for OpenSolaris

Preferably both need to be open source with no licenses attached.

Ideally, zfs lawsuit will be put to rest and Oracle will commit for continuing 
to support OpenSolaris.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



[zfs-discuss] How to identify user-created zfs filesystems?

2010-08-04 Thread Peter Taps
Folks,

In my application, I need to present user-created filesystems. For my test, I 
created a zfs pool called mypool and two file systems called cifs1 and cifs2. 
However, when I run zfs list, I see a lot more entries:

# zfs list
NAME USED  AVAIL  REFER  MOUNTPOINT
mypool  1.31M  1.95G33K  /volumes/mypool
mypool/cifs11.12M  1.95G  1.12M  /volumes/mypool/cifs1
mypool/cifs2  44K  1.95G44K  /volumes/mypool/cifs2
syspool 3.58G  4.23G  35.5K  legacy
syspool/dump 716M  4.23G   716M  -
syspool/rootfs-nmu-000  1.85G  4.23G  1.36G  legacy
syspool/rootfs-nmu-001  53.5K  4.23G  1.15G  legacy
syspool/swap1.03G  5.19G  71.4M  -

I just need to present cifs1 and cifs2 to the user. Is there a property on the 
filesystem that I can use to determine user-created filesystems?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] How to identify user-created zfs filesystems?

2010-08-05 Thread Peter Taps
Thank you all for your help. It turns out that I just need to ignore the ones 
that have their mount points either not defined or are marked as legacy.

It is good to learn about history command. Could come in handy.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Raidz - what is stored in parity?

2010-08-10 Thread Peter Taps
Hi,

I am going through understanding the fundamentals of raidz. From the man pages, 
a raidz configuration of P disks and N parity provides (P-N)*X storage space 
where X is the size of the disk. For example, if I have 3 disks of 10G each and 
I configure it with raidz1, I will have 20G of usable storage. In addition, I 
continue to work even if 1 disk fails.

First, I don't understand why parity takes so much space. From what I know 
about parity, there is typically one parity bit per byte. Therefore, the parity 
should be taking 1/8 of storage, not 1/3 of storage. What am I missing?

Second, if one disk fails, how is my lost data reconstructed? There is no 
duplicate data as this is not a mirrored configuration. Somehow, there should 
be enough information in the parity disk to reconstruct the lost data. How is 
this possible?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Raidz - what is stored in parity?

2010-08-10 Thread Peter Taps
Hi Eric,

Thank you for your help. At least one part is clear now.

I still am confused about how the system is still functional after one disk 
fails.

Consider my earlier example of 3 disks zpool configured for raidz-1. To keep it 
simple let's not consider block sizes.

Let's say I send a write value abcdef to the zpool.

As the data gets striped, we will have 2 characters per disk.

disk1 = ab + some parity info
disk2 = cd + some parity info
disk3 = ef + some parity info

Now, if disk2 fails, I lost cd. How will I ever recover this? The parity info 
may tell me that something is bad but I don't see how my data will get 
recovered.

The only good thing is that any newer data will now be striped over two disks.

Perhaps I am missing some fundamental concept about raidz.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Raidz - what is stored in parity?

2010-08-11 Thread Peter Taps
Thank you all for your help. It appears my understanding of parity was rather 
limited. I kept on thinking about parity in memory where the extra bit would be 
used to ensure that the total of all 9 bits is always even. 

In case of zfs, the above type of checking is actually moved into checksum. 
What zfs calls parity is much more than a simple check. No wonder it takes more 
space.

One question though. Marty mentioned that raidz parity is limited to 3. But in 
my experiment, it seems I can get parity to any level.

You create a raidz zpool as:

# zpool create mypool raidzx disk1 diskk2 

Here, x in raidzx is a numeric value indicating the desired parity.

In my experiment, the following command seems to work:

# zpool create mypool raidz10 disk1 disk2 ...

In my case, it gives an error that I need at least 11 disks (which I don't) but 
the point is that raidz parity does not seem to be limited to 3. Is this not 
true?

Thank you once again for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Raidz - what is stored in parity?

2010-08-11 Thread Peter Taps
I am running ZFS file system version 5 on Nexenta.

Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Raidz - what is stored in parity?

2010-08-11 Thread Peter Taps
Thank you, Eric. Your explanation is clear to understand.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] How to obtain vdev information for a zpool?

2010-08-11 Thread Peter Taps
Folks,

When I create a zpool, I get to specify the vdev type - mirror, raidz1, raidz2, 
etc. How do I get back this information for an existing pool? The status 
command does not reveal this information:

# zpool status mypool

When this command is run, I can see the disks in use. However, I don't see my 
configuration information (raidz, mirror, etc.). 

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] How to obtain vdev information for a zpool?

2010-08-12 Thread Peter Taps
Hi James,

Appreciate your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Dedup - Does on imply sha256?

2010-08-24 Thread Peter Taps
Folks,

One of the articles on the net says that the following two commands are exactly 
the same:

# zfs set dedup=on tank
# zfs set dedup=sha256 tank

Essentially, on is just a pseudonym for sha256 and verify is just a 
pseudonym for sha256,verify.

Can someone please confirm if this is true?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Dedup - Does on imply sha256?

2010-08-25 Thread Peter Taps
Thank you all for your help.

It appears it is better to use on instead of sha256. This way, you are 
letting zfs decide the best option.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Possible to save custom properties on a zfs file system?

2010-09-20 Thread Peter Taps
Thank you all for your help.

Can properties be set on file systems as well as pools? When I try zpool set 
command with a local property, I can an error invalid property.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] What is l2cache setting?

2010-09-22 Thread Peter Taps
Folks,

While going through zpool source code, I see a configuration option called 
l2cache. What is this option for? It doesn't seem to be documented.

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] What is l2cache setting?

2010-09-22 Thread Peter Taps
Neil,

Thank you for your help.

However, I don't see anything about l2cache under Cache devices man pages.

To be clear, there are two different vdev types defined in zfs source code - 
cache and l2cache. I am familiar with cache devices. I am curious about 
l2cache devices.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Feature differences between Solaris 10 9/10 and build 147

2010-09-22 Thread Peter Taps
Folks,

Here is the list of ZFS enhancements as mentioned for the latest Solaris 10 
update:

*  ZFS device replacement enhancements - namely autoexpand
* some changes to the zpool list command
* Holding ZFS snapshots
* Triple parity RAID-Z (raidz3)
* The logbias property
* Log device removal - at last
* ZFS storage pool recovery
* New ZFS system process – In this release, each storage pool has an 
associated process, zpool-poolname
* Splitting a mirrored ZFS storage pool (zpool split)

I am wondering if any of these enhancements are not available under build 147.

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Dedup relationship between pool and filesystem

2010-09-23 Thread Peter Taps
Folks,

I am a bit confused on the dedup relationship between the filesystem and its 
pool.

The dedup property is set on a filesystem, not on the pool.

However, the dedup ratio is reported on the pool and not on the filesystem.

Why is it this way?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] What is dedupditto property on zpool?

2010-09-24 Thread Peter Taps
Folks,

One of the zpool properties that is reported is dedupditto. However, there is 
no documentation available, either in man pages or anywhere else on the 
Internet. What exactly is this property?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Any zfs fault injection tools?

2010-09-24 Thread Peter Taps
Folks,

Command zpool status reports disk status that includes read errors, write 
errors, and checksum errors. These values have always been 0 in our test 
environment. Is there any tool out there that can corrupt the state? At the 
very least, we should be able to write to the disk directly and mess up the 
checksum.

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] Any zfs fault injection tools?

2010-09-24 Thread Peter Taps
Freddie,

Thank you very much for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] When is it okay to turn off the verify option.

2010-10-04 Thread Peter Taps
Folks,

As I understand, the hash generated by sha256 is almost guaranteed not to 
collide. I am thinking it is okay to turn off verify property on the zpool. 
However, if there is indeed a collision, we lose data. Scrub cannot recover 
such lost data.

I am wondering in real life when is it okay to turn off verify option? I 
guess for storing business critical data (HR, finance, etc.), you cannot afford 
to turn this option off. 

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Optimal raidz3 configuration

2010-10-13 Thread Peter Taps
Folks,

If I have 20 disks to build a raidz3 pool, do I create one big raidz vdev or do 
I create multiple raidz3 vdevs? Is there any advantage of having multiple 
raidz3 vdevs in a single pool?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] How does dedup work over iSCSI?

2010-10-22 Thread Peter Taps
Folks,

Let's say I have a volume being shared over iSCSI. The dedup has been turned on.

Let's say I copy the same file twice under different names at the initiator 
end. Let's say each file ends up taking 5 blocks.

For dedupe to work, each block for a file must match the corresponding block 
from the other file. Essentially, each pair of block being compared must have 
the same start location into the actual data.

For a shared filesystem, ZFS may internally ensure that the block starts match. 
However, over iSCSI, the initiator does not even know about the whole block 
mechanism that zfs has. It is just sending raw bytes to the target. This makes 
me wonder if dedup actually works over iSCSI. 

Can someone please enlighten me on what I am missing?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] How does dedup work over iSCSI?

2010-10-22 Thread Peter Taps
Hi Neil,

if the file offset does not match, the chances that the checksum would match, 
especially sha256, is almost 0.

May be I am missing something. Let's say I have a file that contains 11 letters 
- ABCDEFGHIJK. Let's say the block size is 5.

For the first file, the block contents are ABCDE, FGHIJ, and K.

For the second file, let's say the blocks are  ABCD, EFGHI, and JK.

The chance that any checksum would match is very less. The chance that any 
checksum+verify would match is even less.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] Any limit on pool hierarchy?

2010-11-08 Thread Peter Taps
Folks,

From zfs documentation, it appears that a vdev can be built from more vdevs. 
That is, a raidz vdev can be built across a bunch of mirrored vdevs, and a 
mirror can be built across a few raidz vdevs.  

Is my understanding correct? Also, is there a limit on the depth of a vdev?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] How to create a checkpoint?

2010-11-08 Thread Peter Taps
Folks,

My understanding is that there is a way to create a zfs checkpoint before 
doing any  system upgrade or installing a new software. If there is a problem, 
one can simply rollback to the stable checkpoint.

I am familiar with snapshots and clones. However, I am not clear on how to 
manage checkpoints. I would appreciate your help in how I can create, destroy 
and roll back to a checkpoint, and how I can list all the checkpoints.

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] How to create a checkpoint?

2010-11-09 Thread Peter Taps
Thank you all for your help. Looks like beadm is the utility I was looking 
for.

When I run beadm list, it gives me the complete list and indicates which one 
is currently active. It doesn't tell me which one is the default boot. Can I 
assume that whatever is active is also the default?

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] How to grow root vdevs?

2010-11-09 Thread Peter Taps
Folks,

I am trying to understand if there is a way to increase the capacity of a 
root-vdev. After reading zpool man pages, the following is what I understand:

1. If you add a new disk by using zpool add, this disk gets added as a new 
root-vdev. The existing root-vdevs are not changed.
2. You can also add a new disk by using zpool attach on any existing disk of 
the pool. However, the existing disk cannot be part of a raidz vdev. Also, if 
the existing disk is part of a mirror, all we are doing is increasing 
redundancy but not growing the capacity of the vdev.

The only option to grow a root-vdev seems to be to use zpool replace and 
replace an existing disk with a bigger disk.

Is my understanding correct?

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] How to grow root vdevs?

2010-11-10 Thread Peter Taps
Thank you for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] How to safely parse zpool get all output?

2010-12-06 Thread Peter Taps
Folks,

Command zpool get all poolName does not provide any option to generate 
parsable output. The returned output contains 4 fields - name, property, value 
and source. These fields seems to be separated by spaces. I am wondering if it 
is safe to assume that there are no spaces in the field values. If this is the 
case, I can split the output across spaces.

Thank you in advance for your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] How to safely parse zpool get all output?

2010-12-06 Thread Peter Taps
Hi,

Thank you for your help.

I actually had the script working. However, I just wanted to make sure that 
spaces are not permitted within the field value itself. Otherwise, the regular 
expression would break.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] (Fletcher+Verification) versus (Sha256+No Verification)

2011-01-06 Thread Peter Taps
Folks,

I have been told that the checksum value returned by Sha256 is almost 
guaranteed to be unique. In fact, if Sha256 fails in some case, we have a 
bigger problem such as memory corruption, etc. Essentially, adding verification 
to sha256 is an overkill.

Perhaps (Sha256+NoVerification) would work 99.99% of the time. But 
(Fletcher+Verification) would work 100% of the time.

Which one of the two is a better deduplication strategy?

If we do not use verification with Sha256, what is the worst case scenario? Is 
it just more disk space occupied (because of failure to detect duplicate 
blocks) or there is a chance of actual data corruption (because two blocks were 
assumed to be duplicate although they are not)?

Or, if I go with (Sha256+Verification), how much is the overhead of 
verification on the overall process?

If I do go with verification, it seems (Fletcher+Verification) is more 
efficient than (Sha256+Verification). And both are 100% accurate in detecting 
duplicate blocks.

Thank you in advance for your help.

Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Fletcher+Verification) versus (Sha256+No Verification)

2011-01-09 Thread Peter Taps
Thank you all for your help. I am the OP.

I haven't looked at the link that talks about the probability of collision. 
Intuitively, I still wonder how the chances of collision can be so low. We are 
reducing a 4K block to just 256 bits. If the chances of collision are so low, 
*theoretically* it is  possible to reconstruct the original block from the 
256-bit signature by using a simple lookup. Essentially, we would now have 
world's best compression algorithm irrespective of whether the data is text or 
binary. This is hard to digest.

Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


[zfs-discuss] How to increase iometer reading?

2011-01-09 Thread Peter Taps
Hello,

We are building a zfs-based storage system with generic but high-quality 
components. We would like to test the new system under various loads. If we 
find that the iometer reading has started to reduce under certain loads, I am 
wondering what performance counters we should look for to identify the 
bottlenecks. Don't want to replace a component just to find that there was no 
improvement in iometer reading.

Thank you in advance for your insight.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Fletcher+Verification) versus (Sha256+No Verification)

2011-01-14 Thread Peter Taps
Ed,

Thank you for sharing the calculations. In lay terms, for Sha256, how many 
blocks of data would be needed to have one collision?

Assuming each block is 4K is size, we probably can calculate the final data 
size beyond which the collision may occur. This would enable us to make the 
following statement:

With Sha256, you need verification to be turned on only if you are dealing 
with more than xxxT of data.

Also, another related question. Why 256 bits was chosen and not 128 bits or 512 
bits? I guess Sha512 may be an overkill. In your formula, how many blocks of 
data would be needed to have one collision using Sha128?

Appreciate your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Fletcher+Verification) versus (Sha256+No Verification)

2011-01-14 Thread Peter Taps
I am posting this once again as my previous post went into the middle of the 
thread and may go unnoticed.

Ed,

Thank you for sharing the calculations. In lay terms, for Sha256, how many 
blocks of data would be needed to have one collision?

Assuming each block is 4K is size, we probably can calculate the final data 
size beyond which the collision may occur. This would enable us to make the 
following statement:

With Sha256, you need verification to be turned on only if you are dealing 
with more than xxxT of data.

Also, another related question. Why 256 bits was chosen and not 128 bits or 512 
bits? I guess Sha512 may be an overkill. In your formula, how many blocks of 
data would be needed to have one collision using Sha128?

Appreciate your help.

Regards,
Peter
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss