Hi all
I wonder if there has been any new development on this matter over the past 6
months.
Today i pondered an idea of zfs-aware mv, capable of doing zero read/write of
file data when moving files between datasets of one pool.
This seems like a (z)cp idea proposed in this thread and seems
Hi all
I wonder if there has been any new development on this matter over the past 6
months.
Today i pondered an idea of zfs-aware mv, capable of doing zero read/write of
file data when moving files between datasets of one pool.
This seems like a (z)cp idea proposed in this thread and seems
Michael,
Your explanation is 100% correct: I am concerned about the effort when managing
quite large files ex. 500MB.
In my specific case we have DVD/BlueRay chapter files 500MB - 2GB (part of
movie) that are concatenated into complete movie (3-20GB).
From my point of view (large files) it is
-Oorspronkelijk bericht-
Van: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
boun...@opensolaris.org] Namens Roland Rambau
Verzonden: donderdag 3 december 2009 16:25
Aan: Per Baatrup
CC: zfs-discuss@opensolaris.org
Onderwerp: Re: [zfs-discuss] file concatenation with ZFS copy-on-write
I was thinking in the same direction about the efficiency of the offset
calculations. Trying to get into the ZFS source code to understand this part,
but did not have time to get there yet.
This issue may be a showstopper for the proposal as it would restrict the
functionality to quite rare
Darren J Moffat wrote:
Per Baatrup wrote:
I would like to to concatenate N files into one big file taking
advantage of ZFS copy-on-write semantics so that the file
concatenation is done without actually copying any (large amount of)
file content.
cat f1 f2 f3 f4 f5 f15
Is this already
Per Baatrup wrote:
I would like to to concatenate N files into one big file taking advantage of
ZFS copy-on-write semantics so that the file concatenation is done without
actually copying any (large amount of) file content.
cat f1 f2 f3 f4 f5 f15
Is this already possible when source and
After reading all the comments it appears that there may be a 'real'
problem with unaligned block sizes that DEDUP simply will not handle.
What you seem to be after, then, is the opposite of sparse files,
'virtual files' that can be chained together as a linked list of
_fragments_ of
Thank you for the feedback Michael.
zcat was my acronym for a special ZFS aware version of cat and I did
not know that it was an existing command. Simply forgot to check. Should
rename if to zfscat or something similar.
Venlig hilsen
Per
Michael Schuster skrev:
Per Baatrup wrote:
dedup
On Fri, 4 Dec 2009, Jeffry Molanus wrote:
Actually, I asked about this a while ago only called it file-level cloning.
Consider you have 100VM's and you want to clone just one?
BTRFS added a specialized IOCTL() call to make the FS aware that it
has to clone this obviously saves copy time and
...@opensolaris.org] Namens Roland Rambau
Verzonden: donderdag 3 december 2009 16:25
Aan: Per Baatrup
CC: zfs-discuss@opensolaris.org
Onderwerp: Re: [zfs-discuss] file concatenation with ZFS copy-on-
write
gang,
actually a simpler version of that idea would be a zcp:
if I just cp a file, I know
The way I see it, a filename is a handle to a specific set of blocks.
For applications
that can handle multiple files, no worries. For applications that
can't (inferring DVD
players?) I sense that a fixing the tail block issue in a file system
is probably not
the best place. This affects
In my experience, cloning is done for basic provisioning, so how would
you get
to the case where you could not clone any particular VM?
-- richard
Well, a situation where this might come in handy is when you have your typical
ISP provider that has multiple ESX hosts with multiple
Per Baatrup wrote:
I would like to to concatenate N files into one big file taking advantage of
ZFS copy-on-write semantics so that the file concatenation is done without
actually copying any (large amount of) file content.
cat f1 f2 f3 f4 f5 f15
Is this already possible when source and
On Thu, Dec 3, 2009 at 12:08 PM, Darren J Moffat
darr...@opensolaris.org wrote:
Per Baatrup wrote:
I would like to to concatenate N files into one big file taking advantage
of ZFS copy-on-write semantics so that the file concatenation is done
without actually copying any (large amount of)
On Thu, 3 Dec 2009, Darren J Moffat wrote:
The answer to this is likely deduplication which ZFS now has.
The reason dedup should help here is that after the 'cat' f15 will be made up
of blocks that match the blocks of f1 f2 f3 f4 f5.
Copy-on-write isn't what helps you here it is dedup.
On 3 déc. 2009, at 13:29, Bob Friesenhahn bfrie...@simple.dallas.tx.u
s wrote:
On Thu, 3 Dec 2009, Darren J Moffat wrote:
The answer to this is likely deduplication which ZFS now has.
The reason dedup should help here is that after the 'cat' f15 will
be made up of blocks that match the
dedup operates on the block level leveraging the existing FFS checksums. Read
What to dedup: Files, blocks, or bytes here
http://blogs.sun.com/bonwick/entry/zfs_dedup
The trick should be that the zcat userland app already knows that it will
generate duplicate files so data read and writes
Per Baatrup wrote:
dedup operates on the block level leveraging the existing FFS
checksums. Read What to dedup: Files, blocks, or bytes here
http://blogs.sun.com/bonwick/entry/zfs_dedup
The trick should be that the zcat userland app already knows that it
will generate duplicate files so data
zcat was my acronym for a special ZFS aware version of cat and the name was
obviously a big mistake as I did not know it was an existing command and simply
forgot to check.
Should rename if to zfscat or something similar?
--
This message posted from opensolaris.org
gang,
actually a simpler version of that idea would be a zcp:
if I just cp a file, I know that all blocks of the new file
will be duplicates; so the cp could take full advantage for
the dedup without a need to check/read/write anz actual data
-- Roland
Per Baatrup schrieb:
dedup operates
Roland Rambau wrote:
gang,
actually a simpler version of that idea would be a zcp:
if I just cp a file, I know that all blocks of the new file
will be duplicates; so the cp could take full advantage for
the dedup without a need to check/read/write anz actual data
I think they call it 'ln'
Bob Friesenhahn wrote:
On Thu, 3 Dec 2009, Darren J Moffat wrote:
The answer to this is likely deduplication which ZFS now has.
The reason dedup should help here is that after the 'cat' f15 will be
made up of blocks that match the blocks of f1 f2 f3 f4 f5.
Copy-on-write isn't what helps
Actually 'ln -s source target' would not be the same zcp source target as
writing to the source file after the operation would change the target file as
well where as for zcp this would only change the source file due to
copy-on-write semantics of ZFS.
--
This message posted from
Per Baatrup wrote:
Actually 'ln -s source target' would not be the same zcp source target
as writing to the source file after the operation would change the
target file as well where as for zcp this would only change the source
file due to copy-on-write semantics of ZFS.
I actually was
michael schuster wrote:
Roland Rambau wrote:
gang,
actually a simpler version of that idea would be a zcp:
if I just cp a file, I know that all blocks of the new file
will be duplicates; so the cp could take full advantage for
the dedup without a need to check/read/write anz actual data
I
On Thu, 3 Dec 2009, Erik Ableson wrote:
Much depends on the contents of the files. Fixed size binary blobs that align
nicely with 16/32/64k boundaries, or variable sized text files.
Note that the default zfs block size is 128K and so that will
therefore be the default dedup block size.
Bob Friesenhahn wrote:
On Thu, 3 Dec 2009, Erik Ableson wrote:
Much depends on the contents of the files. Fixed size binary blobs
that align nicely with 16/32/64k boundaries, or variable sized text
files.
Note that the default zfs block size is 128K and so that will therefore
be the
On Thu, Dec 3, 2009 at 9:58 AM, Bob Friesenhahn
bfrie...@simple.dallas.tx.us wrote:
On Thu, 3 Dec 2009, Erik Ableson wrote:
Much depends on the contents of the files. Fixed size binary blobs that
align nicely with 16/32/64k boundaries, or variable sized text files.
Note that the default zfs
On Thu, 3 Dec 2009, Jason King wrote:
Well it could be done in a way such that it could be fs-agnostic
(perhaps extending /bin/cat with a new flag such as -o outputfile, or
detecting if stdout is a file vs tty, though corner cases might get
tricky). If a particular fs supported such a
Michael,
michael schuster schrieb:
Roland Rambau wrote:
gang,
actually a simpler version of that idea would be a zcp:
if I just cp a file, I know that all blocks of the new file
will be duplicates; so the cp could take full advantage for
the dedup without a need to check/read/write anz
Roland,
Clearly an extension of cp would be very nice when managing large files.
Today we are relying heavily on snapshots for this, but this requires disipline
on storing files in separate zfs'es avioding to snapshot too many files that
changes frequently.
The reason I was speaking about cat
On Thu, Dec 03, 2009 at 09:36:23AM -0800, Per Baatrup wrote:
The reason I was speaking about cat in stead of cp is that in
addition to copying a single file I would like also to concatenate
several files into a single file. Can this be accomplished with your
(z)cp?
Unless you have special
On Thu, Dec 03, 2009 at 03:57:28AM -0800, Per Baatrup wrote:
I would like to to concatenate N files into one big file taking
advantage of ZFS copy-on-write semantics so that the file
concatenation is done without actually copying any (large amount of)
file content.
cat f1 f2 f3 f4 f5 f15
Per,
Per Baatrup schrieb:
Roland,
Clearly an extension of cp would be very nice when managing large files.
Today we are relying heavily on snapshots for this, but this requires disipline
on storing files in separate zfs'es avioding to snapshot too many files that
changes frequently.
The
if any of f2..f5 have different block sizes from f1
This restriction does not sound so bad to me if this only refers to changes to
the blocksize of a particular ZFS filesystem or copying between different ZFSes
in the same pool. This can properly be managed with a -f switch on the
userlan app
Btw. I would be surprised to hear that this can be implemented
with current APIs;
I agree. However it looks like an opportunity to dive into the Z-source code.
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
On Thu, Dec 03, 2009 at 12:44:16PM -0800, Per Baatrup wrote:
if any of f2..f5 have different block sizes from f1
This restriction does not sound so bad to me if this only refers to
changes to the blocksize of a particular ZFS filesystem or copying
between different ZFSes in the same pool.
Isn't this only true if the file sizes are such that the concatenated
blocks are perfectly aligned on the same zfs block boundaries they used
before? This seems unlikely to me.
Yes that would be the case.
While eagerly awaiting b128 to appear in IPS, I have been giving this issue
On Thu, Dec 03, 2009 at 12:44:16PM -0800, Per Baatrup wrote:
any of f1..f5's last blocks are partial
Does this mean that f1,f2,f3,f4 needs to be exact multiplum of the ZFS
blocksize? This is a severe restriction that will fail unless in very
special cases. Is this related to the disk format
Nicolas Williams wrote:
On Thu, Dec 03, 2009 at 12:44:16PM -0800, Per Baatrup wrote:
if any of f2..f5 have different block sizes from f1
This restriction does not sound so bad to me if this only refers to
changes to the blocksize of a particular ZFS filesystem or copying
between different
41 matches
Mail list logo
