Edward Ned Harvey wrote:
I use the excellent pbzip2
zfs send ... | tee (md5sum) | pbzip2 | ssh remote ...
Utilizes those 8 cores quite well :)
This (pbzip2) sounds promising, and it must be better than what I wrote.
;-) But I don't understand the syntax you've got above, using
If feasible, you may want to generate MD5 sums on the streamed output
and then use these for verification.
That's actually not a bad idea. It should be kinda obvious, but I hadn't
thought of it because it's sort-of duplicating existing functionality.
I do have a multipipe script that behaves
Where exactly do you get zstreamdump?
I found a link to zstreamdump.c ... but is that it? Shouldn't it be part of
a source tarball or something?
Does it matter what OS? Every reference I see for zstreamdump is about
opensolaris. But I'm running solaris.
On Sat, Dec 5, 2009 at 17:17, Richard Elling richard.ell...@gmail.comwrote:
On Dec 4, 2009, at 4:11 PM, Edward Ned Harvey wrote:
Depending of your version of OS, I think the following post from Richard
Elling
will be of great interest to you:
-
On Sun, 6 Dec 2009, Edward Ned Harvey wrote:
I also have a threadzip script, because gzip is invariably the bottleneck
in the data stream. Utilize those extra cores!!! ;-)
Gzip can be a bit slow. Luckily there is 'lzop' which is quite a lot
more CPU efficient on i386 and AMD64, and even
Edward Ned Harvey wrote:
If feasible, you may want to generate MD5 sums on the streamed output
and then use these for verification.
That's actually not a bad idea. It should be kinda obvious, but I hadn't
thought of it because it's sort-of duplicating existing functionality.
I do
Bob Friesenhahn wrote:
On Sun, 6 Dec 2009, Edward Ned Harvey wrote:
I also have a threadzip script, because gzip is invariably the
bottleneck
in the data stream. Utilize those extra cores!!! ;-)
Gzip can be a bit slow. Luckily there is 'lzop' which is quite a lot
more CPU efficient on
Gzip can be a bit slow. Luckily there is 'lzop' which is quite a lot
more CPU efficient on i386 and AMD64, and even on SPARC. If the
compressor is able to keep up with the network and disk, then it is
fast enough. See http://www.lzop.org/;.
In my development/testing this week, I did time
On Sun, 6 Dec 2009, Edward Ned Harvey wrote:
Threadzip performed 10x faster (hardly a performance I expect from lzop) and
compressed about 2-3% smaller than gzip. Also hardly a performance I could
expect from lzop.
The key is multiple cores. I'm on an 8-core xeon.
I am glad to see that you
Edward Ned Harvey wrote:
I use the excellent pbzip2
zfs send ... | tee (md5sum) | pbzip2 | ssh remote ...
Utilizes those 8 cores quite well :)
This (pbzip2) sounds promising, and it must be better than what I wrote.
;-) But I don't understand the syntax you've got above, using
AcghhjkkNnmuUiui
- Original Message -
From: zfs-discuss-boun...@opensolaris.org zfs-discuss-boun...@opensolaris.org
To: Edward Ned Harvey sola...@nedharvey.com
Cc: ZFS discuss zfs-discuss@opensolaris.org
Sent: Sun Dec 06 10:54:11 2009
Subject: Re: [zfs-discuss] ZFS send | verify | receive
Depending of your version of OS, I think the following post from Richard
Elling will be of great interest to you:
Where exactly do you get zstreamdump?
I found a link to zstreamdump.c ... but is that it? Shouldn't it be part of
a source tarball or something?
Does it matter what OS? Every
OS means Operating System, or OpenSolaris. This is in the second
meaning I wrote OS in my answer. It was not obvious you were using
Solaris 10 though. Sorry about that.
(FYI, zstreamdump seems to be an addition to build 125.)
Oh - I never connected OS to OpenSolaris. ;-)
So I gather
I see 3.6X less CPU
consumption from 'lzop -3' than from 'gzip -3'.
Where do you get lzop from? I don't see any binaries on their site, nor
blastwave, nor opencsw. And I am having difficulty building it from source.
___
zfs-discuss mailing list
On Sun, 6 Dec 2009, Edward Ned Harvey wrote:
I see 3.6X less CPU
consumption from 'lzop -3' than from 'gzip -3'.
Where do you get lzop from? I don't see any binaries on their site, nor
blastwave, nor opencsw. And I am having difficulty building it from source.
I just built it from source.
On Dec 5, 2009, at 11:03 AM, Mike Gerdts wrote:
On Sat, Dec 5, 2009 at 11:32 AM, Bob Friesenhahn
bfrie...@simple.dallas.tx.us wrote:
On Sat, 5 Dec 2009, dick hoogendijk wrote:
On Sat, 2009-12-05 at 09:22 -0600, Bob Friesenhahn wrote:
You can also stream into a gzip or lzop wrapper in order
On Dec 6, 2009, at 3:35 PM, Edward Ned Harvey wrote:
OS means Operating System, or OpenSolaris. This is in the second
meaning I wrote OS in my answer. It was not obvious you were using
Solaris 10 though. Sorry about that.
(FYI, zstreamdump seems to be an addition to build 125.)
Oh - I
Oh well. I built LZO, and can't seem to link it in the lzop build, despite
correctly setting the FLAGS variables they say in the INSTALL file. I'd
love to provide an lzop comparison, but can't get it. I give up ... Also,
can't build python-lzo. Also would be sweet, but hey.
For whoever
cat my_log_file | tee (gzip my_log_file.gz) (wc -l) (md5sum) |
sort | uniq -c
That is great. ;-) Thank you very much.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Well what does _that_ verify?
It will verify that no bits provably broke during transport.
It will still leave the chance of getting an incompatible stream, an
incomplete stream (kill the dump), or plain corrupted data. Of course,
the chance of the latter should be extremely small in
On Sat, 5 Dec 2009, Sriram Narayanan wrote:
If feasible, you may want to generate MD5 sums on the streamed output
and then use these for verification.
You can also stream into a gzip or lzop wrapper in order to obtain the
benefit of incremental CRCs and some compression as well. As long as
On Dec 4, 2009, at 4:11 PM, Edward Ned Harvey wrote:
Depending of your version of OS, I think the following post from
Richard
Elling
will be of great interest to you:
-
http://richardelling.blogspot.com/2009/10/check-integrity-of-zfs-send-streams
.
html
Thanks! :-)
No, wait!
On Sat, 2009-12-05 at 09:22 -0600, Bob Friesenhahn wrote:
You can also stream into a gzip or lzop wrapper in order to obtain the
benefit of incremental CRCs and some compression as well.
Can you give an example command line for this option please?
Bob Friesenhahn wrote:
On Sat, 5 Dec 2009, Sriram Narayanan wrote:
If feasible, you may want to generate MD5 sums on the streamed output
and then use these for verification.
You can also stream into a gzip or lzop wrapper in order to obtain the
benefit of incremental CRCs and some
On Sat, 5 Dec 2009, dick hoogendijk wrote:
On Sat, 2009-12-05 at 09:22 -0600, Bob Friesenhahn wrote:
You can also stream into a gzip or lzop wrapper in order to obtain the
benefit of incremental CRCs and some compression as well.
Can you give an example command line for this option please?
If there were a ³zfs send² datastream saved someplace, is there a way to
verify the integrity of that datastream without doing a ³zfs receive² and
occupying all that disk space?
I am aware that ³zfs send² is not a backup solution, due to vulnerability of
even a single bit error, and lack of
If there were a “zfs send” datastream saved someplace, is there a way to
verify the integrity of that datastream without doing a “zfs receive” and
occupying all that disk space?
Depending of your version of OS, I think the following post from Richard Elling
will be of great interest to you:
-
Depending of your version of OS, I think the following post from Richard
Elling
will be of great interest to you:
-
http://richardelling.blogspot.com/2009/10/check-integrity-of-zfs-send-streams.
html
Thanks! :-)
No, wait!
According to that page, if you zfs receive -n then you should
If feasible, you may want to generate MD5 sums on the streamed output
and then use these for verification.
-- Sriram
On 12/5/09, Edward Ned Harvey sola...@nedharvey.com wrote:
Depending of your version of OS, I think the following post from Richard
Elling
will be of great interest to you:
-
29 matches
Mail list logo
