Re: Is it true that `dd` is almost not needed?
On Sun, Dec 12, 2021 at 03:09:37PM +0100, evh wrote: > On Sat, Dec 11, 2021 at 06:06:43PM +0200, u...@mailo.com wrote: > > Another reason to prefer the cat variant is that it lets you actually > > string together a normal shell pipeline. For instance, if you want > > progress information with cat you can combine it with the pv command: > > > > # Cat version with progress meter > > cat image.iso | pv >/dev/sdb > But you can with dd as well, and still get the useful options. Or you can just send a sig INFO, (normally bound to ^T), to the dd process, and get the records in, records out, bytes transferred, etc.
Re: Is it true that `dd` is almost not needed?
On Sat, Dec 11, 2021 at 06:06:43PM +0200, u...@mailo.com wrote: > Another reason to prefer the cat variant is that it lets you actually > string together a normal shell pipeline. For instance, if you want > progress information with cat you can combine it with the pv command: > > # Cat version with progress meter > cat image.iso | pv >/dev/sdb But you can with dd as well, and still get the useful options. $ pv image.iso | dd of=/dev/sdb bs=4M Last time I wrote a disk image I used a zcat ...img.gz | dd pipeline, could easily have included pv. Besides, "almost not needed" applies to cat as well, between ed, sed and even dd. cat is just better^Wmore convenient than those. Creating large zeroed files? I'd make them sparse with truncate(2) long before using non-standard cat options.
Re: Is it true that `dd` is almost not needed?
"The era of magnetic tapes" has not ended. It is just that some people mysteriously believe their data is safer "in the cloud" where they cannot monitor it, than on a tape in a fire safe under their own supervision. I have read back tapes I wrote myself 30 years earlier. Have you tried getting your data back from a deceased "cloud provider"? That is why we get all these stories of ransomware attacks. Also, for many of us that have spent half a century learning Unix, we do not want our well proven tools snatched from our hands. There is room for more than one knife in a virtual tool box. On Sat, 11 Dec 2021 at 23:58, Stuart Longland wrote: > On Sat, 11 Dec 2021 18:06:43 +0200 > wrote: > > > The Cult of DD > > Mar 17, 2017 > > You'll often see instructions for creating and using disk images on Unix > > systems making use of the dd command. This is a strange program of > > [obscure provenance](https://en.wikipedia.org/wiki/Dd_(Unix)) that > > somehow, still manages to survive in the 21st century. > > > > Actually, using dd is almost never necessary, and due to its highly > > nonstandard syntax is usually just an easy way to mess things up. For > > instance, you'll see instructions like this asking you to run commands > > like: > > > > […snip…] > > > > > > End of article and my questions: > > > > Is the author right in general? > > Is the author right for Linux environment? > > Is the author right for OpenBSD environment? > > Can `cat`/`tail` et all, create a "sparse" file? > > vk4msl-gap$ dd if=/dev/zero bs=1M count=1 seek=9 of=sparsefile > > 1+0 records in > > 1+0 records out > > 1048576 bytes transferred in 0.009 secs (108639200 bytes/sec) > > vk4msl-gap$ ls -lh sparsefile > > -rw-r--r-- 1 stuartl stuartl 10.0M Dec 12 08:42 sparsefile > > vk4msl-gap$ du -hs sparsefile > > 1.0Msparsefile > > Very useful for "thin provisioning" of raw disk images with virtual > machines. > > Can `cat`/`tail` et all read bytes from the middle of a file? > > vk4msl-gap$ echo -n '000102030405060708090a0b0c0d0e0f' > test > > vk4msl-gap$ dd if=test of=test.part bs=2 skip=4 count=4 > > 4+0 records in > > 4+0 records out > > 8 bytes transferred in 0.000 secs (66541 bytes/sec) > > vk4msl-gap$ cat test.part > > 04050607 > > Can `cat`/`tail` et all overwrite specific bytes in the middle of a file? > > vk4msl-gap$ echo -n 'aaabacad' | dd of=test bs=2 count=4 seek=8 > conv=notrunc > > 4+0 records in > > 4+0 records out > > 8 bytes transferred in 0.000 secs (98985 bytes/sec) > > vk4msl-gap$ cat test > > 0001020304050607aaabacad0c0d0e0f > > I think you'll find `dd` was written in the era of magnetic tapes as a > storage medium, and so the ability to seek to a specific part of a tape > then perform a read or write, was seen as a critical feature of the day. > > That same feature is handy when doing various low-level disk operations > as well (e.g. backing-up/restoring the boot sector/partition table). > -- > Stuart Longland (aka Redhatter, VK4MSL) > > I haven't lost my mind... > ...it's backed up on a tape somewhere. > >
Re: Is it true that `dd` is almost not needed?
u...@mailo.com [u...@mailo.com] wrote: > The article: > https://eklitzke.org/the-cult-of-dd > > The content of the article: > > The Cult of DD > Mar 17, 2017 > You'll often see instructions for creating and using disk images on Unix > systems making use of the dd command. This is a strange program of > [obscure provenance](https://en.wikipedia.org/wiki/Dd_(Unix)) that > somehow, still manages to survive in the 21st century. > The guy's complaints are somewhat valid. dd can be replaced with cat in many cases for the simple use case of dd if=file of=disk. Setting a higher block size (bs= argument) does make dd more efficient for writing whole disk images, and that's by design. It works with very small "blocks" by default. The dd tool comes in handy when you are trying to do something more than just write a whole disk image. I don't think there will be much conversion from EBCDIC to ASCII these days, but you can grab specific locations out of something instead of the whole thing, that's the more sophisticated use of dd. count=, seek=, skip= all depend on the desired block size bs=. conv=noerror, fsync are both occasionally useful and i'm sure someone else has another interesting use case aside from these things. Sure, the use of dd for everything related to disk is straight up cargo-cult, but also a good opportunity to learn the ins and outs of dd :) Chris
Re: Is it true that `dd` is almost not needed?
u...@mailo.com said on Sat, 11 Dec 2021 18:06:43 +0200 >The article: >https://eklitzke.org/the-cult-of-dd The ddrescue program is a different story entirely. Using ddrescue, you can almost perfectly digitize the scratched and stained garage sale music CD that won't even play on a normal player. ddrescue is the standard way to recover date from disks with lots of bad sectors. SteveT Steve Litt Spring 2021 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques
Re: Is it true that `dd` is almost not needed?
> On Dec 11, 2021, at 11:22 AM, Brian Brombacher wrote: > > >> On Dec 11, 2021, at 11:12 AM, u...@mailo.com wrote: >> >> The article: >> https://eklitzke.org/the-cult-of-dd >> >> The content of the article: >> >> The Cult of DD >> Mar 17, 2017 >> You'll often see instructions for creating and using disk images on Unix >> systems making use of the dd command. This is a strange program of >> [obscure provenance](https://en.wikipedia.org/wiki/Dd_(Unix)) that >> somehow, still manages to survive in the 21st century. >> >> Actually, using dd is almost never necessary, and due to its highly >> nonstandard syntax is usually just an easy way to mess things up. For >> instance, you'll see instructions like this asking you to run commands >> like: >> >> # Obscure dd version >> dd if=image.iso of=/dev/sdb bs=4M >> Guess what? This is exactly equivalent to a regular shell pipeline using >> cat and shell redirection: >> >> # Equivalent cat version >> cat image.iso >/dev/sdb >> That weird bs=4M argument in the dd version isn't actually doing >> anything special---all it's doing is instructing the dd command to use a >> 4 MB buffer size while copying. But who cares? Why not just let the >> command figure out the right buffer size automatically? >> >> Another reason to prefer the cat variant is that it lets you actually >> string together a normal shell pipeline. For instance, if you want >> progress information with cat you can combine it with the pv command: >> >> # Cat version with progress meter >> cat image.iso | pv >/dev/sdb >> There's an obscure option to GNU dd to get it to display a progress >> meter as well. But why bother memorizing that? If you learn the pv trick >> once, you can use it with any program. >> >> If you want to create a file of a certain size, you can do so using >> other standard programs like head. For instance, here are two ways to >> create a 100 MB file containing all zeroes: >> >> # Obscure dd version >> dd if=/dev/zero of=image.iso bs=4MB count=25 >> >> # Regular head version >> head -c 100MB /dev/zero >image.iso >> The head command is useful for lots of things, not just creating disk >> images. Therefore it's a better investment of your time to learn head >> than it is to learn dd. In fact, you probably already know how to use it. >> >> I will confess: there are some interesting options that dd has, which >> aren't easily replicated with cat or head. For instance, you can use dd >> to convert a file between ASCII and EBCDIC encodings. So if you find >> yourself doing that a lot, I won't blame you for reaching for dd. But >> otherwise, try to stick to more standard Unix tools. >> >> >> End of article and my questions: >> >> Is the author right in general? > > No. > >> Is the author right for Linux environment? > > No. > >> Is the author right for OpenBSD environment? > > No. I’ll clarify. Change No to Maybe, only for the examples provided in the article. Otherwise, dd is useful for other actions.
Re: Is it true that `dd` is almost not needed?
> On Dec 11, 2021, at 11:12 AM, u...@mailo.com wrote: > > The article: > https://eklitzke.org/the-cult-of-dd > > The content of the article: > > The Cult of DD > Mar 17, 2017 > You'll often see instructions for creating and using disk images on Unix > systems making use of the dd command. This is a strange program of > [obscure provenance](https://en.wikipedia.org/wiki/Dd_(Unix)) that > somehow, still manages to survive in the 21st century. > > Actually, using dd is almost never necessary, and due to its highly > nonstandard syntax is usually just an easy way to mess things up. For > instance, you'll see instructions like this asking you to run commands > like: > > # Obscure dd version > dd if=image.iso of=/dev/sdb bs=4M > Guess what? This is exactly equivalent to a regular shell pipeline using > cat and shell redirection: > > # Equivalent cat version > cat image.iso >/dev/sdb > That weird bs=4M argument in the dd version isn't actually doing > anything special---all it's doing is instructing the dd command to use a > 4 MB buffer size while copying. But who cares? Why not just let the > command figure out the right buffer size automatically? > > Another reason to prefer the cat variant is that it lets you actually > string together a normal shell pipeline. For instance, if you want > progress information with cat you can combine it with the pv command: > > # Cat version with progress meter > cat image.iso | pv >/dev/sdb > There's an obscure option to GNU dd to get it to display a progress > meter as well. But why bother memorizing that? If you learn the pv trick > once, you can use it with any program. > > If you want to create a file of a certain size, you can do so using > other standard programs like head. For instance, here are two ways to > create a 100 MB file containing all zeroes: > > # Obscure dd version > dd if=/dev/zero of=image.iso bs=4MB count=25 > > # Regular head version > head -c 100MB /dev/zero >image.iso > The head command is useful for lots of things, not just creating disk > images. Therefore it's a better investment of your time to learn head > than it is to learn dd. In fact, you probably already know how to use it. > > I will confess: there are some interesting options that dd has, which > aren't easily replicated with cat or head. For instance, you can use dd > to convert a file between ASCII and EBCDIC encodings. So if you find > yourself doing that a lot, I won't blame you for reaching for dd. But > otherwise, try to stick to more standard Unix tools. > > > End of article and my questions: > > Is the author right in general? No. > Is the author right for Linux environment? No. > Is the author right for OpenBSD environment? No.
Is it true that `dd` is almost not needed?
The article: https://eklitzke.org/the-cult-of-dd The content of the article: The Cult of DD Mar 17, 2017 You'll often see instructions for creating and using disk images on Unix systems making use of the dd command. This is a strange program of [obscure provenance](https://en.wikipedia.org/wiki/Dd_(Unix)) that somehow, still manages to survive in the 21st century. Actually, using dd is almost never necessary, and due to its highly nonstandard syntax is usually just an easy way to mess things up. For instance, you'll see instructions like this asking you to run commands like: # Obscure dd version dd if=image.iso of=/dev/sdb bs=4M Guess what? This is exactly equivalent to a regular shell pipeline using cat and shell redirection: # Equivalent cat version cat image.iso >/dev/sdb That weird bs=4M argument in the dd version isn't actually doing anything special---all it's doing is instructing the dd command to use a 4 MB buffer size while copying. But who cares? Why not just let the command figure out the right buffer size automatically? Another reason to prefer the cat variant is that it lets you actually string together a normal shell pipeline. For instance, if you want progress information with cat you can combine it with the pv command: # Cat version with progress meter cat image.iso | pv >/dev/sdb There's an obscure option to GNU dd to get it to display a progress meter as well. But why bother memorizing that? If you learn the pv trick once, you can use it with any program. If you want to create a file of a certain size, you can do so using other standard programs like head. For instance, here are two ways to create a 100 MB file containing all zeroes: # Obscure dd version dd if=/dev/zero of=image.iso bs=4MB count=25 # Regular head version head -c 100MB /dev/zero >image.iso The head command is useful for lots of things, not just creating disk images. Therefore it's a better investment of your time to learn head than it is to learn dd. In fact, you probably already know how to use it. I will confess: there are some interesting options that dd has, which aren't easily replicated with cat or head. For instance, you can use dd to convert a file between ASCII and EBCDIC encodings. So if you find yourself doing that a lot, I won't blame you for reaching for dd. But otherwise, try to stick to more standard Unix tools. End of article and my questions: Is the author right in general? Is the author right for Linux environment? Is the author right for OpenBSD environment?
