On 6/4/21 8:22 AM, Dan Brown wrote: > Hi Rob- thanks. I get the impression that I would not be able to help on the > things you're working on - they sound like they involve knowledge that is too > deep for me to learn for a side project.
That's just what _I'm_ working on. The project has plenty of surface area. Let's see... If somebody wanted to write an awk, I'd thank that person profusely. Android uses kernighan's original awk: https://en.wikipedia.org/wiki/AWK#Versions_and_implementations But it uses a bespoke bsd license variant (https://github.com/onetrueawk/awk/blob/master/LICENSE) and toybox uses a public domain equivalent license, so I can't take code directly from that. Plus I'm told it doesn't support utf8: https://github.com/landley/toybox/issues/67#issuecomment-587347070 I have the very start of an ar.c that doesn't do anything yet (attached). That's used not just by builds but inside of debian's package management and a few other places. There's a bunch of stuff in toys/pending/*.c. I got about halfway through mkdosfs.c before getting pulled away and never getting back to it (attached that too; turns out you MUST support all the fat12/16/32 variants because it uses the image size to determine file type. I was vaguely thinking I could make a genvfatfs.c that worked like an archiver (ala genvfatfs dir | gzip | ssh remote.sys "cat > img.gz"), and maybe even implement the whole mtools suite). The mke2fs.c in pending is also half-finished and I haven't touched that in YEARS (that I was working on for busybox, and copied what I had over into toybox but it was at the end of the todo heap, and adding ext3 was just a magic file in a reserved inode number but since then ext4 happened and I haven't had a chance to look at that at all)... For diff.c I really wanted to do a "patience diff" algorithm implementation, but got a conventional one submitted instead and never had a chance to wrap my head around the pro/con of the version they did. I bookmarked an algorithm comparison article: https://blog.jcoglan.com/2017/09/19/the-patience-diff-algorithm/ And have various old bookmarks about patience diff: https://bramcohen.livejournal.com/37690.html https://bramcohen.livejournal.com/73318.html https://en.wikipedia.org/wiki/Patience_sorting I need to write a less.c (probably using watch.c as a model, and definitely implementing -R support so colors work). But then I need to do similar code for the shell's command line editing... If somebody who actualy _uses_ modules could review modprobe, that would be nice. My main blocker there is the embedded systems I put together all have static kernels. I've always been slightly unclear on what getty.c _does_ and why it's separate from login.c. (Is it related to stty?) I want to redo route.c to use the netlink api to add multi-table support (ala https://android.googlesource.com/platform/external/toybox/+/48e1f81151f6). > After reading through your state-of-the-toybox message, the commands "tr" and > "stdbuf" are interesting to me. If you think it is worth a shot for me to give > it a go, let me know which one you'd recommend. It'd be great if somebody could tackle stdbuf, I have no idea how that one works. (The buffering is in libc, how does one program tell the next executable to use different libc default parameters? Export an environment variable? Is doing that portable between glibc/musl/bionic? "strings" on the stdbuf binary is showing _STDBUF_ and LD_PRELOAD and libstdbuf.so and that just sounds ugly...) My main todo item on tr is I'd like to have utf8/unicode support, but haven't figured out how to do so yet at a conceptual level because it's matching a source string with a destination string by position, and things like [:alnum:] expand to different numbers of entries in ascii vs unicode. We had a thread on that here: http://lists.landley.net/pipermail/toybox-landley.net/2020-December/012158.html Rich Felker said he had a simple way to do it, but we've never sat down to have him explain it to me. > Dan Rob
/* ar.c - original unix library archive format * * Copyright 2018 Rob Landley <[email protected]> * * See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ar.html * * What counts as printable? linkable? * * deviations from posix: -T is a NOP USE_AR(NEWTOY(ar, "&<1TcCuvsa:b:i:dmpqrtx[!dmpqrtx][!abidpqtx]", TOYFLAG_USR|TOYFLAG_BIN)) config AR bool "ar" default n help usage: ar [-drmstxvpqc] [-abi FILE] ARCHIVE [FILE...] Create and maintain lib.a archives. -a add file(s) after other FILE -b add file(s) before other FILE -c quiet -C don't overwrite existing files -d delete file(s) -i insert files(s) before other FILE (same as -b) -p print fils to stdout (default: all) -r replace (add) file(s) -m move file(s) in archive (to -ab or end) -q quick (append file(s) without removing duplicates) -s rebuild symbol table (for static library) -t list file(s) -u update (only replace newer files) -v verbose -x extract file(s) */ #define FOR_ar #include "toys.h" GLOBALS( char *i, *b, *a; ) void ar_main(void) { char *name = 0; int fd, fdout, flags = O_RDWR; if (toys.optflags&(FLAG_a|FLAG_b|FLAG_i) && !(toys.optflags&FLAG_m)) toys.optflags |= FLAG_r; // read: -x -t // copy: -a -b -i -d -r -m -s // in place: -q or if (toys.optflags&(FLAG_x|FLAG_t)) xopen(*toys.optargs, O_RDONLY); else { if (toys.optflags&FLAG_q) fd = fdout = open(*toys.optargs, O_RDWR|O_CREAT); else { fd = open(*toys.optargs, O_RDONLY); if (fd==-1) fd = -0600; fdout = copy_tempfile(fd, *toys.optargs, &name, 0664); } bidmpqrtxs if (toys.optflags&(FLAG_x|FLAG_t|FLAG_m)) fd = xopen(*toys.optargs, O_RDONLY); else if (toys.optflags&(FLAG_d|FLAG_r|FLAG_m|FLAG_s) fd = open(*toys.optargs, O_ if (!(toys.optflags&FLAG_x)) error_exit("nope"); if (toys.optflags&(FLAG_x|FLAG_t) int fd = xopen(*toys.optargs, O_RDONLY) } // files are basename() }
/* mkdosfs.c - Make a vfat filesystem. * * Copyright 2018 Rob Landley <[email protected]> * * No obvious standard. Lots of BAD standards. * linux/msdos_fs.h * * mkdosfs -F 12/16/32 -O android path * 12 = sec<32m, 16<2g, else 32 * biggest fat12 is 16404 sectors USE_DUMPVFAT(NEWTOY(dumpvfat, "<1", TOYFLAG_USR|TOYFLAG_BIN)) USE_MKDOSFS(NEWTOY(mkdosfs, "\300N#a#b#c#e#i#k#C#F#I#S#T#B:L:O:[!NC]", TOYFLAG_USR|TOYFLAG_BIN)) USE_MKDOSFS(OLDTOY(newfs_msdos, mkdosfs, TOYFLAG_USR|TOYFLAG_BIN)) USE_MKDOSFS(OLDTOY(mkfs.vfat, mkdosfs, TOYFLAG_USR|TOYFLAG_BIN)) config MKDOSFS bool "mkdosfs" default n help usage: mkdosfs [-@A] [-abceikCFIST NUMBER] [-BLO STRING] FILE -@ Start filesystem at OFFSET bytes -A Align root directory -N No create -B Use boot sector from FILE -C Create file with SIZE -F Type (12, 16, or 32, default based on size) -I Set volume ID (32 bits) -L Set LABEL (11 chars) -O Set OEM string (8 chars) -S Bytes per sector (power of 2, 512-32768) -T Set creation TIMESTAMP (unix time, or path to file) -a Number of sectors in filesystem -b Block size (sectors per block) -c Cluster size (sectors per cluster) -e Number of root directory entries -i Fat32 Info sector location (default 0xffff) -s File system size -f Floppy format -h Drive heads -k Fat32 backup boot sector (default 0xffff) -n Number of File Allocation Tables (1-16, default 2) -m Media descriptor (0xf0 to 0xff) -o Hidden sectors -r Reserved sectors -u Sectors per track config DUMPVFAT bool "dumpvfat" default n help usage: dumpvfat blah blah */ #define FOR_mkdosfs #include "toys.h" // The union lets lib/args.c store arguments for either command. // It's customary to put a space between argument variables and other globals. GLOBALS( char *O, *L, *B; long T, S, I, F, C, k, i, e, c, b, a, at; char *fat; unsigned spc, fds; ) // These fields are out of order. On disk half unaligned, all little endian. struct bootsect { char jmp[3], oem[8], sectors_per_cluster, fat_count, media_type, sig, volume_label[11], system_id[8], end; unsigned short bytes_per_sector, reserved_sectors, root_entries, small_sectors, sectors_per_fat, sectors_per_track, number_of_heads; unsigned hidden_sectors, large_sectors, serial_number; }; struct fat_dirent { char name[11], attr, lcase, ctime_cs; unsigned short ctime, cdate, adate, starthi, time, date, start; unsigned size; }; // Round up to next power of 2. unsigned long long p2roundup(unsigned long long ull) { unsigned long long ull2 = --ull; while (ull) ull2 |= (ull>>=12); return ++ull2; } // cluster to sector static unsigned clus2sec(unsigned cluster) { return (cluster-2)*TT.spc+TT.fds; } // then at 0x200 F8 FF FF // then at 0x600 F8 FF FF // Read/calculate values from a FAT boot sector static void measure_fat(char *boot) { unsigned bytes_per_sector = peek_le(boot+11, 2), sectors_per_cluster = boot[13], reserved_sectors = peek_le(boot+14, 2), fat_count = boot[16], root_entries = peek_le(boot+17, 2), total_sectors = peek_le(boot+19, 2), fat_size = peek_le(boot+22, 2), root_sectors = ((root_entries*32)+511)/512; if (!total_sectors) total_sectors = peek_le(boot+32, 4); if (!fat_size) fat_size = peek_le(boot+36, 4); // first data sector, data sector count unsigned fds = boot[14] + 2*fat_size + root_sectors; unsigned dscount = total_sectors - (boot[14] + root_sectors + 2*fat_size); unsigned clusters = dscount/boot[13]; // starting at 2. // <4085 = fat12, <65525 fat16 // fat_sectors = 256*TT.spc printf("bytes_per_sector=%u\n", bytes_per_sector); printf("sectors_per_cluster=%d\n", sectors_per_cluster); printf("reserved_sectors=%u\n", reserved_sectors); printf("fat_count=%d\n", fat_count); printf("root_entries=%d\n", root_entries); printf("total_sectors=%d\n", total_sectors); printf("fat_size=%d\n", fat_size); printf("first data sector=%d\n", fds); printf("data sector count=%d\n", dscount); printf("clusters=%d\n", clusters); } // Fill out FAT boot sector and allocate TT.fat for a filesystem of this size static void format(char *fat, long long bytes, char *oem) { unsigned sectors = bytes/512, spc, is32 = 28*(bytes >= 1LL<<32), root_ent = 512, root_sectors = ((root_ent*32)+511)/512, fat_sectors; char *s; // <= 16404 sectors = biggest fat12: 8 sectors/cluster memset(fat, 0, 512); // Original PC bios would load the boot sector and execute it. This jumped // past the filesystem data. It's still there because USB flash sticks // perform wear leveling when they detect a FAT filesystem, and how they // detect it varies by vendor. So err on the side of "looking like FAT". fat[0] = 0xeb; fat[1] = is32 ? 0x58 : 0x3c; fat[2] = 0x90; strncpy(fat+3, oem, 8); // OEM fat[11] = 2; // 512 bytes per sector, high byte of little endian # fat[13] = spc = is32 ? 8 : p2roundup((sectors/65536)+1);// sectors per cluster fat[14] = is32 ? 32 : 1; // reserved sectors fat[16] = 2; // how many file allocation tables if (!is32) poke_le(fat+18, root_ent, 2); // fat16 root filesystem entries // small or large sector count if (sectors < 65536) poke_le(fat+19, sectors, 2); // unaligned else *(unsigned *)(fat+32) = sectors; // aligned! fat[21] = 0xf8; // media type "fixed disk", must match first byte of each FAT // sectors per File Allocation Table // fat_size = TODO; if (!is32) *(unsigned short *)(fat+22) = fat_size; else *(unsigned *)(fat+36) = fat_size; fat[24] = 63; // sectors per track, obsolete fat[26] = 64; // number of read/write heads, obsolete if (is32) { fat[44] = 2; // first cluster of root dir fat[48] = 1; // fsinfo starting cluster fat[50] = 6; // backup boot sector } fat[36+is32] = 128; // drive number, obsolete fat[38+is32] = 0x29; poke_le(fat+39+is32, time(0), 4); // Volume ID strcpy(fat+43+is32, "NO NAME FAT%02d ", is32 ? 32 : 16); // type magic fat[510] = 0x55; fat[511] = 0xAA; // first data sector, data sector count TT.fds = fat[14] + 2*fat_size + root_sectors; TT.dscount = sectors - (fat[14] + root_sectors + 2*fat_size); clusters = TT.dscount/fat[13]; // starting at 2. // <4085 = fat12, <65525 fat16 fat_sectors = 256*TT.spc // Initialize everything through root_entries strcpy(fat->magic, "\353\74\220Toybox\0\0\0\2\4\1\0\2\0\2", 19); fat->media_type = 0xf8; fat->physical_disk = 128; fat->signature = 0x29; //*fat->sectors_per_track = 63; //*fat->number_of_heads = 64; // small_sectors, large_sectors // sectors_per_fat, sectors_per_track, number_of_heads // reserved_sectors if (fat32) { fat->magic[1] = 0x58; fat->sectors_per_cluster = 8; fat->reserved_sectors[0] = 0x20; zero root_entries, sectors_per_fat, fat->root_entries[1] = 0; } } void mkdosfs_main(void) { struct fat_bootsect *fat = (void *)toybuf; long long bsz; int bs; // Initialize everything through root_entries strcpy(fat->magic, "\353\74\220Toybox\0\0\0\2\4\1\0\2\0\2", 19); fat->media_type = 0xf8; fat->physical_disk = 128; fat->signature = 0x29; //*fat->sectors_per_track = 63; //*fat->number_of_heads = 64; // small_sectors, large_sectors // sectors_per_fat, sectors_per_track, number_of_heads // reserved_sectors if (fat32) { fat->magic[1] = 0x58; fat->sectors_per_cluster = 8; fat->reserved_sectors[0] = 0x20; zero root_entries, sectors_per_fat, fat->root_entries[1] = 0; } // reserved sectors, hidden sectors // sanity check string(s) OEM, volume label // sanity check media descriptor 0xf0-0xff // info sector or backup sector require fat32 // bytes/sec must be power of 2, at least 512 // block size must be power of 2, at least bytes/sec, at most 128*bytes/sec // sectors/cluster must be power of 2 // create or open, truncate to size, seek to offset xioctl(fd, BLKSSZGET, &bs); xioctl(fd, BLKGETSIZE64, bsz); // sectors per track = 63 heads = 64 // sectors per cluster = 1<8192, 8 <= 1<<17, 16 <= 1<<19 32 <= 1<<21 else 64 // minimum fat16 clusters: 4096-11, mininum fat32: 65536-11 printf("%ld\n", TT.at); } void dumpvfat_main(void) { read(xopen(toys.optargs[1]), toybuf, sizeof(toybuf)); measure_fat(toybuf); }
_______________________________________________ Toybox mailing list [email protected] http://lists.landley.net/listinfo.cgi/toybox-landley.net
