Re: Cannot find announcement that min supported i386 CPU is now i686
15.09.2020 9:42, Eugene Grosbein wrote: >> I tried that on my workstation (dual Xeon, lots of RAM) but couldn't get >> it to compile for i586 - various things kept bailing out. Note that I use multiple WITHOUT_XXX knobs documented in src.conf(5) to decrease both build time and size of resulting image, maybe some excluded code really has problems for i386, can't tell. For CONF_BUILD (addition to src.conf for buildworld): WITHOUT_KERNEL_SYMBOLS= WITHOUT_DEBUG_FILES= WITHOUT_TESTS= MODULES_WITH_WORLD= WITHOUT_ACCT= WITHOUT_AMD= WITHOUT_APM= WITHOUT_ASSERT_DEBUG= WITHOUT_ATM= WITHOUT_AUDIT= WITHOUT_AUTHPH= WITHOUT_AUTOFS= WITHOUT_BHYVE= WITHOUT_BOOTPARAMD= WITHOUT_BOOTPD= WITHOUT_BSDINSTALL= WITHOUT_BSNMP= WITHOUT_CALENDAR= WITHOUT_CCD= #WITHOUT_CDDL= WITHOUT_CLANG_FULL= WITHOUT_CPP= WITHOUT_CTM= WITHOUT_DEBUG_FILES= WITHOUT_DICT= WITHOUT_EXAMPLES= WITHOUT_FLOPPY= WITHOUT_FREEBSD_UPDATE= WITHOUT_GAMES= WITHOUT_GCOV= WITHOUT_GDB= WITHOUT_GROFF= WITHOUT_HAST= WITGOUT_HESIOD= WITHOUT_HTML= WITHOUT_HYPERV= WITHOUT_IPFILTER= WITHOUT_IPX= WITHOUT_ISCSI= WITHOUT_JAIL= WITHOUT_KERNEL_SYMBOLS= WITHOUT_LLDB= WITHOUT_LLVM_TARGET_ALL= WITH_LLVM_TARGET_X86= WITHOUT_LOCATE= WITHOUT_LS_COLORS= WITHOUT_MAN= WITH_MAN_UTILS= WITHOUT_NCP= WITHOUT_NDIS= WITHOUT_NETCAT= WITHOUT_NIS= WITHOUT_NLS= WITHOUT_NLS_CATALOGS= WITHOUT_NS_CACHING= WITH_OPENSSH_NONE_CIPHER= WITHOUT_PC_SYSINSTALL= WITHOUT_PF= WITHOUT_PKGTOOLS= WITHOUT_PMC= WITHOUT_PORTSNAP= WITHOUT_PPP= WITHOUT_PROFILE= WITHOUT_QUOTAS= WITHOUT_RBOOTD= WITHOUT_RCS= WITHOUT_RESCUE= WITHOUT_ROUTED= WITHOUT_SHAREDOCS= WITHOUT_SVNLITE= WITHOUT_TALK= WITHOUT_SYSINSTALL= WITHOUT_TESTS= WITHOUT_TEXTPROC= WITHOUT_TIMED= WITHOUT_VT= MODULES_OVERRIDE=cryptodev zfs opensolaris And: CONF_INSTALL=" $CONF_BUILD WITHOUT_BINUTILS= WITHOUT_CLANG= WITHOUT_CLANG_FULL= WITHOUT_CXX= WITHOUT_TOOLCHAIN= WITHOUT_INSTALLLIB= " >> It would compile i386 for i686, but not i586. Have you tried it with 11.4? Not sure what you meant here. ARCH may be i386 (32 bit) or amd64 (64 bit), however i586 is not a name for ARCH but for CPUTYPE only (not same thing). ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
14.09.2020 12:08, Charles Lecklider via freebsd-stable wrote: > On 2020-09-14 04:41, Eugene Grosbein wrote: >> Build time for modern FreeBSD version is too gross and needs way too much >> memory, >> so I stopped building image for my i586 hardware "in place" quite long ago. > > It would take about a week to compile, but since it was stable <11.4 I > didn't really care. > >> I use my i7-based desktop to build NanoBSD image for its upgrade, it works >> just fine >> by default setting ARCH/TARGET_ARCH to i386 and correct CPUTYPE. > > I tried that on my workstation (dual Xeon, lots of RAM) but couldn't get > it to compile for i586 - various things kept bailing out. It would > compile i386 for i686, but not i586. Have you tried it with 11.4? Just updated that my i586 AMD Geode system upto latest 11.4-STABLE, all went OK. I build it with standard nanobsd(8) utility in base system, my build configuration contains: NANO_PMAKE="make -j3" NANO_KERNEL=GWOLD CONF_BUILD=' TARGET=i386 TARGET_ARCH=i386 NANO_ARCH=i386 CPUTYPE?=k6-3 ... ' And kernel configuration has: machine i386 cpu I586_CPU cpu I686_CPU options CPU_GEODE options CPU_SOEKRIS options SCHED_4BSD ... For UP system, I find SCHED_4BSD behave better than default SCHED_ULE. >>> Don't try -Os or -Oz (which really do help on a real i586) - ZFS will >>> wedge quickly. >> >> Currently i586-class CPU needs as many speed improvements as possible >> and I was told that -Os might drop performance comparing with -O2, >> so why bother? > > It depends on your workload. This machine spends most of its life > running rsync over ssh, so having more useful code in L2 cache makes a > difference even if it takes a few more clock cycles to run. The > bottleneck is IO - mainly disc, but also network (way better with > polling - don't know why it's not in GENERIC) so those extra clock > cycles aren't as important. > > If you're doing real number-crunching stuff with it, a) why?!, Isn't ssh encryption/decryption "real" number-crunching? My i586 system did IPSec encryption for tunnels and despite of having AES-CBC-128 onboard hardware accelerator suitable for IPSec, it was quite limited by CPU horsepower, so I used hmac-md5 instead of SHA variants to speed-up things a bit and still, it was able to saturate 100Mbps with unencrypted PPPoE+ipfw nat but only 33Mbps with IPSec AES+MD5 and less with SHA. > and b) yes, -O2 would be better. > >>> 11.4 i386 doesn't honour `vfs.zfs.arc_max` in any meaningful way >>> resulting in one of the `find`s in periodic security wedging ZFS. >> >> Been there, seen that. The problem pre-dates 11.x series and appeared in >> 10.x. >> It was much better in times of 9.x, though. > > Yes, I've also seen it before, but 11.4 is particularly egregious with > its disregard on i386 - amd64 is behaving itself on everything I've tried. > >>> options KSTACK_PAGES=4 >> >> This is default for 12.x/i386 but was not merged to stable/11. > > That's good to know. > >> You might find useful these also: >> >> vfs.zfs.vdev.cache.size="8M" >> vfs.zfs.prefetch_disable="1" > > Prefetch is disabled on i386 by default; I played with the vdev cache > size too but it made no discernable difference. > >> And for kernel config: >> >> options KVA_PAGES=512 >> >> This makes ZFS more stable giving it bigger kernel virtual area with less >> utilization for it. > > Yes, that's the theory, but when I tried it on earlier versions the > kernel would just blow up on boot. I may try it again next time I update > the kernel. It worked for me with KVA_PAGES=768 even, so kernel and ZFS had 3GB for virtual area and only 1GB for userland but that was i386 virtual machine with 512M or 1GB RAM only (cheapest hoster provided some years ago), so 1GB for userland was just fine :-) >>> TL;DR: I'd avoid 11.4 i386 as it doesn't appear to have been tested on >>> i486/i586 at all. >> >> Finally I gave up running ZFS for vmem-contrained systems, >> e.g. I moved my i386 virtual machines that benefit from ZFS compression to >> amd64. > > Oh, sure - I'd never dream of doing anything fancy with ZFS on this > machine - it just does backups. > > My point was that 11.4 doesn't "feel" right - it doesn't seem to have > been tested on of FreeBSD since 2.x - some "feel" right, some "feel" dodgy (5.3, 6.x, > some 8.x IIRC). On amd64 it "feels" great, but I don't really trust it > on i386. Can't see any difference with early 11.x versions but I build it all by myself for many major releases. ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
On 2020-09-14 04:41, Eugene Grosbein wrote: > Build time for modern FreeBSD version is too gross and needs way too much > memory, > so I stopped building image for my i586 hardware "in place" quite long ago. It would take about a week to compile, but since it was stable <11.4 I didn't really care. > I use my i7-based desktop to build NanoBSD image for its upgrade, it works > just fine > by default setting ARCH/TARGET_ARCH to i386 and correct CPUTYPE. I tried that on my workstation (dual Xeon, lots of RAM) but couldn't get it to compile for i586 - various things kept bailing out. It would compile i386 for i686, but not i586. Have you tried it with 11.4? >> Don't try -Os or -Oz (which really do help on a real i586) - ZFS will >> wedge quickly. > > Currently i586-class CPU needs as many speed improvements as possible > and I was told that -Os might drop performance comparing with -O2, > so why bother? It depends on your workload. This machine spends most of its life running rsync over ssh, so having more useful code in L2 cache makes a difference even if it takes a few more clock cycles to run. The bottleneck is IO - mainly disc, but also network (way better with polling - don't know why it's not in GENERIC) so those extra clock cycles aren't as important. If you're doing real number-crunching stuff with it, a) why?!, and b) yes, -O2 would be better. >> 11.4 i386 doesn't honour `vfs.zfs.arc_max` in any meaningful way >> resulting in one of the `find`s in periodic security wedging ZFS. > > Been there, seen that. The problem pre-dates 11.x series and appeared in 10.x. > It was much better in times of 9.x, though. Yes, I've also seen it before, but 11.4 is particularly egregious with its disregard on i386 - amd64 is behaving itself on everything I've tried. >> options KSTACK_PAGES=4 > > This is default for 12.x/i386 but was not merged to stable/11. That's good to know. > You might find useful these also: > > vfs.zfs.vdev.cache.size="8M" > vfs.zfs.prefetch_disable="1" Prefetch is disabled on i386 by default; I played with the vdev cache size too but it made no discernable difference. > And for kernel config: > > options KVA_PAGES=512 > > This makes ZFS more stable giving it bigger kernel virtual area with less > utilization for it. Yes, that's the theory, but when I tried it on earlier versions the kernel would just blow up on boot. I may try it again next time I update the kernel. >> TL;DR: I'd avoid 11.4 i386 as it doesn't appear to have been tested on >> i486/i586 at all. > > Finally I gave up running ZFS for vmem-contrained systems, > e.g. I moved my i386 virtual machines that benefit from ZFS compression to > amd64. Oh, sure - I'd never dream of doing anything fancy with ZFS on this machine - it just does backups. My point was that 11.4 doesn't "feel" right - it doesn't seem to have been tested on https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
14.09.2020 4:30, Charles Lecklider via freebsd-stable wrote: > On 2020-09-01 03:20, Eugene Grosbein wrote: >> CPU: Geode(TM) Integrated Processor by AMD PCS (499.91-MHz 586-class CPU) >> Origin="AuthenticAMD" Id=0x5a2 Family=0x5 Model=0xa Stepping=2 >> Features=0x88a93d >> AMD Features=0xc040 >> >> Of course, its NanoBSD image is built with CPUTYPE=i586 as well as installed >> packages. > > You've got CMOV so you'll be fine with 11.4-RELEASE, whereas I don't: > > CPU: Pentium/P55C (232.68-MHz 586-class CPU) > Origin="GenuineIntel" Id=0x543 Family=0x5 Model=0x4 Stepping=3 > Features=0x8003bf > > Getting 11.4 to work has been a *major* undertaking: > > Setting CPUTYPE in `make.conf` doesn't set -march for the kernel build > tools, so while you can compile world and kernel for i386, you can't > actually link it to create a working kernel on a real i586 (I found that > out with `/usr/obj` and `/usr/src` mounted over NFS). Build time for modern FreeBSD version is too gross and needs way too much memory, so I stopped building image for my i586 hardware "in place" quite long ago. I use my i7-based desktop to build NanoBSD image for its upgrade, it works just fine by default setting ARCH/TARGET_ARCH to i386 and correct CPUTYPE. > Don't try -Os or -Oz (which really do help on a real i586) - ZFS will > wedge quickly. Currently i586-class CPU needs as many speed improvements as possible and I was told that -Os might drop performance comparing with -O2, so why bother? > 11.4 i386 doesn't honour `vfs.zfs.arc_max` in any meaningful way > resulting in one of the `find`s in periodic security wedging ZFS. Been there, seen that. The problem pre-dates 11.x series and appeared in 10.x. It was much better in times of 9.x, though. > By ZFS wedging I mean something gets stuck on zio->i having eaten all of > kmem (I presume - on the real i586 I expect ~200M in ARC would do that > and that's what `top` shows) and that's it for the machine - it doesn't > panic, but there's no more disk IO either. > This also happens in VirtualBox with 4GB RAM, so it's not a hardware > problem. > > To get 11.4 to behave itself (and it's only been 72 hours so still early > days) I've had to set: > > options KSTACK_PAGES=4 This is default for 12.x/i386 but was not merged to stable/11. > and > > kern.maxvnode="1024" > vm.kmem_size="330M" > vm.kmem_size_max="330M" > vfs.zfs.arc_max="125M" > vfs.zfs.arc_meta_limit="16M" > > I had to set kmem for 11.3 but not the rest. You might find useful these also: vfs.zfs.vdev.cache.size="8M" vfs.zfs.prefetch_disable="1" And for kernel config: options KVA_PAGES=512 This makes ZFS more stable giving it bigger kernel virtual area with less utilization for it. > TL;DR: I'd avoid 11.4 i386 as it doesn't appear to have been tested on > i486/i586 at all. Finally I gave up running ZFS for vmem-contrained systems, e.g. I moved my i386 virtual machines that benefit from ZFS compression to amd64. As for real i586-class hardware uncapable of Long Mode, I do not use ZFS at all, UFS2/async+gjournal works just fine there including WiFi, IPSec etc. but no ZFS. ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
On 2020-09-01 03:20, Eugene Grosbein wrote: > CPU: Geode(TM) Integrated Processor by AMD PCS (499.91-MHz 586-class CPU) > Origin="AuthenticAMD" Id=0x5a2 Family=0x5 Model=0xa Stepping=2 > Features=0x88a93d > AMD Features=0xc040 > > Of course, its NanoBSD image is built with CPUTYPE=i586 as well as installed > packages. You've got CMOV so you'll be fine with 11.4-RELEASE, whereas I don't: CPU: Pentium/P55C (232.68-MHz 586-class CPU) Origin="GenuineIntel" Id=0x543 Family=0x5 Model=0x4 Stepping=3 Features=0x8003bf Getting 11.4 to work has been a *major* undertaking: Setting CPUTYPE in `make.conf` doesn't set -march for the kernel build tools, so while you can compile world and kernel for i386, you can't actually link it to create a working kernel on a real i586 (I found that out with `/usr/obj` and `/usr/src` mounted over NFS). clang is hypersensitive to command line arguments - if you just fiddle with CFLAGS etc you'll get a duplicate -O2 and/or -march and that results in a kernel/world that can't build itself (ZFS wedges). You must set NO_CPU_CFLAGS= and then each of CFLAGS/CXXFLAGS/COPTFLAGS. Don't try -Os or -Oz (which really do help on a real i586) - ZFS will wedge quickly. 11.4 i386 doesn't honour `vfs.zfs.arc_max` in any meaningful way resulting in one of the `find`s in periodic security wedging ZFS. By ZFS wedging I mean something gets stuck on zio->i having eaten all of kmem (I presume - on the real i586 I expect ~200M in ARC would do that and that's what `top` shows) and that's it for the machine - it doesn't panic, but there's no more disk IO either. This also happens in VirtualBox with 4GB RAM, so it's not a hardware problem. To get 11.4 to behave itself (and it's only been 72 hours so still early days) I've had to set: options KSTACK_PAGES=4 and kern.maxvnode="1024" vm.kmem_size="330M" vm.kmem_size_max="330M" vfs.zfs.arc_max="125M" vfs.zfs.arc_meta_limit="16M" I had to set kmem for 11.3 but not the rest. TL;DR: I'd avoid 11.4 i386 as it doesn't appear to have been tested on i486/i586 at all. Unfortunately I'm not sure 12 is an option yet either - 12.0 chokes on my SiI3124 SATA controllers, and I've yet to compile 12.1 to test (RELEASE has the same problem as 11.4). -C ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
01.09.2020 6:50, Charles Lecklider via freebsd-stable wrote: > On 2020-08-31 21:51, Christoph Moench-Tegeder wrote: >> Given that the hardware notes >> https://www.freebsd.org/releases/11.4R/hardware.html#proc-i386 >> explicitely claim 80486 compatibility, I'd guess this was not a >> consciuous decision but a blunder which wasn't caught > > Hmm I'd assumed that each release was at least *booted* on a real > 486, but obviously not. > >> (perhaps as >> hardware of that vintage is getting rare these days. That box >> must be old enough to buy alcohol by now?). > > Yes, I first built it in 1998 IIRC; it ran Windows for a few years > (desktop, gaming) then FreeBSD from about 2002. It's been in nearly > continuous operation since, although I did have to replace the PSU last > year ;-) Just for the record, I still run AMD Geode-based FreeBSD 11.3-STABLE as my home WiFi Access Point. This fanless system has 500Mhz 32-bit only i586-class CPU, 1GB RAM (no swap), two 100M wired NICs vr0/vr1 onboard and Atheros 5212 ath0 wireless mini-PCI card (not PCI-E) that supports multiple WLANs. It has CompactFlash slot for ada0 occupied with 16GB CF card at boot device (nanobsd, r/o mounted) plus ada1 2.5" PATA HDD WDC WD2500BEVE (250G) as additional storage. Until recently is also served as PPPoE client (mpd5 kernel level) for uplink with ipfw nat being able to saturate 100MBit/s uplink connection despite of NAT+PPPoE overhead. CPU: Geode(TM) Integrated Processor by AMD PCS (499.91-MHz 586-class CPU) Origin="AuthenticAMD" Id=0x5a2 Family=0x5 Model=0xa Stepping=2 Features=0x88a93d AMD Features=0xc040 Of course, its NanoBSD image is built with CPUTYPE=i586 as well as installed packages. ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
On 2020-08-31 21:51, Christoph Moench-Tegeder wrote: > Given that the hardware notes > https://www.freebsd.org/releases/11.4R/hardware.html#proc-i386 > explicitely claim 80486 compatibility, I'd guess this was not a > consciuous decision but a blunder which wasn't caught Hmm I'd assumed that each release was at least *booted* on a real 486, but obviously not. > (perhaps as > hardware of that vintage is getting rare these days. That box > must be old enough to buy alcohol by now?). Yes, I first built it in 1998 IIRC; it ran Windows for a few years (desktop, gaming) then FreeBSD from about 2002. It's been in nearly continuous operation since, although I did have to replace the PSU last year ;-) -C ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
On 2020-08-31 22:59, Eugene Grosbein wrote: > https://www.freebsd.org/releases/12.1R/relnotes.html section 11.1 discusses > the topic > in context of upcoming FreeBSD 13. And that makes sense - 12.x EOL is a few years out. > There was some discussion on the topic in the freebsd-arch mailing list about > a year ago: > https://docs.freebsd.org/cgi/getmsg.cgi?fetch=75083+0+archive/2019/freebsd-arch/20191006.freebsd-arch Yes, and there was a commitment to an announcement in that thread too. > It seems there was no announcement on the change for 11.x branch. No, and that's the problem: had I know I'd have just compiled it up, but I didn't and now I need to rebuild it > I wonder how much RAM does your Pentium 233 MMX have and what part of the RAM > is cacheable by the chipset? :-) 512MB ECC, and all of it, at least according to the BIOS. -C ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
01.09.2020 1:00, Charles Lecklider via freebsd-stable wrote: > Having just trashed my dual Pentium 233 MMX by installing 11.4 I went > digging to find the cause - and I found it. > > As you can see from objdump below, clearly a decision was made that 11.4 > would no longer support anything before a i686. > > Ignoring for a moment the wisdom of making breaking changes to a minor > version, please could someone tell me where the announcement was made? https://www.freebsd.org/releases/12.1R/relnotes.html section 11.1 discusses the topic in context of upcoming FreeBSD 13. There was some discussion on the topic in the freebsd-arch mailing list about a year ago: https://docs.freebsd.org/cgi/getmsg.cgi?fetch=75083+0+archive/2019/freebsd-arch/20191006.freebsd-arch It seems there was no announcement on the change for 11.x branch. I wonder how much RAM does your Pentium 233 MMX have and what part of the RAM is cacheable by the chipset? :-) ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
On 2020-08-31 21:54, Warner Losh wrote: > The default is to compile for i686. That does seem to be the case for 11.4, but that wasn't the case for 11.3, and I cannot find an announcement for that change anywhere. Had I seen one ... > However, you can still roll your own if you need anything back to i486. ... that's what I would have done. -C ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
## Warner Losh ([email protected]): > The default is to compile for i686. That's really not obvious from the obvious documentation. Both Hardware Notes and the Handbook ("2.2. Minimum Hardware Requirements") state support for 486 and up, but there's no hint that you had to roll you own distribution set for anything below 686. Also, GENERIC's conf on i386 has flags I486_CPU and up. Regards, Christoph -- Spare Space ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
On Mon, Aug 31, 2020 at 2:51 PM Christoph Moench-Tegeder wrote: > ## Charles Lecklider via freebsd-stable ([email protected]): > > > As you can see from objdump below, clearly a decision was made that 11.4 > > would no longer support anything before a i686. > > Given that the hardware notes > https://www.freebsd.org/releases/11.4R/hardware.html#proc-i386 > explicitely claim 80486 compatibility, I'd guess this was not a > consciuous decision but a blunder which wasn't caught (perhaps as > hardware of that vintage is getting rare these days. That box > must be old enough to buy alcohol by now?). > The default is to compile for i686. However, you can still roll your own if you need anything back to i486. Warner ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Cannot find announcement that min supported i386 CPU is now i686
## Charles Lecklider via freebsd-stable ([email protected]): > As you can see from objdump below, clearly a decision was made that 11.4 > would no longer support anything before a i686. Given that the hardware notes https://www.freebsd.org/releases/11.4R/hardware.html#proc-i386 explicitely claim 80486 compatibility, I'd guess this was not a consciuous decision but a blunder which wasn't caught (perhaps as hardware of that vintage is getting rare these days. That box must be old enough to buy alcohol by now?). Regards, Christoph -- Spare Space ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
