Re: [xmail] Tip: Speeding up xmail by pointing envvar XMAIL_TEMP to a ramdisk
Are you sure this will actually speed up XMail? Have you been able to compare, specifically for XMail? As far as I understand how XMail processes e-mails, it actually only creates a file one time and then moves that file around between temp-folder, spool folder and local POP3 folders. Moving a file on the same filesystem means the file isn't re-rewritten. If you move a file between different filesystems, it does need to be re-written. That could actually mean that having the temp folder on a RAM-disk might slow things down if there would be an event in which XMail moves a file from spool to temp to local POP3 folder. When they are all on the same filesystem, this would mean the file would need to be written only once (and then moved around). If the temp folder is on a different file-system, this would cause the file to be writting to the hard drive twice and to the RAM-drive once. All mails will not only pass through the temp-folder, but will also end up in the spool-folder. If most mails only pass through the server (and don't need to be stored locally) then you might be able to see a speed improvement by also placing the spool folder on a ramdisk. But this would mean that all mails pending delivery are lost at server restart (so this isn't really recommended). Sincerely, Bart Mortelmans U.Mutlu via xmail schreef op 25/05/15 om 01:09: Tip: Speeding up xmail by pointing envvar XMAIL_TEMP to a ramdisk (here on Linux using a 64 MB ramdisk mounted to /mnt/ramdisk ) mkdir -p /mnt/ramdisk in /etc/fstab: tmpfs /mnt/ramdisk tmpfs nodev,size=64M 0 0 then either reboot or use this cmd: mount /mnt/ramdisk in xmail start script: export XMAIL_ROOT=/var/MailRoot export XMAIL_TEMP=/mnt/ramdisk export PATH=$XMAIL_ROOT/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ... and restart xmail. The ramdisk size should be bigger than MaxMessageSize (unit KB) in server.tab. This will speed up mail processing (at least incoming mails) significantly. Here's more info about speed gains: http://www.jamescoyle.net/knowledge/951-the-difference-between-a-tmpfs-and-ramfs-ram-disk The major benefit to memory based file systems is that they are very fast – 10s of times faster than modern SSDs. Read and write performance is massively increased for all workload types. One can even copy the mostly called programs (for example filters) onto the ramdisk, but then one must use such a PATH line (instead of the above one): export PATH=/mnt/ramdisk:$XMAIL_ROOT/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ___ xmail mailing list xmail@xmailserver.org http://xmailserver.org/mailman/listinfo/xmail
Re: [xmail] Tip: Speeding up xmail by pointing envvar XMAIL_TEMP to a ramdisk
Bart Mortelmans via xmail wrote on 05/25/2015 02:24 PM: Are you sure this will actually speed up XMail? Have you been able to compare, specifically for XMail? It will help for files xmail creates in /tmp and doesn't move to an other drive (for example when the realtime filters get fired during mail acceptance, the passed file always lies in /tmp). Since a /tmp on a ramdisk is ultra-fast I concluded that this will speed things up. But I'm not sure yet if xmail moves that file to main disk. As far as I understand how XMail processes e-mails, it actually only creates a file one time and then moves that file around between temp-folder, spool folder and local POP3 folders. Moving a file on the same filesystem means the file isn't re-rewritten. If you move a file between different filesystems, it does need to be re-written. That could actually mean that having the temp folder on a RAM-disk might slow things down if there would be an event in which XMail moves a file from spool to temp to local POP3 folder. When they are all on the same filesystem, this would mean the file would need to be written only once (and then moved around). If the temp folder is on a different file-system, this would cause the file to be writting to the hard drive twice and to the RAM-drive once. Hmm. yes, true. But there are use cases where ramdisk will be w/o question useful, for example starting the filter-programs from ramdisk. All mails will not only pass through the temp-folder, but will also end up in the spool-folder. If most mails only pass through the server (and don't need to be stored locally) then you might be able to see a speed improvement by also placing the spool folder on a ramdisk. But this would mean that all mails pending delivery are lost at server restart (so this isn't really recommended). Yes, true, putting spool on ramdisk is not that a good idea :-) cu Uenal Sincerely, Bart Mortelmans U.Mutlu via xmail schreef op 25/05/15 om 01:09: Tip: Speeding up xmail by pointing envvar XMAIL_TEMP to a ramdisk (here on Linux using a 64 MB ramdisk mounted to /mnt/ramdisk ) mkdir -p /mnt/ramdisk in /etc/fstab: tmpfs /mnt/ramdisk tmpfs nodev,size=64M 0 0 then either reboot or use this cmd: mount /mnt/ramdisk in xmail start script: export XMAIL_ROOT=/var/MailRoot export XMAIL_TEMP=/mnt/ramdisk export PATH=$XMAIL_ROOT/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ... and restart xmail. The ramdisk size should be bigger than MaxMessageSize (unit KB) in server.tab. This will speed up mail processing (at least incoming mails) significantly. Here's more info about speed gains: http://www.jamescoyle.net/knowledge/951-the-difference-between-a-tmpfs-and-ramfs-ram-disk The major benefit to memory based file systems is that they are very fast – 10s of times faster than modern SSDs. Read and write performance is massively increased for all workload types. One can even copy the mostly called programs (for example filters) onto the ramdisk, but then one must use such a PATH line (instead of the above one): export PATH=/mnt/ramdisk:$XMAIL_ROOT/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ___ xmail mailing list xmail@xmailserver.org http://xmailserver.org/mailman/listinfo/xmail
[xmail] BUG : XMAIL 1.27 with SSL
I found the bug on XMAIL 1.27 with SSL (POP3S is main), SSMTP servicemaybe has asame bugtoo 1. it will have too many FIN_WAIT_2on POP3Swith two many requests, It is becauseof some non-popular WINDOWS CLIENT e.g. tencent foxmail , outlook will not have this issue. 2. If you're using XMAIL'Sinternal SSL POP3S with any openssl lib. (tested 0.9.8za 1.0.0, tested gcc 4.0.2/4.0.4/4.3.6)and the email accountcontain too many emails (1000-2000 emails),the mobile clientPOP3such as Samsung internal Email Client is using SSL, K9-MAIL on any android devicesusing SSL, some iPhone devices (connect with XMAIL SSL port 995), will not download them successfully (Android Error - Java Exception Error) and some iPhonedevicewill stuck with it, it will not download all emails successfully. But if you disable XMAIL'S internal POP3Swith -B-startup option, and switch stunnel5.17 port 995 tunnel toport 110, it will solve this problem and kick outFIN_WAIT_2 * stunnel 5.17 config * [pop3s]accept = 995connect = 110cert = /etc/stunnel/stunnel.pem *** ___ xmail mailing list xmail@xmailserver.org http://xmailserver.org/mailman/listinfo/xmail
Re: [xmail] BUG : XMAIL 1.27 with SSL
Oversea Site via xmail wrote on 05/25/2015 09:23 PM: I found the bug on XMAIL 1.27 with SSL (POP3S is main), SSMTP service maybe has a same bug too 1. it will have too many FIN_WAIT_2 on POP3S with two many requests, It is because of some non-popular WINDOWS CLIENT e.g. tencent foxmail , outlook will not have this issue. 2. If you're using XMAIL'S internal SSL POP3S with any openssl lib. (tested 0.9.8za 1.0.0, tested gcc 4.0.2/4.0.4/4.3.6) and the email account contain too many emails (1000-2000 emails), the mobile client POP3 such as Samsung internal Email Client is using SSL, K9-MAIL on any android devices using SSL, some iPhone devices (connect with XMAIL SSL port 995), will not download them successfully (Android Error - Java Exception Error) and some iPhone device will stuck with it, it will not download all emails successfully. But if you disable XMAIL'S internal POP3S with -B- startup option, and switch stunnel 5.17 port 995 tunnel to port 110, it will solve this problem and kick out FIN_WAIT_2 * stunnel 5.17 config * [pop3s] accept = 995 connect = 110 cert = /etc/stunnel/stunnel.pem *** What do the logs (server, client) say about the errors? I think you need a faster server HW and/or need to optimize the resources, for example the number of threads, and possibly also RAM. An excerpt from the doc: If you have an heavily loaded server remember to setup the best number of XMAIL threads by specifying the '-Qn nthreads' option (you must do some tentatives to find the best value for your needs). Also you can limit the number of SMTP, POP3 and CTRL service threads by specifying the options '-SX maxthreads', '-PX maxthreads' and '-CX maxthreads'. If there are mail filters, you can put them into a ramdisk and set PATH accordingly, or use an absolute path to them. See also XMAIL_TEMP in the doc. In task manager / system monitor you can see how the resource usage is, especially CPU, memory, and network. -- cu Uenal ___ xmail mailing list xmail@xmailserver.org http://xmailserver.org/mailman/listinfo/xmail
Re: [xmail] BUG : XMAIL 1.27 with SSL
U.Mutlu via xmail wrote on 05/26/2015 07:06 AM: Oversea Site via xmail wrote on 05/26/2015 05:14 AM: No Log Error. No Fatal Error log. No Core Dump It's not RAM or HW problem. It's XMAIL internal issue, because when I switch to stunnel, all problems solved. And then when I test K9-MAIL with XMAIL+STUNNEL, K9 Client will not have any JAVA somethings Exception Error Two tests 1. Very Important Thing in Xmail 1.27 Native SSL If the emailbox has many emails such as 1, Microsoft Outlook with SSL or NON-SSL download them, it WILL NOT have this issue. Some Tencent Foxmail for Windows, after finish ssl session, it will take the server be FIN_WAIT2 (some version, not all) But K9-MAIL for Android, Download first 1000, it will appear the error in K9-MAIL randomly. 2. Xmail 1.27 + Stunnel Proxy If the emailbox has many emails such as 1, Microsoft Outlook with SSL or NON-SSL download them, it WILL NOT have this issue. Some Tencent Foxmail for Windows, after finish ssl session, it WILL NOT have any problem. Server WILL NOT TAKE FIN_WAIT_2 K9-MAIL for Android, Download first 1000, it will NOT DISPLAY any error when it finished download emails. I would need full description about the server HW and SW (OS version, xmail start params etc., any xmail patches applied etc.). There is of course also the possibility that the bug lies in the said mobile client program, especially if it goes online via unstable mobile links as it seems to be a link latency problem, or the client does not close the socket cleanly. Take a look at these: https://github.com/Automattic/socket.io/issues/1380 http://stackoverflow.com/questions/5328155/preventing-fin-wait2-when-closing-socket Check this too: # cat /proc/sys/net/ipv4/tcp_fin_timeout 60 The following page is informative too: http://httpd.apache.org/docs/2.0/misc/fin_wait_2.html -- cu Uenal ___ xmail mailing list xmail@xmailserver.org http://xmailserver.org/mailman/listinfo/xmail
Re: [xmail] BUG : XMAIL 1.27 with SSL
FEDORA 4 X86_64 GLIBC 2.3.6-3 (I can't upgrade it to the lastest version. Because I have a chiliasp 3.6.2 32-bit run ASP VBSCRIPT on APACHE 1.3 32-bit software, it need GLIBC 2.3.x and Kernel 2.6) KERNEL 3.3 MOD FROM FC15 (It will display KERNEL 2.6.43 Number). Kernel is not a main problem. Because I have been test 2.6.32 or other. same problem was found. REAL HARDWARE - INTEL B85M i3-4150 16G RAM SATA3 RAID 1 LINUX SOFTRAID1 MDADM Corp. Email System is not heavily use when I test it in HK TIME midnight. Anyway I will test this issue in other linux distribution in VM, such as CentOS. (You can give me any linux distribution, let's build the same platform in VM. I use virtualbox. [root@bravo logs]# uname -a Linux bravo 2.6.43-gcc402 #3 SMP Sun May 17 23:10:29 HKT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@bravo logs]# [root@bravo logs]# ldd /var/MailRoot/bin/XMail127v3 linux-vdso.so.1 = (0x77f0e000) libssl.so.0.9.8 = /usr/local/ssl/lib/libssl.so.0.9.8 (0x7f43b00bc000) libcrypto.so.0.9.8 = /usr/local/ssl/lib/libcrypto.so.0.9.8 (0x7f43afd35000) libdl.so.2 = /lib64/libdl.so.2 (0x003e1820) libpthread.so.0 = /lib64/libpthread.so.0 (0x003e1860) libstdc++.so.6 = /usr/lib64/libstdc++.so.6 (0x7f43afa37000) libm.so.6 = /lib64/libm.so.6 (0x003e1800) libgcc_s.so.1 = /lib64/libgcc_s.so.1 (0x003e1940) libc.so.6 = /lib64/libc.so.6 (0x003e17d0) /lib64/ld-linux-x86-64.so.2 (0x003e17b0) *** /var/MailRoot/bin/XMail127v3 -Md -Ms /var/MailRoot -Sl -Pl -Cl \ -SI 0.0.0.0:8025 -SI 0.0.0.0:25 \ -PI 0.0.0.0:8110 -PI 0.0.0.0:110 \ -BI 0.0.0.0:995 \ -XI 0.0.0.0:465 \ -Ql -Ln 256 -Pt 60 -St 150 -F- -SX 2048 -Mx 64 \ -Qn 64 -PX 2048 -Qr 1 -Qt 1 -QT 150 -Qg -CX 256 *** Original Message From: U.Mutlu um4...@mutluit.com To: t...@os.st, XMail Users Mailing List xmail@xmailserver.org Sent: Tue, May 26, 2015, 1:06 PM Subject: Re: [xmail] BUG : XMAIL 1.27 with SSL Oversea Site via xmail wrote on 05/26/2015 05:14 AM: No Log Error. No Fatal Error log. No Core Dump It's not RAM or HW problem. It's XMAIL internal issue, because when I switch to stunnel, all problems solved. And then when I test K9-MAIL with XMAIL+STUNNEL, K9 Client will not have any JAVA somethings Exception Error Two tests 1. Very Important Thing in Xmail 1.27 Native SSL If the emailbox has many emails such as 1, Microsoft Outlook with SSL or NON-SSL download them, it WILL NOT have this issue. Some Tencent Foxmail for Windows, after finish ssl session, it will take the server be FIN_WAIT2 (some version, not all) But K9-MAIL for Android, Download first 1000, it will appear the error in K9-MAIL randomly. 2. Xmail 1.27 + Stunnel Proxy If the emailbox has many emails such as 1, Microsoft Outlook with SSL or NON-SSL download them, it WILL NOT have this issue. Some Tencent Foxmail for Windows, after finish ssl session, it WILL NOT have any problem. Server WILL NOT TAKE FIN_WAIT_2 K9-MAIL for Android, Download first 1000, it will NOT DISPLAY any error when it finished download emails. I would need full description about the server HW and SW (OS version, xmail start params etc., any xmail patches applied etc.). There is of course also the possibility that the bug lies in the said mobile client program, especially if it goes online via unstable mobile links as it seems to be a link latency problem, or the client does not close the socket cleanly. Take a look at these: https://github.com/Automattic/socket.io/issues/1380 http://stackoverflow.com/questions/5328155/preventing-fin-wait2-when-closing-socket cu Uenal Original Message From: U.Mutlu um4...@mutluit.com To: t...@os.st, XMail Users Mailing List xmail@xmailserver.org Sent: Tue, May 26, 2015, 10:11 AM Subject: Re: [xmail] BUG : XMAIL 1.27 with SSL Oversea Site via xmail wrote on 05/25/2015 09:23 PM: I found the bug on XMAIL 1.27 with SSL (POP3S is main), SSMTP service maybe has a same bug too 1. it will have too many FIN_WAIT_2 on POP3S with two many requests, It is because of some non-popular WINDOWS CLIENT e.g. tencent foxmail , outlook will not have this issue. 2. If you're using XMAIL'S internal SSL POP3S with any openssl lib. (tested 0.9.8za 1.0.0, tested gcc 4.0.2/4.0.4/4.3.6) and the email account contain too many emails (1000-2000 emails), the mobile client POP3 such as Samsung internal Email Client is using SSL, K9-MAIL on any android devices using SSL, some iPhone devices (connect with XMAIL SSL port 995), will not download them successfully (Android Error - Java Exception Error) and some iPhone device will stuck with it, it will not download all emails successfully. But if you disable XMAIL'S internal POP3S with -B- startup option, and
Re: [xmail] BUG : XMAIL 1.27 with SSL
No Log Error. No Fatal Error log. No Core Dump It's not RAM or HW problem. It's XMAIL internal issue, because when I switch to stunnel, all problems solved. And then when I test K9-MAIL with XMAIL+STUNNEL, K9 Client will not have any JAVA somethings Exception Error Two tests 1. Very Important Thing in Xmail 1.27 Native SSL If the emailbox has many emails such as 1, Microsoft Outlook with SSL or NON-SSL download them, it WILL NOT have this issue. Some Tencent Foxmail for Windows, after finish ssl session, it will take the server be FIN_WAIT2 (some version, not all) But K9-MAIL for Android, Download first 1000, it will appear the error in K9-MAIL randomly. 2. Xmail 1.27 + Stunnel Proxy If the emailbox has many emails such as 1, Microsoft Outlook with SSL or NON-SSL download them, it WILL NOT have this issue. Some Tencent Foxmail for Windows, after finish ssl session, it WILL NOT have any problem. Server WILL NOT TAKE FIN_WAIT_2 K9-MAIL for Android, Download first 1000, it will NOT DISPLAY any error when it finished download emails. Original Message From: U.Mutlu um4...@mutluit.com To: t...@os.st, XMail Users Mailing List xmail@xmailserver.org Sent: Tue, May 26, 2015, 10:11 AM Subject: Re: [xmail] BUG : XMAIL 1.27 with SSL Oversea Site via xmail wrote on 05/25/2015 09:23 PM: I found the bug on XMAIL 1.27 with SSL (POP3S is main), SSMTP service maybe has a same bug too 1. it will have too many FIN_WAIT_2 on POP3S with two many requests, It is because of some non-popular WINDOWS CLIENT e.g. tencent foxmail , outlook will not have this issue. 2. If you're using XMAIL'S internal SSL POP3S with any openssl lib. (tested 0.9.8za 1.0.0, tested gcc 4.0.2/4.0.4/4.3.6) and the email account contain too many emails (1000-2000 emails), the mobile client POP3 such as Samsung internal Email Client is using SSL, K9-MAIL on any android devices using SSL, some iPhone devices (connect with XMAIL SSL port 995), will not download them successfully (Android Error - Java Exception Error) and some iPhone device will stuck with it, it will not download all emails successfully. But if you disable XMAIL'S internal POP3S with -B- startup option, and switch stunnel 5.17 port 995 tunnel to port 110, it will solve this problem and kick out FIN_WAIT_2 * stunnel 5.17 config * [pop3s] accept = 995 connect = 110 cert = /etc/stunnel/stunnel.pem *** What do the logs (server, client) say about the errors? I think you need a faster server HW and/or need to optimize the resources, for example the number of threads, and possibly also RAM. An excerpt from the doc: If you have an heavily loaded server remember to setup the best number of XMAIL threads by specifying the '-Qn nthreads' option (you must do some tentatives to find the best value for your needs). Also you can limit the number of SMTP, POP3 and CTRL service threads by specifying the options '-SX maxthreads', '-PX maxthreads' and '-CX maxthreads'. If there are mail filters, you can put them into a ramdisk and set PATH accordingly, or use an absolute path to them. See also XMAIL_TEMP in the doc. In task manager / system monitor you can see how the resource usage is, especially CPU, memory, and network. -- cu Uenal ___ xmail mailing list xmail@xmailserver.org http://xmailserver.org/mailman/listinfo/xmail
