[SOGo] Request Failed when replying to message
We have a LMS student which can be used to send e-mail messages to our students. When a student clicks to reply to any of these messages Sogo displays "Request Failed" and the logs show a 501 error on the reply get request. We are using v4.3.2 of Sogo. The messages themselves come from a "do-not-reply" address with a reply-to header containing a real e-mail address. If tested this by sending a test message with reply-to header to my test account, replies work fine. Has anyone seen a problem like this? I assume there is some header which is confusing Sogo. This problem impacts all messages sent from the LMS. An example of the source of one of these messages is: Return-Path: Delivered-To: ... Received: from barracuda.luzerne.edu (unknown [10.3.1.4]) by stmail.luzerne.edu (Postfix) with ESMTPS id 4BhSdX6p7Mz5HYlXL for <...>; Wed, 2 Sep 2020 11:33:36 -0400 (EDT) X-ASG-Debug-ID: 1599060816-072430105047ceb0001-QGxhuQ Received: from outbound-ip107a.ess.barracuda.com (outbound-ip107a.ess.barracuda.com [209.222.82.249]) by barracuda.luzerne.edu with ESMTP id tEa4Basao090jgfX (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 02 Sep 2020 11:33:36 -0400 (EDT) X-Barracuda-Envelope-From: do-not-re...@blackboard.com X-Barracuda-RBL-Trusted-Forwarder: 209.222.82.249 Received: from mail-relay1-va2.blackboard.com (mail-relay1-va2.blackboard.com [69.196.241.1]) by mx118.us-east-2a.ess.aws.cudaops.com (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 02 Sep 2020 15:33:35 + Received: from ip-10-146-215-243.ec2.internal (ec2-54-197-63-203.compute-1.amazonaws.com [54.197.63.203]) by mail-relay1-va2.blackboard.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id 082FUAPl032471; Wed, 2 Sep 2020 15:30:10 GMT X-Barracuda-RBL-IP: 69.196.241.1 X-Barracuda-Effective-Source-IP: mail-relay1-va2.blackboard.com[69.196.241.1] X-Barracuda-Apparent-Source-IP: 69.196.241.1 X-Barracuda-BBL-IP: 69.196.241.1 DKIM-Filter: OpenDKIM Filter v2.7.5 mail-relay1-va2.blackboard.com 082FUAPl032471 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=blackboard.com; s=sep2018; t=1599060610; bh=HRzUC/Y5wCgAU7qzNW1Rkv0B8Tl11/fjmymZNNaj1aE=; h=Date:From:Reply-To:To:Subject; b=u5JplCqOs99seYFv5xOWSwqx6IHXKOGMZPAH/u/5uA9DDTK8Hg/QasVzm5QaXpRM0 AXt5ldmEE44g6hVI5klLfUGJC/gvXFoOTd024HHMzDyjQaMRfgWJ8imHOYrOCtmHRa pm6tCvRfOjB9VHNAirhNSxxZIDsbNPDNMGpCGvx0Z4cjX1NOk2iORshkcCuQPpsvu6 BLz9049hcG981A9AL5+q+4Nr9jIoxMSzOtdB1pRSQfL0fqRdBBVZXNOXHMofDCKeIn dDHrd6HTowJpATUNWYVWnhoaSqfs7o8MJbcF9tZNitFwHuDNR0eLFyJsVYqK27g4ih MSxgt99DDHH3A== Date: Wed, 2 Sep 2020 11:33:34 -0400 (EDT) From: "Joseph Nester - <...>" Reply-To: "Joseph Nester - <...>" <...> To: "EMAIL-TEST-000":; Message-ID: <71744606.2296.1599060814381.JavaMail.bbuser@ip-10-146-215-243> Subject: [EXTERNAL] Email Test Course: Test Message 3 MIME-Version: 1.0 X-ASG-Orig-Subj: Email Test Course: Test Message 3 Content-Type: multipart/alternative; boundary="=_Part_2294_716594908.1599060814121" X-BESS-ID: 1599060815-893234-3420-15812-1 X-BESS-VER: 2019.1_20200831.2045 X-BESS-Apparent-Source-IP: 69.196.241.1 X-Barracuda-Connect: outbound-ip107a.ess.barracuda.com[209.222.82.249] X-Barracuda-Start-Time: 1599060816 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Virus-Scanned: by bsmtpd at luzerne.edu X-Barracuda-Scan-Msg-Size: 35 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: Yes, SCORE=0.00 using global scores of TAG_LEVEL=0.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=4.5 tests=HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.84348 Rule breakdown below pts rule name description -- -- 0.00 HTML_MESSAGE BODY: HTML included in message X-Barracuda-Spam-Flag: YES --=_Part_2294_716594908.1599060814121 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Test message 3 --=_Part_2294_716594908.1599060814121 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit Test message 3 --=_Part_2294_716594908.1599060814121-- In the above source I've replaced e-mail addresses with ... Any suggestions on how to fix this problem would be appreciated. Thanks, Bob -- users@sogo.nuhttps://inverse.ca/sogo/lists
Re: [SOGo] Users can't configure clients to use ActiveSync
Ludovic,
I'm running SOGo 4.0.8 and SOPE 4.9.
I have found a solution to this problem. I'm running SOGO (both instances) using systemd. The EAS instance has been the only one with this problem.
My EAS instance service file it sogod-eas.service. I've created the file override.conf within /etc/systemd/system/sogod-eas.service.d. The file contains:
[Service]
LimitNOFILE=infinity
TasksMax=infinity
Once this file was in place I was able to create more than 508 child processes without a problem. I'm currently running with 700 eas processes.
Thanks,
Bob
Bob Dushok
Director of Enterprise Systems and Computer Labs
Luzerne County Community College
1-800-377-5222 ext 7327
bdus...@luzerne.edu
- Original message -
From: ""Ludovic Marcotte (lmarco...@inverse.ca)"
Sent by: users-requ...@sogo.nu
To: users@sogo.nu
Cc:
Subject: Re: [SOGo] Users can't configure clients to use ActiveSync
Date: Mon, Nov 11, 2019 11:17 AM
Hi Bob,
Which version of SOGo and SOPE are you using?
Thanks,
On 2019-11-11 10:42 a.m., bdus...@luzerne.edu wrote:
Michel,
Thanks for your response.
The Sogo logs don't appear to address the problem. I am seeing the error "No child available to handle incoming request!" around the time of the connection attempt, but I can't match it with the configuration attempt.
The users aren't using Office, they are using the ActiveSync support included with iOS and Android.
I'm not certain if I understand your third point. The URL I'm using is https. We have a CA signed cert installed in Nginx.
Since I posted my original message I've done a lot of additional troubleshooting. It seems like the "No child available to handle incoming request!" error may indicate there aren't enough child processes available. I've tried increasing WOWorkerCount unsuccessfully. When I attempt to increase this count I begin obtaining the "sogod [1716]: [ERROR] <0x0x5595d0273e50[WOHttpAdaptor]> failure notifying watchdog we are ready during events registration: NAME:NGSocketShutdownDuringWriteException REASON:the socket was shutdown INFO:{errno = 32; error = "Broken pipe"; stream = "{object = 0x5595d03818f0;}"; }" error.
I'm able to start exactly 508 workers. When I attempt 509 workers the error appears and SOGo shutdown quickly after startup.
I believe this may be OS related but I can't solve it. I've changed /etc/security/limits.d/20-nproc.conf to include:
sogo soft nproc unlimited
sogoeas soft nproc unlimited
root soft nproc unlimited
No difference.
I’ve also tried changing /etc/security/limits.conf to include:
sogo - stack 32768
sogo - nproc 8192
sogoeas - stack 32768
sogoeas - nproc 8192
Again, no change.
I'm researching thread/process limits per user in Linux. So far I haven't had any success.
Thanks again,
Bob
Bob Dushok
Director of Enterprise Systems and Computer Labs
Luzerne County Community College
1-800-377-5222 ext 7327
bdus...@luzerne.edu
- Original message -
From: ""Support (supp...@foxserv.be)"
Sent by: users-requ...@sogo.nu
To: users@sogo.nu
Cc:
Subject: Re: [SOGo] Users can't configure clients to use ActiveSync
Date: Sat, Nov 9, 2019 7:27 AM
Hello
3 things
First thing, what does Sogo's logs say?
Second thing, the Office version, is that the last one?
3rd thing, when you add the URL with the subdomain with /Microsoft-Server-ActiveSync, you add it without the "https", because if you add it, was that my mistake, it actually adds an error message.
Have a good day
Michel
Le Mercredi, Novembre 06, 2019 16:19 CET, "bdus...@luzerne.edu" a écrit:
We've recently begun receiving reports from some of our students indicating they can't configure their phones to access Sogo. I've verified this using iOS and an encountering the problem as well.
Users select to add a Microsoft Exchange account, enter their e-mail address, select to configure manually when prompted, enter their password when prompted, then click next. iOS displays "Verifying..." for a few seconds and then prompts for more information. They enter the correct server, username, and password then click next. After a pause of about a minute with "Verifying" displayed the error "Unable to verify account information" appears.
I've been checking both my nginx and sogo logs and can't find any details on the problem.
I'm running a separate instance for eas traffic as specified at https://sogo.nu/support/faq/
Re: [SOGo] Users can't configure clients to use ActiveSync
Michel,
Thanks for your response.
The Sogo logs don't appear to address the problem. I am seeing the error "No child available to handle incoming request!" around the time of the connection attempt, but I can't match it with the configuration attempt.
The users aren't using Office, they are using the ActiveSync support included with iOS and Android.
I'm not certain if I understand your third point. The URL I'm using is https. We have a CA signed cert installed in Nginx.
Since I posted my original message I've done a lot of additional troubleshooting. It seems like the "No child available to handle incoming request!" error may indicate there aren't enough child processes available. I've tried increasing WOWorkerCount unsuccessfully. When I attempt to increase this count I begin obtaining the "sogod [1716]: [ERROR] <0x0x5595d0273e50[WOHttpAdaptor]> failure notifying watchdog we are ready during events registration: NAME:NGSocketShutdownDuringWriteException REASON:the socket was shutdown INFO:{errno = 32; error = "Broken pipe"; stream = "{object = 0x5595d03818f0;}"; }" error.
I'm able to start exactly 508 workers. When I attempt 509 workers the error appears and SOGo shutdown quickly after startup.
I believe this may be OS related but I can't solve it. I've changed /etc/security/limits.d/20-nproc.conf to include:
sogo soft nproc unlimited
sogoeas soft nproc unlimited
root soft nproc unlimited
No difference.
I’ve also tried changing /etc/security/limits.conf to include:
sogo - stack 32768
sogo - nproc 8192
sogoeas - stack 32768
sogoeas - nproc 8192
Again, no change.
I'm researching thread/process limits per user in Linux. So far I haven't had any success.
Thanks again,
Bob
Bob Dushok
Director of Enterprise Systems and Computer Labs
Luzerne County Community College
1-800-377-5222 ext 7327
bdus...@luzerne.edu
- Original message -
From: ""Support (supp...@foxserv.be)"
Sent by: users-requ...@sogo.nu
To: users@sogo.nu
Cc:
Subject: Re: [SOGo] Users can't configure clients to use ActiveSync
Date: Sat, Nov 9, 2019 7:27 AM
Hello
3 things
First thing, what does Sogo's logs say?
Second thing, the Office version, is that the last one?
3rd thing, when you add the URL with the subdomain with /Microsoft-Server-ActiveSync, you add it without the "https", because if you add it, was that my mistake, it actually adds an error message.
Have a good day
Michel
Le Mercredi, Novembre 06, 2019 16:19 CET, "bdus...@luzerne.edu" a écrit:
We've recently begun receiving reports from some of our students indicating they can't configure their phones to access Sogo. I've verified this using iOS and an encountering the problem as well.
Users select to add a Microsoft Exchange account, enter their e-mail address, select to configure manually when prompted, enter their password when prompted, then click next. iOS displays "Verifying..." for a few seconds and then prompts for more information. They enter the correct server, username, and password then click next. After a pause of about a minute with "Verifying" displayed the error "Unable to verify account information" appears.
I've been checking both my nginx and sogo logs and can't find any details on the problem.
I'm running a separate instance for eas traffic as specified at https://sogo.nu/support/faq/dedicated-separate-sogo-instance-for-activesync.html
The nginx logs are showing a LOT of ActiveSync traffic. I believe this problem is related to something in my configuration, not users ability to access our server. It appears users who have preexisting configs are fine. New users can’t seem to configure their devices.
I am seeing the following errors within my SOGo log:
Nov 06 10:05:45 sogod [32707]: [ERROR] <0x0x55720946ebf0[WOWatchDog]> No child available to handle incoming request!
We have a large number of users accessing the EAS instance. I initially had the WOWorkersCount for the eas instance set to 400. I have it set to 500 now. When I try setting it higher I encounter the following error in the SOGo log:
sogod [1716]: [ERROR] <0x0x5595d0273e50[WOHttpAdaptor]> failure notifying watchdog we are ready during events registration: NAME:NGSocketShutdownDuringWriteException REASON:the socket was shutdown INFO:{errno = 32; error = "Broken pipe"; stream = "{object = 0x5595d03818f0;}"; }
I’m going to investigate any OS related limits which may be causing this problem. Could lack of threads be preventing new users from
[SOGo] Users can't configure clients to use ActiveSync
We've recently begun receiving reports from some of our students indicating they can't configure their phones to access Sogo. I've verified this using iOS and an encountering the problem as well.
Users select to add a Microsoft Exchange account, enter their e-mail address, select to configure manually when prompted, enter their password when prompted, then click next. iOS displays "Verifying..." for a few seconds and then prompts for more information. They enter the correct server, username, and password then click next. After a pause of about a minute with "Verifying" displayed the error "Unable to verify account information" appears.
I've been checking both my nginx and sogo logs and can't find any details on the problem.
I'm running a separate instance for eas traffic as specified at https://sogo.nu/support/faq/dedicated-separate-sogo-instance-for-activesync.html
The nginx logs are showing a LOT of ActiveSync traffic. I believe this problem is related to something in my configuration, not users ability to access our server. It appears users who have preexisting configs are fine. New users can’t seem to configure their devices.
I am seeing the following errors within my SOGo log:
Nov 06 10:05:45 sogod [32707]: [ERROR] <0x0x55720946ebf0[WOWatchDog]> No child available to handle incoming request!
We have a large number of users accessing the EAS instance. I initially had the WOWorkersCount for the eas instance set to 400. I have it set to 500 now. When I try setting it higher I encounter the following error in the SOGo log:
sogod [1716]: [ERROR] <0x0x5595d0273e50[WOHttpAdaptor]> failure notifying watchdog we are ready during events registration: NAME:NGSocketShutdownDuringWriteException REASON:the socket was shutdown INFO:{errno = 32; error = "Broken pipe"; stream = "{object = 0x5595d03818f0;}"; }
I’m going to investigate any OS related limits which may be causing this problem. Could lack of threads be preventing new users from configuring their devices?
Does anyone have any suggestions on what I should look for while troubleshooting this problem?
Thanks,
Bob
Bob Dushok
Director of Enterprise Systems and Computer Labs
Luzerne County Community College
1-800-377-5222 ext 7327
bdus...@luzerne.edu
-- users@sogo.nuhttps://inverse.ca/sogo/lists
Re: [SOGo] Performance Tuning
Ludovic, Thanks for the information. I've followed the link you've provided and have created a second instance of sogod. Unfortunately, my new instance still attempted to listen on tcp 2. I have created the info.plist file at /var/lib/sogoeas/GNUstep/Defaults as described in step 3. I believe my new instance of sogod isn't reading this file. To correct this problem I modified the service file for my original instance as well as the new instance. Within each of these files I added the -WOPort option to the ExecStart line. I specified port 2 for my original instance and 3 for the new instance. Within this change I was able to start both instances and each listened on the correct port. Should this change be avoided? If so, can you suggest any troubleshooting steps to determine why my plist file for the new instance is being ignored? Thanks, Bob Bob Dushok Director of Enterprise Systems and Computer Labs Luzerne County Community College 1-800-377-5222 ext 7327 bdus...@luzerne.edu - Original message - From: ""Ludovic Marcotte (lmarco...@inverse.ca)" Sent by: users-requ...@sogo.nu To: users@sogo.nu Cc: Subject: Re: [SOGo] Performance Tuning Date: Wed, Sep 4, 2019 11:44 AM Hi, If you've got over 650 EAS users, a PREFORK of 400 will never work. EAS clients will fight to get sogod workers and will leave nothing to the SOGo web interface. Dedicate a SOGo instance to EAS and use a handful of workers for the web interface. See https://sogo.nu/support/faq/dedicated-separate-sogo-instance-for-activesync.html for details. Ludovic On 2019-09-04 11:30 a.m., bdus...@luzerne.edu wrote: A few months ago we began providing SOGo for our students. It was summer and the number of students using the server was limited. I found tuning was needed, especially the prefork value. I was able to find good values for our installation within a day or two. We started a new semester a few days ago and our full student body is using the product. We've encountered performance problems several times a day. I've handled these problems by making a tuning change and rebooting the server. Within a few hours the problem returns. I haven't been able to find appropriate configuration settings for our environment. We have a little over 14,000 users. I wrote some code which attempts to determine how many users are using ActiveSync. Searching the nginx access logs I'm finding about 650 unique authenticated accounts posting to /Microsoft-Server-ActiveSync?User=. I'm only searching logs from the past seven days. I've also attempted to get a rough estimate of current web users by using: netstat | grep http | wc -l I get about 700 returned. I know this isn't completely accurate, but it gives me an estimate. The server running SOGo is Centos 7.6. It's virtualized with 8 CPU and 16 GB of ram. We're a VMware environment. Checking the server performance doesn't show anything significant. CPU is usually around 10% with some spikes. RAM usage is <10GB. Currently my SOGO sysconfig file contains: PREFORK = 400 SOGoMaximumPingInterval = 3540 SOGoMaximumSyncInterval = 3540 SOGoInternalSyncInterval = 30 WOWatchDogRequestTimeout = 5 WOListenQueueSize = 50 I've modified the OS kernel parameters as follows in the sysctl.conf file: net.ipv4.tcp_max_syn_backlog=8192 net.ipv4.tcp_fin_timeout=25 net.core.somaxconn=1280 I run NetData on this server. It was indicating problem with tcp accept queue overflows and drops. This is why I made the changes to sysctl.conf. It's now indicating tcp syn cookies queue is overflowing. I'm going to do some research on this and will make appropriate changes. Does anyone have any suggestions? I'm making changes, but the performance doesn't appear to be improving much. Thanks, Bob Bob Dushok Director of Enterprise Systems and Computer Labs Luzerne County Community College 1-800-377-5222 ext 7327 bdus...@luzerne.edu -- users@sogo.nu https://inverse.ca/sogo/lists -- Ludovic Marcottelmarco...@inverse.ca :: +1.514.755.3630 :: https://inverse.caInverse inc. :: Leaders behind SOGo (https://sogo.nu), PacketFence (https://packetfence.org) and Fingerbank (https://fingerbank.org) -- users@sogo.nuhttps://inverse.ca/sogo/lists
[SOGo] Performance Tuning
A few months ago we began providing SOGo for our students. It was summer and the number of students using the server was limited. I found tuning was needed, especially the prefork value. I was able to find good values for our installation within a day or two. We started a new semester a few days ago and our full student body is using the product. We've encountered performance problems several times a day. I've handled these problems by making a tuning change and rebooting the server. Within a few hours the problem returns. I haven't been able to find appropriate configuration settings for our environment. We have a little over 14,000 users. I wrote some code which attempts to determine how many users are using ActiveSync. Searching the nginx access logs I'm finding about 650 unique authenticated accounts posting to /Microsoft-Server-ActiveSync?User=. I'm only searching logs from the past seven days. I've also attempted to get a rough estimate of current web users by using: netstat | grep http | wc -l I get about 700 returned. I know this isn't completely accurate, but it gives me an estimate. The server running SOGo is Centos 7.6. It's virtualized with 8 CPU and 16 GB of ram. We're a VMware environment. Checking the server performance doesn't show anything significant. CPU is usually around 10% with some spikes. RAM usage is <10GB. Currently my SOGO sysconfig file contains: PREFORK = 400 SOGoMaximumPingInterval = 3540 SOGoMaximumSyncInterval = 3540 SOGoInternalSyncInterval = 30 WOWatchDogRequestTimeout = 5 WOListenQueueSize = 50 I've modified the OS kernel parameters as follows in the sysctl.conf file: net.ipv4.tcp_max_syn_backlog=8192 net.ipv4.tcp_fin_timeout=25 net.core.somaxconn=1280 I run NetData on this server. It was indicating problem with tcp accept queue overflows and drops. This is why I made the changes to sysctl.conf. It's now indicating tcp syn cookies queue is overflowing. I'm going to do some research on this and will make appropriate changes. Does anyone have any suggestions? I'm making changes, but the performance doesn't appear to be improving much. Thanks, Bob Bob Dushok Director of Enterprise Systems and Computer Labs Luzerne County Community College 1-800-377-5222 ext 7327 bdus...@luzerne.edu -- users@sogo.nuhttps://inverse.ca/sogo/lists
