Re: SMTPUTF8 support
Le 2019-04-04 03:28, André a écrit : I have it working, only with the user part of the email address. It works well, as long as the whole chain of software is correctly configured. I have not tested with a whole domain in UTF8 yet. More details and working examples here: https://github.com/progmaticltd/homebox Good luck. Homebox looks quite interesting, but I'm not sure I want to redo my entire e-mail stack for this. May I ask what did you do make Dovecot support UTF-8 characters in the user part of the e-mail address? Did you patch it? -- Sylvhem
Re: Solr connection timeout hardwired to 60s
On 4/4/2019 6:42 PM, M. Balridge via dovecot wrote: What is a general rule of thumb for RAM and SSD disk requirements as a fraction of indexed document hive size to keep query performance at 200ms or less? How do people deal with the JAVA GC world-stoppages, other than simply doubling or tripling every instance? There's no hard and fast rule for exactly how much memory you need for a search engine. Some installs work well with half the index cached, others require more, some require less. For ideal performance, you should have enough memory over and above your program requirements to cache the entire index. That can be problematic with indexes that are hundreds of gigabytes, or even terabytes. Achieving the ideal is rarely necessary, though. With a large enough heap, it is simply impossible to avoid long stop-the-world GC. With proper tuning, those full garbage collections can happen far less frequently. I've got another page about that. https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr To handle extremely large indexes with good performance, I would recommend many servers running SolrCloud, and a sharded index. That way each individual server will not be required to handle terabytes of data. This can get very expensive very quickly. You will also need a load balancer, to eliminate single points of failure. I am wondering how well alternatives to Solr work in these situations (ElasticSearch, Xapian, and any others I may have missed). Assuming they are configured as similarly as possible, ElasticSearch and Solr will have nearly identical requirements, and perform similarly to each other. They are both Lucene-based, and it is Lucene that primarily drives the requirements. I know nothing about any other solutions. With the extremely large index you have described, memory will be your achilles heel no matter what solution you find. It is not Java that needs the extreme amounts of memory for very large indexes. It is the operating system -- the disk cache. You might also need a fairly large heap, but the on-disk size of the index will have less of an impact on heap requirements than the number of documents in the index. Thanks, Shawn
Re: Solr connection timeout hardwired to 60s
> I'm a denizen of the solr-u...@lucene.apache.org mailing list. > [...] > Here's a wiki page that I wrote about that topic. This wiki is going > away next month, but for now you can still access it: > > https://wiki.apache.org/solr/SolrPerformanceProblems That's a great resource, Shawn. I am about to put together a test case to provide a comprehensive FTS setup around Dovecot with a goal towards exposing proximity keyword searching, with email silos containing tens of terabytes (most of the "bulk" is represented by attachments, each of which get processed down to plaintext, if possible). Figure thousands of users with decades of email (80,000 to 750,000) emails per user). My main background is in software engineering (C/C++/Python/Assembler), but I have been forced into system admin tasks during many stretches of my work. I do vividly remember the tedium of dealing with JAVA and GC, tuning it to avoid stalls, and its ravenous appetite for RAM. It looks like those problems are still with us, many versions later. For corporations with infinite budgets, throwing lots of crazy money at the problem is "fine" (>1TB RAM, all PCIe SSDs, etc), but I am worried that I will be shoved forcefully into a wall of having to spend a fortune just to keep FTS performing reasonably well before I even get to the 10,000 user mark. I realise the only way to keep performance reasonable is to heavily shard the index database, but I am concerned about how well the process works in practice without needing a great deal of sysadmin hand-holding. I would ideally prefer the decisions of how/where to shard be based on volume/heuristics than something that is done manually. I realise that a human will be necessary to add more hardware to the pools, but what are my options for scaling the system by orders of magnitude? What is a general rule of thumb for RAM and SSD disk requirements as a fraction of indexed document hive size to keep query performance at 200ms or less? How do people deal with the JAVA GC world-stoppages, other than simply doubling or tripling every instance? I am wondering how well alternatives to Solr work in these situations (ElasticSearch, Xapian, and any others I may have missed). Regards, =M=
Re: Solr connection timeout hardwired to 60s
On 4/4/19 6:47 PM, dovecot-requ...@dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(m...@example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter
Re: Solr connection timeout hardwired to 60s
On 4/4/2019 2:21 AM, Peter Mogensen via dovecot wrote: What's the recommended way to handling timeouts on large mailboxes given the hardwired request timeout of 60s in solr-connection.c: http_set.request_timeout_msecs = 60*1000; I'm a denizen of the solr-u...@lucene.apache.org mailing list. For a typical Solr index, 60 seconds is an eternity. Most people aim for query times of 100 milliseconds or less, and they often achieve that goal. If you have an index where queries really are taking longer than 60 seconds, you're most likely going to need to get better hardware for Solr. Memory is the resource that usually has the greatest impact on Solr performance. Putting the index on SSD can help, but memory will help more. Here's a wiki page that I wrote about that topic. This wiki is going away next month, but for now you can still access it: https://wiki.apache.org/solr/SolrPerformanceProblems There's a section in that wiki page about asking for help on performance issues. It describes how to create a particular process listing for a screenshot. If you can get that screenshot and share it using a file sharing site (dropbox is usually a good choice), I may be able to offer some insight. Thanks, Shawn
Re: Solr connection timeout hardwired to 60s
Hi Shawn Am 04.04.19 um 16:12 schrieb Shawn Heisey via dovecot: > On 4/4/2019 2:21 AM, Peter Mogensen via dovecot wrote: > Here's a wiki page that I wrote about that topic. This wiki is going > away next month, but for now you can still access it: > > https://wiki.apache.org/solr/SolrPerformanceProblems https://web.archive.org/web/20190404143817/https://wiki.apache.org/solr/SolrPerformanceProblems That one will last longer :). Best Daniel
Re: Solr connection timeout hardwired to 60s
On 4/4/2019 2:21 AM, Peter Mogensen via dovecot wrote: What's the recommended way to handling timeouts on large mailboxes given the hardwired request timeout of 60s in solr-connection.c: http_set.request_timeout_msecs = 60*1000; I'm a denizen of the solr-u...@lucene.apache.org mailing list. For a typical Solr index, 60 seconds is an eternity. Most people aim for query times of 100 milliseconds or less, and they often achieve that goal. If you have an index where queries really are taking longer than 60 seconds, you're most likely going to need to get better hardware for Solr. Memory is the resource that usually has the greatest impact on Solr performance. Putting the index on SSD can help, but memory will help more. Here's a wiki page that I wrote about that topic. This wiki is going away next month, but for now you can still access it: https://wiki.apache.org/solr/SolrPerformanceProblems There's a section in that wiki page about asking for help on performance issues. It describes how to create a particular process listing for a screenshot. If you can get that screenshot and share it using a file sharing site (dropbox is usually a good choice), I may be able to offer some insight. Thanks, Shawn
Re: doveadm backup + ISO 8859-1 mailbox name
> On 4 Apr 2019, at 11.03, Michael Goth via dovecot wrote: > > That too includes the byte in its output. But I've found that only a > couple of mailboxes are affected, so i just renamed the folders. > > Thanks for your help! The legacy server is then not RFC 3501 compliant. 8-bit folder names should be encoded in mUTF7. But if you already solved the issue by renaming folders then I guess you are already fine. Sami
Re: Autosubscribe to public folder
# in the public namespace
namespace {
...
mailbox name {
auto=subscribe
}
}
?
Aki
On 4.4.2019 12.45, Рустам via dovecot wrote:
> Hello everyone! I am trying to migrate from Dovecot 1.2 to Dovecot 2.2
> and I can`t find information how to subscribe all users to public
> folder. In Dovecot 1.2 I have a few strings kind of "autosubscribeX =
> Public.Documents" and every users get subscribed to this Public
> folders and can`t unsubscribe. This is a goal. Is any posibility to
> make the same in Dovecot 2.2 or each user have to susbscribe to public
> folders themselves and this is kind of policy of Dovecot 2.2?
>
> --
> С уважением, Рустам Багов.
Autosubscribe to public folder
Hello everyone! I am trying to migrate from Dovecot 1.2 to Dovecot 2.2 and I can`t find information how to subscribe all users to public folder. In Dovecot 1.2 I have a few strings kind of "autosubscribeX = Public.Documents" and every users get subscribed to this Public folders and can`t unsubscribe. This is a goal. Is any posibility to make the same in Dovecot 2.2 or each user have to susbscribe to public folders themselves and this is kind of policy of Dovecot 2.2? -- С уважением, Рустам Багов.
Re: doveadm backup + ISO 8859-1 mailbox name
On 4/3/19 2:23 PM, Sami Ketola via dovecot wrote: On 3 Apr 2019, at 13.40, Michael Goth via dovecot wrote: Hi! I'm trying to migrate a mailbox from an old IMAP server (not Dovecot) to a new Dovecot server with 'doveadm backup'. The command fails because of an unexpected character in a mailbox name: doveadm -o imapc_user=a...@example.org \ -o imapc_password=XXX \ -o imapc_host=old-mailserver.webflow.de \ -o imapc_port=993 \ -o imapc_ssl=imaps \ -o imapc_ssl_verify=no \ -o imapc_features=rfc822.size,fetch-headers \ -o ssl_client_ca_dir=/etc/ssl \ backup -R -f -x ~* -u migration...@example.org imapc: dsync(migration...@example.org): Info: imapc(old-mailserver.webflow.de:993): Connected to 1.1.1.1:993 (local 2.2.2.2:46154) dsync(migration...@example.org): Error: Failed to access mailbox Entw?rfe: mailbox does not exist The problem is with "Entw?rfe" here. It's actually Entw<0xfc>rfe, which translates to "Entwürfe" in ISO-8859-1. Why can't doveadm find that mailbox? Is the old mailserver misbehaving (I hope not because I have no idea how to change that...), or does this happen on the new server? Can I tell doveadm something that makes this work? Can you log on the remote imap server with telnet or nc and ask for: A LIST "" "*" I think your legacy server does not encode the folder names correctly. That too includes the byte in its output. But I've found that only a couple of mailboxes are affected, so i just renamed the folders. Thanks for your help! Michael
Solr connection timeout hardwired to 60s
Hi, What's the recommended way to handling timeouts on large mailboxes given the hardwired request timeout of 60s in solr-connection.c: http_set.request_timeout_msecs = 60*1000; /Peter
Re: SMTPUTF8 support
I have it working, only with the user part of the email address. It works well, as long as the whole chain of software is correctly configured. I have not tested with a whole domain in UTF8 yet. More details and working examples here: https://github.com/progmaticltd/homebox Good luck. On 4 April 2019 07:03:10 BST, sylvhem--- via dovecot wrote: >Le 2019-04-04 01:36, Michael Slusarz a écrit : >>> On April 3, 2019 at 10:12 PM sylvhem--- via dovecot >>> wrote: >>> >>> I'm currently trying to set up SMTPUTF8 on my mail stack, but I >can't >>> find any information on Dovecot's RFC 6531 support. Has it been >>> implemented yet? >> >> No. >> >> michael >Thank you for the answer Michael. >Does anybody know if it's been worked on? > >-- >Sylvhem -- André
Re: SMTPUTF8 support
Le 2019-04-04 01:36, Michael Slusarz a écrit : On April 3, 2019 at 10:12 PM sylvhem--- via dovecot wrote: I'm currently trying to set up SMTPUTF8 on my mail stack, but I can't find any information on Dovecot's RFC 6531 support. Has it been implemented yet? No. michael Thank you for the answer Michael. Does anybody know if it's been worked on? -- Sylvhem
