Re: doveadm_allowed_commands doesn't work as expected
Err...! I thought it is matching the API commands, instead, matching
doveadm cli commands.
Thanks you so much,
Ronald
On 4/1/2019 5:02 AM, Aki Tuomi wrote:
On 03 January 2019 at 22:45 Ronald Poon wrote:
Trying to limit the API calls to doveadm-http-api by configure allowed
commands, but once the commands added to the list, the RestAPI no longer
work.
1) Return correct reply when doveadm_allowed_commands is empty
# curl -k -H "Content-Type: application/json" -H "Authorization:
X-Dovecot-API " https://localhost:9088/doveadm/v1
-d'[["quotaGet",{"user":"us...@mydomain.com"},"c01"]]'
[["doveadmResponse",[{"root":"User
quota","type":"STORAGE","value":"0","limit":1024","percent":"0"},{"root":"Userquota","type":"MESSAGE","value":"0","limit":"-","percent":"0"}],"c01"]]
2) Return unAuthorized when doveadm_allowed_commands =
quotaGet,quotaRecalc,expunge
# curl -k -H "Content-Type: application/json" -H "Authorization:
X-Dovecot-API " https://localhost:9088/doveadm/v1
-d'[["quotaGet",{"user":"us...@mydomain.com"},"c01"]]'
[["error",{"type":"unAuthorized", "exitCode":0},"c01"]]
This is mostly because v1 API is not so fantastic. Try
doveadm_allowed_commands="quota get,quota recalc,expunge"
Aki
Solr -> Xapian ?
What about consedering linking Dovecot with Xapian librairies instead of going to nightmare Solr ? https://xapian.org/features On 2019-01-02 17:10, John Tulp wrote: On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is : After some time of indexing from Dovecot, Dovecot returns errors (invalid SID, etc...) and Solr return "out of range indexes" errors I've been watching the progress of this thread with no small concern, mainly because I've been tasked with providing a server-side email search facility with a budget and manpower level that comes down to mainly *1*, i.e., me. I was expecting, given the strongly worded language about "just use lucene/SOLR" and "ignore squat", that I should invest time + effort into this JAVA nightmare that is SOLR. I started with squat and another word-indexor system that used out-of-band (not a dovecot plugin) software to provide rapid (sub-second) searches through tens-of-GB-scale mailboxes. Unlike what I was led to believe, the squat indexes worked surprisingly well, once you sorted out the odd resource size (ulimit-related) issues (vsz & friends) limitations. I did notice the "worst-case" search performance have worryingly high O(x) increases in time, but I'd not seen anything that was a dealbreaker. It goes without saying that various substring searches worked as expected, for the most part. My experiences with SOLR were similar to Messr. Moreau's: lots of startup errors with provided schemata files. Lots of JAVA nonsense issues. Lots of sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM, version of SOLR, and dovecot to find the "best" working combination, only to find that the searches didn't work out as expected. I expected to be able to do date-ranging based searches. Didn't work. I expected to search CONTENTS of emails, and despite many days of tweaks, I couldn't get it to index even the basics like filenames/types of attachments, so I could exposed attachment-based searching to my users. So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality: 1) The ability to search for a string within any of the structured fields (from/subject) that returns correct results? 2) The ability to search for any string within the BODY of emails, including the MIME attachment boundaries? 3) The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data? OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines: 4) The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika? - SOLR seems to have "brand cachet", so presumably it actually works (for somebody). Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I am no stranger to the "costs" of "free" software, in that you sacrifice your own blood, sweat, and tears just to get these disparate pieces to work together. I *DO* respect that Timo has to keep the lights (and sauna) on in Finland. Maybe there's a super-secret (no advertised prices, "carrier-only" price list) with _Dovecot, Oy_ wherein the above ARE actually available for something less than 6.022 x 10^23 Euros per centi-second of licencing fees. But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?) I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have. He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them. Respectfully, =M= Fascinating... SOLR says the following are powered by SOLR... https://wiki.apache.org/solr/PublicServers Perhaps if you could find out from that list which of them are using SOLR in conjunction with Dovecot... food for thought...
Solr - complete setup (update)
Hi
This is the summary of my work with SOLR-Dovecot, in my QUEST TO
REPRODUCE THE PREVIOULSY EXCELLENT WORK OF FTS_SQUAT
@Aki : Based on the time I have spent on this, I would love to see you
updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
- INSTALLATION:
-> Create a clean install using the default, (at least in the Archlinux
package), and do a "sudo -u solr solr create -c dovecot ". The config
files are then in /opt/solr/server/solr/dovecot/conf and datafiles in
/opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change false to
true
* around line 147, set 2000
(or above)
* around line 696 : uncomment hdr
* around line 1127, before , add
* around line 1161, delete the whole name="add-schema-fields">
* around line 1192, remove the whole name="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior
(equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note :
such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition
with *space*, ideally ext4 or faster file system (it looks like Solr is
not considering using a simple mysql database, which would make sense to
avoid all the fuzz and let it transit to a non-java state, but that is
another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see
below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have
quite a space on my server, but you may adapt it as per your specs) : in
/opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in
your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u )
-> wait for a big while to let the system re-index all your mail boxes
- BUGS SO FAR
-> Line 620 of fts_solr dovecot plugin : the size oof header is
improperly calculated ("huge header" warning for a simple email, which
kilss the index of that considered email, so basically MOST emails as
the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is
maybe the source of problem of the "out of bound" errors in fts_solr
dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java.
But, with increased memory, it seems not crashing, even if complaining
quite a lot in the logs
---SCHEMA.XML IN /OPT/SOLR/SERVER/SOLR/DOVECOT/CONF
id
-- DOVECOT.CONF
mail_plugins = fts fts_solr
plugin {
plugin = fts fts_solr managesieve sieve
fts = solr
fts_autoindex = yes
fts_enforced = yes
fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external
server)
(...)
}
-- /ETC/SYSTEMD/SYSTEM/MULTI-USER.TARGET.WANTS/SOLR.SERVICE
[Unit]
Description=Solr full text search engine
After=network.target
[Service]
Type=simple
User=solr
Group=solr
PrivateTmp=yes
WorkingDirectory=/opt/solr
LIMITNOFILE=65000
LIMITNPROC=65000
ExecStart=/opt/solr/bin/solr start -f
[Install]
WantedBy=multi-user.target
Solr - complete setup
Hi
This is the summary of my work with SOLR-Dovecot, in my QUEST TO
REPRODUCE THE PREVIOULSY EXCELLENT WORK OF FTS_SQUAT
@Aki : Based on the time I have spent on this, I would love to see you
updating the Wiki with those improvements, and adding my name somewhere
@All : Hope it helps
- INSTALLATION:
-> Create a clean install using the default, (at least in the Archlinux
package), and do a "sudo -u solr solr create -c dovecot ". The config
files are then in /opt/solr/server/solr/dovecot/conf and datafiles in
/opt/solr/server/solr/dovecot/data
-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:
* around line 313, change false to
true
* around line 147, set 2000
(or above)
* around line 1127, before , add
* around line 1161, delete the whole name="add-schema-fields">
* around line 1192, remove the whole name="add-unknown-fields-to-the-schema" ... />
-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema
-> Change "schema.xml" by the one below to reproduce fts_squat behavior
(equivalent to " fts_squat = partial=3 full=25" in dovecot.conf) (note :
such a huge trouble to replace a single line setup, anyway...)
-> Move /opt/solr/server/solr (or the subfolder data) to a partition
with *space*, ideally ext4 or faster file system (it looks like Solr is
not considering using a simple mysql database, which would make sense to
avoid all the fuzz and let it transit to a non-java state, but that is
another story)
-> Config of dovecot.conf is as below
-> The systemd unit shall specify high ulimit for files and proc (see
below)
-> Increase the memory available for the JavaVM (I put 12Gb as I have
quite a space on my server, but you may adapt it as per your specs) : in
/opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"
-> As Solr is complaining a lot, you may consider a filter for it in
your syslog-ng or journald as it pollutes greatly your audit files
-> (re)Start solr (first) and dovecot by systemctl
-> Launch redindex ( doveadm fts rescan -u )
-> wait for a big while to let the system re-index all your mail boxes
- BUGS SO FAR
-> Line 620 of fts_solr dovecot plugin : the size oof header is
improperly calculated ("huge header" warning for a simple email, which
kilss the index of that considered email, so basically MOST emails as
the calculation is wrong)
-> The UID returned by SOlr is to be considered as a STRING (and that is
maybe the source of problem of the "out of bound" errors in fts_solr
dovecot, as "long" is not enough)
-> Java errors : A lot of non sense for me, I am not expert in Java.
But, with increased memory, it seems not crashing, even if complaining
quite a lot in the logs
---SCHEMA.XML IN /OPT/SOLR/SERVER/SOLR/DOVECOT/CONF
id
-- DOVECOT.CONF
mail_plugins = fts fts_solr
plugin {
plugin = fts fts_solr managesieve sieve
fts = solr
fts_autoindex = yes
fts_enforced = yes
fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
(replace 127.0.0.1 by your solr server if you want to use an external
server)
(...)
}
-- /ETC/SYSTEMD/SYSTEM/MULTI-USER.TARGET.WANTS/SOLR.SERVICE
[Unit]
Description=Solr full text search engine
After=network.target
[Service]
Type=simple
User=solr
Group=solr
PrivateTmp=yes
WorkingDirectory=/opt/solr
LIMITNOFILE=65000
LIMITNPROC=65000
ExecStart=/opt/solr/bin/solr start -f
[Install]
WantedBy=multi-user.target
Re: doveadm_allowed_commands doesn't work as expected
> On 03 January 2019 at 22:45 Ronald Poon wrote:
>
>
> Trying to limit the API calls to doveadm-http-api by configure allowed
> commands, but once the commands added to the list, the RestAPI no longer
> work.
>
>
> 1) Return correct reply when doveadm_allowed_commands is empty
>
> # curl -k -H "Content-Type: application/json" -H "Authorization:
> X-Dovecot-API " https://localhost:9088/doveadm/v1
> -d'[["quotaGet",{"user":"us...@mydomain.com"},"c01"]]'
> [["doveadmResponse",[{"root":"User
> quota","type":"STORAGE","value":"0","limit":1024","percent":"0"},{"root":"Userquota","type":"MESSAGE","value":"0","limit":"-","percent":"0"}],"c01"]]
>
>
> 2) Return unAuthorized when doveadm_allowed_commands =
> quotaGet,quotaRecalc,expunge
>
> # curl -k -H "Content-Type: application/json" -H "Authorization:
> X-Dovecot-API " https://localhost:9088/doveadm/v1
> -d'[["quotaGet",{"user":"us...@mydomain.com"},"c01"]]'
> [["error",{"type":"unAuthorized", "exitCode":0},"c01"]]
This is mostly because v1 API is not so fantastic. Try
doveadm_allowed_commands="quota get,quota recalc,expunge"
Aki
doveadm_allowed_commands doesn't work as expected
Trying to limit the API calls to doveadm-http-api by configure allowed
commands, but once the commands added to the list, the RestAPI no longer
work.
1) Return correct reply when doveadm_allowed_commands is empty
# curl -k -H "Content-Type: application/json" -H "Authorization:
X-Dovecot-API " https://localhost:9088/doveadm/v1
-d'[["quotaGet",{"user":"us...@mydomain.com"},"c01"]]'
[["doveadmResponse",[{"root":"User
quota","type":"STORAGE","value":"0","limit":1024","percent":"0"},{"root":"Userquota","type":"MESSAGE","value":"0","limit":"-","percent":"0"}],"c01"]]
2) Return unAuthorized when doveadm_allowed_commands =
quotaGet,quotaRecalc,expunge
# curl -k -H "Content-Type: application/json" -H "Authorization:
X-Dovecot-API " https://localhost:9088/doveadm/v1
-d'[["quotaGet",{"user":"us...@mydomain.com"},"c01"]]'
[["error",{"type":"unAuthorized", "exitCode":0},"c01"]]
Here is my config,
# uname -a
Linux ad92422d8e94 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47
UTC 2018 x86_64 Linux
# free -m
total used free shared buffers cached
Mem: 15885 7133 8751 0 1 4374
-/+ buffers/cache: 2758 13126
Swap: 0 0 0
/ # dovecot -n
# 2.3.2.1 (0719df592): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.2 (7704de5e)
# OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 xfs
# Hostname: ad92422d8e94
auth_mechanisms = plain login
doveadm_allowed_commands = quotaGet,quotaRecalc,expunge
doveadm_api_key = # hidden, use -P to show it
hostname = mailhost.mydomain.com
info_log_path = /dev/stdout
lda_mailbox_autosubscribe = yes
log_path = /dev/stderr
login_greeting = Dovecot ready.
mail_gid = vmail
mail_home = /var/vmail/%d/%n
mail_location = maildir:/var/vmail/%d/%n/Maildir
mail_plugins = " quota zlib"
mail_privileged_group = vmail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
imapsieve_mailbox1_before =
file:/etc/dovecot/sieve/global/learn-spam.sieve
imapsieve_mailbox1_causes = COPY
imapsieve_mailbox1_name = Junk
imapsieve_mailbox2_before =
file:/etc/dovecot/sieve/global/learn-ham.sieve
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_from = Junk
imapsieve_mailbox2_name = *
quota = maildir:User quota
quota_exceeded_message = User %u has exhausted allowed storage space.
recipient_delimiter = -
sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_before = /etc/dovecot/sieve/global/spam-to-folder.sieve
sieve_global_extensions = +vnd.dovecot.pipe
sieve_pipe_bin_dir = /usr/lib/dovecot/sieve
sieve_pipe_exec_timeout = 60s
sieve_plugins = sieve_imapsieve sieve_extprograms
zlib_save = gz
zlib_save_level = 6
}
postmaster_address = postmas...@mydomain.com
protocols = lmtp imap pop3 sieve
recipient_delimiter = -
service auth {
inet_listener {
port = 9000
}
}
service doveadm {
client_limit = 1
drop_priv_before_exec = no
executable = doveadm-server
extra_groups = $default_internal_group
inet_listener http {
port = 9088
ssl = yes
}
service_count = 1
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
ssl_cert =
Re: Compiling Dovecot on Solaris 11 fails
> On 3 Jan 2019, at 18.45, Andrew Watkins wrote: > > > Hi, > > Tried to build dovecot-2.3.4 on Solaris 11 x86 and it fails at configure > part. I just went checked and it last version it works on is v2.3.2.1 does work just fine on my solaris 11 x86 box. configure:22610: checking whether fd passing works configure:22685: gcc -o conftest -std=gnu99 -g -O2 -fstack-protector -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I./src/lib ./src/lib/fdpass.c conftest.c -lsocket -lnsl -lresolv >&5 configure:22685: $? = 0 configure:22685: ./conftest configure:22685: $? = 0 configure:22708: result: yes What is your Solaris release and which version of gcc you are using? my test system is solaris 11.3 and I have gcc 4.5.2. Sami
Re: Solr
On 1/3/2019 10:56 AM, Tanstaafl wrote: On 12/21/2018, 11:19:42 AM, Daniel Miller via dovecot wrote: There is a *huge* difference between a functional Solr setup & squat Interesting. Care to elaborate? This is one of those things that has to be experienced to be understood. When you can perform an FTS search across (pause while I check current stats...): du -c -h /var/mail 136G Solr numDocs: 520102 and using any IMAP client that supports server-side searches (like Thunderbird & AquaMail) the results are basically instantaneous...it's worth the effort. And that's searching a Dovecot virtual folder defined as "* all", including all my archives, all my list subscriptions, and all the shared Inbox/Sent folders from my other users. But I certainly wish it was easier to setup. -- Daniel
Re: Solr
On 1/1/2019 3:49 PM, Joan Moreau via dovecot wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the
systemd installation script is included (and it is launching
/opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this
creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
I'm using that nasty word *should*...in that the above installation
*should* yield working results. But...since I don't use Arch and have
no insight into it I suggest downloading a binary tarball from the Solr
site and do a clean install. It may behave identically...or maybe
something will be different.
--
Daniel
Re: Solr
On 1/2/2019 12:59 AM, M. Balridge wrote: So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality: 1) The ability to search for a string within any of the structured fields (from/subject) that returns correct results? Yes. 2) The ability to search for any string within the BODY of emails, including the MIME attachment boundaries? Yes. 3) The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data? Dunno - I don't think I've needed that and I'm not sure how to do it. My mail clients are Thunderbird and AquaMail (on Android). If you'll give me either the desired Thunderbird steps or telnet-based IMAP command I'm happy to test. OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines: 4) The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika? Haven't tried. SOLR seems to have "brand cachet", so presumably it actually works (for somebody). It works - just sometimes needs more effort to setup than it should. Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I think we're all in agreement here. But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?) I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have. He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them. Short answer - it actually works. Longer answer - I've gone through a hate/love/hate/like relationship with Solr myself. The transition from v3 to v4 was a major headache - and I gave up for a while. But versions 6 & 7 have been pretty good for me. I'm neither a Dovecot nor a Solr developer - just enough of a fiddler to get them working to fulfill my own needs. If my unreliable memory serves I believe the Dovecot fts-solr plugin hasn't needed to change much (I recall one significant change required when Solr changed it's protocol - I think an XML/JSON thing). So having a stable interface let's Timo & Co. forget about on-going FTS development and continue focusing on things not provided by other tools. Hopefully they'll revisit SIS... I recall reading something about the Lucene library (which Squat & Solr are based on) and again my memory is the C version(s) weren't getting maintained as well as might be desired. I think having the Solr/Lucene team focusing on Java development was another point of consideration for Dovecot's squat - but I could be totally off here. Based on the errors reported by Joan I believe that system's problems are due to configuration - either Solr, Dovecot, or both. They don't sound like Java related issues (which are a *major* pain to deal with!). I've provided a copy of what is a working configuration *for me*. I'm happy to continue helping as best I can - and if Joan, you, or anyone else would like my aid I'll do my best. If you're crazy I-mean-trusting enough to have me SSH or remote view to your system I'm willing to take a look. I've had enough people help me over the years for various packages that I'd like to pay it forward where I can. -- Daniel
Re: Solr
I'm running 7.5.0. The solrconfig.xml file is what I've modified over
time - I haven't started one from scratch for a while but perhaps I'll try.
Have you tried using the complete config that I sent you? With *all*
the files I included - and *none* of yours?
--
Daniel
On 1/1/2019 4:12 PM, Joan Moreau wrote:
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a
"manage-schema" and starts complaining about useless types (tdates,
booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things
as they are (yeah !), does not complains about those useless types and
startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her
own use.
The main problem is : After some time of indexing from Dovecot,
Dovecot returns errors (invalid SID, etc...) and Solr return "out of
range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the
systemd installation script is included (and it is launching
/opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this
creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote:
On 12/29/2018 4:49 PM, Joan Moreau wrote:
Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the
solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation
and unpacked it)
How did you create the dovecot instance? (I've provided explicit
instructions for how I did it - did you follow those exactly or
something different)?
How are you starting Solr? (I use the provided "solr/bin/solr
start" command, wrapped inside a systemd service).
--
Daniel
--
--
Daniel
Re: Compiling Dovecot on Solaris 11 fails
> On 03 January 2019 at 21:07 James wrote: > > > On 03/01/2019 16:45, Andrew Watkins wrote: > > Tried to build dovecot-2.3.4 on Solaris 11 x86 and it fails at configure > > part. I just went checked and it last version it works on is v2.3.2.1 > > > ... > > > > checking Linux compatible mremap()... no > > checking whether shared mmaps get updated by write()s... no > > checking whether fd passing works... no > > configure: error: fd passing is required for Dovecot to work > > > > In think it's thunk (but I forget exactly what aspect). > > Try: > ./configure --disable-hardening ... > > Or build with cc. > > (Sorry, it's too late for me to do another test today.) It seems we need to improve the thunk code to check that the binaries generated are actually able to run, and not just check that the flag is accepted by compiler. Aki
Re: Compiling Dovecot on Solaris 11 fails
On 03/01/2019 16:45, Andrew Watkins wrote: Tried to build dovecot-2.3.4 on Solaris 11 x86 and it fails at configure part. I just went checked and it last version it works on is v2.3.2.1 ... checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work In think it's thunk (but I forget exactly what aspect). Try: ./configure --disable-hardening ... Or build with cc. (Sorry, it's too late for me to do another test today.)
Re: Solr
On 12/21/2018, 11:19:42 AM, Daniel Miller via dovecot wrote: > There is a *huge* difference between a functional Solr setup & squat Interesting. Care to elaborate?
Re: Compiling Dovecot on Solaris 11 fails
> On 03 January 2019 at 18:45 Andrew Watkins wrote: > > > > Hi, > > Tried to build dovecot-2.3.4 on Solaris 11 x86 and it fails at configure > part. I just went checked and it last version it works on is v2.3.2.1 > > Version 2.3.2.1 (OK) > # ./configure > > checking Linux compatible mremap()... no > checking whether shared mmaps get updated by write()s... yes > checking whether fd passing works... yes > ... > > Version 2.3.4 (FAIL) > > checking Linux compatible mremap()... no > checking whether shared mmaps get updated by write()s... no > checking whether fd passing works... no > configure: error: fd passing is required for Dovecot to work > > > config.log shows: > configure:22538: checking whether shared mmaps get updated by write()s > configure:22585: gcc -o conftest -std=gnu99 -g -O2 > -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 > -mfunction-return=thunk -mindirect-branch=thunk -Wall -W > -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith > -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime > -Wstrict-aliasing=2 conftest.c >&5 > configure:22585: $? = 0 > configure:22585: ./conftest > ./configure[2026]: eval: line 1: 29183: Memory fault(coredump) > > onfigure:22610: checking whether fd passing works > configure:22685: gcc -o conftest -std=gnu99 -g -O2 > -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 > -mfunction-return=thunk -mindirect-branch=thunk -Wall -W > -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith > -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime > -Wstrict-aliasing=2 -I./src/lib ./src/lib/fdpass.c conftest.c >&5 > configure:22685: $? = 0 > configure:22685: ./conftest > ./configure[2026]: eval: line 1: 29206: Memory fault(coredump) > ... > > $ ulimit -a > core file size (blocks, -c) unlimited > data seg size (kbytes, -d) unlimited > file size (blocks, -f) unlimited > open files (-n) 512 > pipe size (512 bytes, -p) 10 > stack size (kbytes, -s) unlimited > cpu time (seconds, -t) unlimited > max user processes (-u) 29995 > virtual memory (kbytes, -v) unlimited > > Any ideas and thanks. > > Andrew > > -- > Andrew Watkins * Birkbeck, University of London * Computer Science * > * http://notallmicrosoft.blogspot.com * > * UKOUG Systems SIG Chair * > * UKOUG Tech committee * > * tel: 020 7631 6720 * > > ::e-mails are only read & actioned between 0900 and 1800 hours (Monday - > Friday) > Backtrace from the core dump could help. Aki
Compiling Dovecot on Solaris 11 fails
Hi, Tried to build dovecot-2.3.4 on Solaris 11 x86 and it fails at configure part. I just went checked and it last version it works on is v2.3.2.1 Version 2.3.2.1 (OK) # ./configure checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... yes checking whether fd passing works... yes ... Version 2.3.4 (FAIL) checking Linux compatible mremap()... no checking whether shared mmaps get updated by write()s... no checking whether fd passing works... no configure: error: fd passing is required for Dovecot to work config.log shows: configure:22538: checking whether shared mmaps get updated by write()s configure:22585: gcc -o conftest -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -mfunction-return=thunk -mindirect-branch=thunk -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 conftest.c >&5 configure:22585: $? = 0 configure:22585: ./conftest ./configure[2026]: eval: line 1: 29183: Memory fault(coredump) onfigure:22610: checking whether fd passing works configure:22685: gcc -o conftest -std=gnu99 -g -O2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -mfunction-return=thunk -mindirect-branch=thunk -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -I./src/lib ./src/lib/fdpass.c conftest.c >&5 configure:22685: $? = 0 configure:22685: ./conftest ./configure[2026]: eval: line 1: 29206: Memory fault(coredump) ... $ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 512 pipe size (512 bytes, -p) 10 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 29995 virtual memory (kbytes, -v) unlimited Any ideas and thanks. Andrew -- Andrew Watkins * Birkbeck, University of London * Computer Science * * http://notallmicrosoft.blogspot.com * * UKOUG Systems SIG Chair * * UKOUG Tech committee * * tel: 020 7631 6720 * ::e-mails are only read & actioned between 0900 and 1800 hours (Monday - Friday)
Re: Request: Pigeonhole - strip CWFS from message-id logs
Op 30-11-2018 om 14:06 schreef Lee Maguire: Normally for a log line containing the contents of a Message-Id, it is logged like the following Nov 29 11:41:27 xxx dovecot[211]: lmtp(lee)<30167>: sieve: msgid=: stored mail into mailbox 'Notifications' However, if there is CFWS other than a space, it changes the format of the log line (potentially making the logs harder to process - eg via “logcheck" rules) Nov 29 11:50:09 xxx dovecot[401]: lmtp(lee)<55724>: sieve: msgid=? : stored mail into mailbox ‘Notifications' A message-id line broken up is valid per RFC 5322 and header whitespace shouldn’t be significant for logging purposes https://tools.ietf.org/html/rfc5322#section-3.6.4 https://tools.ietf.org/html/rfc5322#section-3.2.2 Yes, that is a bug. Regards, Stephan.
Re: Fwd: Re: gcc -> clang
The arithmetic stuff has already been fixed in master with https://github.com/dovecot/core/commit /5cccb4af850bb3ba81e73a8fb4f6881c3e1d4046.patch https://github.com/dovecot/core/commit /ac7aa955db4c77bbb169baa5d104a4c128674646.patch I have not seen the second error ever, it would need more information, can you gdb to it and see what the actual value is? Aki > On 03 January 2019 at 13:38 Rupert Gallagher via dovecot > wrote: > > > The compiler returns many warnings, and the test returns two IPv6-related > errors. I am attaching both logs as reference. > > > ‐‐‐ Original Message ‐‐‐ > > On Thursday, January 3, 2019 9:53 AM, Aki Tuomi > > wrote: > > > >> We compile all core code with both gcc and clang. What sort of interesting > >> things did you find? > >> > >> Aki > >> > >>> On 03 January 2019 at 11:50 Rupert Gallagher via dovecot < > >>> dovecot@dovecot.org> wrote: > >>> > >>> Please, use clang instead of gcc. Code quality can only profit from it. I > >>> just compiled 2.3.4 and compiler stderr is full of interesting problems. > >> > >> --- > >> Aki Tuomi
Re: gcc -> clang
> On 3 Jan 2019, at 14.04, Ruben Safir wrote: > > On 1/3/19 4:50 AM, Rupert Gallagher via dovecot wrote: >> Please, use clang instead of gcc. Code quality can only profit from it. I >> just compiled 2.3.4 and compiler stderr is full of interesting problems. >> > > > oh please This is not very helpful. Sami
Re: Moving Alternate Storage to another disk.
Okay, thank you. On 3-1-2019 11:25, Urban Loesch via dovecot wrote: Hi, if you have the new disk installed on the same server you can try: - mount new disk for example in /mnt/temp - rsync -vaWH all files + directories from the old to the new disk - stop dovecot so no changes will happen on disks - make a final rsync again -> should not take many time - umout the old disk - mount new disk to the original "alt-storage" path, so you don't have to change each soft-link in each users directory. - start dovecot Not tested, but in theory it should work. Best Urban Am 31.12.18 um 15:02 schrieb bOnK: Hello, Dovecot 2.3.4_3 on FreeBSD 11.2. I am using mdbox Alternate Storage since about two years without any problems. However, the disk containing this storage is almost full and I have to move this data to another disk, probably zpool. Would it be okay to do the following? 1) Shut down dovecot (and mail server) so no new mail comes in. 2) Copy/move all files in ALT location to new disk, using shell commands like cp/mv/cpdup. 3) Change the path to ALT in dovecot-conf mail_location. 4) Change the 'dbox-alt-root' soft-links in each users main (INBOX) directory to point to this new location. 5) Start up dovecot and mail server. Am I missing something or maybe there is a better way?
Re: gcc -> clang
Hi all! On 03/01/2019 10:50, Rupert Gallagher via dovecot wrote: > Please, use clang instead of gcc. Code quality can only profit from it. > I just compiled 2.3.4 and compiler stderr is full of interesting problems. But obviously not interesting enough to share the clang command line and the output from it. MfG, Bernd -- "I dislike type abstraction if it has no real reason. And saving on typing is not a good reason - if your typing speed is the main issue when you're coding, you're doing something seriously wrong." - Linus Torvalds pEpkey.asc Description: application/pgp-keys
Re: gcc -> clang
On 1/3/19 4:50 AM, Rupert Gallagher via dovecot wrote: > Please, use clang instead of gcc. Code quality can only profit from it. I > just compiled 2.3.4 and compiler stderr is full of interesting problems. > oh please
Re: Sieve script unable to file into public folder after dovecot upgrade to version 2.3.4
Hi all, it has turned out that it actually was an ACL issue. I had to add "p" to the rule: authenticated lrsp Now fileinto is working again :-) Best regards Andreas Am 2. Januar 2019 13:10:03 schrieb Andreas Oster : Hi all, Wishing you and your families a Happy New Year 2019. I am currently struggling with an odd sieve issue after upgrading dovecot from 2.2.32 to the latest 2.3.4 version. OS is Ubuntu 18.04. For one of our users I have a sieve rule in place which should copy (keep + fileinto) incoming mails from the INBOX to a public namespace folder. Unfortunately this does not work anymore. In the sieve log file of the user I get error messages like this: sieve: info: started log at Jan 02 12:41:31. error: msgid=: failed to store into mailbox 'Public/Verteiler': Permission denied. This is the content of the sieve script : require ["fileinto"]; keep; fileinto "Public/Verteiler"; stop; At first I thought it might be caused by a faulty IMAP ACL but this does not seem to be the case. When I do a manual copy in Thunderbird from the INBOX to this public folder there are no problems at all. Does anybody have an idea what could be the cause of the issue ? Thank you very much for your kind help. Best regards Andreas
Re: Moving Alternate Storage to another disk.
Hi, if you have the new disk installed on the same server you can try: - mount new disk for example in /mnt/temp - rsync -vaWH all files + directories from the old to the new disk - stop dovecot so no changes will happen on disks - make a final rsync again -> should not take many time - umout the old disk - mount new disk to the original "alt-storage" path, so you don't have to change each soft-link in each users directory. - start dovecot Not tested, but in theory it should work. Best Urban Am 31.12.18 um 15:02 schrieb bOnK: Hello, Dovecot 2.3.4_3 on FreeBSD 11.2. I am using mdbox Alternate Storage since about two years without any problems. However, the disk containing this storage is almost full and I have to move this data to another disk, probably zpool. Would it be okay to do the following? 1) Shut down dovecot (and mail server) so no new mail comes in. 2) Copy/move all files in ALT location to new disk, using shell commands like cp/mv/cpdup. 3) Change the path to ALT in dovecot-conf mail_location. 4) Change the 'dbox-alt-root' soft-links in each users main (INBOX) directory to point to this new location. 5) Start up dovecot and mail server. Am I missing something or maybe there is a better way?
Re: gcc -> clang
We compile all core code with both gcc and clang. What sort of interesting things did you find? Aki On 03 January 2019 at 11:50 Rupert Gallagher via dovecot < dovecot@dovecot.org> wrote: Please, use clang instead of gcc. Code quality can only profit from it. I just compiled 2.3.4 and compiler stderr is full of interesting problems. --- Aki Tuomi
gcc -> clang
Please, use clang instead of gcc. Code quality can only profit from it. I just compiled 2.3.4 and compiler stderr is full of interesting problems.
