Bug Tracker item #2896879, was opened at 2009-11-12 15:19
Message generated for change (Comment added) made by aredridel
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=2896879&group_id=250683
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.9.0
Status: Open
Resolution: None
Priority: 9
Private: No
Submitted By: Aria Stewart (aredridel)
Assigned to: Stevan Bajic (sbajic)
Summary: Crash with webui training a fallbackDomain user
Initial Comment:
(gdb) bt
#0 0x000000310987bc70 in strcasecmp () from /lib64/libc.so.6
#1 0x0000000000407458 in ctx_init (ATX=0x7fffbf5e8d40, username=0x0)
at dspam.c:2429
#2 0x000000000040a5bf in process_message (ATX=0x7fffbf5e8d40,
message=0x1005ad0, username=0x0, result_string=0x7fffbf5e8cd8)
at dspam.c:358
#3 0x000000000040c069 in process_users (ATX=0x7fffbf5e8d40, message=0x1001a60)
at dspam.c:1865
#4 0x000000000040cbb8 in main (argc=6, argv=0x7fffbf5e9cd8) at dspam.c:265
this is with a user that's a shared group in the groups file, and is a fallback
user in the preferences.
3.9.0 git
----------------------------------------------------------------------
>Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 17:59
Message:
It also /does/ retrain even so, now, so I could theoretically throw this
into production. What's it not doing that it should, if it is in fact
retraining the message?
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 17:58
Message:
Ah, excellent. I do!
Nov 20 19:56:08 24-75-24-220 dspam[24648]: process_users(): Can not
fallback to domains for username 'aria.hydra.blumenthals.com' without
@domain part.
I don't entirely understand what it's saying, but at least it flags it.
It's implying the username has to be @domain? (Is there any reason this has
to be the case?)
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-20 17:48
Message:
This here is not going to work with fallback domains:
--user aria.hydra.blumenthals.com
I submitted yesterday a bunch of patches dealing with that in the proper
way since I suspected something to happen in your setup and I just wanted
to handle that and print out a message informing the user that he/she is
doing wrong. Here is the submitted code:
http://dspam.git.sourceforge.net/git/gitweb.cgi?p=dspam/dspam;a=commit;h=5183294f11e985b242d3817ae47eef9f1d8e9d2c
Do you get another message if you compile DSPAM again from GIT and re-run
the test?
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 17:39
Message:
That's running on the commandline, but the same command that dspam.cgi
executes with system()
> How do you run the CGI's in the WebUI (SuEXEC? Other way?)
Directly as CGI. apache is in dspam.conf's Trust list.
- With what user?
domain.com -- the domain user. It's got an entry for @domain.com with the
same uid in dspam_virtual_uids. If I don't, the web UI won't show the
history for the user.
- With what user/group does the WebUI run?
apache
- Can you post the output of "ls -lah" against the dspam binary or post
the output of: ls -lah $(whereis dspam|awk '{print $2}')
-rwx--s--- 1 dspam mail 274K Nov 19 00:06 /usr/bin/dspam
dspam: /usr/bin/dspam /etc/dspam /usr/share/dspam
/usr/share/man/man1/dspam.1.gz
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-20 17:25
Message:
The errors you posted are when executing the commands from the command line
or when executing from within the WebUI?
Can you answer the following questions:
- How do you run the CGI's in the WebUI (SuEXEC? Other way?)
- With what user?
- With what user/group does the WebUI run?
- Can you post the output of "ls -lah" against the dspam binary or post
the output of: ls -lah $(whereis dspam|awk '{print $2}')
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 17:15
Message:
Pardon. I mean that that fixes the asking the wrong question problem, but
leaves the greater problem -- not being able t train from the webui.
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 17:13
Message:
I had that in my dspam.conf, same issue.
I've tried with pgsql (now that the compilation bug is fixed, I got that
working) -- same issue. Looks like it's not in any of the db-specific
code:
dspam --source=error --class=innocent
--signature=3,4b072ebc91645760821076 --user aria.hydra.blumenthals.com
gives me
Nov 20 19:08:05 24-75-24-220 dspam[9489]: Unable to determine the
destination user
Nov 20 19:08:05 24-75-24-220 dspam[9489]: Context initialization failed
Nov 20 19:08:05 24-75-24-220 dspam[9489]: process_message returned error
-2. dropping message.
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-20 13:28
Message:
Please add in your dspam.conf this here:
MySQLUIDInSignature on
Restart the daemon and then try again.
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 08:53
Message:
Things are added to the database with uid = 4, signature =
4b06b9f741961536915273, for example, but if I strace dspam when retraining
with --signature=4,4b06b9f741961536915273, I see that dspam makes this
query:
write(3, "b\0\0\0\3SELECT data,length FROM dspam_signature_data WHERE
uid=4 AND signature=\"4,4b06b9f741961536915273\"", 102) = 102
Any idea why it'd ask the wrong question?
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 11:52
Message:
[r...@spamfilter ~]# dspam --version
DSPAM Anti-Spam Suite 3.9.0-BETA4 (agent/library)
Copyright (c) 2002-2009 DSPAM Project
http://dspam.sourceforge.net.
DSPAM may be copied only under the terms of the GNU General Public
License,
a copy of which can be found with the DSPAM distribution kit.
Configuration parameters: '--host=x86_64-redhat-linux-gnu'
'--build=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--localstatedir=/var'
'--sharedstatedir=/usr/com' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--sysconfdir=/etc/dspam'
'--with-dspam-home=/var/spool/dspam' '--with-logdir=/var/log/dspam'
'--enable-shared' '--enable-static' '--enable-daemon'
'--enable-external-lookup' '--enable-clamav' '--enable-debug'
'--enable-syslog' '--enable-long-usernames' '--enable-domain-scale'
'--enable-virtual-users'
'--with-storage-driver=hash_drv,mysql_drv,pgsql_drv'
'--with-mysql-includes=/usr/include/mysql'
'--with-mysql-libraries=/usr/lib64/mysql' 'CFLAGS=-O2 -g -m64
-mtune=generic' 'CXXFLAGS=-O2 -g -m64 -mtune=generic' 'FFLAGS=-O2 -g -m64
-mtune=generic' 'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux'
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 11:51
Message:
dspam.conf:
Home /var/spool/dspam
StorageDriver /usr/lib64/dspam/libmysql_drv.so
DeliveryHost 127.0.0.1
DeliveryPort 10025
DeliveryIdent localhost
DeliveryProto SMTP
FallbackDomains on
OnFail error
Trust root
Trust dspam
Trust apache
Trust mail
Trust mailnull
Trust smmsp
Trust daemon
Debug *
DebugOpt process spam fp classify
ClassAliasSpam spam
TrainingMode teft
TestConditionalTraining on
Feature whitelist
Algorithm graham burton
Tokenizer chain
PValue bcr
WebStats on
ImprobabilityDrive on
Preference "spamAction=quarantine"
Preference "signatureLocation=message"
Preference "showFactors=on"
Preference "fallbackDomain=on"
AllowOverride trainingMode
AllowOverride spamAction spamSubject
AllowOverride statisticalSedation
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride signatureLocation
AllowOverride showFactors
AllowOverride optIn optOut
AllowOverride whitelistThreshold
AllowOverride dailyQuarantineSummary
AllowOverride fallbackDomain
MySQLServer /var/lib/mysql/mysql.sock
MySQLUser dspam
MySQLPass redacted
MySQLDb dspam
MySQLReconnect true
MySQLConnectionCache 10
PgSQLServer 127.0.0.1
PgSQLPort 5432
PgSQLUser dspam
PgSQLDb dspam
PgSQLConnectionCache 3
PgSQLUIDInSignature on
HashRecMax 98317
HashAutoExtend on
HashMaxExtents 0
HashExtentSize 49157
HashPctIncrease 10
HashMaxSeek 10
HashConnectionCache 10
Notifications on
PurgeSignatures 14
PurgeNeutral 90
PurgeUnused 90
PurgeHapaxes 30
PurgeHits1S 15
PurgeHits1I 15
LocalMX 127.0.0.1
SystemLog on
UserLog on
Opt out
ParseToHeaders on
ChangeModeOnParse on
ChangeUserOnParse full
ClamAVPort 3310
ClamAVHost 127.0.0.1
ClamAVResponse reject
ServerDomainSocketPath /var/run/dspam/dspam.sock
ServerMode auto
ServerQueueSize 32
ServerMode auto
ServerPass.Relay1 redacted
ServerParameters "--deliver=innocent -d %u"
ServerIdent "spamfilter.blumenthals.com"
ClientHost /var/run/dspam/dspam.sock
ClientIdent redac...@relay1
ProcessorURLContext on
ProcessorBias on
StripRcptDomain off
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 11:33
Message:
Well, it logs and stops the segfault with that patch. Now to determine why
it's null ...
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 11:18
Message:
Looking at it now. Still happens with current git, but I'm trying other
usernames in the group to see if that's the problem.
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-15 15:41
Message:
Hallo Aredridel,
do you have any update on this case?
--
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-13 04:48
Message:
Hallo Aredridel,
when looking at your stack trace I see that ctx_init() get's called with
an NULL username. This should NOT happen. I could trap that case in
ctx_init(). Better: I must trap that case. Can you apply this patch to the
latest GIT source and try if the crash still presists:
------------------------
diff --git a/src/dspam.c b/src/dspam.c
index 7bfe301..b97f4a5 100644
--- a/src/dspam.c
+++ b/src/dspam.c
@@ -1,4 +1,4 @@
-/* $Id: dspam.c,v 1.378 2009/11/13 02:26:19 sbajic Exp $ */
+/* $Id: dspam.c,v 1.379 2009/11/13 12:44:59 sbajic Exp $ */
/*
DSPAM
@@ -2367,6 +2367,12 @@ NEXT:
*/
DSPAM_CTX *ctx_init(AGENT_CTX *ATX, const char *username) {
+
+ if (username == NULL) {
+ LOG (LOG_CRIT, ERR_AGENT_USER_UNDEFINED);
+ return NULL;
+ }
+
DSPAM_CTX *CTX;
char filename[MAX_FILENAME_LENGTH];
char ctx_group[128] = { 0 };
------------------------
I would say that the crash is gone but you will have a run time error
because DSPAM can not work on a undefined user name.
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-13 04:34
Message:
Hallo Aredridel,
I can not reproduce this here. What I have done so far:
dspam.conf:
FallbackDomains on
MySQL:
insert into dspam_virtual_uids (uid,username) values
(null,'@domain.com');
Shell (I run a DSPAM in OptIn mode):
dspam_admin change preference @domain.com fallbackDomain on
dspam_admin change preference @domain.com optOut off
dspam_admin change preference @domain.com optIn on
After that my data in MySQL looks like this:
dspam_virtual_uids:
+------+-------------+
| uid | username |
+------+-------------+
| 2087 | @domain.com |
+------+-------------+
dspam_preferences:
+------+----------------+-------+
| uid | preference | value |
+------+----------------+-------+
| 2087 | fallbackDomain | on |
| 2087 | optIn | on |
| 2087 | optOut | off |
+------+----------------+-------+
group file:
@domain.com:shared:*...@domain.com
Forced a learning for a bunch of messages. All of them resulted in the
same output:
nyx ~ # dspam --user @domain.com --deliver=summary --stdout
--source=corpus --class=spam < path/to/message1.txt
X-DSPAM-Result: @domain.com; result="Spam"; class="Spam";
probability=1.0000; confidence=1.00; signature=N/A
nyx ~ #
Tokens where added to the MySQL table:
> select count(*) from dspam_token_data where uid=2087;
+----------+
| count(*) |
+----------+
| 36377 |
+----------+
Now did a test with a user that does NOT exist in dspam_virtual_uids:
nyx ~ # dspam --user t...@domain.com --deliver=summary --stdout --process
< path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: @domain.com; result="Spam"; class="Spam";
probability=1.0000; confidence=0.99; signature=2087,4afd3dd512221635636504
X-DSPAM-Result: Spam
X-DSPAM-Processed: Fri Nov 13 12:07:01 2009
X-DSPAM-Confidence: 0.9899
X-DSPAM-Improbability: 1 in 9809 chance of being ham
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2087,4afd3dd512221635636504
nyx ~ #
Looks okay to me. The uid used for that user is the one from the user
"@domain.com". Okay. Now let's try to retrain that message:
nyx ~ # dspam --user t...@domain.com --stdout --source=error
--class=innocent --signature=2087,4afd3dd512221635636504
nyx ~ #
Works. I see in the Web-UI that the message got changed. Now retrain it
back (undo) to spam:
nyx ~ # dspam --user t...@domain.com --stdout --source=error --class=spam
--signature=2087,4afd3dd512221635636504
nyx ~ #
Works as well.
The only difference I see in your and my setup is that I used "@domain" as
the group name for the shared group. You used "domain.com" but "domain.com"
does not exist as user in the dspam_virtual_uids table so I used
"@domain.com". Anyway... going back to just use "domain.com" for the group
(@domain.com:shared:*...@domain.com) lead in the same result. Except that this
time the result from DSPAM was innocent instead of spam but that is
probably because the used user "domain.com" for the shared group does not
exist (only @domain.com exists as an user):
nyx ~ # dspam --user t...@domain.com --deliver=summary --stdout --process
< path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: @domain.com; result="Innocent"; class="Innocent";
probability=0.0000; confidence=0.70; signature=2087,4afd3f6b15271305324895
nyx ~ #
==> The strange thing here is that I get a summary result but "--stdout"
should FORCE the whole message to be displayed to the console (incl. a
summary). Something is strange here. I can however force deliver to show up
on screen by instructing DSPAM to deliver innocent or spam mails (which
should have been selected by the "--stdout" switch already):
nyx ~ # dspam --user t...@domain.com --deliver=innocent,spam --stdout
--process < path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: Innocent
X-DSPAM-Processed: Fri Nov 13 12:22:48 2009
X-DSPAM-Confidence: 0.6701
X-DSPAM-Improbability: 1 in 204 chance of being spam
X-DSPAM-Probability: 0.0000
X-DSPAM-Signature: 2087,4afd418820052092820584
nyx ~ #
--> BINGO! That is the right output. It should have been that output from
the beginning. Anyway... this is another issue I need to look after later.
I went ahead and retrained that message to be spam:
nyx ~ # dspam --user t...@domain.com --stdout --source=error --class=spam
--signature=2087,4afd3f6b15271305324895
nyx ~ #
The Web-UI changed and the message was switched to spam.
What have I done on my setup that you have not? Can you attach your
dspam.conf and post the output of "dspam --version"?
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-12 23:29
Message:
I do have the rest as you suggested.
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-12 16:40
Message:
I have:
domain.com:shared:*...@domain.com
The docs were a little unclear how it all fits together, the username
munging, etc.
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-12 15:38
Message:
Hallo Aredridel,
how can you mirror that setup on my end? What do I need to have in the
group file and what in the preference extension?
If I have understood your setup then it's +/- like this:
group file:
@domain.com:shared,managed:*...@another.com
preferences for domain.com:
fallbackDomain=on
dspam.conf:
FallbackDomains on
Right?
Kind Regards from Switzerland,
Stevan Bajic
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=2896879&group_id=250683
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Dspam-devel mailing list
Dspam-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspam-devel
