Bug#771341: segfaults in sqlite3_value_type while using from Python

[Marc F. Clemente]

On 10/24/2015 12:12 PM, László Böszörményi (GCS) wrote:

On Mon, Mar 16, 2015 at 3:10 PM, Marc F. Clemente  wrote:

One is an Intel i7.  Four others are xen virtual machines running on an Intel 
Xeon.

  On which computers do you get the segfaults? Can you try recent
SQLite3 package uploads, especially 3.9.1?


When I look at my logs, I have not had any segfaults since I wrote that 
email on 16 March 2015.  I completely forgot about this bug.


Currently running 3.8.11.1-1.


Do you have a minimal installation maybe where the crash happens?


Not any more.

Marc

Bug#771341: segfaults in sqlite3_value_type while using from Python

[GCS]

On Mon, Mar 16, 2015 at 3:10 PM, Marc F. Clemente  wrote:
> One is an Intel i7.  Four others are xen virtual machines running on an Intel 
> Xeon.
 On which computers do you get the segfaults? Can you try recent
SQLite3 package uploads, especially 3.9.1?
Do you have a minimal installation maybe where the crash happens?

Laszlo/GCS

Bug#771341: segfaults in sqlite3_value_type while using from Python

[GCS]

On Mon, Mar 16, 2015 at 1:11 PM, Marc F. Clemente m...@mclemente.net wrote:
 For what it's worth, I am still getting segfaults in version 3.8.7.4-1.
 Multiple different computers, all amd64.
 What kind of CPU do you have? Intel or AMD?

Laszlo/GCS


-- 
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#771341: segfaults in sqlite3_value_type while using from Python

[Marc F. Clemente]

 On Mar 16, 2015, at 8:43 AM, László Böszörményi (GCS) g...@debian.org wrote:
 
 On Mon, Mar 16, 2015 at 1:11 PM, Marc F. Clemente m...@mclemente.net wrote:
 For what it's worth, I am still getting segfaults in version 3.8.7.4-1.
 Multiple different computers, all amd64.
 What kind of CPU do you have? Intel or AMD?

One is an Intel i7.  Four others are xen virtual machines running on an Intel 
Xeon.

Marc


--
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#771341: segfaults in sqlite3_value_type while using from Python

[Marc F. Clemente]

For what it's worth, I am still getting segfaults in version 3.8.7.4-1. 
 Multiple different computers, all amd64.


kernel: [413524.044820] fail2ban-server[10402]: segfault at 8 ip 
7fbdb22b2350 sp 7fbdb1a5d808 error 4 in 
libsqlite3.so.0.8.6[7fbdb22a1000+c3000]


and

kernel: [663408.925224] fail2ban-server[23549]: segfault at 8 ip 
7f115455d350 sp 7f1153d08808 error 4 in 
libsqlite3.so.0.8.6[7f115454c000+c3000]


Marc


--
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#771341: segfaults in sqlite3_value_type while using from Python

[GCS]

On Sat, Nov 29, 2014 at 4:36 PM, Yaroslav Halchenko y...@debian.org wrote:
 On Sat, 29 Nov 2014, László Böszörményi (GCS) wrote:
  May you give me some details how it happened in Sid?

 ok -- I will try to replicate again under sid (trickier since I have no sid on
 publicly bombarded servers)
 You may also backport the recent, SQLite3 3.8.7.2-1 upload to your box.

  unfortunately we haven't logged the sql queries so not sure on exact one, 
  but I
  think it was this one, which if executed from the shell seems to not cause 
  the
  segfault...

  n {1..100}; do sqlite3 -list fail2ban.sqlite3 'SELECT ip, timeofban, data 
  FROM bans WHERE 1 AND jail=sshd AND ip=111.74.239.35 ORDER BY ip, 
  timeofban' /dev/null  echo success; done
 You may try the same loop with a Python script doing the same SQLite3 query.

 FWIW -- I am the Fail2ban maintainer... but fail2ban is just a 'trigger'
 here -- it is a purely Python implementation so either it is sqlite3 or python
 bindings which are at fault here
 Then I do wonder why do you use an unofficial fail2ban backport
instead of doing it yourself.

Cheers,
Laszlo/GCS


--
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#771341: segfaults in sqlite3_value_type while using from Python

[Yaroslav Halchenko]

On Wed, 03 Dec 2014, László Böszörményi (GCS) wrote:

 On Sat, Nov 29, 2014 at 4:36 PM, Yaroslav Halchenko y...@debian.org wrote:
  On Sat, 29 Nov 2014, László Böszörményi (GCS) wrote:
   May you give me some details how it happened in Sid?

  ok -- I will try to replicate again under sid (trickier since I have no sid 
  on
  publicly bombarded servers)
  You may also backport the recent, SQLite3 3.8.7.2-1 upload to your box.

good idea... meanwhile #771549 confirmed issue on sid
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771341#20

   unfortunately we haven't logged the sql queries so not sure on exact 
   one, but I
   think it was this one, which if executed from the shell seems to not 
   cause the
   segfault...

   n {1..100}; do sqlite3 -list fail2ban.sqlite3 'SELECT ip, timeofban, 
   data FROM bans WHERE 1 AND jail=sshd AND ip=111.74.239.35 ORDER BY 
   ip, timeofban' /dev/null  echo success; done
  You may try the same loop with a Python script doing the same SQLite3 query.

yet another good idea- - I will do when I hit it again (I have managed
not to keep a copy of that DB as of during segfault :-/)

  FWIW -- I am the Fail2ban maintainer... but fail2ban is just a 'trigger'
  here -- it is a purely Python implementation so either it is sqlite3 or 
  python
  bindings which are at fault here
  Then I do wonder why do you use an unofficial fail2ban backport
 instead of doing it yourself.

doing what myself? backport?  this my backport ;)

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik


-- 
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#771341: segfaults in sqlite3_value_type while using from Python

[GCS]

On Fri, Nov 28, 2014 at 5:50 PM, Yaroslav Halchenko
deb...@onerussian.com wrote:
 Package: libsqlite3-0
 Version: 3.7.13-1+deb7u1
 Severity: serious

 Originally detected I believe on sid installation, but forgot to capture the
 version, I will try to replicate/report later.  But very consistent with 
 wheezy
 (from which I am reporting now).
 May you give me some details how it happened in Sid?

 Triggered by the backport fail2ban 0.9.1-1~nd70+1 (available from
 http://neuro.debian.net/debian-devel/ wheezy/main amd64 Packages  apt repo) it
 gets to
[...]
 The problem is, I don't see the segfault in the mentioned gdb output.

 unfortunately we haven't logged the sql queries so not sure on exact one, but 
 I
 think it was this one, which if executed from the shell seems to not cause the
 segfault...

 n {1..100}; do sqlite3 -list fail2ban.sqlite3 'SELECT ip, timeofban, data 
 FROM bans WHERE 1 AND jail=sshd AND ip=111.74.239.35 ORDER BY ip, 
 timeofban' /dev/null  echo success; done
 Then how often do you get segfault? Do you have any additional
information if it happens in a given daytime, when there are several
bots try to get into your system or anything else?

 Please help me to troubleshoot this one if more information is necessary
 to point the issue
 I'm the SQLite3 maintainer and not the fail2ban one. But please note
the changelog the version you use[1]:
 - 0.9 series is quite a big leap in development, especially since 0.8.6
   which made it to previous Debian stable wheezy.  Please consult upstream
   ChangeLog about changes

Did you check it, reviewed your configuration? Does a segfault happen
in other applications that link to SQLite3?

Regards,
Laszlo/GCS
[1] https://packages.qa.debian.org/f/fail2ban/news/20141028T041852Z.html


-- 
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#771341: segfaults in sqlite3_value_type while using from Python

[Yaroslav Halchenko]

Hi László,

Thanks for getting back to me

On Sat, 29 Nov 2014, László Böszörményi (GCS) wrote:
  Package: libsqlite3-0
  Version: 3.7.13-1+deb7u1
  Severity: serious

  Originally detected I believe on sid installation, but forgot to capture the
  version, I will try to replicate/report later.  But very consistent with 
  wheezy
  (from which I am reporting now).
  May you give me some details how it happened in Sid?

ok -- I will try to replicate again under sid (trickier since I have no sid on
publicly bombarded servers)

  Triggered by the backport fail2ban 0.9.1-1~nd70+1 (available from
  http://neuro.debian.net/debian-devel/ wheezy/main amd64 Packages  apt repo) 
  it
  gets to
 [...]
  The problem is, I don't see the segfault in the mentioned gdb output.

well -- I didn't think it would be useful since backtrace has that call
on top and you would just trust me ;) but here you go:

# gdb --args /usr/bin/python-dbg /usr/bin/fail2ban-server -s 
/var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -f
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type show copying
and show warranty for details.
This GDB was configured as x86_64-linux-gnu.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/python-dbg...done.
(gdb) c
The program is not being run.
(gdb) r
Starting program: /usr/bin/python-dbg /usr/bin/fail2ban-server -s 
/var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library /lib/x86_64-linux-gnu/libthread_db.so.1.
2014-11-24 08:52:18,796 fail2ban.server [29794]: INFOStarting 
Fail2ban v0.9.1
2014-11-24 08:52:26,467 fail2ban.server [29794]: INFOStopping all 
jails
[New Thread 0x752a4700 (LWP 29812)]
[New Thread 0x74aa3700 (LWP 29813)]
[New Thread 0x7fffe700 (LWP 29814)]
[Thread 0x752a4700 (LWP 29812) exited]
[Thread 0x7fffe700 (LWP 29814) exited]
[Thread 0x74aa3700 (LWP 29813) exited]
[New Thread 0x74aa3700 (LWP 29895)]
[New Thread 0x7fffe700 (LWP 29896)]
[New Thread 0x752a4700 (LWP 29897)]
[Thread 0x74aa3700 (LWP 29895) exited]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x752a4700 (LWP 29897)]
sqlite3_value_type (pVal=0x0) at sqlite3.c:63805
63805   sqlite3.c: No such file or directory.
(gdb)


  unfortunately we haven't logged the sql queries so not sure on exact one, 
  but I
  think it was this one, which if executed from the shell seems to not cause 
  the
  segfault...

  n {1..100}; do sqlite3 -list fail2ban.sqlite3 'SELECT ip, timeofban, data 
  FROM bans WHERE 1 AND jail=sshd AND ip=111.74.239.35 ORDER BY ip, 
  timeofban' /dev/null  echo success; done
  Then how often do you get segfault? 

once in few days

 Do you have any additional
 information if it happens in a given daytime, when there are several
 bots try to get into your system or anything else?

I will keep monitoring/analyzing more -- so far haven't spotted 

  Please help me to troubleshoot this one if more information is necessary
  to point the issue
  I'm the SQLite3 maintainer and not the fail2ban one

FWIW -- I am the Fail2ban maintainer... but fail2ban is just a 'trigger'
here -- it is a purely Python implementation so either it is sqlite3 or python
bindings which are at fault here

. But please note
 the changelog the version you use[1]:
  - 0.9 series is quite a big leap in development, especially since 0.8.6
which made it to previous Debian stable wheezy.  Please consult 
 upstream
ChangeLog about changes

 Did you check it, reviewed your configuration? 

configuration of ... fail2ban?  it is somewhat minimalistic -- just a single
jail, that is why my original suspicion of possibly incorrect locking for the
same DB across threads didn't get much of my mental support ;) Although there
is still also few threads  involved :

  Id   Target Id Frame 
* 7Thread 0x752a4700 (LWP 29897) python-dbg sqlite3_value_type 
(pVal=0x0) at sqlite3.c:63805
  6Thread 0x7fffe700 (LWP 29896) python-dbg 0x00525338 in 
PyEval_EvalCodeEx (co=0xbe9ca0, globals=
{'logMultiprocessing': 1, '__path__': ['/usr/lib/python2.7/logging'], 
'LogRecord': type at remote 0xc29c70, 'logProcesses': 1, 'addLevelName': 
function at remote 0xc6e6f0, '_addHandlerRef': function at remote 0xc70798, 
'WARNING': 30, 'fatal': function at remote 0xc751b0, 'currentframe': 
function at remote 0xc6e648, 'INFO': 20, '_startTime': float at remote 
0x9d38f8, '__file__': '/usr/lib/python2.7/logging/__init__.pyc', 
'BufferingFormatter': type at remote 0xc2aca0, 'NOTSET': 0, '_levelNames': 
{0: 'NOTSET', 'NOTICE': 25, 10: 'DEBUG', 

Bug#771341: segfaults in sqlite3_value_type while using from Python

[Yaroslav Halchenko]

Package: libsqlite3-0
Version: 3.7.13-1+deb7u1
Severity: serious

Originally detected I believe on sid installation, but forgot to capture the
version, I will try to replicate/report later.  But very consistent with wheezy
(from which I am reporting now).

Triggered by the backport fail2ban 0.9.1-1~nd70+1 (available from
http://neuro.debian.net/debian-devel/ wheezy/main amd64 Packages  apt repo) it
gets to

#0  sqlite3_value_type (pVal=0x0) at sqlite3.c:63805
#1  0x7570ee2e in sqlite3_column_type (pStmt=0x114eec8, i=optimized 
out) at sqlite3.c:64447
#2  0x7599b3d4 in _pysqlite_fetch_one_row (self=0x11e3540) at 
/home/jmm/py/python2.7-2.7.3/Modules/_sqlite/cursor.c:363
#3  0x7599d71d in pysqlite_cursor_iternext (self=0x11e3540) at 
/home/jmm/py/python2.7-2.7.3/Modules/_sqlite/cursor.c:939
#4  0x0046d967 in listextend (self=0x107cb28, b=sqlite3.Cursor at 
remote 0x11e3540) at ../Objects/listobject.c:872
#5  0x00471932 in list_init (self=0x107cb28, args=(sqlite3.Cursor at 
remote 0x11e3540,), kw=0x0) at ../Objects/listobject.c:2458
#6  0x004b2092 in type_call (type=0x86dde0, args=(sqlite3.Cursor at 
remote 0x11e3540,), kwds=0x0) at ../Objects/typeobject.c:737
#7  0x004291af in PyObject_Call (func=type at remote 0x86dde0, 
arg=(sqlite3.Cursor at remote 0x11e3540,), kw=0x0) at 
../Objects/abstract.c:2529
#8  0x0052977e in do_call (func=type at remote 0x86dde0, 
pp_stack=0x752a0490, na=1, nk=0) at ../Python/ceval.c:4239
#9  0x00528b3b in call_function (pp_stack=0x752a0490, oparg=1) at 
../Python/ceval.c:4044
#10 0x00523733 in PyEval_EvalFrameEx (f=
Frame 0x10f0ba0, for file 
/usr/lib/python2.7/dist-packages/fail2ban/server/database.py, line 467, in 
getBansMerged 
(self=Fail2BanDb(_dbFilename='/var/lib/fail2ban/fail2ban.sqlite3', 
_db=sqlite3.Connection at remote 0x11178c0, _lock=thread.lock at remote 
0xf2ad10, _purgeAge=86400, _bansMergedCache={('113.12.202.186', 
Jail(_Jail__actions=Actions(active=True, _Thread__ident=140737306576640, 
run=function at remote 0x1056840, _jail=..., 
_Thread__block=_Condition(_Verbose__verbose=False, 
_Condition__lock=thread.lock at remote 0x106f0e0, acquire=built-in method 
acquire of thread.lock object at remote 0x106f0e0, _Condition__waiters=[], 
release=built-in method release of thread.lock object at remote 0x106f0e0) at 
remote 0x1070290, _Thread__name='Thread-5', _Thread__daemonic=False, 
_actions=OrderedDict(_OrderedDict__root=[[[...], [...], 'iptables-multiport'], 
[...], None], _OrderedDict__map={'iptables-multiport': [...]}) at remote 
0x11181c0, sleeptime=1, _Thread__start
 ed=_Event(_Verbose__verbose=F...(truncated), throwflag=0) at 
../Python/ceval.c:2666
#11 0x00526121 in PyEval_EvalCodeEx (co=0xf18670, globals=
{'locale': module at remote 0xe4f7b0, 'commitandrollback': function at 
remote 0xf2bc30, 'MyTime': classobj at remote 0xd51520, 'Fail2BanDb': type 
at remote 0xf3a820, 'shutil': module at remote 0xf1be98, 'wraps': function 
at remote 0xcc1258, '__package__': 'fail2ban.server', 'json': module at 
remote 0xeb6f00, 'sqlite3': module at remote 0xf1b3a0, 'FailTicket': 
classobj at remote 0xd51228, '__doc__': None, '__license__': 'GPL', 
'__builtins__': {'bytearray': type at remote 0x85fd20, 'IndexError': type at 
remote 0x866360, 'all': built-in function all, 'help': _Helper at remote 
0xb25fb0, 'vars': built-in function vars, 'SyntaxError': type at remote 
0x865ce0, 'unicode': type at remote 0x87d040, 'UnicodeDecodeError': type at 
remote 0x866c80, 'memoryview': type at remote 0x870c00, 'isinstance': 
built-in function isinstance, 'copyright': 
_Printer(_Printer__data='Copyright (c) 2001-2012 Python Software 
Foundation.\nAll Rights Reserved.\n\nCopyright (c
 ) 2000 BeOpen.com.\nAll Rights...(truncated), locals=0x0, args=0x11556b8, 
argcount=1, kws=
0x11556c0, kwcount=2, defs=0xf28c08, defcount=3, closure=0x0) at 
../Python/ceval.c:3253
#12 0x00528f36 in fast_function (func=function at remote 0xf58450, 
pp_stack=0x752a0950, n=5, na=1, nk=2) at ../Python/ceval.c:4117
#13 0x00528b1f in call_function (pp_stack=0x752a0950, oparg=512) at 
../Python/ceval.c:4042
#14 0x00523733 in PyEval_EvalFrameEx (f=
Frame 0x1155520, for file 
/usr/lib/python2.7/dist-packages/fail2ban/server/actions.py, line 274, in 
lambda (jail=Jail(_Jail__actions=Actions(active=True, 
_Thread__ident=140737306576640, run=function at remote 0x1056840, 
_jail=..., _Thread__block=_Condition(_Verbose__verbose=False, 
_Condition__lock=thread.lock at remote 0x106f0e0, acquire=built-in method 
acquire of thread.lock object at remote 0x106f0e0, _Condition__waiters=[], 
release=built-in method release of thread.lock object at remote 0x106f0e0) at 
remote 0x1070290, _Thread__name='Thread-5', _Thread__daemonic=False, 
_actions=OrderedDict(_OrderedDict__root=[[[...], [...], 'iptables-multiport'], 
[...], None], _OrderedDict__map={'iptables-multiport': [...]}) at