Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

It wasn't memory.  Or at least nothing obvious.  I ran Memtest86+ 5.01 in
SMP mode for 3 full passes (about 10 hours) with no errors.

On reflection, it didn't seem likely to be a kernel freeze anyway.  It
wasn't that the light on the keyboard was unresponsive, or not just that,
but that all the lights were off including the mouse laser.  The sort of
general "on" light in the tower was on, and the power supply fan was going,
but that's all.  I now wish I had looked at the motherboard readouts.
Anyway, that suggested a power distribution or power supply problem to me.
I just don't know how to bifurcate between the power supply and something
on the mobo USB interface.  And it hasn't happened again.

On Wed, Dec 7, 2016 at 5:18 PM, Kevin O'Gorman <kevinogorm...@gmail.com>
wrote:

> Good feedback.  I haven't done a memory check on that machine in a
> while
>
> Next on my list.
>
> On Sun, Dec 4, 2016 at 11:25 AM, Keith Medcalf <kmedc...@dessus.com>
> wrote:
>
>>
>> If pressing the CAPS LOCK or NUM LOCK keys on the keyboard does not
>> toggle the light on the keyboard then you have lost the all interrupt
>> processing since those keypresses have to be processed by the kernel mode
>> keyboard driver toggling the internal state of the keyboard driver, and
>> then the kernel driver sends output to the keyboard to change the status
>> LEDs.  Typically (all Operating Systems) this means you have suffered a
>> complete kernel crash (or halt) and the system is not running.
>>
>> Since the system must be running in order to output indicator status, all
>> indicators will stay "stuck" in their last known position (hold output).
>>
>> Only a power-cycle (or hardware reset -- assuming the RESET is a hardware
>> reset and not just a request to reset which will be ignored) triggering a
>> reboot will restart the system.
>>
>> The most frequent cause is a Parity Check.  Or in these latter days of
>> not having ECC or even Parity checked memory, just an undetected memory
>> fault which will cause random AHTBL.
>>
>> > -Original Message-
>> > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org
>> ]
>> > On Behalf Of Kevin O'Gorman
>> > Sent: Sunday, 4 December, 2016 09:21
>> > To: SQLite mailing list
>> > Subject: Re: [sqlite] I keep getting seg faults building my database
>> using
>> > python sqlite3
>> >
>> > Well, the i7 system failed again, but this time it was quite different.
>> > And peculiar.
>> > The system wasn't doing anything, but it should have been.  So I tried
>> > something. It didn't matter what, because I could not get the mouse or
>> > keyboard to work -- it was like they weren't plugged in.  Really like
>> it,
>> > because the caps lock light wasn't on, nor was the laser light visible
>> in
>> > the mouse.  Even when I changed mouse, keyboard and USB slot.  I
>> couldn't
>> > get in with SSH from elsewhere either.  But the computer's "I'm running"
>> > light was on.
>> > So I'm suspecting a partial power failure.  I don't know enough about
>> > mobos
>> > and USB to diagnose whether the problem was on the mobo or the power
>> > supply.
>> >
>> > Creepty.  I had to do a hard reset  to get thing going again, and it's
>> > been
>> > running fine for a day now.
>> >
>> > On Mon, Nov 21, 2016 at 9:51 AM, Kevin O'Gorman <
>> kevinogorm...@gmail.com>
>> > wrote:
>> >
>> > > On Mon, Nov 21, 2016 at 9:41 AM, Roger Binns <rog...@rogerbinns.com>
>> > > wrote:
>> > >
>> > >> On 19/11/16 08:08, Kevin O'Gorman wrote:
>> > >> > System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
>> > >> [...]
>> > >> > System without this problem: Running Ubuntu Linux 14.04.5, Python
>> > 3.4.3.
>> > >>
>> > >> You are good on Python versions then.  My remaining recommendation is
>> > to
>> > >> make the process that does SQLite be a child process (ie no making
>> its
>> > >> own children).  That will eliminate an entire class of potential
>> > >> problems, although it appears unlikely you are experiencing any of
>> > them.
>> > >>
>> > >> The final option is to run the process under valgrind.  That will
>> > >> definitively show the cause.  Do note however that you may want to
>> > >> change some of the default options since you have nice 

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

Good feedback.  I haven't done a memory check on that machine in a
while

Next on my list.

On Sun, Dec 4, 2016 at 11:25 AM, Keith Medcalf <kmedc...@dessus.com> wrote:

>
> If pressing the CAPS LOCK or NUM LOCK keys on the keyboard does not toggle
> the light on the keyboard then you have lost the all interrupt processing
> since those keypresses have to be processed by the kernel mode keyboard
> driver toggling the internal state of the keyboard driver, and then the
> kernel driver sends output to the keyboard to change the status LEDs.
> Typically (all Operating Systems) this means you have suffered a complete
> kernel crash (or halt) and the system is not running.
>
> Since the system must be running in order to output indicator status, all
> indicators will stay "stuck" in their last known position (hold output).
>
> Only a power-cycle (or hardware reset -- assuming the RESET is a hardware
> reset and not just a request to reset which will be ignored) triggering a
> reboot will restart the system.
>
> The most frequent cause is a Parity Check.  Or in these latter days of not
> having ECC or even Parity checked memory, just an undetected memory fault
> which will cause random AHTBL.
>
> > -Original Message-
> > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org]
> > On Behalf Of Kevin O'Gorman
> > Sent: Sunday, 4 December, 2016 09:21
> > To: SQLite mailing list
> > Subject: Re: [sqlite] I keep getting seg faults building my database
> using
> > python sqlite3
> >
> > Well, the i7 system failed again, but this time it was quite different.
> > And peculiar.
> > The system wasn't doing anything, but it should have been.  So I tried
> > something. It didn't matter what, because I could not get the mouse or
> > keyboard to work -- it was like they weren't plugged in.  Really like it,
> > because the caps lock light wasn't on, nor was the laser light visible in
> > the mouse.  Even when I changed mouse, keyboard and USB slot.  I couldn't
> > get in with SSH from elsewhere either.  But the computer's "I'm running"
> > light was on.
> > So I'm suspecting a partial power failure.  I don't know enough about
> > mobos
> > and USB to diagnose whether the problem was on the mobo or the power
> > supply.
> >
> > Creepty.  I had to do a hard reset  to get thing going again, and it's
> > been
> > running fine for a day now.
> >
> > On Mon, Nov 21, 2016 at 9:51 AM, Kevin O'Gorman <kevinogorm...@gmail.com
> >
> > wrote:
> >
> > > On Mon, Nov 21, 2016 at 9:41 AM, Roger Binns <rog...@rogerbinns.com>
> > > wrote:
> > >
> > >> On 19/11/16 08:08, Kevin O'Gorman wrote:
> > >> > System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
> > >> [...]
> > >> > System without this problem: Running Ubuntu Linux 14.04.5, Python
> > 3.4.3.
> > >>
> > >> You are good on Python versions then.  My remaining recommendation is
> > to
> > >> make the process that does SQLite be a child process (ie no making its
> > >> own children).  That will eliminate an entire class of potential
> > >> problems, although it appears unlikely you are experiencing any of
> > them.
> > >>
> > >> The final option is to run the process under valgrind.  That will
> > >> definitively show the cause.  Do note however that you may want to
> > >> change some of the default options since you have nice big systems.
> > For
> > >> example I like to set --freelist-vol and related to very big numbers
> > >> (several gigabytes) which ensures that freed memory is not reused for
> a
> > >> long time.  You could also set the valgrind option so that only one
> > >> thread is allowed - it will catch inadvertent threading you may note
> be
> > >> aware of.
> > >>
> > >> Roger
> > >>
> > >
> > > Thanks for that.  I may do the valgrind thing -- it sounds useful.  But
> > > just to add
> > > to my annoyance about this whole things, I've been having both systems
> > > running
> > > for a couple of days now with no problems or interruptions.  Remember,
> > the
> > > i7 system was failing after 2 hours at most.  I did tweak the code a
> > > little, but
> > > the only thing that seems likely to have stopped the problem is that I
> > put
> > > in
> > > code to do a commit after every 10,000 INSERT statements.  The two
> > systems
> > &

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Keith Medcalf]

If pressing the CAPS LOCK or NUM LOCK keys on the keyboard does not toggle the 
light on the keyboard then you have lost the all interrupt processing since 
those keypresses have to be processed by the kernel mode keyboard driver 
toggling the internal state of the keyboard driver, and then the kernel driver 
sends output to the keyboard to change the status LEDs.  Typically (all 
Operating Systems) this means you have suffered a complete kernel crash (or 
halt) and the system is not running.  

Since the system must be running in order to output indicator status, all 
indicators will stay "stuck" in their last known position (hold output).

Only a power-cycle (or hardware reset -- assuming the RESET is a hardware reset 
and not just a request to reset which will be ignored) triggering a reboot will 
restart the system.

The most frequent cause is a Parity Check.  Or in these latter days of not 
having ECC or even Parity checked memory, just an undetected memory fault which 
will cause random AHTBL.

> -Original Message-
> From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org]
> On Behalf Of Kevin O'Gorman
> Sent: Sunday, 4 December, 2016 09:21
> To: SQLite mailing list
> Subject: Re: [sqlite] I keep getting seg faults building my database using
> python sqlite3
> 
> Well, the i7 system failed again, but this time it was quite different.
> And peculiar.
> The system wasn't doing anything, but it should have been.  So I tried
> something. It didn't matter what, because I could not get the mouse or
> keyboard to work -- it was like they weren't plugged in.  Really like it,
> because the caps lock light wasn't on, nor was the laser light visible in
> the mouse.  Even when I changed mouse, keyboard and USB slot.  I couldn't
> get in with SSH from elsewhere either.  But the computer's "I'm running"
> light was on.
> So I'm suspecting a partial power failure.  I don't know enough about
> mobos
> and USB to diagnose whether the problem was on the mobo or the power
> supply.
> 
> Creepty.  I had to do a hard reset  to get thing going again, and it's
> been
> running fine for a day now.
> 
> On Mon, Nov 21, 2016 at 9:51 AM, Kevin O'Gorman <kevinogorm...@gmail.com>
> wrote:
> 
> > On Mon, Nov 21, 2016 at 9:41 AM, Roger Binns <rog...@rogerbinns.com>
> > wrote:
> >
> >> On 19/11/16 08:08, Kevin O'Gorman wrote:
> >> > System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
> >> [...]
> >> > System without this problem: Running Ubuntu Linux 14.04.5, Python
> 3.4.3.
> >>
> >> You are good on Python versions then.  My remaining recommendation is
> to
> >> make the process that does SQLite be a child process (ie no making its
> >> own children).  That will eliminate an entire class of potential
> >> problems, although it appears unlikely you are experiencing any of
> them.
> >>
> >> The final option is to run the process under valgrind.  That will
> >> definitively show the cause.  Do note however that you may want to
> >> change some of the default options since you have nice big systems.
> For
> >> example I like to set --freelist-vol and related to very big numbers
> >> (several gigabytes) which ensures that freed memory is not reused for a
> >> long time.  You could also set the valgrind option so that only one
> >> thread is allowed - it will catch inadvertent threading you may note be
> >> aware of.
> >>
> >> Roger
> >>
> >
> > Thanks for that.  I may do the valgrind thing -- it sounds useful.  But
> > just to add
> > to my annoyance about this whole things, I've been having both systems
> > running
> > for a couple of days now with no problems or interruptions.  Remember,
> the
> > i7 system was failing after 2 hours at most.  I did tweak the code a
> > little, but
> > the only thing that seems likely to have stopped the problem is that I
> put
> > in
> > code to do a commit after every 10,000 INSERT statements.  The two
> systems
> > are running identical Python code on the same inputs.  I had intended
> this
> > to
> > verify that one fails and the other does not.  What I got is something
> > different,
> > but on balance I like it best when my processes do not fail out.  Maybe
> > this
> > time the code will finish (at this rate it will be at least a week,
> maybe
> > three.
> >
> > --
> > #define QUESTION ((bb) || (!bb)) /* Shakespeare */
> >
> 
> 
> 
> --
> #define QUESTION ((bb) || (!bb)) /* Shakespeare */
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

Well, the i7 system failed again, but this time it was quite different.
And peculiar.
The system wasn't doing anything, but it should have been.  So I tried
something. It didn't matter what, because I could not get the mouse or
keyboard to work -- it was like they weren't plugged in.  Really like it,
because the caps lock light wasn't on, nor was the laser light visible in
the mouse.  Even when I changed mouse, keyboard and USB slot.  I couldn't
get in with SSH from elsewhere either.  But the computer's "I'm running"
light was on.
So I'm suspecting a partial power failure.  I don't know enough about mobos
and USB to diagnose whether the problem was on the mobo or the power supply.

Creepty.  I had to do a hard reset  to get thing going again, and it's been
running fine for a day now.

On Mon, Nov 21, 2016 at 9:51 AM, Kevin O'Gorman 
wrote:

> On Mon, Nov 21, 2016 at 9:41 AM, Roger Binns 
> wrote:
>
>> On 19/11/16 08:08, Kevin O'Gorman wrote:
>> > System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
>> [...]
>> > System without this problem: Running Ubuntu Linux 14.04.5, Python 3.4.3.
>>
>> You are good on Python versions then.  My remaining recommendation is to
>> make the process that does SQLite be a child process (ie no making its
>> own children).  That will eliminate an entire class of potential
>> problems, although it appears unlikely you are experiencing any of them.
>>
>> The final option is to run the process under valgrind.  That will
>> definitively show the cause.  Do note however that you may want to
>> change some of the default options since you have nice big systems.  For
>> example I like to set --freelist-vol and related to very big numbers
>> (several gigabytes) which ensures that freed memory is not reused for a
>> long time.  You could also set the valgrind option so that only one
>> thread is allowed - it will catch inadvertent threading you may note be
>> aware of.
>>
>> Roger
>>
>
> Thanks for that.  I may do the valgrind thing -- it sounds useful.  But
> just to add
> to my annoyance about this whole things, I've been having both systems
> running
> for a couple of days now with no problems or interruptions.  Remember, the
> i7 system was failing after 2 hours at most.  I did tweak the code a
> little, but
> the only thing that seems likely to have stopped the problem is that I put
> in
> code to do a commit after every 10,000 INSERT statements.  The two systems
> are running identical Python code on the same inputs.  I had intended this
> to
> verify that one fails and the other does not.  What I got is something
> different,
> but on balance I like it best when my processes do not fail out.  Maybe
> this
> time the code will finish (at this rate it will be at least a week, maybe
> three.
>
> --
> #define QUESTION ((bb) || (!bb)) /* Shakespeare */
>



-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Mon, Nov 21, 2016 at 9:41 AM, Roger Binns  wrote:

> On 19/11/16 08:08, Kevin O'Gorman wrote:
> > System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
> [...]
> > System without this problem: Running Ubuntu Linux 14.04.5, Python 3.4.3.
>
> You are good on Python versions then.  My remaining recommendation is to
> make the process that does SQLite be a child process (ie no making its
> own children).  That will eliminate an entire class of potential
> problems, although it appears unlikely you are experiencing any of them.
>
> The final option is to run the process under valgrind.  That will
> definitively show the cause.  Do note however that you may want to
> change some of the default options since you have nice big systems.  For
> example I like to set --freelist-vol and related to very big numbers
> (several gigabytes) which ensures that freed memory is not reused for a
> long time.  You could also set the valgrind option so that only one
> thread is allowed - it will catch inadvertent threading you may note be
> aware of.
>
> Roger
>

Thanks for that.  I may do the valgrind thing -- it sounds useful.  But
just to add
to my annoyance about this whole things, I've been having both systems
running
for a couple of days now with no problems or interruptions.  Remember, the
i7 system was failing after 2 hours at most.  I did tweak the code a
little, but
the only thing that seems likely to have stopped the problem is that I put
in
code to do a commit after every 10,000 INSERT statements.  The two systems
are running identical Python code on the same inputs.  I had intended this
to
verify that one fails and the other does not.  What I got is something
different,
but on balance I like it best when my processes do not fail out.  Maybe this
time the code will finish (at this rate it will be at least a week, maybe
three.

-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Roger Binns]

On 19/11/16 08:08, Kevin O'Gorman wrote:
> System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
[...]
> System without this problem: Running Ubuntu Linux 14.04.5, Python 3.4.3.

You are good on Python versions then.  My remaining recommendation is to
make the process that does SQLite be a child process (ie no making its
own children).  That will eliminate an entire class of potential
problems, although it appears unlikely you are experiencing any of them.

The final option is to run the process under valgrind.  That will
definitively show the cause.  Do note however that you may want to
change some of the default options since you have nice big systems.  For
example I like to set --freelist-vol and related to very big numbers
(several gigabytes) which ensures that freed memory is not reused for a
long time.  You could also set the valgrind option so that only one
thread is allowed - it will catch inadvertent threading you may note be
aware of.

Roger




signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

On 19 Nov 2016, at 10:22pm, Kevin O'Gorman  wrote:

> Ran Memtest86+ 5.01, two complete passes, with no errors.

Well, thanks for doing it anyway.  It removes one possible source of problems.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Fri, Nov 18, 2016 at 10:18 AM, James K. Lowden 
wrote:

> On Fri, 18 Nov 2016 08:55:11 -0800
> "Kevin O'Gorman"  wrote:
>
> > All of the python code is a single thread.  The closest I come
> > is a few times where I use subprocess.Popen to create what amounts to
> > a pipeline, and one place where I start a number of copies of a C
> > program in parallel, but each is a separate process with its own
> > input and output files.  These C programs have been in use for a
> > number of months for earlier stages of this project, and I regard
> > them as quite reliable.  None of them uses threads, and they are
> > mostly very simple filters.
>
> As you know, a process started with Popen cannot corrupt the Python
> process's memory.  If you're not doing anything to defeat the GIL, a
> segfault inside the Python interpreter would be considered a bug.
>
> But is it happening in the interpreter, or in SQLite for that matter?
> ISTM that's what you need to know.  To know that, you're going to need
> to run a debug version of the interpreter under gdb.  When it faults, a
> backtrace will tell you where.  That's not definititive proof; memory
> corruption is often detected far from where it was caused.  But if the
> fault is at a consistent place in SQLite code, for example, you can
> use a hardware watchpoint to discover what's writing to it.
>
> I'm game to try that, but unsure how to get such a thing.  If I have to
build
it, it's gonna take a while to assemble all the pieces. because it's gotta
have
all the parts I use, and be arranged so as not to interfere with normal use
of python 3.  Seems pretty error-prone itself, but as I said, I'm game.


> I don't know what more to suggest.  I would be surprised if you find a
> fault in Python, in the Python standard library, or in SQLite.  I'm
> sure it won't be in anything on the other side of a popen call.  Are
> there non-standard libraries or Python modules in use that you haven't
> mentioned?
>
> The most likely culprit in my mind is RAM.  You're exercising new memory
> pretty hard, running a bunch of processes at it at full tilt.  Any
> defect in the chips or DMA could explain what you're seeing.  An easy
> test, not necessarily cheap, would be to replace the RAM (or, if
> possible, run with some removed).
>
> I have two war stories related to rotten I/O hardware, where the device
> appeared to work for all intents and purposes, but was actually a
> high-speed bit munger. Those were both over 20 years ago.  It will
> be interesting to hear if that turns out to be your issue.
>
> HTH.
>
> --jkl
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

Ran Memtest86+ 5.01, two complete passes, with no errors.



On Sat, Nov 19, 2016 at 8:19 AM, Kevin O'Gorman 
wrote:

>
>
> On Sat, Nov 19, 2016 at 8:11 AM, Simon Slavin 
> wrote:
>
>>
>> On 19 Nov 2016, at 4:08pm, Kevin O'Gorman 
>> wrote:
>>
>> > I have two different machines running this stuff.  Only one is having
>> the
>> > seg faults, but they are different enough that this does not convince me
>> > to blame hardware.
>>
>> Could you, anyway, run a memory test on the computer which is failing ?
>> I don't how one does that under Xubuntu but perhaps you do, or perhaps
>> there's one built into the BIOS.
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>
> Sure.  Memtest86+ is an option on boot.  I just have to find a time slot
> because it takes a while.  Later today.
>
> --
> #define QUESTION ((bb) || (!bb)) /* Shakespeare */
>



-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Roger Binns]

On 18/11/16 15:19, James K. Lowden wrote:
> Good catch, Roger.  It's a liability, but I slightly disagree with your
> characterization.  
> 
>> - Running any Python code (destructors can be called which then run in
>> the parent and child)
> 
> Yes, if those destructors affect shared resources.  The OP did say the
> processes on the other side of popen were C programs.  

You missed a subtlety.  The fork call is made from Python code.  The
parent and child keep running Python code.  Eventually the child calls
exec and similar.  In the child process between the fork call return and
the exec call, the Python code that runs can include the destructors for
Python wrapped SQLite objects.

>> - Not having file descriptors closed so the child process trashes them
>> (close_fds Popen argument is False in python 2.x but True in python
>> 3.x).  
> 
> The child process can't "trash" the parent's descriptors.

Lets say the SQLite database is on fd 3 in the parent process.  That
makes it fd 3 in the child process too after the fork.  And if not
closed, it will be fd 3 in whatever gets execed.  This scenario is the
very first one listed in how to corrupt:

 https://www.sqlite.org/howtocorrupt.html

>> Also python 2.x subprocess module is broken in many ways.
> 
> My foray into Unicode in Python convinced me once and for all that
> Python 3 is the only way to go.  But would you care to elaborate on the
> problems with 2.x subprocess?  

The Python 2.x subprocess module is implemented in Python.  It has race
conditions and other problems when the parent is multi-threaded.  They
aren't theoretical, and affected the project I work on.  When debugging
you find yourself in "impossible" situations.  My memory is a bit fuzzy
now, but it was things like file descriptors between parent and child
not being as expected.

Python 3 includes a pure C implementation of code that runs from fork
through exec.  Someone at Google backported that to 2.x, so they must
have been affected too.

  https://github.com/google/python-subprocess32

Roger




signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Sat, Nov 19, 2016 at 8:11 AM, Simon Slavin  wrote:

>
> On 19 Nov 2016, at 4:08pm, Kevin O'Gorman  wrote:
>
> > I have two different machines running this stuff.  Only one is having the
> > seg faults, but they are different enough that this does not convince me
> > to blame hardware.
>
> Could you, anyway, run a memory test on the computer which is failing ?  I
> don't how one does that under Xubuntu but perhaps you do, or perhaps
> there's one built into the BIOS.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

Sure.  Memtest86+ is an option on boot.  I just have to find a time slot
because it takes a while.  Later today.

-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

On 19 Nov 2016, at 4:08pm, Kevin O'Gorman  wrote:

> I have two different machines running this stuff.  Only one is having the
> seg faults, but they are different enough that this does not convince me
> to blame hardware.

Could you, anyway, run a memory test on the computer which is failing ?  I 
don't how one does that under Xubuntu but perhaps you do, or perhaps there's 
one built into the BIOS.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Fri, Nov 18, 2016 at 3:19 PM, James K. Lowden 
wrote:

> On Fri, 18 Nov 2016 10:56:37 -0800
> Roger Binns  wrote:
>
> > Popen calls fork (it seems like you are doing Unix/Mac, not Windows).
> > fork() duplicates the process including all open file descriptors.
> > One or more of those descriptors belong to open SQLite databases and
> > ancillary files.
>
> Good catch, Roger.  It's a liability, but I slightly disagree with your
> characterization.
>
> > - Running any Python code (destructors can be called which then run in
> > the parent and child)
>
> Yes, if those destructors affect shared resources.  The OP did say the
> processes on the other side of popen were C programs.
>
> > - Not having file descriptors closed so the child process trashes them
> > (close_fds Popen argument is False in python 2.x but True in python
> > 3.x).
>
> The child process can't "trash" the parent's descriptors.  When the
> child exits, the OS will close its descriptors, that's all.  But, yes,
> if the child process is making some assumption about open descriptors
> it receives at startup, that could be lead to problems.  Especially if
> it scribbles on the SQLite database.
>
> > Also python 2.x subprocess module is broken in many ways.
>
> My foray into Unicode in Python convinced me once and for all that
> Python 3 is the only way to go.  But would you care to elaborate on the
> problems with 2.x subprocess?
>
>
> You can all put aside worries about Python 2.  I only started using Python
a couple of months ago, and went with 3.

I have two different machines running this stuff.  Only one is having the
seg faults, but they are different enough that this does not convince me
to blame hardware.

System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
CPU: core i7, MOBO: Asus Z97 Deluxe, 32GB ram.  4 cores, hyperthreaded
to 8.  Home-built.

System without this problem: Running Ubuntu Linux 14.04.5, Python 3.4.3.
This is a System76 Silverback.  CPU: dual Intel Xeon, 256 GB RAM(!)
16 cores, hyperthreaded to 32.

On both: Filesystem is an 11-TB software RAID (Linux LVM) on top of
three 4-TB hard drives.

--
Python modules are all pretty standard (I think):
stat argparse datetime fileinput glob os re sqlite3 subprocess sys time

children on the other side of Popen are either standard Linux utilities like
split/grep/sort or my own C code, which is mostly very simple with one
exception,
and that is a pretty well-tested implementation of minimax search through
a game tree.  But even in that one, the I/O is very simple: one input line
yields one output line on one of two output streams.

I never set close_fds=False, so the Python3 default protects the database.

Questions, though: I don't bother to commit in the midst of anything, so
you'll notice my journal gets pretty big.  That's not a problem for my
filesystem,
but could it be a problem internally to Python?
-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[James K. Lowden]

On Fri, 18 Nov 2016 10:56:37 -0800
Roger Binns  wrote:

> Popen calls fork (it seems like you are doing Unix/Mac, not Windows).
> fork() duplicates the process including all open file descriptors.
> One or more of those descriptors belong to open SQLite databases and
> ancillary files.

Good catch, Roger.  It's a liability, but I slightly disagree with your
characterization.  

> - Running any Python code (destructors can be called which then run in
> the parent and child)

Yes, if those destructors affect shared resources.  The OP did say the
processes on the other side of popen were C programs.  

> - Not having file descriptors closed so the child process trashes them
> (close_fds Popen argument is False in python 2.x but True in python
> 3.x).  

The child process can't "trash" the parent's descriptors.  When the
child exits, the OS will close its descriptors, that's all.  But, yes,
if the child process is making some assumption about open descriptors
it receives at startup, that could be lead to problems.  Especially if
it scribbles on the SQLite database.  

> Also python 2.x subprocess module is broken in many ways.

My foray into Unicode in Python convinced me once and for all that
Python 3 is the only way to go.  But would you care to elaborate on the
problems with 2.x subprocess?  

--jkl
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Roger Binns]

On 18/11/16 08:55, Kevin O'Gorman wrote:
>> I am not.  All of the python code is a single thread.  The closest I come
> is a few times where I use subprocess.Popen to create what amounts to a
> pipeline, and one place where I start a number of copies of a C program in
> parallel, but each is a separate process with its own input and output
> files.

Ding ding ding.  You didn't mention which version of Python.

Popen calls fork (it seems like you are doing Unix/Mac, not Windows).
fork() duplicates the process including all open file descriptors.  One
or more of those descriptors belong to open SQLite databases and
ancillary files.

If the child process does virtually anything, it will result in crashes.
 Examples of doing things include:

- Running any Python code (destructors can be called which then run in
the parent and child)

- Not having file descriptors closed so the child process trashes them
(close_fds Popen argument is False in python 2.x but True in python
3.x).  Also python 2.x subprocess module is broken in many ways.

There are three methods for addressing this assuming it is the cause.

0: Python 3 is safer if you aren't already using it.  The subprocess32
module for Python 2 backports Python 3 subprocess if you are stuck on
Python 2

1: Make sure that the process that does SQLite work is a "leaf".  ie it
doesn't call subprocess or make any child processes.  Instead a master
process feeds it data on its stdin/out

2: Switch to APSW which has a fork checker built in.  This will catch
the fork problem no matter how it happens, and whatever is going on in
other libraries you may use:


https://rogerbinns.github.io/apsw/apsw.html?highlight=fork#apsw.fork_checker

Disclaimer:  I am the author of APSW

Roger



signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

On 18 Nov 2016, at 6:18pm, James K. Lowden  wrote:

> The most likely culprit in my mind is RAM.  You're exercising new memory
> pretty hard, running a bunch of processes at it at full tilt.  Any
> defect in the chips or DMA could explain what you're seeing.  An easy
> test, not necessarily cheap, would be to replace the RAM (or, if
> possible, run with some removed).  

Or borrow someone else's computer.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[James K. Lowden]

On Fri, 18 Nov 2016 08:55:11 -0800
"Kevin O'Gorman"  wrote:

> All of the python code is a single thread.  The closest I come
> is a few times where I use subprocess.Popen to create what amounts to
> a pipeline, and one place where I start a number of copies of a C
> program in parallel, but each is a separate process with its own
> input and output files.  These C programs have been in use for a
> number of months for earlier stages of this project, and I regard
> them as quite reliable.  None of them uses threads, and they are
> mostly very simple filters.

As you know, a process started with Popen cannot corrupt the Python
process's memory.  If you're not doing anything to defeat the GIL, a
segfault inside the Python interpreter would be considered a bug.  

But is it happening in the interpreter, or in SQLite for that matter?
ISTM that's what you need to know.  To know that, you're going to need
to run a debug version of the interpreter under gdb.  When it faults, a
backtrace will tell you where.  That's not definititive proof; memory
corruption is often detected far from where it was caused.  But if the
fault is at a consistent place in SQLite code, for example, you can
use a hardware watchpoint to discover what's writing to it.  

I don't know what more to suggest.  I would be surprised if you find a
fault in Python, in the Python standard library, or in SQLite.  I'm
sure it won't be in anything on the other side of a popen call.  Are
there non-standard libraries or Python modules in use that you haven't
mentioned?  

The most likely culprit in my mind is RAM.  You're exercising new memory
pretty hard, running a bunch of processes at it at full tilt.  Any
defect in the chips or DMA could explain what you're seeing.  An easy
test, not necessarily cheap, would be to replace the RAM (or, if
possible, run with some removed).  

I have two war stories related to rotten I/O hardware, where the device
appeared to work for all intents and purposes, but was actually a
high-speed bit munger. Those were both over 20 years ago.  It will
be interesting to hear if that turns out to be your issue.  

HTH.  

--jkl

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Fri, Nov 18, 2016 at 8:38 AM, Roger Binns  wrote:

> On 17/11/16 19:14, Kevin O'Gorman wrote:
> > SO: I need help bifurcating this problem.  For instance, how can I tell
> if
> > the fault lies in SQLite, or in python? Or even in the hardware, given
> that
> > the time to failure is so variable?
>
> Are you using threads, threading related settings etc in any way?  The
> python sqlite3 module (aka pysqlite) is not threadsafe.  That doesn't
> stop people "working around" it, which can lead to crashes.
>
> Roger
>
>
> I am not.  All of the python code is a single thread.  The closest I come
is a few times where I use subprocess.Popen to create what amounts to a
pipeline, and one place where I start a number of copies of a C program in
parallel, but each is a separate process with its own input and output
files.  These C programs have been in use for a number of months for
earlier stages of this project, and I regard them as quite reliable.  None
of them uses threads, and they are mostly very simple filters.

The one that runs in parallel cannot be the culprit, however, because the
code has not reached the point where it would come into play.  That is the
step where the results get "cached" (in flat files) and all of the early
results are in those files.  This thing is dying before reaching unknown
territory where new results are needed.  The reason the results exist is
that they were generated by previous versions of the software that did not
use Python.  I am switching because (a) I want a database instead of flat
files for speed reasons and (b) the C code was getting too hard to maintain.

All of this is a hobby project, and I can share any parts of it that you
care to see.
-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Fri, Nov 18, 2016 at 3:11 AM, Simon Slavin  wrote:

> Forgot to say ...
>
> Most of these problems result from attempting to reuse memory you've
> already released.  Even if the error is happening inside a SQLite routine,
> it will be because you passed it a pointer to an SQLite connection which
> had already been _close()'d or a statement which had already been
> _finalize()'d.
>
>
> Well, the entire process runs under a single connection that is never
closed. Although cursors are opened and closed within some of the Python
functions, the close is always at the end of the function.  And I have no
idea how things get finalized under python, or if that's an issue.  Anyway,
most of the time I use patterns like
  for row in c1.execute(...)
or
  with dbfile.connect() as conn:
and as I understand it, these protect me from most errors of that kind.

Moreover, it seems to me that these would all lead to failure at exactly
the same point, rather than at three points up to an hour different from
each other.

I was thinking it more likely to be one of
a) random hardware malfunction
b) some bug in memory handling that was subject to influence by other
activity in the machine)
c) some bug in hanling pointers to the python structures (it would have to
be a python or sqlite3 bug.)




-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Roger Binns]

On 17/11/16 19:14, Kevin O'Gorman wrote:
> SO: I need help bifurcating this problem.  For instance, how can I tell if
> the fault lies in SQLite, or in python? Or even in the hardware, given that
> the time to failure is so variable?

Are you using threads, threading related settings etc in any way?  The
python sqlite3 module (aka pysqlite) is not threadsafe.  That doesn't
stop people "working around" it, which can lead to crashes.

Roger




signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

Forgot to say ...

Most of these problems result from attempting to reuse memory you've already 
released.  Even if the error is happening inside a SQLite routine, it will be 
because you passed it a pointer to an SQLite connection which had already been 
_close()'d or a statement which had already been _finalize()'d.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

On 18 Nov 2016, at 3:14am, Kevin O'Gorman  wrote:

> SO: I need help bifurcating this problem.  For instance, how can I tell if
> the fault lies in SQLite, or in python? Or even in the hardware, given that
> the time to failure is so variable?

Normal recommendation is to use GDB.  You can find several tutorials on how to 
use GDB to spot seg faults on the web.

Some other deveopment environments have shells around GBD or tools which will 
spot misused memory, for example Apple's Xcode environment.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users