Re: [sqlite] Fwd: Bug in the docs

[Keith Medcalf]

The fifth parameter is a pointer into which the address of the error message is 
written.  If you pass NULL (0), then no error message is returned, and no 
memory is allocated to contain the message.

On the gripping hand, if you pass a pointer (non NULL / non 0) then memory is 
allocated to contain the error message and the address of that memory block is 
written to the location specified by the 5th parameter.


eg:

 char* errmsg;

 sqlite_exec(p1, p2, p3, p4, );

now errmsg points to the error message and you call sqlite_free(errmsg) when 
you are done with the error message.

 sqlite_exec(p1, p2, p3, p4, 0);

no memory is allocated for an error message and therefore no pointer is written 
to *0 ...

> -Original Message-
> From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org]
> On Behalf Of Igor Korot
> Sent: Saturday, 19 November, 2016 15:59
> To: Discussion of SQLite Database; General Discussion of SQLite Database
> Subject: [sqlite] Fwd: Bug in the docs
> 
> Hi, ALL,
> On the page https://www.sqlite.org/c3ref/exec.html following phrase is
> written:
> 
> [quote]
> If the 5th parameter to sqlite3_exec() is not NULL then any error
> message is written into memory obtained from sqlite3_malloc() and
> passed back through the 5th parameter.
> [/quote]
> 
> Shouldn't it start with:
> 
> "If the 5th parameter to sqlite3_exec() is NULL"
> 
> ?
> 
> Thank you.
> 
> P.S.: Sorry if it appears twice.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

On 19 Nov 2016, at 10:22pm, Kevin O'Gorman  wrote:

> Ran Memtest86+ 5.01, two complete passes, with no errors.

Well, thanks for doing it anyway.  It removes one possible source of problems.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Fwd: Bug in the docs

[Igor Korot]

Hi, ALL,
On the page https://www.sqlite.org/c3ref/exec.html following phrase is written:

[quote]
If the 5th parameter to sqlite3_exec() is not NULL then any error
message is written into memory obtained from sqlite3_malloc() and
passed back through the 5th parameter.
[/quote]

Shouldn't it start with:

"If the 5th parameter to sqlite3_exec() is NULL"

?

Thank you.

P.S.: Sorry if it appears twice.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Fri, Nov 18, 2016 at 10:18 AM, James K. Lowden 
wrote:

> On Fri, 18 Nov 2016 08:55:11 -0800
> "Kevin O'Gorman"  wrote:
>
> > All of the python code is a single thread.  The closest I come
> > is a few times where I use subprocess.Popen to create what amounts to
> > a pipeline, and one place where I start a number of copies of a C
> > program in parallel, but each is a separate process with its own
> > input and output files.  These C programs have been in use for a
> > number of months for earlier stages of this project, and I regard
> > them as quite reliable.  None of them uses threads, and they are
> > mostly very simple filters.
>
> As you know, a process started with Popen cannot corrupt the Python
> process's memory.  If you're not doing anything to defeat the GIL, a
> segfault inside the Python interpreter would be considered a bug.
>
> But is it happening in the interpreter, or in SQLite for that matter?
> ISTM that's what you need to know.  To know that, you're going to need
> to run a debug version of the interpreter under gdb.  When it faults, a
> backtrace will tell you where.  That's not definititive proof; memory
> corruption is often detected far from where it was caused.  But if the
> fault is at a consistent place in SQLite code, for example, you can
> use a hardware watchpoint to discover what's writing to it.
>
> I'm game to try that, but unsure how to get such a thing.  If I have to
build
it, it's gonna take a while to assemble all the pieces. because it's gotta
have
all the parts I use, and be arranged so as not to interfere with normal use
of python 3.  Seems pretty error-prone itself, but as I said, I'm game.


> I don't know what more to suggest.  I would be surprised if you find a
> fault in Python, in the Python standard library, or in SQLite.  I'm
> sure it won't be in anything on the other side of a popen call.  Are
> there non-standard libraries or Python modules in use that you haven't
> mentioned?
>
> The most likely culprit in my mind is RAM.  You're exercising new memory
> pretty hard, running a bunch of processes at it at full tilt.  Any
> defect in the chips or DMA could explain what you're seeing.  An easy
> test, not necessarily cheap, would be to replace the RAM (or, if
> possible, run with some removed).
>
> I have two war stories related to rotten I/O hardware, where the device
> appeared to work for all intents and purposes, but was actually a
> high-speed bit munger. Those were both over 20 years ago.  It will
> be interesting to hear if that turns out to be your issue.
>
> HTH.
>
> --jkl
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

Ran Memtest86+ 5.01, two complete passes, with no errors.



On Sat, Nov 19, 2016 at 8:19 AM, Kevin O'Gorman 
wrote:

>
>
> On Sat, Nov 19, 2016 at 8:11 AM, Simon Slavin 
> wrote:
>
>>
>> On 19 Nov 2016, at 4:08pm, Kevin O'Gorman 
>> wrote:
>>
>> > I have two different machines running this stuff.  Only one is having
>> the
>> > seg faults, but they are different enough that this does not convince me
>> > to blame hardware.
>>
>> Could you, anyway, run a memory test on the computer which is failing ?
>> I don't how one does that under Xubuntu but perhaps you do, or perhaps
>> there's one built into the BIOS.
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>
> Sure.  Memtest86+ is an option on boot.  I just have to find a time slot
> because it takes a while.  Later today.
>
> --
> #define QUESTION ((bb) || (!bb)) /* Shakespeare */
>



-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Bug in the docs

[Igor Korot]

Hi, ALL,
On the page https://www.sqlite.org/c3ref/exec.html following phrase is written:

[quote]
If the 5th parameter to sqlite3_exec() is not NULL then any error
message is written into memory obtained from sqlite3_malloc() and
passed back through the 5th parameter.
[/quote]

Shouldn't it start with:

"If the 5th parameter to sqlite3_exec() is NULL"

?

Thank you.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Roger Binns]

On 18/11/16 15:19, James K. Lowden wrote:
> Good catch, Roger.  It's a liability, but I slightly disagree with your
> characterization.  
> 
>> - Running any Python code (destructors can be called which then run in
>> the parent and child)
> 
> Yes, if those destructors affect shared resources.  The OP did say the
> processes on the other side of popen were C programs.  

You missed a subtlety.  The fork call is made from Python code.  The
parent and child keep running Python code.  Eventually the child calls
exec and similar.  In the child process between the fork call return and
the exec call, the Python code that runs can include the destructors for
Python wrapped SQLite objects.

>> - Not having file descriptors closed so the child process trashes them
>> (close_fds Popen argument is False in python 2.x but True in python
>> 3.x).  
> 
> The child process can't "trash" the parent's descriptors.

Lets say the SQLite database is on fd 3 in the parent process.  That
makes it fd 3 in the child process too after the fork.  And if not
closed, it will be fd 3 in whatever gets execed.  This scenario is the
very first one listed in how to corrupt:

 https://www.sqlite.org/howtocorrupt.html

>> Also python 2.x subprocess module is broken in many ways.
> 
> My foray into Unicode in Python convinced me once and for all that
> Python 3 is the only way to go.  But would you care to elaborate on the
> problems with 2.x subprocess?  

The Python 2.x subprocess module is implemented in Python.  It has race
conditions and other problems when the parent is multi-threaded.  They
aren't theoretical, and affected the project I work on.  When debugging
you find yourself in "impossible" situations.  My memory is a bit fuzzy
now, but it was things like file descriptors between parent and child
not being as expected.

Python 3 includes a pure C implementation of code that runs from fork
through exec.  Someone at Google backported that to 2.x, so they must
have been affected too.

  https://github.com/google/python-subprocess32

Roger




signature.asc
Description: OpenPGP digital signature
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Sat, Nov 19, 2016 at 8:11 AM, Simon Slavin  wrote:

>
> On 19 Nov 2016, at 4:08pm, Kevin O'Gorman  wrote:
>
> > I have two different machines running this stuff.  Only one is having the
> > seg faults, but they are different enough that this does not convince me
> > to blame hardware.
>
> Could you, anyway, run a memory test on the computer which is failing ?  I
> don't how one does that under Xubuntu but perhaps you do, or perhaps
> there's one built into the BIOS.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

Sure.  Memtest86+ is an option on boot.  I just have to find a time slot
because it takes a while.  Later today.

-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Simon Slavin]

On 19 Nov 2016, at 4:08pm, Kevin O'Gorman  wrote:

> I have two different machines running this stuff.  Only one is having the
> seg faults, but they are different enough that this does not convince me
> to blame hardware.

Could you, anyway, run a memory test on the computer which is failing ?  I 
don't how one does that under Xubuntu but perhaps you do, or perhaps there's 
one built into the BIOS.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] I keep getting seg faults building my database using python sqlite3

[Kevin O'Gorman]

On Fri, Nov 18, 2016 at 3:19 PM, James K. Lowden 
wrote:

> On Fri, 18 Nov 2016 10:56:37 -0800
> Roger Binns  wrote:
>
> > Popen calls fork (it seems like you are doing Unix/Mac, not Windows).
> > fork() duplicates the process including all open file descriptors.
> > One or more of those descriptors belong to open SQLite databases and
> > ancillary files.
>
> Good catch, Roger.  It's a liability, but I slightly disagree with your
> characterization.
>
> > - Running any Python code (destructors can be called which then run in
> > the parent and child)
>
> Yes, if those destructors affect shared resources.  The OP did say the
> processes on the other side of popen were C programs.
>
> > - Not having file descriptors closed so the child process trashes them
> > (close_fds Popen argument is False in python 2.x but True in python
> > 3.x).
>
> The child process can't "trash" the parent's descriptors.  When the
> child exits, the OS will close its descriptors, that's all.  But, yes,
> if the child process is making some assumption about open descriptors
> it receives at startup, that could be lead to problems.  Especially if
> it scribbles on the SQLite database.
>
> > Also python 2.x subprocess module is broken in many ways.
>
> My foray into Unicode in Python convinced me once and for all that
> Python 3 is the only way to go.  But would you care to elaborate on the
> problems with 2.x subprocess?
>
>
> You can all put aside worries about Python 2.  I only started using Python
a couple of months ago, and went with 3.

I have two different machines running this stuff.  Only one is having the
seg faults, but they are different enough that this does not convince me
to blame hardware.

System with problems: Running Xubuntu Linux 16.04.1, Python 3.5.2.
CPU: core i7, MOBO: Asus Z97 Deluxe, 32GB ram.  4 cores, hyperthreaded
to 8.  Home-built.

System without this problem: Running Ubuntu Linux 14.04.5, Python 3.4.3.
This is a System76 Silverback.  CPU: dual Intel Xeon, 256 GB RAM(!)
16 cores, hyperthreaded to 32.

On both: Filesystem is an 11-TB software RAID (Linux LVM) on top of
three 4-TB hard drives.

--
Python modules are all pretty standard (I think):
stat argparse datetime fileinput glob os re sqlite3 subprocess sys time

children on the other side of Popen are either standard Linux utilities like
split/grep/sort or my own C code, which is mostly very simple with one
exception,
and that is a pretty well-tested implementation of minimax search through
a game tree.  But even in that one, the I/O is very simple: one input line
yields one output line on one of two output streams.

I never set close_fds=False, so the Python3 default protects the database.

Questions, though: I don't bother to commit in the midst of anything, so
you'll notice my journal gets pretty big.  That's not a problem for my
filesystem,
but could it be a problem internally to Python?
-- 
#define QUESTION ((bb) || (!bb)) /* Shakespeare */
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Anybody know why mono SQLite uses SqliteConnection but Windows uses SQLiteConnection?

[Drago, William @ CSG - NARDA-MITEQ]

> -Original Message-
> From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On
> Behalf Of Jim Henderson
> Sent: Saturday, November 19, 2016 12:16 AM
> To: SQLite mailing list 
> Subject: Re: [sqlite] Anybody know why mono SQLite uses SqliteConnection
> but Windows uses SQLiteConnection?
>
> Thanks for the suggestion, Bill.
>
> Will this run on a Linux system? The zip file has Install.exe and Test.exe 
> inside
> it.

I've never used the installer or the test program. I don't know what they do. I 
just use the .dll.

I thought Mono was .NET for Linux, so I just assumed that the SQLite 
Precompiled Binaries for Mono was the Linux version of System.Data.SQLite. 
Sorry if I led you in the wrong direction.

Assuming this doesn't work for you, is it possible for you to edit and compile 
the library you are presently using so that it has the correct case?

--
Bill Drago
Staff Engineer
L3 Narda-MITEQ
435 Moreland Road
Hauppauge, NY 11788
631-272-5947 / william.dr...@l-3com.com

CONFIDENTIALITY, EXPORT CONTROL AND DISCLAIMER NOTE:This e-mail and any 
attachments are solely for the use of the addressee and may contain information 
that is privileged or confidential. Any disclosure, use or distribution of the 
information contained herein is prohibited. In the event this e-mail contains 
technical data within the definition of the International Traffic in Arms 
Regulations or Export Administration Regulations, it is subject to the export 
control laws of the U.S.Government. The recipient should check this e-mail and 
any attachments for the presence of viruses as L-3 does not accept any 
liability associated with the transmission of this e-mail. If you have received 
this communication in error, please notify the sender by reply e-mail and 
immediately delete this message and any attachments.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users