[sqlite] Patch for build on ppc64le systems

[Paul G Crumley]

Hello,

Today, while trying to install the apsw package for python on an IBM POWER 
system I found I needed a new version of sqlite.

I tried to build sqlite from sqlite-configure-3801001.tar.gz on an IBM 
POWER system running ubuntu and the build failed due to a lack of 
knowledge of ppc64le in the embedded config.guess / config.sub scripts.

On my target system "uname -a" gives:
Linux pip-test 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 
03:50:31 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

The commands I used (as suggested on the sqlite download page of 
http://www.sqlite.org/cgi/src/doc/trunk/README.md ) are:

mkdir test_area
cd test_area
mkdir sqlite
cd sqlite
wget https://www.sqlite.org/2015/sqlite-autoconf-3081001.tar.gz
tar xvf *gz
mkdir bld
cd bld
../sqlite-autoconf-3081001/configure 

Here I am told that the config.guess command in the source does not know 
about ppc64le.

I then tried the same commands on an x86 system and all went smoothly. 
Back on the POWER system I downloaded new versions for config.guess and 
config.sub, overlaid the versions in the sqlite directory, then tried 
again:

cd ..
find . -name config.guess
find . -name config.sub
cd ..
wget  -O config.guess 
'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD'
wget  -O config.sub 
'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD'
chmod +x con*
./config.guess 
cd sqlite/sqlite-autoconf-3081001/
ls config.*
mv config.guess config.guess.orig
mv config.sub config.sub.orig
cp ../../config.* .
../bld/
../sqlite-autoconf-3081001/configure
make
make sqlite3.c
make test
sudo make install

The test target is not present in the Makefile but all the other 
operations succeeded on my ppc64le target.

Before submitting this report I found 
http://article.gmane.org/gmane.comp.db.sqlite.general/85660/match=config+guess 
which may be related.  Replacing config.guess and config.sub worked for 
me. 

I am happy to validate operation on the POWER system for you once the 
latest config.guess and config.sub files are in the sqlite source.

Thank you,
Paul


Paul G Crumley
STSM & Manager
IBM Research

Re: [sqlite] Extrange files

[Paul G]

- Original Message - 
From: "Nuno Lucas" <[EMAIL PROTECTED]>

To: 
Sent: Wednesday, March 01, 2006 10:36 PM
Subject: Re: [sqlite] Extrange files


The only problem is that it can be a race condition between the
closing of the handle and the actual delete command (because you can't
delete an open file on win world).


you lie!  if the file is opened with the delete sharemode set, it can be 
'deleted' while open. in this case, it will be *really* deleted when the 
refcount drops to 0, iirc. besides, these hoops don't need to be jumped 
through if you're the process that issued the original CreateFile(), again 
iirc.


-p 

Re: [sqlite] Proposed 3.3.0 changes. Was: 5/2==2

[Paul G]

- Original Message - 
From: "Jay Sprenkle" <[EMAIL PROTECTED]>
To: 
Sent: Tuesday, November 01, 2005 2:08 PM
Subject: Re: [sqlite] Proposed 3.3.0 changes. Was: 5/2==2

> I think it's a bit misleading to call the library "SQL-Lite: if you're
> going to redesign
> and not be like SQL. What do you think about creating a separate
> project for a fast
> light database engine that's not "SQL Like"?

strict typing is not, to most people at least, a core feature of sql - the
syntax and semantics of operations (ie selects+joins etc) are. as drh
stated, strict typing was originally a performance
enhancement/implementation complexity reduction measure. the difference
between what sql prescribes and what sql does will be apparent to developers
at two points: a) schema definition and b) operation semantics expectation
formulation.

imo, the former is a non-issue, since pretty much each rdbms will have
significant differences there; sqlite would indeed probably be one of the
easiest one to adapt to in this regard due to the short list of possible
types.

the latter issue is the one that needs to be discussed and indeed the
one drh is trying to get at with his question. is asking developers to
change their mindset when working with sqlite and expect, say, a number with
a fraction back from a division of two integers too much? this question
should also be posed to the authors of language-specific bindings, since
this could be a hassle in strictly typed languages.

regardless, sqlite's fudging of type handling does not make it not look like
sql, although it's pretty obvious it's noncompliant in those respects. this
is not a bad thing - following braindead standards faithfully is a braindead
design. (disclaimer: assuming that the deviation is less braindead).
besides, if you want to get into linguistic interpretations, 'sqlite' could
be interpreted both as 'lightweight sql rdbms' and as 'rdbms conforming to a
reduced, lightweight subset of sql'.

imo, provided that there is a prominenent 'You are standing on a chair and
wearing a noose around your neck!' notice given to folks, i don't see this
as a problem.

-p

Re: [sqlite] Multi-threading.

[Paul G]

- Original Message - 
From: "Ben Clewett" <[EMAIL PROTECTED]>
To: 
Sent: Wednesday, July 20, 2005 12:26 PM
Subject: Re: [sqlite] Multi-threading.


> Dr Hipp,
>
> I am just playing devils advocate here because I have completed much
> Java programming in a multi-threaded application. :)
>
> I understand the problems of multi-threading.  I am reminded that it
> took nearly 20 years of development to get multi-processor support in a
> modern OS stable.  Much success for this can be attributed to Semaphore
> Flags.  With CPU hardware support to ensure that the semaphore it's self
> cannot be controlled by more than one process.

whatcha talkin bout willis? all you need to fence access to a shared
resource is a set of atomic operations which carry the semantics you are
looking for. this has been around for quite some time.

> Multi-thread applications suffer the same problems.  Without semaphore
> flags or 20 years of development.

say what?

> A novice programmer can happily
> create a second thread and quickly create terribly applications.

yes. non-novices too, hence richard's advice.

> As another user also mentioned, a Windows system works better with few
> processes with many threads.

this has to do with process creation being a very expensive operation in
windows. just because a certain platform cripples something doesn't make it
bad.

> I believe the issue is not whether to use threads, but to use them
> correctly.  Which is not a hard thing to do with a little support.

you've got no idea. there are *very* few threading interfaces which make
writing correct code reasonably easy. java and .net are certainly not among
them. erlang is, but the trouble is they're implemented as microthreads, so
you're still stuck with problems making blocking syscalls.

> This is where Java (and .NET) work well.  If you use them correctly.
> They provide thread-safe objects.  Which have been designed to use
> semaphore flags internally.  If the programmer uses these thread-safe
> objects correctly, they will not encounter thread issues.  For instance,
> all communication between threads should be exclusively through these
> thread-safe objects.

stop misusing 'semaphore' already, semaphore is not a synonym for a
synchronization object - it is a kind of a synchronization object and has a
very narrow definition.

> Further, Java and .NET provide Sycronisation methods.  The defining of a
> method to be synchronised automatically creates the locks to ensure
> thread safe access.

you don't get the point. this (as opposed to more intuitive primitives
*around* threading, such as actors) is just a couple lines of code here and
there in a language which does not have syntax for it. coders have to
understand the issues and use the locking primitives they are given
correctly. i'd say your opinions, as expressed in your message, demonstrate
that the mere presence of higher level sycnhronization primitives in a
development platform does not magically make users of said platform
understand threading issues.

> I am also interested in your comments on Pointers and GoTo.  I note that
> Java is 100% pointers.  Apart from basic types, all object access is by
> pointer.

this is why teaching java in cs 101 is a bad idea, people have no
understanding of how things work 'under the hood' anymore. java uses
references, which are implemented as primitives in the jvm itself.

> Using Exceptions correctly, I have never felt the need for a GoTo.
> Exceptions do the same as GoTo, accept, maybe, in a slightly more
> developed and useful way.

then you haven't coded anything complex enough to require them. i can tell
you that they are an absolute necessity when dealing with a lot of nesting,
which may or may not deal with error handling. dijkstra was preaching
against using goto to create spaghetti code; many experienced coders use
gotos to *improve* readability and hence maintainability of their code.
'using'-like constructs give you *some* of this, but you will still want
gotos even in languages where both exceptions and using are available when
dealing with non-trivial logic.

> These are just my opinions :)

no offense, but you probably want to have the correct data from which to
derive your opinions ;)

-p

Re: [sqlite] Multi-threading.

[Paul G]

- Original Message - 
From: "Andrew Piskorski" <[EMAIL PROTECTED]>
To: 
Sent: Friday, July 15, 2005 1:05 PM
Subject: Re: [sqlite] Multi-threading.


> On Fri, Jul 15, 2005 at 04:21:05PM +0300, Cariotoglou Mike wrote:
>
> > memory and cpu-wise. on Linux, this is nothing, it can handle it easily.
> > otoh, 500 threads for windows is business as usual, but threading on
> > Linux, is , I hear, iffy at best.
>
> Linux runs multi-threaded apps (e.g., AOLserver) quite well, and has
> for many years - since at least 2000 or so, probably earlier.  My
> understanding is that the old LinuxThreads implementation had some
> pretty ugly bits, but it worked.  NPTL is much better, and is standard
> with the Linux 2.6.x kernels.

the issue wasn't necessarily the thread implementation per se, but the fact
that threads were treated as processes for scheduling purposes and hence
scheduled with the regular process scheduler, which was not efficient for a
large number of processes. these problems went away when ingo molnar's O(1)
scheduler was adopted (not sure when it was merged into linus' 2.4, but
distros adopted it quite fast).

-p

Re: Re[2]: [sqlite] Sqlite low level and Speed.

[Paul G]

- Original Message - 
From: "Christian Smith" <[EMAIL PROTECTED]>
To: "Christian Smith" ; "Yuriy" <[EMAIL PROTECTED]>
Sent: Monday, June 20, 2005 12:01 PM
Subject: Re[2]: [sqlite] Sqlite low level and Speed.


> On Mon, 20 Jun 2005, Yuriy wrote:
>
> >CS> When creating testtable, specify val as unique, and specify what to
do
> >CS> with conflicts:
> >CS> CREATE TABLE testtable(val TEXT UNIQUE ON CONFLICT REPLACE);
> >CS> The conflict clauses are documented here:
> >CS> http://www.sqlite.org/lang_conflict.html
> >Try it for ~1,000,000 UNIQUE records very slow :(
> >and me need allso count this UNIQUE records. now i try keep count in
> >the memory.
>
>
> What are you actually trying to do? And can you quantify "very slow" and
> tell us what you actually expect or what would be acceptable?
>
>
> > [snip sample]
>
>
> Is this representitive of what you are trying to do? Are you storing IP
> addresses, and you want to discard duplicates? Using the "on conflict"
> resolution is probably your fastest course of action.

if all he's doing is discarding duplicate strings, with no requirement for
persistent storage, it is easily done with a primitive hash table
implementation. could probably be done efficiently in less than a hundred
lines of c, most of which could be adapted from some example code.

-p

Re: [sqlite] Training opportunity: The Inner Workings Of SQLite

[Paul G]

- Original Message - 
From: "Robert L Cochran" <[EMAIL PROTECTED]>
To: 
Sent: Sunday, June 19, 2005 8:19 PM
Subject: Re: [sqlite] Training opportunity: The Inner Workings Of SQLite


> I'd like to ask about this too. I'd be very happy to pay for DVDs. I'd
> like to go to OSCON but the costs involved make it tough for me,
> especially with 2 kids in university.

i'll second that. financial considerations are not an issue in my case, but
i have other obligations which will prevent me from being able to travel to
oregon at that time.

-p

Re: [sqlite] license to hack: pager in shmem

[Paul G]

- Original Message - 
From: "D. Richard Hipp" <[EMAIL PROTECTED]>
To: <sqlite-users@sqlite.org>
Sent: Friday, June 17, 2005 9:42 PM
Subject: Re: [sqlite] license to hack: pager in shmem


> On Fri, 2005-06-17 at 21:21 -0400, Paul G wrote:
> > i might put in some work on this, although it's a bit more work than i'd
> > hoped (shm would have been faster afair). will this patch, assuming it
looks
> > decent and works (for a reasonable threshold of 'works'), be accepted
into
> > trunk? it's a bit too much to do just for myself, but assuming others
would
> > benefit as well, i might be up for it.
> >
>
> Anything that increases the library footprint needs to be
> surrounded with #ifndef SQLITE_OMIT_

already in place in my working copy.

> See also http://www.sqlite.org/copyright.html

no issue there. i'll take your mail as a yes. is there a separate -dev list
i can moot my questions (if any) to or are they on-topic for this one?

-p

Re: [sqlite] license to hack: pager in shmem

[Paul G]

richard,

comments inline:

- Original Message - 
From: "D. Richard Hipp" <[EMAIL PROTECTED]>
To: <sqlite-users@sqlite.org>
Sent: Friday, June 17, 2005 8:54 PM
Subject: Re: [sqlite] license to hack: pager in shmem


> On Fri, 2005-06-17 at 20:16 -0400, Paul G wrote:
> > i'd like to hack sqlite3 to use shared memory for the page cache.
> >
>
> A more useful hack, perhaps, would be to change the pager to use
> memmap() (when available) instead of read() and write().  This

i'm going to assume you meant mmap() ;] indeed, i've already looked at doing
this (and have the beginnings of an implementation) but was put off by the
intrusiveness of the change.

> The downside:  You'll need to change the pager API around some, which
> means changes to the BTree layer.  The current pager allocates some
> extra memory immediately before and after the data page.  And it tracks
> each page using just a pointer to the data - since it can always find
> the header and tailer using pointer arithmetic.  That won't work with
> memmap().  You'd have to relocate this extra stuff into a separate
> allocation which you then track separately.

right. when i looked at it, it just didn't seem like it would be all that
time-efficient (wrt implementation work) to decouple the page header and
EXTRA from the page itself.

> Also, you'll have to take
> out an exclusive lock on the database before making any changes to the
> data buffers in memory, not just before calling write() as in the
> current
> solution.  That will cut your currency some because it means taking an
> exclusive lock much earlier in the write process.

personally, i don't specifically care, since my usage pattern will be
heavily trended towards reads (if i do any writes at all). however, this is
indeed a valid concern. i'm sure something can be thought up here if there's
user demand, but i don't think it makes sense to deal with this from the
start (premature optimization and all that).

> A new pager API that works for memmap() would also continue to work
> for the old read()/write() method.  So presumably you could create
> a new pragma that lets the programmer decide at run-time which method
> to use.  Use memmap() for faster reads and less memory usage or
> use read/write for better concurrency.

i was actually going to do something like ":mmap:/path/to/db", in keeping
with the :memory: feature, since this whole excercise started with me trying
to determine whether i can tweak the :memory: implementation to do what i
want.

i might put in some work on this, although it's a bit more work than i'd
hoped (shm would have been faster afair). will this patch, assuming it looks
decent and works (for a reasonable threshold of 'works'), be accepted into
trunk? it's a bit too much to do just for myself, but assuming others would
benefit as well, i might be up for it.

cheers,
-p

[sqlite] license to hack: pager in shmem

[Paul G]

folks,

for better or for worse, i'm going to use sqlite3 as a local cache for some
of my preciousss objects (permanent persistence target is mysql). this is in
a php app, so the obvious problem is having multiple sqlite page caches (one
per apache process).

therefore, unless well informed people tell me i'm crazy, i'd like to hack
sqlite3 to use shared memory for the page cache. please note that, at least
for now, i'm quite happy with a read-only solution, which should postpone
having to deal with locking. i'm not going to do this to any other
structures, because despite the obvious sexiness of doing so, i don't think
i'm going to win much in the memory usage or performance departments
(assuming db stored on tmpfs).

the million shekel question is, of course, what non-obvious problems you can
think of with this approach. any useful info/pointers are welcome. i've
perused the source (which is amazingly readable), so feel free to assume
cursory familiarity with the internals.

-p