[sqlite] Newbie issue - Linux error malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *)

[George]

On Sat, 06 Jun 2015 21:14:46 +0700
Dan Kennedy  wrote:

> On 06/06/2015 03:19 AM, George wrote:
> > Hello everyone,
> >
> > I am new to the list. I am working on an application in which I
> > will be embedding SQLite as the database engine. The application is
> > written in C.
> >
> > I am currently having an issue which I am not able to resolve at the
> > moment so I thought I would ask here since I am just starting out
> > with SQLite.
> >
> > My problem is, from my point of view, that I am not able to perform
> > an action to the same database file in the following manner:
> >
> > 1) I open a database via:
> > sqlite3_initialize()
> > sqlite3_open_v2
> > 2) I do some work on getting metadata from the database like table
> > names and their fields and then
> > 3) I close the connection via:
> > sqlite3_close_v2
> > sqlite3_shutdown
> > 4) After all of this is done I wish to process an import file so I
> > need to open another connection to the same database file and run
> > some statements but when I try to do that I get this on the open
> > call in step 1 (above):
> 
> I guess that assert() failing means the heap is corrupted. Which
> might be SQLite related or might not.
> 
> Running the app under [valgrind] might tell you more. Post its
> complete output here if there are errors but it's not obvious what
> the problem is.
> 
> Dan.

Thanks to everyone who answered!

I am doing this on Linux not on an embedded system, Ubuntu 14.04 LTS to
be more precise.

I removed the initialize and shutdown which I had wrapped into my
connect and close methods after reading the "Using SQLite" book and
probably misunderstanding it...

The problem is still there even after I removed the 2 function calls.

I ran the whole app under the valgrind memory checker and it and
valgrind crashed at the end when the explosion happens. Here is the
summary:

==6013== HEAP SUMMARY:
==6013== in use at exit: 0 bytes in 0 blocks
==6013==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==6013== 
==6013== All heap blocks were freed -- no leaks are possible
==6013== 
==6013== For counts of detected and suppressed errors, rerun with: -v
==6013== Use --track-origins=yes to see where uninitialised values come
from ==6013== ERROR SUMMARY: 2113 errors from 137 contexts (suppressed:
0 from 0)
Aborted (core dumped)
make: *** [vg-check-import] Error 134

I do check the return codes and print any errors with sqlite3_errmsg to
see if I can move into another direction.

I also checked my code and I do finalize my statement before the close.

It seems that when I go second time around things go bad...

Not sure what is going on here...
Looking ...
Thanks guys,
George


> 
> 
> 
> 
> >
> > malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr)
> > (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof
> > (struct malloc_chunk, fd && old_size == 0) || ((unsigned long)
> > (old_size)
> >> = (unsigned long)__builtin_offsetof (struct malloc_chunk,
> >> fd_nextsize))+((2 *(sizeof(size_t)) < __alignof__ (long double) ?
> >> __alignof__ (long double) : 2 *(sizeof(size_t))) - 1)) & ~((2
> >> *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long
> >> double) : 2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) &&
> >> ((unsigned long) old_end & pagemask) == 0)' failed. Aborted (core
> >> dumped)
> > This happens in on line 17149 when calling:
> >
> > p = SQLITE_MALLOC( nByte+8 )
> >
> > nByte is 64000
> >
> > in sqlite3.c (amalgamation latest version
> > sqlite-amalgamation-3081002.zip)
> >
> > I am compiling and running the code on:
> > Linux x140e 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:28 UTC
> > 2015 i686 athlon i686 GNU/Linux
> >
> > NAME="Ubuntu"
> > VERSION="14.04.2 LTS, Trusty Tahr"
> > ID=ubuntu
> > ID_LIKE=debian
> > PRETTY_NAME="Ubuntu 14.04.2 LTS"
> > VERSION_ID="14.04"
> > HOME_URL="http://www.ubuntu.com/;
> > SUPPORT_URL="http://help.ubuntu.com/;
> > BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/;
> >
> > I have compiled sqlite.o with the following:
> >
> > gcc -c -Wall -O0 -g -std=c99 -Dlinux -I/usr/local/include sqlite3.c
> > \ -DSQLITE_THREADSAFE=1 -DSQLITE_OMIT_LOAD_EXTENSION -o
> > obj/sqlite3.o
> >
> > Any suggestions or directions greatly appreciated.
> > TIA,
> > George
> > ___
> > sqlite-users mailing list
> > sqlite-users at mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> 
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] sqlite web server on port 80 down? (IPv6)

[Richard Hipp]

On 6/7/15, Mark Martinec  wrote:
> For a web browser that does not implement a 'happy eyeballs' IPv6 ->
> IPv4
> fallback mechanism, the www.sqlite.org web server on port 80 is
> unreachable,
> although it does respond to https on port 443, and ping6 is fine too.
>
> $ curl -6 http://www.sqlite.org/
> curl: (7) Failed to connect to www.sqlite.org port 80: Connection
> refused

Unknown cause.  Problem cleared by running "service xinetd restart".


-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] User-defined types -- in Andl

[da...@andl.org]

I've been reading this thread with great interest. It parallels the project
I've been working on: Andl.

Andl is A New Database Language.

Andl does what SQL does, but it is not SQL. Andl has been developed as a
fully featured database programming language following the principles set
out by Date and Darwen in The Third Manifesto. It includes a full
implementation of the Relational Model published by E.F. Codd in 1970, an
advanced extensible type system, database updates and other SQL-like
capabilities in a novel and highly expressive syntax.

The intended role of Andl is to be the implementation language for the data
model of an application. It is already possible to code the business model
of an application in an SQL dialect, but few people do this because of
limitations in SQL.  Andl aims to provide a language free of these problems
that works on all these platforms.

The current implementation on SQLite uses a mixture of generated SQL and a
runtime VM. User-defined types are blobs, which the VM understands. A future
implementation could generate SQLite VM code directly instead of SQL, which
would save some overhead. 

The website is andl.org. The GitHub project is
https://github.com/davidandl/Andl. It's a work in progress. Any feedback
welcomed.

Regards
David M Bennett FACS

Andl - A New Database Language - andl.org

-Original Message-
From: sqlite-users-boun...@mailinglists.sqlite.org
[mailto:sqlite-users-bounces at mailinglists.sqlite.org] On Behalf Of Darko
Volaric
Sent: Thursday, 4 June 2015 8:55 AM
To: General Discussion of SQLite Database; ott at mirix.org
Subject: Re: [sqlite] User-defined types

I've tackled this problem from a couple of different angles. My goal was to
allow arbitrary user defined types, based on the builtin types (essentially
subtypes of the existing types), with a minimum of work and minimum
disruption of the normal/existing use of the database and API.

The approaches I considered were:

- encoding the user type codes for each data column in a separate column
dedicated to the purpose. This is a low impact but cumbersome, for instance
using a function that interprets the user type would have to have the user
type passed in for each argument, along with the actual data.

- modifying the data file format to carry user type information. There is
space in the record/row header where you can encode this information in a
backwards compatible way, but the source code for data record access is not
friendly, basically a dense blob of code with a lot of integer literals
which are all very important, but it's hard to be sure what they entail and
that you haven't introduced a subtle bug and ultimately data corruption.
Additionally the user type would have to be passed around internally - for
example in the sqlite3_value object - and tracking down all of those
reliably is a bit of work.

- using blobs. Although using text representation is friendly when looking
at the data with standard tools, it's slower and takes up more memory in
various places. I found that encoding some user types as blobs with a type
marker at their start (a single byte with extensions) and interpreting them
was a simple and low impact approach. I also split the standard integer type
four ways (negative and positive, odd and even) to get the scalar user types
I needed. User defined functions and collations need to be defined for
interpreting these user types of course.

The first option isn't very practical. The second option is the fastest and
most robust solution and my long term approach which I will be going back to
after development has progressed a bit more. Currently I'm using the third
approach as an interim measure. I'm supporting arbitrary prec ints and
reals, arrays and tuples and other types this way.



On Wed, May 27, 2015 at 3:48 AM, Matthias-Christian Ott 
wrote:

> I want to define user-defined types, i.e. types not SQLite has not 
> built-in and make sure that I didn't overlook something. Is it correct 
> that values of user-defined types should be stored as text and have a 
> collation defined if there is an order relation for the type if the 
> type cannot be represented as a subset of integer or float?
>
> Example:
> Suppose I want to store arbitrary precision integers in SQLite. I 
> would create a column with text affinity, (uniquely) serialize and 
> deserialize the integers to text (e.g. by converting them into decimal
> representation) and define and declare a collation that deserializes 
> the texts to arbitrary integers and compares the integers.
>
> Is there another way to define user-defined types despite this method 
> and virtual tables?
>
> - Matthias-Christian
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org

[sqlite] User-defined types -- with Andl

[da...@andl.org]

This is the challenge that I accept, with Andl.

SQL has been astonishingly successful, partly because of sound foundations
and partly because it's a monopoly. It's not a bad language, but on the
other hand it many ways it's not a language at all. Up until the 1992
version and including the SQLite dialect, there are many things that a
programming language should provide that it does not.

I have some specific criticisms of SQL, but I'm not here to bury it. I'd
just like to offer something better. That's the point of Andl.

I have reviewed the SQL 'challenge' you posted some little while back, and
there is no doubt that Andl can handle it, and (IMHO) the code is somewhat
shorter and somewhat cleaner than the SQL. I'll see what I can do to respond
to the challenge. I assume there is an SQLite database somewhere I can check
it on.

Regards
David M Bennett FACS

Andl - A New Database Language - andl.org

-Original Message-
From: sqlite-users-boun...@mailinglists.sqlite.org
[mailto:sqlite-users-bounces at mailinglists.sqlite.org] On Behalf Of Richard
Hipp
Sent: Friday, 5 June 2015 9:11 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] User-defined types

On 6/4/15, Darko Volaric  wrote:
>
> What is motivating this for me is that I generate many unique queries 
> in my code for almost any operation. Converting those to SQL is error 
> prone and uses a lot of memory compared to the operation involved. The 
> database engine is so fast and efficient yet I'm wasting resources making
SQL!
>

You are welcomed to go off and try to come up with a new and better
interface.  That's the beauty of open-source.  Maybe you will come up with
some new and innovative ideas that will change the industry!
It's happened before!

I just want to ensure that if, after working on your new approach for a
while, you eventually decide that SQL isn't quite as bad a language as you
originally thought it was, that you don't come back and say I didn't warn
you.

--
D. Richard Hipp
drh at sqlite.org
___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] User-defined types -- in Andl

[Simon Slavin]

On 7 Jun 2015, at 6:51pm, Scott Doctor  wrote:

> Do you have a PDF that explains the language?

There are plenty of blog entries which explain the language.  I spent more time 
looking for some examples (I understand better from examples) and eventually 
found one.

Simon.

[sqlite] json_* functions in sqlite

[David Barrett]

For what it's worth, I'd also love some official JSON support, with JSON
indexes (eg, a function index that pulls a JSON value).  However, I agree
it doesn't make sense to add to the main codebase -- I was more thinking an
official plugin (so we don't just keep writing our own over and over).

David
On Apr 22, 2015 10:57 PM, "James K. Lowden" 
wrote:

> On Tue, 21 Apr 2015 18:09:33 -0700
> Ashish Sharma  wrote:
>
> > Many times I store JSON data in sqlite. It will be useful if sqlite
> > came with functions which understand JSON. Presto has a nice set
> > https://prestodb.io/docs/current/functions/json.html
>
> In case you don't know, you could implement functions such as Presto
> provides yourself with a set of user-defined functions, without any
> help from the SQLite project.
>
> As to whether JSON should be a supported datatype in SQLite, the answer
> is clearly No.  SQLite, let us note, barely recognizes datatypes at
> all, and lacks even a date type.  But more than that, JSON as "native"
> datatype has two fundamental problems: definition and complexity.
>
> Definitionally, we should require any datatype have meaningful
> operations comparable to those of numbers and strings.  What does it
> mean to "add" two JSON objects?  When is one less than another?  Do you
> seriously want to propose as a *type* a thing that can't be a primary
> key?
>
> The problem of complexity is that everything in the JSON blob can,
> fundamentally, be represented as tables in SQLite.  I realize "modern"
> tools read/write JSON, that it's the OODBMS of the Javascript set.  But
> that doesn't change the fact that the JSON tree is a graph, and we know
> every graph can be represented with tables.
>
> Why does that matter?  Because a tree-as-datatype would add a whole new
> theoretical structure (graph theory) that is 100% redundant to the
> relational model embodied in SQLite.  You get a bunch of new functions
> and ways to get at the data.  What you do *not* get is additional query
> power.  In fact you get less, because graph theory gives you less: no
> subsets and no joins, to name just two.
>
> That's not to say there should be some rule preventing you from storing
> JSON in your SQLite database.  You may find it convenient, especially if
> supported with some functions that make it possible to compare (or
> perhaps update) components of it, because it represents some giant
> state-property that for most purposes can be treated as an integral
> unit.  It is to say that every JSON-specific feature you add duplicates
> one already present (in a different form) in SQLite.  Add enough of
> them and you'll replace the DBMS with itself, if you see what I mean.
>
> --jkl
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] User-defined types -- in Andl

[Scott Doctor]

So we are supposed to learn this new language by osmosis?


Scott Doctor
scott at scottdoctor.com

On 6/7/2015 11:00 AM, Simon Slavin wrote:
> On 7 Jun 2015, at 6:51pm, Scott Doctor  wrote:
>
>> Do you have a PDF that explains the language?
> There are plenty of blog entries which explain the language.  I spent more 
> time looking for some examples (I understand better from examples) and 
> eventually found one.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>

[sqlite] User-defined types -- in Andl

[Scott Doctor]

Do you have a PDF that explains the language?

My opinion is that I have seen many languages come and go. Consider 
general programing languages. C is far superior to just about any 
language available. In fact the underlying  code for most languages is 
written in C. So the question becomes, why does everyone see a need to 
keep creating new languages? Beyond the assembler instruction compiler 
for a new processor, a C compiler is usually the first compiler written 
for that processor. It is simple and straight forward to do, and I have 
done so many times. If you look at the evolution of the C language (even 
considering that bastard child C++) it has changed very little in 35 
years, is available for all architectures, properly written code will 
compile for any architecture unmodified, and it has every hook needed to 
do any programming task.

Many decades ago, an attempt was made to make a more efficient keyboard 
to replace the QWERTY keyboard. Some of the fledgling computer companies 
in the 1970's and 80's tried to get them accepted. From a straight 
technical perspective, they are more efficient. They all failed to be 
accepted.

Consider what is easier. To train a new generation on the old stuff? or 
to re-train half a dozen generations on the new stuff? Choices are 
usually made on which is easier now versus the long term benefits later.

What I find interesting is how many of these "New" languages are so 
similar to C. Java, PHP, and such, take the base constructs of C, then 
add "Special" additions to do what is basically just a C function. They 
change some syntax to make it more BASIC like, but the general technique 
still follows C. So why not just use C?

In my current project, I debated just writing a bunch of C functions to 
handle my data. Once I got into it, I realized that beyond simply 
reading/writing some data structures, the code was getting complex very 
fast. SQLite lets me offload the low level details and just write a 
query with SQLite handling the parsing and search. Is it ideal, hardly. 
But the alternative is much more complicated and not worth the effort. 
If your data is just a few simple data structures, sure, just write some 
C code. But the reality is that most well developed programs quickly 
branch into ever increasing complexity.

Regarding SQL, many companies are attempting to replace SQL with their 
flavor of an interface. Embarcadero (the old Borland) has in their 
development system a "Universal" database interface to make accessing 
databases "Universal". The idea being that a database designer just 
wants their data and does not care about the underlying mechanisms. 
Wait, that is the entire concept behind every programming language. If 
programmers cared about the underlying mechanism at every level and just 
wanted to write the most optimal code possible (which is a far off 
concept no longer desired for some reason) then all programs would be 
written in assembler.  I used their system for a while. Now I just write 
the SQL directly and just link in SQLite instead of using Embarcadero's 
stuff. Although some of their constructs "seemed" to simplify some 
tasks, the program as a whole was actually more complex.

My opinion why SQL has endured is that it actually hides from the 
programmer the internal complexity required to implement a task. While 
some of the syntax may be a bit quirky, so is talking to a teenager, but 
we adapt. If SQL did not do what is needed then people would not use it. 
The reality is that SQL actually is a well thought out language, even if 
the syntax can be a bit bulky and awkward.

If you consider "Who" will be using the language, I find that many of 
the "Users" are not computer scientists, but people that are trying to 
make a database for their business purposes, and their specialty is not 
the intricacies and philosophies of coding architecture. Some of us are 
those brainiac computer geek types. But I see a large number of users as 
people who "Learned Enough" to do their job.

Another issue is the ability for someone a decade, or two, or three, 
later, who is handed the database code without any documentation, to be 
able to decipher the program. (remember the Y2K bug). As you develop 
your language, consider if someone without having read the manual, but 
has a background in programming, could decipher a program written in 
your language.



Scott Doctor
scott at scottdoctor.com

On 6/7/2015 2:17 AM, david at andl.org wrote:
> I've been reading this thread with great interest. It parallels the project
> I've been working on: Andl.
>
> Andl is A New Database Language.
>
> Andl does what SQL does, but it is not SQL. Andl has been developed as a
> fully featured database programming language following the principles set
> out by Date and Darwen in The Third Manifesto. It includes a full
> implementation of the Relational Model published by E.F. Codd in 1970, an
> advanced extensible type system, database updates