[sqlite] Request: Metadata about C API constants and functions

[James K. Lowden]

On Thu, 16 Apr 2015 00:40:28 +0100
Simon Slavin  wrote:

> > It is a very productive time when you
> > get to delete code :-)
> 
> 

"[If] we wish to count lines of code, we should not regard
them as "lines produced" but as "lines spent": the current conventional
wisdom is so foolish as to book that count on the wrong side of the
ledger."
--EWD, On the cruelty of really teaching computing science
http://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/EWD1036.html

[sqlite] Request: Metadata about C API constants and functions

[Simon Slavin]

On 2015-04-10 04:11 PM, Richard Hipp wrote:

> https://www.sqlite.org/toc.db

I download this and do the following:

164:~ simon$ sqlite3 /Users/simon/Desktop/toc.db 
SQLite version 3.8.5 2014-08-15 22:37:57
Enter ".help" for usage hints.
sqlite> .mode column
sqlite> select * from toc limit 10;
sqlite3 object  0   Database Connection Handle  
c3ref/sqlite3.html
sqlite3_in  object  0   64-Bit Integer Types
c3ref/int64.html  
sqlite3_ui  object  0   64-Bit Integer Types
c3ref/int64.html  
sqlite_int  object  0   64-Bit Integer Types
c3ref/int64.html  
sqlite_uin  object  0   64-Bit Integer Types
c3ref/int64.html  
sqlite3_fi  object  0   OS Interface Open File Han  c3ref/file.html 
  
sqlite3_io  object  0   OS Interface File Virtual   
c3ref/io_methods.h
sqlite3_mu  object  0   Mutex Handle
c3ref/mutex.html  
sqlite3_vf  object  0   OS Interface Object c3ref/vfs.html  
  
sqlite3_me  object  0   Memory Allocation Routines  
c3ref/mem_methods.


I know that the .width command exists, but why is the first column so short ?  
Is the shell tool set to use width 10 for the first column ?  All the columns 
except for the numeric one are defined as TEXT but they're given different 
widths.

Simon.

[sqlite] Request: Metadata about C API constants and functions

[Simon Slavin]

On 16 Apr 2015, at 12:31am, Roger Binns  wrote:

> It is a very productive time when you
> get to delete code :-)



See the paragraph "How do you measure programmer productivity?" half way down 
the page, just before the Epilogue.

Simon.

[sqlite] Request: Metadata about C API constants and functions

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 04/10/2015 07:11 AM, Richard Hipp wrote:
> On 4/9/15, Roger Binns  wrote:
>> 
>> So the URL for this SQLite database is?  :-)
>> 
> 
> https://www.sqlite.org/toc.db

Thank you for this.  I've now updated my doc build tools, and even
added some more features.  All the regular expressions are gone.  My
doc tool has ~15% less code.  It is a very productive time when you
get to delete code :-)

Roger

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlUu9N0ACgkQmOOfHg372QSvmgCePLoTdlfDYPxoBqPeF1k9oW9t
f+0AoMwRfmOe5WUJ1y3XEI+0jUqEh0WK
=+NlH
-END PGP SIGNATURE-

[sqlite] Request: Metadata about C API constants and functions

[Kevin Youren]

Thanks, Richard

I found (https://www.sqlite.org/docsrc/timeline) &
https://www.sqlite.org/docsrc/artifact/5c48dd261dbe5804  very useful.

Last year I was parsing 100M of XML stuff and loading it into an Sqlite
database. A bit of hammering required as the XML had minor errors like
missing end tags. "malformed" 


In a meeting, we were discussing a very specific piece of functionality.
"How many times is this function used?" One guy said "About 300 times"
by I was able to answer 2365 times, within 30 seconds, just some simple
SQL.

Incidentally, it was far easier to regenerate the XML from the Sqlite,
similar to a comma delimited format, without all the "malformed"
problems. 

The malformed XML was targeted to be stored in Git. 


regs,

Kev

[sqlite] Request: Metadata about C API constants and functions

[R.Smith]

On 2015-04-10 04:11 PM, Richard Hipp wrote:
> https://www.sqlite.org/toc.db

Thank you Richard!
Will this link always have the latest DB?


This db format is preferred, but in case anyone is still interested in 
the webby formats, here is the DB in XML:
http://sqlc.rifin.co.za/toc.xml

and JSON:
http://sqlc.rifin.co.za/toc.json

[sqlite] Request: Metadata about C API constants and functions

[Simon Slavin]

On 10 Apr 2015, at 5:00pm, R.Smith  wrote:

> On 2015-04-10 04:11 PM, Richard Hipp wrote:
>> https://www.sqlite.org/toc.db
> 
> Thank you Richard!
> Will this link always have the latest DB?

If I may suggest ... in that database, introduce a new column which holds the 
version in which that call was introduced.  This would allow people to 
understand why FOREIGN KEYs, for example, don't work in the version of SQLite3 
that they're stuck with.

Don't really care what you do for initial values.  You could leave them all 
null which means 'was already implemented by version 3.8.9.  Or you could find 
some people who would want to put the correct values in.

Simon.

[sqlite] Request: Metadata about C API constants and functions

[R.Smith]

On 2015-04-10 02:09 PM, Simon Slavin wrote:
> On 10 Apr 2015, at 12:53pm, R.Smith  wrote:
>
>> It doesn't matter, C API calls or pragma, whatever works best as long as 
>> there is /some/ way to get to that info.  Usually though, most C API calls 
>> valuable to end users too, ends up having a pragma that can produce similar 
>> output.
> I think there are two different things being discussed here:
>
> The original user was asking for details about the calls built into the 
> SQLite API like sqlite3_step().
>
> The other user was asking for details about what loadable extensions are 
> currently loaded.

Right you are, my bad, both issues are important though: I see Roger 
already asked for a compiled version of the db containing the 
information as Richard suggested exists - which would make documentation 
linking much easier for any wrapper or gui programmer and would 
definitely be a boon - I'd like this very much too. (And I'd prefer it 
in SQLite DB format as I'm sure Roger would, just did not realise that 
is even an option, usually documenty things are all XMLed up).

And then there is the ability to query available function calls 
registered (or embedded) with the SQLite engine you are using at the 
moment (which might well be linked via a library plus loadable 
extensions). This would be fantastic to have too.

As for the usual C API calls supported, I think knowing the version 
number (which you can already obtain via query or API) suffices, so 
happy there if the DB mentioned above is available.

[sqlite] Request: Metadata about C API constants and functions

[R.Smith]

On 2015-04-10 09:48 AM, Zsb?n Ambrus wrote:
> On 4/10/15, Dominique Devienne  wrote:
>> But that's build time. There is no way (i.e. pragma) to list registered SQL
>> functions at runtime.
>>
>> This is something that I've asked for before, and I'm having trouble
>> understanding why nobody cares.
>>
>> When you run an SQLite "shell", the host program can have enabled/disabled
>> built-in functions at build-time, and/or added any number of custom SQL
>> functions. But you have no way to know.
> Um, you're talking about SQL functions.  But I think Roger asked for C
> functions in the public C api of Sqlite.

It doesn't matter, C API calls or pragma, whatever works best as long as 
there is /some/ way to get to that info.  Usually though, most C API 
calls valuable to end users too, ends up having a pragma that can 
produce similar output.

I would very much like this too.

[sqlite] Request: Metadata about C API constants and functions

[Simon Slavin]

On 10 Apr 2015, at 12:53pm, R.Smith  wrote:

> It doesn't matter, C API calls or pragma, whatever works best as long as 
> there is /some/ way to get to that info.  Usually though, most C API calls 
> valuable to end users too, ends up having a pragma that can produce similar 
> output.

I think there are two different things being discussed here:

The original user was asking for details about the calls built into the SQLite 
API like sqlite3_step().

The other user was asking for details about what loadable extensions are 
currently loaded.

Simon.

[sqlite] Request: Metadata about C API constants and functions

[Dominique Devienne]

On Fri, Apr 10, 2015 at 9:48 AM, Zsb?n Ambrus  wrote:

> On 4/10/15, Dominique Devienne  wrote:
> > But that's build time. There is no way (i.e. pragma) to list registered
> SQL functions at runtime.
>
> Um, you're talking about SQL functions.  But I think Roger asked for C
> functions in the public C api of Sqlite.


Right. Silly me... Thanks for getting that straight. And sorry to be OT.

Hopefully all my (valid to me) points will be considered though, and we can
get that runtime introspection of functions/modules, despite being
off-topic for this thread. Obviously I care about this :). --DD

[sqlite] Request: Metadata about C API constants and functions

[Richard Hipp]

On 4/9/15, Roger Binns  wrote:
>
> So the URL for this SQLite database is?  :-)
>

https://www.sqlite.org/toc.db
-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Request: Metadata about C API constants and functions

[Zsbán Ambrus]

On 4/10/15, Dominique Devienne  wrote:
> But that's build time. There is no way (i.e. pragma) to list registered SQL
> functions at runtime.
>
> This is something that I've asked for before, and I'm having trouble
> understanding why nobody cares.
>
> When you run an SQLite "shell", the host program can have enabled/disabled
> built-in functions at build-time, and/or added any number of custom SQL
> functions. But you have no way to know.

Um, you're talking about SQL functions.  But I think Roger asked for C
functions in the public C api of Sqlite.

Ambrus

[sqlite] Request: Metadata about C API constants and functions

[Dominique Devienne]

On Fri, Apr 10, 2015 at 2:42 AM, Richard Hipp  wrote:

> On 4/9/15, Roger Binns  wrote:
> > For each function, its name...
>

> As it happens, the SQLite documentation build process already parses
> out most of this and puts it into a database already.  If you
> check-out the documentation sources
> (https://www.sqlite.org/docsrc/timeline) and successfully build the
> documentation, it creates a
> docinfo.db file that contains a lot of what you have requested above.
> Probably it just needs to be enhanced a little.  Most of the relevant
> information is extracted from sqlite3.h using the
> https://www.sqlite.org/docsrc/artifact/5c48dd261dbe5804 script.


But that's build time. There is no way (i.e. pragma) to list registered SQL
functions at runtime.

This is something that I've asked for before, and I'm having trouble
understanding why nobody cares.

When you run an SQLite "shell", the host program can have enabled/disabled
built-in functions at build-time, and/or added any number of custom SQL
functions. But you have no way to know.

Even in the sqlite3 shell, the .load of extensions often triggers
registrations of new functions, and again you cannot know about this new
function(s) (or vtable module(s)).

You have to a-priori know all this, yet tools (the sqlite3 shell, the
myriad GUI shells out there) can't allow you to introspect all that. That's
a shame IMHO.

The info exists inside the runtime, all that's missing are the pragmas to
extract them. Can we please add these pragmas? The "truth" is in the
"code", especially with SQLite where features (and thus functions) can be
turned on/off at build-time. Neither the website or the doc are the "truth"
of what you are running, which is why we need a "programmatic" way to get
the info at runtime.

My $0.02. --DD

PS: Roger could then just *run* these pragmas and with some good naming
convention fetch additional info from the website to embed the 1-liner doc
he wants. (sqlite3 the shell tool should ideally provide built-in 1-liner
doc for the functions as well, but I already hear all the "Lite-in-SQLite
is not for nothing" if I even proposed to include that in the library
itself, or allow custom function to provide such a doc snippet during
registration...).

[sqlite] Request: Metadata about C API constants and functions

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 04/10/2015 12:33 AM, Dominique Devienne wrote:
> But that's build time. There is no way (i.e. pragma) to list
> registered SQL functions at runtime.

There is a ticket for this:

  https://www.sqlite.org/src/tktview/5896edbe463b6a1f0f14

It was rejected without explanation a year ago, after being open for 8
years.  Sadly there is no workaround as the information is just not
exposed in any useful way.

Roger

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlUn0SsACgkQmOOfHg372QS9HgCeJS+65yV4iagsRqCpSXs8+em/
hawAoNPy4Oqw4t2iEbJzuSSOrFq3IqWa
=zlHT
-END PGP SIGNATURE-

[sqlite] Request: Metadata about C API constants and functions

[Simon Slavin]

On 10 Apr 2015, at 1:42am, Richard Hipp  wrote:

> As it happens, the SQLite documentation build process already parses
> out most of this and puts it into a database already.  If you
> check-out the documentation sources
> (https://www.sqlite.org/docsrc/timeline) and successfully build the
> documentation, it creates a
> docinfo.db file that contains a lot of what you have requested above.

Hmm.  We could add an output mode to the shell tool which outputs in JSON 
format.  And by "We" I mean someone who can write good C, which isn't me.

Simon.

[sqlite] Request: Metadata about C API constants and functions

[Richard Hipp]

On 4/9/15, Roger Binns  wrote:
> These are the details of what I need:
>
> For each function, its name, the doc page and fragment, and ideally a
> few word description of it (the title on the doc page is fine).
>
> For each family of constants (eg result codes, extended result codes,
> run time limits), the doc page, and a few word description (of the family)
>
> A JSON or XML file with that would make my life a lot easier.

JSON/XML?  Seriously?  We eat our own dogfood here, thank you!  When
you get the data it will be in an SQLite database file!  JSON/XML
Good grief...

As it happens, the SQLite documentation build process already parses
out most of this and puts it into a database already.  If you
check-out the documentation sources
(https://www.sqlite.org/docsrc/timeline) and successfully build the
documentation, it creates a
docinfo.db file that contains a lot of what you have requested above.
Probably it just needs to be enhanced a little.  Most of the relevant
information is extracted from sqlite3.h using the
https://www.sqlite.org/docsrc/artifact/5c48dd261dbe5804 script.


-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Request: Metadata about C API constants and functions

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 04/09/2015 06:43 PM, Simon Slavin wrote:
> Hmm.  We could add an output mode to the shell tool which outputs
> in JSON format.  And by "We" I mean someone who can write good C,
> which isn't me.

Get yourself a time machine, go back 6 years, and enjoy the shell
which has had that all along:

  http://rogerbinns.github.io/apsw/shell.html

The shell is in Python, but you don't need to know any to use it.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlUnNdQACgkQmOOfHg372QRiPQCgp3OWyyxIhnKvUfOSYoQG8Mnz
RvkAnRFOXOukgmWyB8TRnEHVtpTYNzIG
=gLhP
-END PGP SIGNATURE-

[sqlite] Request: Metadata about C API constants and functions

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 04/09/2015 05:42 PM, Richard Hipp wrote:
> On 4/9/15, Roger Binns  wrote:
>> A JSON or XML file with that would make my life a lot easier.
> 
> JSON/XML?  Seriously?  We eat our own dogfood here, thank you!
> When you get the data it will be in an SQLite database file!
> JSON/XML Good grief...

So the URL for this SQLite database is?  :-)

> As it happens, the SQLite documentation build process already
> parses out most of this and puts it into a database already.

Any chance you can put a copy of the database on the website that gets
updated along with documentation updates?

Roger

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlUnNLwACgkQmOOfHg372QSr3gCfcR77qmpDg0ZOntG0BHXFtHY8
EbwAnA3DjcRx1aCvJj31DbNpyV/XVgSe
=Puuy
-END PGP SIGNATURE-

[sqlite] Request: Metadata about C API constants and functions

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I am the author of a Python wrapper for SQLite.  As part of my
documentation build process I link function names to the relevant page
in the SQLite doc.  I also have to wrap all constants - eg making sure
that all SQLITE_LIMIT_ ones are wrapped and pointing to their doc.  (I
point to most C functions and wrap virtually all constants.)

In order to point to the correct doc pages, and verify that I pick up
all constants including new ones with each release, I have code that
parses the SQLite website building up lists of functions and constants
and which page is the correct one for them.  That means regular
expressions like r"""(sqlite3_.+?)<"""

This is a little brittle as it breaks every time there are formatting
changes.  Would it be possible to have some data file on the website
instead that I could use?  Perhaps other wrapper authors have the same
needs.

These are the details of what I need:

For each function, its name, the doc page and fragment, and ideally a
few word description of it (the title on the doc page is fine).

For each family of constants (eg result codes, extended result codes,
run time limits), the doc page, and a few word description (of the family)

A JSON or XML file with that would make my life a lot easier.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlUnF1UACgkQmOOfHg372QT6wACggLHovwyZvwyYQJgD2UPmhH+N
g7YAn08hjxOP5Qol/jy0QZekxPgWMy4v
=261V
-END PGP SIGNATURE-