Re: sqlite3 update

[James Turner]

On Thu, Jun 23, 2016 at 09:03:43AM +0100, Stuart Henderson wrote:
> On 2016/06/23 00:25, Ted Unangst wrote:
> > Stuart Henderson wrote:
> > > Turns out I forgot about the pthread stubs, without which there
> > > are build failures in mandoc and various ports things unless we link
> > > them with -lpthread. This broke a few things in my first ports test
> > > build, but thanks to guenther's work it should now be valid to pull
> > > this in via an inter-library dependency rather than having to
> > > sprinkle it over mandoc and lots of the ports tree.
> > 
> > 
> > > Index: Makefile
> > > -# so that it works with NO THREADS
> > > -SRCS +=  pthread_stub.c
> > > -
> > > -#mem3.c mem5.c
> > > +SRCS =   sqlite3.c
> > > +LDADD =  -lpthread
> > 
> > Why are we dropping the stubs in favor of linking with libpthread?
> 
> Trying to avoid too many changes to upstream's source. But I suppose
> we can keep that as a separate file - just tested that with 3 things
> where I ran into the problem (mandoc, www/kcaldav, www/nsh,static)
> and they're happy. And I've tried build usr.bin/sqlite3 both with
> and without -pthread and both versions of that also work OK for the
> things I've tested.
> 
> So, new diff: https://junkpile.org/sqlite-3.11.0.diff.gz,2
> 
> pthread_stub.c moves from lib/libsqlite3/tsrc/ to lib/libsqlite3/
> but is otherwise unchanged from the in-tree version.
> 
> The only change to the Makefile from my previous diff is adding
> pthread_stub.c to SRCS.
> 

Like I've stated before, I think this is the best way to move forward
for the time being, so you have my ok.

-- 
James Turner

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/23 00:25, Ted Unangst wrote:
> Stuart Henderson wrote:
> > Turns out I forgot about the pthread stubs, without which there
> > are build failures in mandoc and various ports things unless we link
> > them with -lpthread. This broke a few things in my first ports test
> > build, but thanks to guenther's work it should now be valid to pull
> > this in via an inter-library dependency rather than having to
> > sprinkle it over mandoc and lots of the ports tree.
> 
> 
> > Index: Makefile
> > -# so that it works with NO THREADS
> > -SRCS +=pthread_stub.c
> > -
> > -#  mem3.c mem5.c
> > +SRCS = sqlite3.c
> > +LDADD =-lpthread
> 
> Why are we dropping the stubs in favor of linking with libpthread?

Trying to avoid too many changes to upstream's source. But I suppose
we can keep that as a separate file - just tested that with 3 things
where I ran into the problem (mandoc, www/kcaldav, www/nsh,static)
and they're happy. And I've tried build usr.bin/sqlite3 both with
and without -pthread and both versions of that also work OK for the
things I've tested.

So, new diff: https://junkpile.org/sqlite-3.11.0.diff.gz,2

pthread_stub.c moves from lib/libsqlite3/tsrc/ to lib/libsqlite3/
but is otherwise unchanged from the in-tree version.

The only change to the Makefile from my previous diff is adding
pthread_stub.c to SRCS.

Re: sqlite3 update

[Ted Unangst]

Stuart Henderson wrote:
> Turns out I forgot about the pthread stubs, without which there
> are build failures in mandoc and various ports things unless we link
> them with -lpthread. This broke a few things in my first ports test
> build, but thanks to guenther's work it should now be valid to pull
> this in via an inter-library dependency rather than having to
> sprinkle it over mandoc and lots of the ports tree.


> Index: Makefile
> -# so that it works with NO THREADS
> -SRCS +=  pthread_stub.c
> -
> -#mem3.c mem5.c
> +SRCS =   sqlite3.c
> +LDADD =  -lpthread

Why are we dropping the stubs in favor of linking with libpthread?

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/22 00:10, Stuart Henderson wrote:
> On 2016/06/08 19:01, James Turner wrote:
> > I prefer option 2. Switch to the amalgamation with our changes on top.
> 
> I've been looking at this. I don't really like any of the options
> but this seems the "least worst" one. I'm not terribly happy about
> this, but I don't see what else we can really do at present.
> 
> Current version of this diff is a horrendous 480K lines / 17MB
> uncompressed, so it's at https://junkpile.org/sqlite-3.11.0.diff.gz
> rather than in this mail, and I am just starting a ports build
> with it.

Results from the ports build are in. pthread is a slight problem;
most things are fine, but there are a couple of ports which statically
link libsqlite3 and those of course don't pull in libpthread.

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/08 19:01, James Turner wrote:
> I prefer option 2. Switch to the amalgamation with our changes on top.

I've been looking at this. I don't really like any of the options
but this seems the "least worst" one. I'm not terribly happy about
this, but I don't see what else we can really do at present.

Current version of this diff is a horrendous 480K lines / 17MB
uncompressed, so it's at https://junkpile.org/sqlite-3.11.0.diff.gz
rather than in this mail, and I am just starting a ports build
with it.

It was constructed like this:

- Remove the old source files and tools

- Extract verbatim sqlite 3.11.0 from upstream,
https://www.sqlite.org/2016/sqlite-amalgamation-311.zip

- Add the local arc4random patch

- Adjust Makefile (diff for this is below, and also in the above
gzipped diff).

At least the library size has dropped by a decent amount by doing
this, from 3.7M to 3.0M (amd64).

> > > > With this the main file is unfortunately huge but would be easier
> > > > by far to update later. Note that we only have a small change to
> > > > the actual source code (replacing the RNG code) which is easy enough
> > > > to carry across. Most of our changes are to the build infrastructure.

Turns out I forgot about the pthread stubs, without which there
are build failures in mandoc and various ports things unless we link
them with -lpthread. This broke a few things in my first ports test
build, but thanks to guenther's work it should now be valid to pull
this in via an inter-library dependency rather than having to
sprinkle it over mandoc and lots of the ports tree.

Index: Makefile
===
RCS file: /cvs/src/lib/libsqlite3/Makefile,v
retrieving revision 1.15
diff -u -p -r1.15 Makefile
--- Makefile12 Sep 2015 02:08:34 -  1.15
+++ Makefile21 Jun 2016 22:52:47 -
@@ -2,40 +2,10 @@
 
 .include 
 
-.if defined(NOPIC)
-CPPFLAGS +=-DSQLITE_OMIT_LOAD_EXTENSION=1
-.endif
-
-CPPFLAGS +=-I${.OBJDIR} -I${.CURDIR}/tsrc -I${.CURDIR}/src \
-   -I${.CURDIR}/ext/rtree -I${.CURDIR}/ext/fts3
 LIB = sqlite3
 
-.PATH: ${.CURDIR}/tsrc ${.CURDIR}/src ${.CURDIR}/ext/fts3 ${.CURDIR}/ext/rtree
-
-SRCS = alter.c analyze.c attach.c auth.c \
-   backup.c bitvec.c btmutex.c btree.c build.c \
-   callback.c complete.c ctime.c date.c dbstat.c delete.c \
-   expr.c fault.c fkey.c \
-   fts3.c fts3_aux.c fts3_expr.c fts3_hash.c fts3_icu.c fts3_porter.c \
-   fts3_snippet.c fts3_tokenize_vtab.c fts3_tokenizer.c fts3_tokenizer1.c \
-   fts3_unicode.c fts3_unicode2.c fts3_write.c \
-   func.c global.c hash.c \
-   insert.c journal.c legacy.c loadext.c \
-   main.c malloc.c mem1.c \
-   memjournal.c \
-   mutex.c mutex_noop.c mutex_unix.c \
-   notify.c opcodes.c os.c os_unix.c \
-   pager.c parse.c pcache.c pcache1.c pragma.c prepare.c printf.c \
-   random.c resolve.c rowset.c rtree.c select.c status.c \
-   table.c threads.c tokenize.c treeview.c trigger.c \
-   update.c util.c vacuum.c \
-   vdbe.c vdbeapi.c vdbeaux.c vdbeblob.c vdbemem.c vdbesort.c \
-   vdbetrace.c wal.c walker.c where.c wherecode.c whereexpr.c utf.c vtab.c
-
-# so that it works with NO THREADS
-SRCS +=pthread_stub.c
-
-#  mem3.c mem5.c
+SRCS = sqlite3.c
+LDADD =-lpthread
 
 FEATURE_FLAGS =-DSQLITE_ENABLE_COLUMN_METADATA \
-DSQLITE_ENABLE_RTREE \
@@ -62,55 +32,14 @@ CPPFLAGS += $(FEATURE_FLAGS) -DSQLITE_T
 
 FILES = sqlite3.h sqlite3ext.h
 
-opcodes.c: opcodes.h ${.CURDIR}/mkopcodec.awk
-   sort -n -b -k 3 opcodes.h | awk -f ${.CURDIR}/mkopcodec.awk >opcodes.c
-
-
-opcodes.h: parse.h ${.CURDIR}/src/vdbe.c \
-${.CURDIR}/mkopcodeh.awk
-   cat parse.h ${.CURDIR}/src/vdbe.c | \
-   awk -f ${.CURDIR}/mkopcodeh.awk >$@
-
-beforedepend: opcodes.h keywordhash.h
-
-keywordhash.h: mkkeywordhash
-   ${.OBJDIR}/mkkeywordhash >$@
-
-mkkeywordhash: tool/mkkeywordhash.c
-   ${HOSTCC} ${LDSTATIC} -o $@ $< ${LDADD}
-
-lemon: tool/lemon.c
-   ${HOSTCC} ${LDSTATIC} -o $@ $< ${LDADD}
-
-parse.c: parse.y lemon
-   ln -sf ${.CURDIR}/src/parse.y
-   ln -sf ${.CURDIR}/src/lempar.c # XXX tweaked parser
-   ${.OBJDIR}/lemon ${FEATURE_FLAGS} parse.y
-   mv parse.h parse.h.temp
-   awk -f ${.CURDIR}/addopcodes.awk parse.h.temp >parse.h
-   rm parse.h.temp
-
-parse.h: parse.c
-
-tokenize.o tokenize.po tokenize.so: keywordhash.h
-
-CLEANFILES += mkkeywordhash opcodes.c opcodes.h keywordhash.h \
-   parse.* lemon lempar.c
-   
 beforeinstall:
${INSTALL} ${INSTALL_COPY} -o root -g ${SHAREGRP} \
-m ${SHAREMODE} ${.CURDIR}/sqlite3.pc ${DESTDIR}/usr/lib/pkgconfig/
 
 includes:
@for i in ${FILES}; do \
-   cmp -s ${.CURDIR}/src/$$i ${DESTDIR}/usr/include/$$i || \
-   ${INSTALL} ${INSTALL_COPY} -m 444 ${.CURDIR}/src/$$i 
${DESTDIR}/usr/include/$$i; \
+   cmp -s 

Re: sqlite3 update

[James Turner]

On Wed, Jun 08, 2016 at 11:33:42PM +0100, Stuart Henderson wrote:
> On 2016/06/03 20:17, James Turner wrote:
> > On Fri, Jun 03, 2016 at 11:24:15PM +0100, Stuart Henderson wrote:
> > > On 2016/06/01 11:22, Stuart Henderson wrote:
> > > > On 2016/06/01 09:09, Landry Breuil wrote:
> > > > > And most importantly, 47 (released next week) requires 3.11.
> > > > > /usr/obj/ports/firefox-47.0beta9/firefox-47.0b9/old-configure:SQLITE_VERSION=3.11.0
> > > > 
> > > > Ok, I'll update the diff this evening.
> > > 
> > > So this was a bit optimistic... 3.11 and 3.13 start making more
> > > changes to the Tcl scripts that replaced the awk scripts and it's
> > > beyond my awk to replicate them. (Also I'm not too sure about the
> > > XXX in Makefile about lempar.c which has changed upstream since
> > > the version where we added it).
> > > 
> > > As I see it there are two options to get us out of the hole.
> > > 
> > > - Pregenerate the files with Tcl and commit them, adapting the
> > > Makefile to use them. This is looking possible for 3.11 though
> > > still a bit of a pain to handle the build system hacks that
> > > will be needed. From a quick look at 3.13 there are more build
> > > system changes and further changes to the Tcl. I for one am
> > > not going to be able to maintain a forked build system with
> > > this many moving pieces.
> > > 
> > > - Switch to the "amalgamation" (4 source files: 3 for the lib,
> > > one for the shell), which is what upstream push people towards
> > > using (and what everybody else that I've seen including
> > > sqlite source uses).
> 
> 
> Does anyone have thoughts about these two as a "for now" option?
> 

I prefer option 2. Switch to the amalgamation with our changes on top.

> 
> > > With this the main file is unfortunately huge but would be easier
> > > by far to update later. Note that we only have a small change to
> > > the actual source code (replacing the RNG code) which is easy enough
> > > to carry across. Most of our changes are to the build infrastructure.
> > > 
> > > As far as Firefox goes, I think sqlite 3.11 should do us until
> > > release, but might be problematic if we want to update in -stable
> > > ports.
> > > 
> > > From what I can tell sqlite upstream *are* careful about ABI.
> > > But it would seem quite a stretch to expect things to work
> > > safely/reliably with two different versions brought into
> > > one address space.
> > > 
> > 
> > So when I talked with Ingo about removing SQLite from base we talked
> > about imported the "amalgamation" into mandoc for local use only and
> > then removing lib/libsqlite3 and usr.bin/sqlite3.
> > 
> > We would then need to updated sqlite3 in ports and update ports to use
> > this instead.
> 
> I think this does make sense. But it's used quite a lot in ports,
> handling a move back to ports is too much work to do before release.
> 

-- 
James Turner

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/03 20:17, James Turner wrote:
> On Fri, Jun 03, 2016 at 11:24:15PM +0100, Stuart Henderson wrote:
> > On 2016/06/01 11:22, Stuart Henderson wrote:
> > > On 2016/06/01 09:09, Landry Breuil wrote:
> > > > And most importantly, 47 (released next week) requires 3.11.
> > > > /usr/obj/ports/firefox-47.0beta9/firefox-47.0b9/old-configure:SQLITE_VERSION=3.11.0
> > > 
> > > Ok, I'll update the diff this evening.
> > 
> > So this was a bit optimistic... 3.11 and 3.13 start making more
> > changes to the Tcl scripts that replaced the awk scripts and it's
> > beyond my awk to replicate them. (Also I'm not too sure about the
> > XXX in Makefile about lempar.c which has changed upstream since
> > the version where we added it).
> > 
> > As I see it there are two options to get us out of the hole.
> > 
> > - Pregenerate the files with Tcl and commit them, adapting the
> > Makefile to use them. This is looking possible for 3.11 though
> > still a bit of a pain to handle the build system hacks that
> > will be needed. From a quick look at 3.13 there are more build
> > system changes and further changes to the Tcl. I for one am
> > not going to be able to maintain a forked build system with
> > this many moving pieces.
> > 
> > - Switch to the "amalgamation" (4 source files: 3 for the lib,
> > one for the shell), which is what upstream push people towards
> > using (and what everybody else that I've seen including
> > sqlite source uses).


Does anyone have thoughts about these two as a "for now" option?


> > With this the main file is unfortunately huge but would be easier
> > by far to update later. Note that we only have a small change to
> > the actual source code (replacing the RNG code) which is easy enough
> > to carry across. Most of our changes are to the build infrastructure.
> > 
> > As far as Firefox goes, I think sqlite 3.11 should do us until
> > release, but might be problematic if we want to update in -stable
> > ports.
> > 
> > From what I can tell sqlite upstream *are* careful about ABI.
> > But it would seem quite a stretch to expect things to work
> > safely/reliably with two different versions brought into
> > one address space.
> > 
> 
> So when I talked with Ingo about removing SQLite from base we talked
> about imported the "amalgamation" into mandoc for local use only and
> then removing lib/libsqlite3 and usr.bin/sqlite3.
> 
> We would then need to updated sqlite3 in ports and update ports to use
> this instead.

I think this does make sense. But it's used quite a lot in ports,
handling a move back to ports is too much work to do before release.

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/03 20:17, James Turner wrote:
> I personally think this is the way to go. Sadly I don't have the time to
> do this right now (We just welcomed our second child recently).

Heh. One doesn't really prepare you for two :-)

Re: sqlite3 update

[James Turner]

On Fri, Jun 03, 2016 at 11:24:15PM +0100, Stuart Henderson wrote:
> On 2016/06/01 11:22, Stuart Henderson wrote:
> > On 2016/06/01 09:09, Landry Breuil wrote:
> > > And most importantly, 47 (released next week) requires 3.11.
> > > /usr/obj/ports/firefox-47.0beta9/firefox-47.0b9/old-configure:SQLITE_VERSION=3.11.0
> > 
> > Ok, I'll update the diff this evening.
> 
> So this was a bit optimistic... 3.11 and 3.13 start making more
> changes to the Tcl scripts that replaced the awk scripts and it's
> beyond my awk to replicate them. (Also I'm not too sure about the
> XXX in Makefile about lempar.c which has changed upstream since
> the version where we added it).
> 
> As I see it there are two options to get us out of the hole.
> 
> - Pregenerate the files with Tcl and commit them, adapting the
> Makefile to use them. This is looking possible for 3.11 though
> still a bit of a pain to handle the build system hacks that
> will be needed. From a quick look at 3.13 there are more build
> system changes and further changes to the Tcl. I for one am
> not going to be able to maintain a forked build system with
> this many moving pieces.
> 
> - Switch to the "amalgamation" (4 source files: 3 for the lib,
> one for the shell), which is what upstream push people towards
> using (and what everybody else that I've seen including
> sqlite source uses).
> 
> With this the main file is unfortunately huge but would be easier
> by far to update later. Note that we only have a small change to
> the actual source code (replacing the RNG code) which is easy enough
> to carry across. Most of our changes are to the build infrastructure.
> 
> As far as Firefox goes, I think sqlite 3.11 should do us until
> release, but might be problematic if we want to update in -stable
> ports.
> 
> From what I can tell sqlite upstream *are* careful about ABI.
> But it would seem quite a stretch to expect things to work
> safely/reliably with two different versions brought into
> one address space.
> 

So when I talked with Ingo about removing SQLite from base we talked
about imported the "amalgamation" into mandoc for local use only and
then removing lib/libsqlite3 and usr.bin/sqlite3.

We would then need to updated sqlite3 in ports and update ports to use
this instead.

I personally think this is the way to go. Sadly I don't have the time to
do this right now (We just welcomed our second child recently).

-- 
James Turner

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/01 11:22, Stuart Henderson wrote:
> On 2016/06/01 09:09, Landry Breuil wrote:
> > And most importantly, 47 (released next week) requires 3.11.
> > /usr/obj/ports/firefox-47.0beta9/firefox-47.0b9/old-configure:SQLITE_VERSION=3.11.0
> 
> Ok, I'll update the diff this evening.

So this was a bit optimistic... 3.11 and 3.13 start making more
changes to the Tcl scripts that replaced the awk scripts and it's
beyond my awk to replicate them. (Also I'm not too sure about the
XXX in Makefile about lempar.c which has changed upstream since
the version where we added it).

As I see it there are two options to get us out of the hole.

- Pregenerate the files with Tcl and commit them, adapting the
Makefile to use them. This is looking possible for 3.11 though
still a bit of a pain to handle the build system hacks that
will be needed. From a quick look at 3.13 there are more build
system changes and further changes to the Tcl. I for one am
not going to be able to maintain a forked build system with
this many moving pieces.

- Switch to the "amalgamation" (4 source files: 3 for the lib,
one for the shell), which is what upstream push people towards
using (and what everybody else that I've seen including
sqlite source uses).

With this the main file is unfortunately huge but would be easier
by far to update later. Note that we only have a small change to
the actual source code (replacing the RNG code) which is easy enough
to carry across. Most of our changes are to the build infrastructure.

As far as Firefox goes, I think sqlite 3.11 should do us until
release, but might be problematic if we want to update in -stable
ports.

>From what I can tell sqlite upstream *are* careful about ABI.
But it would seem quite a stretch to expect things to work
safely/reliably with two different versions brought into
one address space.

Re: sqlite3 update

[Mark Kettenis]

> Date: Wed, 1 Jun 2016 07:44:23 +0200
> From: Landry Breuil 
> 
> > (I don't really understand how things are working in Firefox on
> > other OS, there are two different versions of sqlite in their
> > tree...even in the one firefox tarball)..
> 
> Nss is considered a third-party library in firefox, and thus has to be
> self-contained. We dont build the copy of sqlite bundled in nss bundled
> in firefox since we build against systemwide nss...

The 46.0.1 firefox binary tarball I just downloaded has libsoftkn3.so
linked against libmozsqlite3.so, i.e. the bundled libsqlite3.
However, Firefox 46.0 as installed on a OpenSuSE Linux system has a
libsoftkn3.so that is linked against libsqlite3.so, i.e. the system
libsqlite3.  The libxul.so in /usr/lib64/firefox on that system does
use libmozsqlite3.so though.

On that OpenSUSE system, the same thing happens as on OpenBSD.  Both
libsqlite3.so and libmozsqlite3.so get loaded.  And on that system the
sizes of sqlite3_version don't match, just as on OpenBSD.  It seems
the linux dynamic linker simply doesn't warn about this.  I vaguely
remmeber that the they only warn if the symbol is bigger in the 2nd
library that gets loaded.  But since libmozsqlite3.so gets loaded
first and has the longest version string ("3.10.2" vs. "3.8.6") the
warning doesn't trigger.

In the end we're just witnessing a broken software development process
on the side of Mozilla.  Perhaps landry can point this out to them.
But they'll probably not see this as a serious problem.  As long as
the sqlite devlopers don't break the ABI, they'll get away with this.
We probably just have to live with this.  As soon as the system
libsqlite3.so gets updated to 3.xx.y, the warning will disappear
again.  In that sense sthen@'s update to 3.10.2 will "fix" the
problem.

Given the current Mozilla development process, I don't think it makes
sense to move back to using the system libsqlite3.so for Mozilla.  It
seems the days of libsqlite3.so in base are numbered.  Perhaps at that
point you can revisit the decision to use the bundled
libmozsqlite3.so.

Re: sqlite3 update

[Landry Breuil]

On Wed, Jun 01, 2016 at 11:08:36AM +0100, Stuart Henderson wrote:
> On 2016/06/01 07:44, Landry Breuil wrote:
> > On Tue, May 31, 2016 at 11:36:35PM +0100, Stuart Henderson wrote:
> > > Thanks for the pointer Ingo..
> > > 
> > > I'll be doing tests (including a bulk ports build) with this,
> > > but here's an update to sqlite 3.10.2 (the version which is
> > > in Firefox 46) in the hope it will let us get rid of the
> > > conflicting sqlite3 versions in firefox.
> > 
> > Back to square one, when i need to pester someone to update sqlite3 in
> > base each time mozilla updates its copy/dependencies ? :)
> 
> Or do it yourself :)

I've done that one for 3.7.17, and.. no thanks. Given that nobody gives
a crap, it's impossible to get okays/testing.

> > Firefox trunk/49 ships 3.13... see
> > http://hg.mozilla.org/mozilla-central/log/tip/db/sqlite3/src/sqlite3.h
> > for the history of updates.
> 
> What's the timeframe for that?

https://wiki.mozilla.org/RapidRelease/Calendar

47 next week
48 beginning of august
49 mid-september

> > > If anyone has anything in particular they'd like to test,
> > > could you try it please.
> > 
> > To test it with firefox, you should just need to comment out
> > MOZILLA_USE_BUNDLED_SQLITE = Yes in www/mozilla-firefox/Makefile.
> > 
> > > A shlib_version update is included, four symbols were removed so
> > > it's a major bump.
> > > 
> > > I modified addopcodes.awk by hand based on the changes in the
> > > upstream version which has now switched to a Tcl script.
> > > 
> > > (I don't really understand how things are working in Firefox on
> > > other OS, there are two different versions of sqlite in their
> > > tree...even in the one firefox tarball)..
> > 
> > Nss is considered a third-party library in firefox, and thus has to be
> > self-contained. We dont build the copy of sqlite bundled in nss bundled
> > in firefox since we build against systemwide nss...
> 
> I tried building nss with the included one but that still pulled
> in the base one..

With NSS_USE_SYSTEM_SQLITE=1 removed from MAKE_ENV ?

Landry

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/01 09:09, Landry Breuil wrote:
> And most importantly, 47 (released next week) requires 3.11.
> /usr/obj/ports/firefox-47.0beta9/firefox-47.0b9/old-configure:SQLITE_VERSION=3.11.0

Ok, I'll update the diff this evening.

Re: sqlite3 update

[Stuart Henderson]

On 2016/06/01 07:44, Landry Breuil wrote:
> On Tue, May 31, 2016 at 11:36:35PM +0100, Stuart Henderson wrote:
> > Thanks for the pointer Ingo..
> > 
> > I'll be doing tests (including a bulk ports build) with this,
> > but here's an update to sqlite 3.10.2 (the version which is
> > in Firefox 46) in the hope it will let us get rid of the
> > conflicting sqlite3 versions in firefox.
> 
> Back to square one, when i need to pester someone to update sqlite3 in
> base each time mozilla updates its copy/dependencies ? :)

Or do it yourself :)

The current thing with conflicting versions is definitely wrong, so if
you have another way to fix it, that would be fine too.

> Firefox trunk/49 ships 3.13... see
> http://hg.mozilla.org/mozilla-central/log/tip/db/sqlite3/src/sqlite3.h
> for the history of updates.

What's the timeframe for that?

> > If anyone has anything in particular they'd like to test,
> > could you try it please.
> 
> To test it with firefox, you should just need to comment out
> MOZILLA_USE_BUNDLED_SQLITE = Yes in www/mozilla-firefox/Makefile.
> 
> > A shlib_version update is included, four symbols were removed so
> > it's a major bump.
> > 
> > I modified addopcodes.awk by hand based on the changes in the
> > upstream version which has now switched to a Tcl script.
> > 
> > (I don't really understand how things are working in Firefox on
> > other OS, there are two different versions of sqlite in their
> > tree...even in the one firefox tarball)..
> 
> Nss is considered a third-party library in firefox, and thus has to be
> self-contained. We dont build the copy of sqlite bundled in nss bundled
> in firefox since we build against systemwide nss...

I tried building nss with the included one but that still pulled
in the base one..

Re: sqlite3 update

[Landry Breuil]

On Wed, Jun 01, 2016 at 07:44:23AM +0200, Landry Breuil wrote:
> On Tue, May 31, 2016 at 11:36:35PM +0100, Stuart Henderson wrote:
> > Thanks for the pointer Ingo..
> > 
> > I'll be doing tests (including a bulk ports build) with this,
> > but here's an update to sqlite 3.10.2 (the version which is
> > in Firefox 46) in the hope it will let us get rid of the
> > conflicting sqlite3 versions in firefox.
> 
> Back to square one, when i need to pester someone to update sqlite3 in
> base each time mozilla updates its copy/dependencies ? :)
> 
> Technically, i could also relax the version check in configure, but i'm
> never sure when firefox really depends on features from newer sqlite. It
> seems development is closely followed for valid reasons...
> 
> > If this is the wrong version to be targetting (Landry?) please
> > let me know.
> 
> Firefox trunk/49 ships 3.13... see
> http://hg.mozilla.org/mozilla-central/log/tip/db/sqlite3/src/sqlite3.h
> for the history of updates.

And most importantly, 47 (released next week) requires 3.11.

/usr/obj/ports/firefox-47.0beta9/firefox-47.0b9/old-configure:SQLITE_VERSION=3.11.0

Landry

Re: sqlite3 update

[Landry Breuil]

On Tue, May 31, 2016 at 11:36:35PM +0100, Stuart Henderson wrote:
> Thanks for the pointer Ingo..
> 
> I'll be doing tests (including a bulk ports build) with this,
> but here's an update to sqlite 3.10.2 (the version which is
> in Firefox 46) in the hope it will let us get rid of the
> conflicting sqlite3 versions in firefox.

Back to square one, when i need to pester someone to update sqlite3 in
base each time mozilla updates its copy/dependencies ? :)

Technically, i could also relax the version check in configure, but i'm
never sure when firefox really depends on features from newer sqlite. It
seems development is closely followed for valid reasons...

> If this is the wrong version to be targetting (Landry?) please
> let me know.

Firefox trunk/49 ships 3.13... see
http://hg.mozilla.org/mozilla-central/log/tip/db/sqlite3/src/sqlite3.h
for the history of updates.

> If anyone has anything in particular they'd like to test,
> could you try it please.

To test it with firefox, you should just need to comment out
MOZILLA_USE_BUNDLED_SQLITE = Yes in www/mozilla-firefox/Makefile.

> A shlib_version update is included, four symbols were removed so
> it's a major bump.
> 
> I modified addopcodes.awk by hand based on the changes in the
> upstream version which has now switched to a Tcl script.
> 
> (I don't really understand how things are working in Firefox on
> other OS, there are two different versions of sqlite in their
> tree...even in the one firefox tarball)..

Nss is considered a third-party library in firefox, and thus has to be
self-contained. We dont build the copy of sqlite bundled in nss bundled
in firefox since we build against systemwide nss...

Landry

Re: sqlite3 update

[James Turner]

On Tue, May 31, 2016 at 11:36:35PM +0100, Stuart Henderson wrote:
> Thanks for the pointer Ingo..
> 
> I'll be doing tests (including a bulk ports build) with this,
> but here's an update to sqlite 3.10.2 (the version which is
> in Firefox 46) in the hope it will let us get rid of the
> conflicting sqlite3 versions in firefox.
> 
> If this is the wrong version to be targetting (Landry?) please
> let me know.
> 
> If anyone has anything in particular they'd like to test,
> could you try it please.
> 
> A shlib_version update is included, four symbols were removed so
> it's a major bump.
> 
> I modified addopcodes.awk by hand based on the changes in the
> upstream version which has now switched to a Tcl script.
> 
> (I don't really understand how things are working in Firefox on
> other OS, there are two different versions of sqlite in their

Stuart,

This looks good. I've tested with mandoc and fossil in ports. ok
jturner.

-- 
James Turner