Re: [sqlite] found a Malware , when download sqlite-shell-win32-x86-3071500.zip

2012-12-18 Thread Miles Liu 
oh,sorry,  i hate 360safe.
在 2012-12-18 下午3:44，"Roy Tam" 写道：

> Hello,
>
> 2012/12/18 Miles Liu :
> > sqlite-shell-win32-x86-3071500
> > when download it , anti-virus said found a Malware.
>
> False positive. VirusTotal report here:
>
> https://www.virustotal.com/file/85fa29a391d7ab8af4736dfd36222a651a9b3af56c5f81bb58c9b106eaa63533/analysis/1355816443/
>
> HTH,
> Roy
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] found a Malware , when download sqlite-shell-win32-x86-3071500.zip

2012-12-17 Thread Roy Tam 
Hello,

2012/12/18 Miles Liu :
> sqlite-shell-win32-x86-3071500
> when download it , anti-virus said found a Malware.

False positive. VirusTotal report here:
https://www.virustotal.com/file/85fa29a391d7ab8af4736dfd36222a651a9b3af56c5f81bb58c9b106eaa63533/analysis/1355816443/

HTH,
Roy
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-26 Thread Duquette, William H (318K) 
On 1/26/12 9:36 AM, "Stephan Beal"  wrote:


>On Thu, Jan 26, 2012 at 2:53 PM, Mohit Sindhwani  wrote:
>
>> Absolutely!  I come home from work and tune in to this thread, gripped
>>:)
>>
>
>+1 to Mohit and the others who's written similar responses. i rarely
>follow
>threads which don't directly affect me, but John Elrick's determination
>and
>stamina have been a treat to watch. IMO he should be made an honorary
>member of the dev team (or at least get a coffee mug).

+1

>
>-- 
>- stephan beal
>http://wanderinghorse.net/home/stephan/
>http://gplus.to/sgbeal
>___
>sqlite-users mailing list
>sqlite-users@sqlite.org
>http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-26 Thread John Elrick 
On Thu, Jan 26, 2012 at 12:36 PM, Stephan Beal wrote:

> On Thu, Jan 26, 2012 at 2:53 PM, Mohit Sindhwani  wrote:
>
> > Absolutely!  I come home from work and tune in to this thread, gripped :)
> >
>
> +1 to Mohit and the others who's written similar responses. i rarely follow
> threads which don't directly affect me, but John Elrick's determination and
> stamina have been a treat to watch. IMO he should be made an honorary
> member of the dev team (or at least get a coffee mug).
>
>
Thank you.  You made my day.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-26 Thread Stephan Beal 
On Thu, Jan 26, 2012 at 2:53 PM, Mohit Sindhwani  wrote:

> Absolutely!  I come home from work and tune in to this thread, gripped :)
>

+1 to Mohit and the others who's written similar responses. i rarely follow
threads which don't directly affect me, but John Elrick's determination and
stamina have been a treat to watch. IMO he should be made an honorary
member of the dev team (or at least get a coffee mug).

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-26 Thread John Elrick 
On Thu, Jan 26, 2012 at 3:29 AM, Jean-Denis MUYS wrote:

>
> On 25 janv. 2012, at 22:30, John Elrick wrote:
>
> >
> > I have to say this has been a fascinating learning experience.  It has
> been
> > enlightening reading the SQLite code and beginning to understand its
> inner
> > workings.
> > John
> > ___
>
> What a thriller! I was holding my breath. Did you think about selling the
> rights to this story to a movie studio?


Steven Spielberg just bought the rights.  He outbid James Cameron by a
million.  Should be in theaters Summer 2014.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-26 Thread Mohit Sindhwani 

On 26/1/2012 5:21 AM, Nico Williams wrote:

On Wed, Jan 25, 2012 at 3:07 PM, Richard Hipp  wrote:

The SQLite byte-code engine was being too conservative and was reparsing
the schema in places where it was not strictly necessary.  The fix was to
restrict the places where the schema was reparsed to situations that really
needed it.

This thread has been awesome, gripping reading.  Thanks to both of you
for sticking with it and figuring it out.



Absolutely!  I come home from work and tune in to this thread, gripped :)

Thanks,
Mohit.
26/1/2012 | 9:53 PM.



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-26 Thread Jean-Denis MUYS 

On 25 janv. 2012, at 22:30, John Elrick wrote:

> 
> I have to say this has been a fascinating learning experience.  It has been
> enlightening reading the SQLite code and beginning to understand its inner
> workings. 
> John
> ___

What a thriller! I was holding my breath. Did you think about selling the 
rights to this story to a movie studio?

Jean-Denis

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 4:21 PM, Nico Williams wrote:

> On Wed, Jan 25, 2012 at 3:07 PM, Richard Hipp  wrote:
> > The SQLite byte-code engine was being too conservative and was reparsing
> > the schema in places where it was not strictly necessary.  The fix was to
> > restrict the places where the schema was reparsed to situations that
> really
> > needed it.
>
> This thread has been awesome, gripping reading.  Thanks to both of you
> for sticking with it and figuring it out.
>
>

Thank God we're...

a) early with this release cycle
b) in maintenance mode
c) working for a client with deep pockets who recognizes the need for a
solid code base

or I would have been an utter wreck by now.

I have to say this has been a fascinating learning experience.  It has been
enlightening reading the SQLite code and beginning to understand its inner
workings.  I feel significantly more confident in the solidity of our code
base and am very grateful that we chose SQLite as our database years ago.
 I doubt I would have had this much cooperation from the Chief Scientist
with any other product.

John
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 4:07 PM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 4:01 PM, John Elrick  >wrote:
>
> > Much better.  I wasn't sure which version to incorporate the changes into
> > (the link is to the source), so I put them into 3.7.9.
> >
> > The total mallocs have dropped to 1.5 million -- twice as high as 3.7.5
> but
> > in line.  I'll have to run some additional performance testing but
> visually
> > the system appears to be a lot closer to expected.
> >
> > What was going on? (My client will want a report on the resolution).
> >
>
> The SQLite byte-code engine was being too conservative and was reparsing
> the schema in places where it was not strictly necessary.  The fix was to
> restrict the places where the schema was reparsed to situations that really
> needed it.
>
> I cannot give more detail without seeing your application, since I still
> don't know exactly what you are doing to trigger the excess reparsing.  The
> patch above comes from a script of my own that I finally managed to put
> together that caused unnecessary schema reparsing.  It took a specific
> combination of events:  A schema change inside of a transaction followed by
> DML statements that had been prepared prior to the schema change.
>
>
My brain missed that.   The sequence I described should have been:

prepare query1
prepare query2
begin transaction
create something -> schema change
query1.step
  - schema invalid so recreate
  - recreation of schema changes the generation
query2.step
  - schema invalid so recreate
  - recreation of schema changes the generation
query1.step
  - schema invalid so recreate
  - recreation of schema changes the generation
query2.step
  - schema invalid so recreate
  - recreation of schema changes the generation
end transaction

...which I believe describes your scenario perfectly.  Odd that the script
didn't replicate it, since it should have had the nested transactions as
well.

Thank you very much for your assistance Richard.  I sincerely appreciate it.

Thanks to everyone else who helped guide me on this investigation.


John
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Nico Williams 
On Wed, Jan 25, 2012 at 3:07 PM, Richard Hipp  wrote:
> The SQLite byte-code engine was being too conservative and was reparsing
> the schema in places where it was not strictly necessary.  The fix was to
> restrict the places where the schema was reparsed to situations that really
> needed it.

This thread has been awesome, gripping reading.  Thanks to both of you
for sticking with it and figuring it out.

Nico
--
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
Thanks.

On Wed, Jan 25, 2012 at 4:11 PM, Petite Abeille wrote:

>
> On Jan 25, 2012, at 10:09 PM, John Elrick wrote:
>
> > DML?
>
> http://en.wikipedia.org/wiki/Data_Manipulation_Language
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
John Elrick
Fenestra Technologies
540-868-1377
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Petite Abeille 

On Jan 25, 2012, at 10:09 PM, John Elrick wrote:

> DML?

http://en.wikipedia.org/wiki/Data_Manipulation_Language


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 4:07 PM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 4:01 PM, John Elrick  >wrote:
>
> > Much better.  I wasn't sure which version to incorporate the changes into
> > (the link is to the source), so I put them into 3.7.9.
> >
> > The total mallocs have dropped to 1.5 million -- twice as high as 3.7.5
> but
> > in line.  I'll have to run some additional performance testing but
> visually
> > the system appears to be a lot closer to expected.
> >
> > What was going on? (My client will want a report on the resolution).
> >
>
> The SQLite byte-code engine was being too conservative and was reparsing
> the schema in places where it was not strictly necessary.  The fix was to
> restrict the places where the schema was reparsed to situations that really
> needed it.
>
> I cannot give more detail without seeing your application, since I still
> don't know exactly what you are doing to trigger the excess reparsing.  The
> patch above comes from a script of my own that I finally managed to put
> together that caused unnecessary schema reparsing.  It took a specific
> combination of events:  A schema change inside of a transaction followed by
> DML statements that had been prepared prior to the schema change.
>
>
DML?  I'm not familiar with that term.  The rest of the description sounds
very familiar.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Richard Hipp 
On Wed, Jan 25, 2012 at 4:01 PM, John Elrick wrote:

> Much better.  I wasn't sure which version to incorporate the changes into
> (the link is to the source), so I put them into 3.7.9.
>
> The total mallocs have dropped to 1.5 million -- twice as high as 3.7.5 but
> in line.  I'll have to run some additional performance testing but visually
> the system appears to be a lot closer to expected.
>
> What was going on? (My client will want a report on the resolution).
>

The SQLite byte-code engine was being too conservative and was reparsing
the schema in places where it was not strictly necessary.  The fix was to
restrict the places where the schema was reparsed to situations that really
needed it.

I cannot give more detail without seeing your application, since I still
don't know exactly what you are doing to trigger the excess reparsing.  The
patch above comes from a script of my own that I finally managed to put
together that caused unnecessary schema reparsing.  It took a specific
combination of events:  A schema change inside of a transaction followed by
DML statements that had been prepared prior to the schema change.


>
> On Wed, Jan 25, 2012 at 3:44 PM, Richard Hipp  wrote:
>
> > On Wed, Jan 25, 2012 at 3:39 PM, Richard Hipp  wrote:
> >
> > > I'm testing a candidate fix for your problem now.  I'll send you a link
> > > once the tests finish (assuming they all work).
> >
> >
> > Please try http://www.sqlite.org/src/info/11f68d997d and let me know if
> it
> > solves your problem.  Thanks.
> >
> > --
> > D. Richard Hipp
> > d...@sqlite.org
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
>
> --
> John Elrick
> Fenestra Technologies
> 540-868-1377
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
Much better.  I wasn't sure which version to incorporate the changes into
(the link is to the source), so I put them into 3.7.9.

The total mallocs have dropped to 1.5 million -- twice as high as 3.7.5 but
in line.  I'll have to run some additional performance testing but visually
the system appears to be a lot closer to expected.

What was going on? (My client will want a report on the resolution).

On Wed, Jan 25, 2012 at 3:44 PM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 3:39 PM, Richard Hipp  wrote:
>
> > I'm testing a candidate fix for your problem now.  I'll send you a link
> > once the tests finish (assuming they all work).
>
>
> Please try http://www.sqlite.org/src/info/11f68d997d and let me know if it
> solves your problem.  Thanks.
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
John Elrick
Fenestra Technologies
540-868-1377
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Richard Hipp 
On Wed, Jan 25, 2012 at 3:39 PM, Richard Hipp  wrote:

> I'm testing a candidate fix for your problem now.  I'll send you a link
> once the tests finish (assuming they all work).


Please try http://www.sqlite.org/src/info/11f68d997d and let me know if it
solves your problem.  Thanks.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
Richard,

I've found a problem that I don't understand.  Looking at the case
OP_VerifyCookie, I attempted to output pOp->opcode, which I expected to
equal OP_VerifyCookie.  However, it doesn't:

 Schema has changed current state = 'db=0025, pOp->opCode=66785672,
db->aDb[pOp->p1].pSchema=02110AB0,
db->aDb[pOp->p1].pSchema->schema_cookie=27,
db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
pOp->p1=1, pOp->p2=27, pOp->p3=0'

I don't see anything which should be modifying pOp.  Any thoughts?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
Richard,

Is it reasonable to say that, assuming:

1.  The schema has not changed
2.  There is only one connection to the database

that the value of iGeneration should always equal pOp->p3 in
OP_VerifyCookie?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 1:13 PM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 10:18 AM, John Elrick  >wrote:
>
> >
> > I do have some more information for you.  I tracking everything which
> could
> > call reset and discovered that all of these are stemming from
> >
> > SQLITE_PRIVATE int sqlite3VdbeHalt(Vdbe *p){
> > 
> >/* Rollback or commit any schema changes that occurred. */
> >if( p->rc!=SQLITE_OK && db->flags_InternChanges ){
> > fenestra_sqlite3_dump_stack_trace("rollback in vdbeHalt");
> >  sqlite3ResetInternalSchema(db, -1);
> >  db->flags = (db->flags | SQLITE_InternChanges);
> >}
> >
> > What is the value of p->rc when these resets occur? Do I understand
> correctly that your application is making a bunch of schema changes, then
> doing other operations, at least one of which fails?
>
>
>
'p->rc=17, db->flags=48100a00, db->flags_InternChanges=200'

"A bunch" is a bit nebulous but essentially yes.  Your description looks
accurate.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Richard Hipp 
On Wed, Jan 25, 2012 at 10:18 AM, John Elrick wrote:

>
> I do have some more information for you.  I tracking everything which could
> call reset and discovered that all of these are stemming from
>
> SQLITE_PRIVATE int sqlite3VdbeHalt(Vdbe *p){
> 
>/* Rollback or commit any schema changes that occurred. */
>if( p->rc!=SQLITE_OK && db->flags_InternChanges ){
> fenestra_sqlite3_dump_stack_trace("rollback in vdbeHalt");
>  sqlite3ResetInternalSchema(db, -1);
>  db->flags = (db->flags | SQLITE_InternChanges);
>}
>
> What is the value of p->rc when these resets occur? Do I understand
correctly that your application is making a bunch of schema changes, then
doing other operations, at least one of which fails?


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
Addendum

On Wed, Jan 25, 2012 at 1:12 PM, John Elrick wrote:

> On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp  wrote:
>
>> On Wed, Jan 25, 2012 at 9:02 AM, John Elrick > >wrote:
>>
>> >
>> > current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
>> > db->aDb[pOp->p1].pSchema->schema_cookie=27,
>> > db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
>> > pOp->p1=1, pOp->p2=27, pOp->p3=0'
>> >
>> > sql = 'update alerter_linksset is_dirty = null '
>> >
>> > Obviously, iGeneration and p3 are out of sync.  So, what exactly is
>> pOp->p3
>> > ?
>> >
>>
>> pOp->p3 is suppose to be the value of iGeneration when the statement was
>> first prepared.  It is set here:
>> http://www.sqlite.org/src/artifact/8e2a4dedad?ln=160-162
>>
>> I'm a little concerned that pOp->p3 is zero when iGeneration is 63.
>>  That's
>> a lot of changes.  When was the statement first prepared?  When the
>> program
>> first starts up?  Or is pOp->p3 getting zeroed somehow?
>
>
> I didn't think of this until now.  The test run starts with no database,
> which means the entire create script must be run.  It is physically
> impossible for pOp->p3 to be legitimately set to a generation of 0.
>
> Something must be zeroing pOp->p3
>


Physically impossible for a SELECT, UPDATE or INSERT prepared statement to
be legitimately set to a generation of 0.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 9:02 AM, John Elrick  >wrote:
>
> >
> > current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
> > db->aDb[pOp->p1].pSchema->schema_cookie=27,
> > db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
> > pOp->p1=1, pOp->p2=27, pOp->p3=0'
> >
> > sql = 'update alerter_linksset is_dirty = null '
> >
> > Obviously, iGeneration and p3 are out of sync.  So, what exactly is
> pOp->p3
> > ?
> >
>
> pOp->p3 is suppose to be the value of iGeneration when the statement was
> first prepared.  It is set here:
> http://www.sqlite.org/src/artifact/8e2a4dedad?ln=160-162
>
> I'm a little concerned that pOp->p3 is zero when iGeneration is 63.  That's
> a lot of changes.  When was the statement first prepared?  When the program
> first starts up?  Or is pOp->p3 getting zeroed somehow?


I didn't think of this until now.  The test run starts with no database,
which means the entire create script must be run.  It is physically
impossible for pOp->p3 to be legitimately set to a generation of 0.

Something must be zeroing pOp->p3
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
Ok.  Possible scenario.  I can't prove it, but somebody tell me if this is
what could be going on:

prepare query1
prepare query2
create something -> schema change
query1.step
  - schema invalid so recreate
  - recreation of schema changes the generation
query2.step
  - schema invalid so recreate
  - recreation of schema changes the generation
query1.step
  - schema invalid so recreate
  - recreation of schema changes the generation
query2.step
  - schema invalid so recreate
  - recreation of schema changes the generation

???
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 10:32 AM, Simon Slavin  wrote:

>
> On 25 Jan 2012, at 3:18pm, John Elrick wrote:
>
> > So, it appears that something is making SQLite think the schema has
> changed.
>
> Can you read the following:
>
> 
>
> If you think this corresponds to what you are seeing in the source code,
> can you log the value returned by
>
> PRAGMA schema_version
>
> at various points ?  Does the value returned by the PRAGMA agree with what
> you see in
>
> db->flags_InternChanges
>
> ?  Perhaps you can find that the PRAGMA value is changing when you see no
> reason for it changing.
>
>
The pragma becomes 63 at the very start of the tracked run and never
changes.  However, vdbeHalt continually reports that the schema has altered.

schema from 56 to 57 changed on = 'create table preferences ( machine_user
varchar(80), section text, value text, primary key(machine_user, section))'
schema from 61 to 62 changed on = 'create table if not exists
passthrough_attributes (Passthrough_attribute_oid integer,
manifest_identifier varchar, element_name varchar, attribute_name varchar,
value varchar)'
schema from 62 to 63 changed on = 'create trigger if not exists
manifest_ad2 after delete on manifests begin delete from
passthrough_attributes where manifest_identifier = old.manifest_oid; end;'
Schema has changed current state = 'db=03FB1188,
db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=0, u.av.iMeta=63, u.av.iGen=0,
pOp->p1=0, pOp->p2=61, pOp->p3=0'
Schema has changed sql = 'INSERT INTO TABSHEETS VALUES (null,
:workbookType, :manifestOid, :surveyOid, :formDefinitionId, :prefKey,
:name, :originalName, :sequence, :columnList)'
vdbeHalt rollback or commit schema changes = 'p->rc=17, db->flags=48100a00,
db->flags_InternChanges=200'
Schema has changed current state = 'db=03FB1188,
db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=1, u.av.iMeta=63, u.av.iGen=1,
pOp->p1=0, pOp->p2=61, pOp->p3=0'
Schema has changed sql = 'select distinct
FORM_DEFINITIONS.FORM_DEFINITION_OID from FORM_DEFINITIONS where
FORM_DEFINITIONS.guid=:FORM_DEFINITIONS_guid order by
FORM_DEFINITIONS.rowid '
vdbeHalt rollback or commit schema changes = 'p->rc=17, db->flags=48100a00,
db->flags_InternChanges=200'
Schema has changed current state = 'db=03FB1188,
db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=2, u.av.iMeta=63, u.av.iGen=2,
pOp->p1=0, pOp->p2=63, pOp->p3=0'
Schema has changed sql = 'select distinct
FORM_DEFINITIONS.FORM_DEFINITION_OID from FORM_DEFINITIONS, SURVEYS where
SURVEYS.SURVEY_OID=:SURVEYS_SURVEY_OID and
FORM_DEFINITIONS.FORM_DEFINITION_OID=:FORM_DEFINITIONS_FORM_DEFINITION_OID
and FORM_DEFINITIONS.definition_parent=SURVEYS.SURVEY_OID order by
FORM_DEFINITIONS.rowid '
vdbeHalt rollback or commit schema changes = 'p->rc=17, db->flags=48100a00,
db->flags_InternChanges=200'
Schema has changed current state = 'db=03FB1188,
db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=3, u.av.iMeta=63, u.av.iGen=3,
pOp->p1=0, pOp->p2=61, pOp->p3=0'
Schema has changed sql = 'insert   into background_constraints values (
:sequence_instance_oid , :sequence_definition_oid ) '
vdbeHalt rollback or commit schema changes = 'p->rc=17, db->flags=48100a00,
db->flags_InternChanges=200'
Schema has changed current state = 'db=03FB1188,
db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=4, u.av.iMeta=63, u.av.iGen=4,
pOp->p1=0, pOp->p2=63, pOp->p3=2'
Schema has changed sql = 'select distinct
FORM_DEFINITIONS.FORM_DEFINITION_OID from FORM_DEFINITIONS where
FORM_DEFINITIONS.guid=:FORM_DEFINITIONS_guid order by
FORM_DEFINITIONS.rowid '
vdbeHalt rollback or commit schema changes = 'p->rc=17, db->flags=48100a00,
db->flags_InternChanges=200'
...

There are 9,775 more messages like these in the queue.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Simon Slavin 

On 25 Jan 2012, at 3:18pm, John Elrick wrote:

> So, it appears that something is making SQLite think the schema has changed.

Can you read the following:



If you think this corresponds to what you are seeing in the source code, can 
you log the value returned by

PRAGMA schema_version

at various points ?  Does the value returned by the PRAGMA agree with what you 
see in

db->flags_InternChanges

?  Perhaps you can find that the PRAGMA value is changing when you see no 
reason for it changing.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 9:50 AM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 9:02 AM, John Elrick  >wrote:
>
> >
> > current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
> > db->aDb[pOp->p1].pSchema->schema_cookie=27,
> > db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
> > pOp->p1=1, pOp->p2=27, pOp->p3=0'
> >
> > sql = 'update alerter_linksset is_dirty = null '
> >
> > Obviously, iGeneration and p3 are out of sync.  So, what exactly is
> pOp->p3
> > ?
> >
>
> pOp->p3 is suppose to be the value of iGeneration when the statement was
> first prepared.  It is set here:
> http://www.sqlite.org/src/artifact/8e2a4dedad?ln=160-162
>
> I'm a little concerned that pOp->p3 is zero when iGeneration is 63.  That's
> a lot of changes.  When was the statement first prepared?  When the program
> first starts up?  Or is pOp->p3 getting zeroed somehow?


As much as possible, we are using lazy initialization.  A query is prepared
the first time it is used, but the prepared query should be retained until
shutdown.


I do have some more information for you.  I tracking everything which could
call reset and discovered that all of these are stemming from

SQLITE_PRIVATE int sqlite3VdbeHalt(Vdbe *p){

/* Rollback or commit any schema changes that occurred. */
if( p->rc!=SQLITE_OK && db->flags_InternChanges ){
fenestra_sqlite3_dump_stack_trace("rollback in vdbeHalt");
  sqlite3ResetInternalSchema(db, -1);
  db->flags = (db->flags | SQLITE_InternChanges);
}

I added a logging item and retrieved the following data:

status = 'p->rc=17, db->flags=48100a00, SQLITE_InternChanges=200'

So, it appears that something is making SQLite think the schema has changed.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Richard Hipp 
On Wed, Jan 25, 2012 at 9:02 AM, John Elrick wrote:

>
> current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
> db->aDb[pOp->p1].pSchema->schema_cookie=27,
> db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
> pOp->p1=1, pOp->p2=27, pOp->p3=0'
>
> sql = 'update alerter_linksset is_dirty = null '
>
> Obviously, iGeneration and p3 are out of sync.  So, what exactly is pOp->p3
> ?
>

pOp->p3 is suppose to be the value of iGeneration when the statement was
first prepared.  It is set here:
http://www.sqlite.org/src/artifact/8e2a4dedad?ln=160-162

I'm a little concerned that pOp->p3 is zero when iGeneration is 63.  That's
a lot of changes.  When was the statement first prepared?  When the program
first starts up?  Or is pOp->p3 getting zeroed somehow?

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
One more bit of interesting information.

if( db->aDb[pOp->p1].pSchema->schema_cookie!=u.av.iMeta ){
fenestra_sqlite3_dump_stack_trace("OP_VerifyCookie");
  sqlite3ResetInternalSchema(db, pOp->p1);
}

The call to fenestra_sqlite3_dump_stack_trace is never invoked.  Whatever
is resetting the internal schema isn't coming from here.

On Wed, Jan 25, 2012 at 9:02 AM, John Elrick wrote:

>
> On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp  wrote:
>
>> On Tue, Jan 24, 2012 at 7:37 PM, John Elrick > >wrote:
>>
>> >
>> > It appears that the value is set here:
>> >
>> >  u.av.pBt = db->aDb[pOp->p1].pBt;
>> >  if( u.av.pBt ){
>> >sqlite3BtreeGetMeta(u.av.pBt, BTREE_SCHEMA_VERSION, (u32
>> *));
>> >u.av.iGen = db->aDb[pOp->p1].pSchema->iGeneration;
>> >  }else{
>> >u.av.iGen = u.av.iMeta = 0;
>> >  }
>> >
>> >
>> > I added a log trace to the else condition and it never fired off, so
>> > something is causing db->aDb[pOp->p1].pSchema->iGeneration to change
>> > relative to pOp->p3.  Give me some guidance as to where to dig and I'll
>> be
>> > glad to do so
>> >
>>
>> iGeneration changes in only one place in the code:
>> http://www.sqlite.org/src/artifact/5069f2248?ln=431
>>
>> May I recommend that you put a printf() before this line
>>
>>   http://www.sqlite.org/src/artifact/fc1b97fa6816?ln=3022
>>
>> And have that printf display the following values:
>>
>>   db  (rendered using %p)
>>   pOp->p1  (using %d)
>>   db->aDb[pOp->p1].pSchema  (using %p)
>>   db->aDb[pOp->p1].pSchema->schema_cookie  (using %d)
>>   db->aDb[pOp->p1].pSchema->iGeneration (using %d)
>>   iMeta  (using %d)
>>   pOp->p2  (using %d)
>>   pOp->p3  (using %d)
>>   p->zSql  (using %s)
>>
>>
> Progress.   A couple of representatives
>
> current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
> db->aDb[pOp->p1].pSchema->schema_cookie=27,
> db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
> pOp->p1=1, pOp->p2=27, pOp->p3=0'
>
> sql = 'update alerter_linksset is_dirty = null '
>
> current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
> db->aDb[pOp->p1].pSchema->schema_cookie=27,
> db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
> pOp->p1=1, pOp->p2=27, pOp->p3=0'
>
> sql = 'select bd.behavior_oid as behavior_oid   from
> computation_definitions cd  , behavior_dependents bd  where
> bd.behavior_oid = cd.behavior_oidand bd.response_definition_oid =
> :response_definition_oid '
>
> current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110B18,
> db->aDb[pOp->p1].pSchema->schema_cookie=63,
> db->aDb[pOp->p1].pSchema->iGeneration=4, u.av.iMeta=63, u.av.iGen=4,
> pOp->p1=0, pOp->p2=63, pOp->p3=2'
>
> sql = 'select distinct FORM_DEFINITIONS.FORM_DEFINITION_OID from
> FORM_DEFINITIONS where FORM_DEFINITIONS.guid=:FORM_DEFINITIONS_guid order
> by FORM_DEFINITIONS.rowid '
>
> current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110B18,
> db->aDb[pOp->p1].pSchema->schema_cookie=63,
> db->aDb[pOp->p1].pSchema->iGeneration=5, u.av.iMeta=63, u.av.iGen=5,
> pOp->p1=0, pOp->p2=63, pOp->p3=2'
>
> sql = 'select distinct FORM_DEFINITIONS.FORM_DEFINITION_OID from
> FORM_FLAGS, FORM_DEFINITIONS, MANIFESTS, SURVEYS where
> MANIFESTS.MANIFEST_OID=:MANIFESTS_MANIFEST_OID and
> FORM_DEFINITIONS.FORM_DEFINITION_OID=:FORM_DEFINITIONS_FORM_DEFINITION_OID
> and FORM_FLAGS.definition_parent=FORM_DEFINITIONS.FORM_DEFINITION_OID and
> FORM_FLAGS.instance_parent=MANIFESTS.MANIFEST_OID and
> FORM_DEFINITIONS.definition_parent=SURVEYS.SURVEY_OID and
> MANIFESTS.definition_parent=SURVEYS.SURVEY_OID order by
> FORM_DEFINITIONS.rowid '
>
> Obviously, iGeneration and p3 are out of sync.  So, what exactly is
> pOp->p3 ?
>
>


-- 
John Elrick
Fenestra Technologies
540-868-1377
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 7:37 PM, John Elrick  >wrote:
>
> >
> > It appears that the value is set here:
> >
> >  u.av.pBt = db->aDb[pOp->p1].pBt;
> >  if( u.av.pBt ){
> >sqlite3BtreeGetMeta(u.av.pBt, BTREE_SCHEMA_VERSION, (u32
> *));
> >u.av.iGen = db->aDb[pOp->p1].pSchema->iGeneration;
> >  }else{
> >u.av.iGen = u.av.iMeta = 0;
> >  }
> >
> >
> > I added a log trace to the else condition and it never fired off, so
> > something is causing db->aDb[pOp->p1].pSchema->iGeneration to change
> > relative to pOp->p3.  Give me some guidance as to where to dig and I'll
> be
> > glad to do so
> >
>
> iGeneration changes in only one place in the code:
> http://www.sqlite.org/src/artifact/5069f2248?ln=431
>
> May I recommend that you put a printf() before this line
>
>   http://www.sqlite.org/src/artifact/fc1b97fa6816?ln=3022
>
> And have that printf display the following values:
>
>   db  (rendered using %p)
>   pOp->p1  (using %d)
>   db->aDb[pOp->p1].pSchema  (using %p)
>   db->aDb[pOp->p1].pSchema->schema_cookie  (using %d)
>   db->aDb[pOp->p1].pSchema->iGeneration (using %d)
>   iMeta  (using %d)
>   pOp->p2  (using %d)
>   pOp->p3  (using %d)
>   p->zSql  (using %s)
>
>
Progress.   A couple of representatives

current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
db->aDb[pOp->p1].pSchema->schema_cookie=27,
db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
pOp->p1=1, pOp->p2=27, pOp->p3=0'

sql = 'update alerter_linksset is_dirty = null '

current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110AB0,
db->aDb[pOp->p1].pSchema->schema_cookie=27,
db->aDb[pOp->p1].pSchema->iGeneration=63, u.av.iMeta=27, u.av.iGen=63,
pOp->p1=1, pOp->p2=27, pOp->p3=0'

sql = 'select bd.behavior_oid as behavior_oid   from
computation_definitions cd  , behavior_dependents bd  where
bd.behavior_oid = cd.behavior_oidand bd.response_definition_oid =
:response_definition_oid '

current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=4, u.av.iMeta=63, u.av.iGen=4,
pOp->p1=0, pOp->p2=63, pOp->p3=2'

sql = 'select distinct FORM_DEFINITIONS.FORM_DEFINITION_OID from
FORM_DEFINITIONS where FORM_DEFINITIONS.guid=:FORM_DEFINITIONS_guid order
by FORM_DEFINITIONS.rowid '

current state = 'db=03FB1188, db->aDb[pOp->p1].pSchema=02110B18,
db->aDb[pOp->p1].pSchema->schema_cookie=63,
db->aDb[pOp->p1].pSchema->iGeneration=5, u.av.iMeta=63, u.av.iGen=5,
pOp->p1=0, pOp->p2=63, pOp->p3=2'

sql = 'select distinct FORM_DEFINITIONS.FORM_DEFINITION_OID from
FORM_FLAGS, FORM_DEFINITIONS, MANIFESTS, SURVEYS where
MANIFESTS.MANIFEST_OID=:MANIFESTS_MANIFEST_OID and
FORM_DEFINITIONS.FORM_DEFINITION_OID=:FORM_DEFINITIONS_FORM_DEFINITION_OID
and FORM_FLAGS.definition_parent=FORM_DEFINITIONS.FORM_DEFINITION_OID and
FORM_FLAGS.instance_parent=MANIFESTS.MANIFEST_OID and
FORM_DEFINITIONS.definition_parent=SURVEYS.SURVEY_OID and
MANIFESTS.definition_parent=SURVEYS.SURVEY_OID order by
FORM_DEFINITIONS.rowid '

Obviously, iGeneration and p3 are out of sync.  So, what exactly is pOp->p3
?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
On Wed, Jan 25, 2012 at 8:02 AM, Richard Hipp  wrote:

> On Wed, Jan 25, 2012 at 7:53 AM, John Elrick  >wrote:
>
> > I started with the place where iGeneration was being changed and found
> > nothing out of the ordinary.  I started an examination and noticed the
> > following.  My "huh?" surrounds the question, why is sqlite3SchemaClear
> > being called if pBt is assigned?
> >
>
> sqlite3SchemaClear is a destructor.  The third parameter to
> sqlite3BtreeSchema() is a pointer to the destructor.  sqlite3SchemaClear()
> is not actually called at this point.  Rather, a pointer to
> sqlite3SchemaClear() is passed into sqlite3BtreeSchema() and
> sqlite3BtreeSchema() then makes arrangements to invoke sqlite3SchemaClear()
> at some later time, when the Schema object it has just created gets
> destroyed.
>
>
Ah, I see it now.  Sorry about that.  In Pascal a procedure can be invoked
without the ().  I sometimes forget to translate.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread Richard Hipp 
On Wed, Jan 25, 2012 at 7:53 AM, John Elrick wrote:

> I started with the place where iGeneration was being changed and found
> nothing out of the ordinary.  I started an examination and noticed the
> following.  My "huh?" surrounds the question, why is sqlite3SchemaClear
> being called if pBt is assigned?
>

sqlite3SchemaClear is a destructor.  The third parameter to
sqlite3BtreeSchema() is a pointer to the destructor.  sqlite3SchemaClear()
is not actually called at this point.  Rather, a pointer to
sqlite3SchemaClear() is passed into sqlite3BtreeSchema() and
sqlite3BtreeSchema() then makes arrangements to invoke sqlite3SchemaClear()
at some later time, when the Schema object it has just created gets
destroyed.

>
> SQLITE_PRIVATE Schema *sqlite3SchemaGet(sqlite3 *db, Btree *pBt){
>  Schema * p;
>  if( pBt ){
>p = (Schema *)sqlite3BtreeSchema(pBt, sizeof(Schema),
> sqlite3SchemaClear);
>  }else{
>p = (Schema *)sqlite3DbMallocZero(0, sizeof(Schema));
>  }
>  if( !p ){
>db->mallocFailed = 1;
>  }else if ( 0==p->file_format ){
>sqlite3HashInit(>tblHash);
>sqlite3HashInit(>idxHash);
>sqlite3HashInit(>trigHash);
>sqlite3HashInit(>fkeyHash);
>p->enc = SQLITE_UTF8;
>  }
>  return p;
> }
>
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
...especially given that the results of sqlite3SchemaClear are being passed
as a parameter and from the declaration, sqlite3SchemaClear doesn't return
anything.

On Wed, Jan 25, 2012 at 7:53 AM, John Elrick wrote:

> I started with the place where iGeneration was being changed and found
> nothing out of the ordinary.  I started an examination and noticed the
> following.  My "huh?" surrounds the question, why is sqlite3SchemaClear
> being called if pBt is assigned?
>
> SQLITE_PRIVATE Schema *sqlite3SchemaGet(sqlite3 *db, Btree *pBt){
>   Schema * p;
>   if( pBt ){
> p = (Schema *)sqlite3BtreeSchema(pBt, sizeof(Schema),
> sqlite3SchemaClear);
>   }else{
> p = (Schema *)sqlite3DbMallocZero(0, sizeof(Schema));
>   }
>   if( !p ){
> db->mallocFailed = 1;
>   }else if ( 0==p->file_format ){
> sqlite3HashInit(>tblHash);
> sqlite3HashInit(>idxHash);
> sqlite3HashInit(>trigHash);
> sqlite3HashInit(>fkeyHash);
> p->enc = SQLITE_UTF8;
>   }
>   return p;
> }
>
>
> On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp  wrote:
>
>> On Tue, Jan 24, 2012 at 7:37 PM, John Elrick > >wrote:
>>
>> >
>> > It appears that the value is set here:
>> >
>> >  u.av.pBt = db->aDb[pOp->p1].pBt;
>> >  if( u.av.pBt ){
>> >sqlite3BtreeGetMeta(u.av.pBt, BTREE_SCHEMA_VERSION, (u32
>> *));
>> >u.av.iGen = db->aDb[pOp->p1].pSchema->iGeneration;
>> >  }else{
>> >u.av.iGen = u.av.iMeta = 0;
>> >  }
>> >
>> >
>> > I added a log trace to the else condition and it never fired off, so
>> > something is causing db->aDb[pOp->p1].pSchema->iGeneration to change
>> > relative to pOp->p3.  Give me some guidance as to where to dig and I'll
>> be
>> > glad to do so
>> >
>>
>> iGeneration changes in only one place in the code:
>> http://www.sqlite.org/src/artifact/5069f2248?ln=431
>>
>> May I recommend that you put a printf() before this line
>>
>>   http://www.sqlite.org/src/artifact/fc1b97fa6816?ln=3022
>>
>> And have that printf display the following values:
>>
>>   db  (rendered using %p)
>>   pOp->p1  (using %d)
>>   db->aDb[pOp->p1].pSchema  (using %p)
>>   db->aDb[pOp->p1].pSchema->schema_cookie  (using %d)
>>   db->aDb[pOp->p1].pSchema->iGeneration (using %d)
>>   iMeta  (using %d)
>>   pOp->p2  (using %d)
>>   pOp->p3  (using %d)
>>   p->zSql  (using %s)
>>
>> And then also add a printf on the line above where iGeneration changes and
>> display its new value every time it changes.
>>
>>
>>
>>
>> --
>> D. Richard Hipp
>> d...@sqlite.org
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
>
> --
> John Elrick
> Fenestra Technologies
> 540-868-1377
>
>


-- 
John Elrick
Fenestra Technologies
540-868-1377
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-25 Thread John Elrick 
I started with the place where iGeneration was being changed and found
nothing out of the ordinary.  I started an examination and noticed the
following.  My "huh?" surrounds the question, why is sqlite3SchemaClear
being called if pBt is assigned?

SQLITE_PRIVATE Schema *sqlite3SchemaGet(sqlite3 *db, Btree *pBt){
  Schema * p;
  if( pBt ){
p = (Schema *)sqlite3BtreeSchema(pBt, sizeof(Schema),
sqlite3SchemaClear);
  }else{
p = (Schema *)sqlite3DbMallocZero(0, sizeof(Schema));
  }
  if( !p ){
db->mallocFailed = 1;
  }else if ( 0==p->file_format ){
sqlite3HashInit(>tblHash);
sqlite3HashInit(>idxHash);
sqlite3HashInit(>trigHash);
sqlite3HashInit(>fkeyHash);
p->enc = SQLITE_UTF8;
  }
  return p;
}


On Tue, Jan 24, 2012 at 7:46 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 7:37 PM, John Elrick  >wrote:
>
> >
> > It appears that the value is set here:
> >
> >  u.av.pBt = db->aDb[pOp->p1].pBt;
> >  if( u.av.pBt ){
> >sqlite3BtreeGetMeta(u.av.pBt, BTREE_SCHEMA_VERSION, (u32
> *));
> >u.av.iGen = db->aDb[pOp->p1].pSchema->iGeneration;
> >  }else{
> >u.av.iGen = u.av.iMeta = 0;
> >  }
> >
> >
> > I added a log trace to the else condition and it never fired off, so
> > something is causing db->aDb[pOp->p1].pSchema->iGeneration to change
> > relative to pOp->p3.  Give me some guidance as to where to dig and I'll
> be
> > glad to do so
> >
>
> iGeneration changes in only one place in the code:
> http://www.sqlite.org/src/artifact/5069f2248?ln=431
>
> May I recommend that you put a printf() before this line
>
>   http://www.sqlite.org/src/artifact/fc1b97fa6816?ln=3022
>
> And have that printf display the following values:
>
>   db  (rendered using %p)
>   pOp->p1  (using %d)
>   db->aDb[pOp->p1].pSchema  (using %p)
>   db->aDb[pOp->p1].pSchema->schema_cookie  (using %d)
>   db->aDb[pOp->p1].pSchema->iGeneration (using %d)
>   iMeta  (using %d)
>   pOp->p2  (using %d)
>   pOp->p3  (using %d)
>   p->zSql  (using %s)
>
> And then also add a printf on the line above where iGeneration changes and
> display its new value every time it changes.
>
>
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
John Elrick
Fenestra Technologies
540-868-1377
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 7:37 PM, John Elrick wrote:

>
> It appears that the value is set here:
>
>  u.av.pBt = db->aDb[pOp->p1].pBt;
>  if( u.av.pBt ){
>sqlite3BtreeGetMeta(u.av.pBt, BTREE_SCHEMA_VERSION, (u32 *));
>u.av.iGen = db->aDb[pOp->p1].pSchema->iGeneration;
>  }else{
>u.av.iGen = u.av.iMeta = 0;
>  }
>
>
> I added a log trace to the else condition and it never fired off, so
> something is causing db->aDb[pOp->p1].pSchema->iGeneration to change
> relative to pOp->p3.  Give me some guidance as to where to dig and I'll be
> glad to do so
>

iGeneration changes in only one place in the code:
http://www.sqlite.org/src/artifact/5069f2248?ln=431

May I recommend that you put a printf() before this line

   http://www.sqlite.org/src/artifact/fc1b97fa6816?ln=3022

And have that printf display the following values:

   db  (rendered using %p)
   pOp->p1  (using %d)
   db->aDb[pOp->p1].pSchema  (using %p)
   db->aDb[pOp->p1].pSchema->schema_cookie  (using %d)
   db->aDb[pOp->p1].pSchema->iGeneration (using %d)
   iMeta  (using %d)
   pOp->p2  (using %d)
   pOp->p3  (using %d)
   p->zSql  (using %s)

And then also add a printf on the line above where iGeneration changes and
display its new value every time it changes.




-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 7:26 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 7:18 PM, John Elrick  >wrote:
>

SNIP


> So the problem was introduced by
>
> http://www.sqlite.org/src/info/36c04dd1695f08
>
> Which is the fix for this bug:
>
> http://www.sqlite.org/src/info/f7b4edece25c99
>
> That helps.  But we still cannot recreate the problem.  Are you using
> shared cache mode and do you have multiple database connections open on the
> same database file?


Nope and nope.

It appears that the value is set here:

  u.av.pBt = db->aDb[pOp->p1].pBt;
  if( u.av.pBt ){
sqlite3BtreeGetMeta(u.av.pBt, BTREE_SCHEMA_VERSION, (u32 *));
u.av.iGen = db->aDb[pOp->p1].pSchema->iGeneration;
  }else{
u.av.iGen = u.av.iMeta = 0;
  }


I added a log trace to the else condition and it never fired off, so
something is causing db->aDb[pOp->p1].pSchema->iGeneration to change
relative to pOp->p3.  Give me some guidance as to where to dig and I'll be
glad to do so (tomorrow, my 9 year old daughter just asked for ice cream).
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 7:18 PM, John Elrick wrote:

> I really found it this time.
>
> While I was looking over the code changes, I noticed a section of code
> dealing with case OP_VerifyCookie:
>
> On a hunch, I replaced this line:
>
>  if( u.av.iMeta!=pOp->p2 || u.av.iGen!=pOp->p3 ){
>
> with the if statement from 3.7.5
>
>  if( u.av.iMeta!=pOp->p2 ){
>
> When I do, the number of mallocs drops back to 14,000 - still a little
> higher than in 3.7.5, but nowhere near 140,000.
>

So the problem was introduced by

 http://www.sqlite.org/src/info/36c04dd1695f08

Which is the fix for this bug:

 http://www.sqlite.org/src/info/f7b4edece25c99

That helps.  But we still cannot recreate the problem.  Are you using
shared cache mode and do you have multiple database connections open on the
same database file?


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
I really found it this time.

While I was looking over the code changes, I noticed a section of code
dealing with case OP_VerifyCookie:

Inside that section is the following in 3.7.6

  if( u.av.iMeta!=pOp->p2 || u.av.iGen!=pOp->p3 ){
sqlite3DbFree(db, p->zErrMsg);
p->zErrMsg = sqlite3DbStrDup(db, "database schema has changed");
/* If the schema-cookie from the database file matches the cookie
** stored with the in-memory representation of the schema, do
** not reload the schema from the database file.
**
** If virtual-tables are in use, this is not just an optimization.
** Often, v-tables store their data in other SQLite tables, which
** are queried from within xNext() and other v-table methods using
** prepared queries. If such a query is out-of-date, we do not want to
** discard the database schema, as the user code implementing the
** v-table would have to be ready for the sqlite3_vtab structure itself
** to be invalidated whenever sqlite3_step() is called from within
** a v-table method.
*/
if( db->aDb[pOp->p1].pSchema->schema_cookie!=u.av.iMeta ){
  sqlite3ResetInternalSchema(db, pOp->p1);
}

On a hunch, I replaced this line:

  if( u.av.iMeta!=pOp->p2 || u.av.iGen!=pOp->p3 ){

with the if statement from 3.7.5

  if( u.av.iMeta!=pOp->p2 ){

When I do, the number of mallocs drops back to 14,000 - still a little
higher than in 3.7.5, but nowhere near 140,000.

Does that help?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 5:58 PM, John Elrick wrote:

> Let me rephrase that...
>
> > > http://www.sqlite.org/fenstra/sqlite-201102040051.zip
>
> 8,191 calls to malloc (consistent with 3.7.5 for this dataset)
> >> > > http://www.sqlite.org/fenstra/sqlite-201102112254.zip
> >
> 14,726 calls to malloc
>
>
> 3.7.6
>
> 148,606 calls to malloc
>

I still believe that the problem is along this graph:


http://www.sqlite.org/src/timeline?from=36d79e6f54=f7e2ea33d5

Do you disagree?  Do you think the problem occurs before or after the set
of changes shown above.  Only you are able to recreate the problem (so
far), so you have to be judge as to when it works and when it does not.
Let me know where we ought to be looking.


>
> --
> John Elrick
> Fenestra Technologies
> 540-868-1377
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
Let me rephrase that...

I ran this test with a small case.  See results below.

On Tue, Jan 24, 2012 at 5:55 PM, John Elrick wrote:

SNIP

> > http://www.sqlite.org/fenstra/sqlite-201102040051.zip
>>
>
> Low malloc count
>

8,191 calls to malloc (consistent with 3.7.5 for this dataset)


>
>
>>
>> > > http://www.sqlite.org/fenstra/sqlite-201102112254.zip
>>
>
> High malloc count
>
>
14,726 calls to malloc


3.7.6

148,606 calls to malloc

So there was a jump, but not to the full level.


> The problem is between the two.
>



-- 
John Elrick
Fenestra Technologies
540-868-1377
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 5:36 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 5:21 PM, John Elrick  >wrote:
>
> > On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp  wrote:
> > > >
> > > > OK.  Please try these:
> > >
> > > http://www.sqlite.org/fenstra/sqlite-201102040051.zip
>

Low malloc count


>
> > > http://www.sqlite.org/fenstra/sqlite-201102112254.zip
>

High malloc count

The problem is between the two.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 5:21 PM, John Elrick wrote:

> On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp  wrote:
> > >
> > > OK.  Please try these:
> >
> > http://www.sqlite.org/fenstra/sqlite-201102040051.zip
> > http://www.sqlite.org/fenstra/sqlite-201102112254.zip
> >
> >
> Both of these are low malloc count.
>

Progress!  Consider the timeline at

http://www.sqlite.org/src/timeline?p=36d79e6f54cdc412=37

You have reported that the top-most check-in exhibits the problem but the
bottom-most check-in does not.  So the problem must be somewhere in between
these two.  Here are two other check-ins in the middle for you to try so
that we can narrow it down further:

http://www.sqlite.org/fenstra/sqlite-201102211146.zip
http://www.sqlite.org/fenstra/sqlite-201103040056.zip


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 3:05 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 1:28 PM, John Elrick  >wrote:
>
> >
> >
> > > http://www.sqlite.org/fenstra/sqlite-201103081639.zip
> >
> >
> > This one gives the high number of mallocs just as does 3.7.6.
> >
> > OK.  Please try these:
>
> http://www.sqlite.org/fenstra/sqlite-201102040051.zip
> http://www.sqlite.org/fenstra/sqlite-201102112254.zip
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>


Both of these are low malloc count.

I happened to think on my long drive from Frederick, are their canonical
procedures used to flag the system, to tell it that the schema needs
rebuilt?  Maybe I should attempt a call stack trace on those (if they
exist).
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
Nope and nope.  They are happening apparently out of the blue.
On Jan 24, 2012 2:40 PM, "Nico Williams"  wrote:

> Maybe a view is getting materialized, or a some other temp table's
> creation under the hood is triggering this?
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 1:28 PM, John Elrick wrote:

>
>
> > http://www.sqlite.org/fenstra/sqlite-201103081639.zip
>
>
> This one gives the high number of mallocs just as does 3.7.6.
>
> OK.  Please try these:

http://www.sqlite.org/fenstra/sqlite-201102040051.zip
http://www.sqlite.org/fenstra/sqlite-201102112254.zip


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Nico Williams 
Maybe a view is getting materialized, or a some other temp table's
creation under the hood is triggering this?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 12:53 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 12:15 PM, John Elrick  >wrote:
>
> > >
> > > Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out
> > > from whence it is being called so often?
> > >
> > >
> >
> > Just did.  Most of the calls are bubbling up from _sqlite3_step which I
> > mentioned before.
>

Richard,

I added some tracking which keeps a log of every query and writes the last
50 things done when the call to sqlite3ResetInternalSchema() takes place.
 Here is a fairly representative entry.  The last SQL is the one which
triggered the call to sqlite3ResetInternalSchema().  Nothing here which
touches the schema of the database that I can see.

last 50 sqls = 'step: select distinct RESPONSES.RESPONSE_OID from
RESPONSES, DATA_ELEMENTS, SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
RESPONSES.rowid
step: SELECT * FROM RESPONSES a where a.soft_deleted_char = 'F' and
a.RESPONSE_OID = :oid
step: UPDATE RESPONSES SET
RESPONSE_NAME = :RESPONSE_NAME,
prelisted_value = :prelisted_value
WHERE RESPONSE_OID = :RESPONSE_OID
step: select distinct DATA_ELEMENTS.DATA_ELEMENT_OID from RESPONSES,
DATA_ELEMENTS, SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
DATA_ELEMENTS.rowid
step: select distinct DATA_ELEMENTS.DATA_ELEMENT_OID from RESPONSES,
DATA_ELEMENTS, SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
DATA_ELEMENTS.rowid
step: select distinct RESPONSES.RESPONSE_OID from RESPONSES, DATA_ELEMENTS,
SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
RESPONSES.rowid
step: select distinct RESPONSES.RESPONSE_OID from RESPONSES, DATA_ELEMENTS,
SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
RESPONSES.rowid
step: SELECT * FROM RESPONSES a where a.soft_deleted_char = 'F' and
a.RESPONSE_OID = :oid
step: UPDATE RESPONSES SET
RESPONSE_NAME = :RESPONSE_NAME,
prelisted_value = :prelisted_value
WHERE RESPONSE_OID = :RESPONSE_OID
step: select distinct DATA_ELEMENTS.DATA_ELEMENT_OID from RESPONSES,
DATA_ELEMENTS, SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
DATA_ELEMENTS.rowid
step: select distinct DATA_ELEMENTS.DATA_ELEMENT_OID from RESPONSES,
DATA_ELEMENTS, SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
DATA_ELEMENTS.rowid
step: select distinct RESPONSES.RESPONSE_OID from RESPONSES, DATA_ELEMENTS,
SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
RESPONSES.rowid
step: select distinct RESPONSES.RESPONSE_OID from RESPONSES, DATA_ELEMENTS,
SEQUENCE_ELEMENTS where
SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID=:SEQUENCE_ELEMENTS_SEQUENCE_ELEMENT_OID
and DATA_ELEMENTS.DATA_ELEMENT_NAME=:DATA_ELEMENTS_DATA_ELEMENT_NAME and
RESPONSES.instance_parent=SEQUENCE_ELEMENTS.SEQUENCE_ELEMENT_OID and
RESPONSES.definition_parent=DATA_ELEMENTS.DATA_ELEMENT_OID order by
RESPONSES.rowid
step: SELECT * FROM RESPONSES a where a.soft_deleted_char = 'F' and
a.RESPONSE_OID = :oid

Re: [sqlite] Found it

2012-01-24 Thread Krystian Bigaj 
On 24 January 2012 19:32, John Elrick  wrote:

> On Tue, Jan 24, 2012 at 1:29 PM, Krystian Bigaj  >wrote:
> > You can compile sqlite3.dll with debug information, and then you can in
> VS
> > attach to your running .exe - you should be able to see sqlite
> > stacks/breakpoints/step-in/out/etc. should also work.
> >
> Thanks very much for the suggestion.  We do not have, nor have any
> intention of using, Visual Studio.  It's also not a DLL, the resulting
> output is a Borland OBJ, which cannot be generated correctly from any other
> compiler.


 So maybe you could try build OBJ with debug/remote options/symbols
(tdebug?). And then try to attach to running process from C++ Builder IDE
(if it's not plain Borland C++ compiler).
I've never used it, but it might work.

PS. I forgot also that VS doesn't support debugging code if there is more
than 64K lines in source file.

Best regards,
Krystian Bigaj
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 1:28 PM, John Elrick wrote:

SNIP

http://www.sqlite.org/fenstra/sqlite-201103241737.zip
>
>
> This one is failing to compile.  It needs _msize.  That sounds like the
> 3.7.10 issue reported recently.
>
>
I opened the sqlite3.c file and it is version 3.7.10.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 1:29 PM, Krystian Bigaj wrote:

> You can compile sqlite3.dll with debug information, and then you can in VS
> attach to your running .exe - you should be able to see sqlite
> stacks/breakpoints/step-in/out/etc. should also work.
>
>
Thanks very much for the suggestion.  We do not have, nor have any
intention of using, Visual Studio.  It's also not a DLL, the resulting
output is a Borland OBJ, which cannot be generated correctly from any other
compiler.


John
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Krystian Bigaj 
You can compile sqlite3.dll with debug information, and then you can in VS
attach to your running .exe - you should be able to see sqlite
stacks/breakpoints/step-in/out/etc. should also work.

Best regards,
Krystian Bigaj


On 24 January 2012 19:25, John Elrick  wrote:

> On Tue, Jan 24, 2012 at 1:18 PM, Dan Kennedy 
> wrote:
>
> > On 01/25/2012 12:15 AM, John Elrick wrote:
> > Unfortunately, no I cannot.  Delphi 2007 doesn't give me accurate
> information from a linked OBJ.  It's roughly equivalent to trying to get
> accurate call stacks from inside a DLL.  I've had to hand add pseudo call
> stack information.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 12:53 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 12:15 PM, John Elrick  >wrote:
>
>
SNIP


> http://www.sqlite.org/fenstra/sqlite-201103081639.zip


This one gives the high number of mallocs just as does 3.7.6.


>
> http://www.sqlite.org/fenstra/sqlite-201103241737.zip


This one is failing to compile.  It needs _msize.  That sounds like the
3.7.10 issue reported recently.


John
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 1:18 PM, Dan Kennedy  wrote:

> On 01/25/2012 12:15 AM, John Elrick wrote:
>
>> On Tue, Jan 24, 2012 at 12:06 PM, Richard Hipp  wrote:
>>
>>  On Tue, Jan 24, 2012 at 11:59 AM, John 
>> Elrick
>>>
 wrote:

>>>
>>>  While reviewing our other thread, I noticed a piece of information
 concerning sqlite3ResetInternalSchema.  Based on the number of times

>>> that I
>>>
 reported it had been called, I believe that something was changed
 between
 3.7.5 and 3.7.6 which causes sqlite3ResetInternalSchema to be called
 when
 it wasn't in the earlier version.   Is there some kind of enumeration of
 what events would cause sqlite3ResetInternalSchema to be called in

>>> version
>>>
 3.7.5 vs 3.7.6?


>>> I agree that the fact that the triggers are being reparsed is a big hint.
>>> But we still don't know why they are being reparsed.  And we still cannot
>>> reproduce the problem in a command-line shell - in only seems to occur
>>> within your Delphi app.
>>>
>>> Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out
>>> from whence it is being called so often?
>>>
>>>
>>>
>> Just did.  Most of the calls are bubbling up from _sqlite3_step which I
>> mentioned before.
>>
>
> Can you see the whole call stack? There are a couple of different
> ways sqlite3ResetInternalSchema() may be called from within
> sqlite3_step(). Knowing which will be a clue as to why it is being
> called.
>
> Unfortunately, no I cannot.  Delphi 2007 doesn't give me accurate
information from a linked OBJ.  It's roughly equivalent to trying to get
accurate call stacks from inside a DLL.  I've had to hand add pseudo call
stack information.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Dan Kennedy 

On 01/25/2012 12:15 AM, John Elrick wrote:

On Tue, Jan 24, 2012 at 12:06 PM, Richard Hipp  wrote:


On Tue, Jan 24, 2012 at 11:59 AM, John Elrick

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 12:15 PM, John Elrick wrote:

> >
> > Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out
> > from whence it is being called so often?
> >
> >
>
> Just did.  Most of the calls are bubbling up from _sqlite3_step which I
> mentioned before.  There are no new changes to the schema immediately prior
> to these calls.  Here is one example where step triggers a call to
> sqlite3ResetInternalSchema() (apologies for using Delphi and our wrappers,
> but I hope it's clear):
>


That information isn't really useful.  I still have no idea what is wrong,
nor idea how to recreate the observed behavior.

Let's bisect to find the specific check-in that is causing your problem.
Here are two amalgamations that are in between 3.7.5 and 3.7.6.  Please let
me know how these work, and based on that will continue to narrow down the
changes.

http://www.sqlite.org/fenstra/sqlite-201103081639.zip
http://www.sqlite.org/fenstra/sqlite-201103241737.zip


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
On Tue, Jan 24, 2012 at 12:06 PM, Richard Hipp  wrote:

> On Tue, Jan 24, 2012 at 11:59 AM, John Elrick  >wrote:
>
> > While reviewing our other thread, I noticed a piece of information
> > concerning sqlite3ResetInternalSchema.  Based on the number of times
> that I
> > reported it had been called, I believe that something was changed between
> > 3.7.5 and 3.7.6 which causes sqlite3ResetInternalSchema to be called when
> > it wasn't in the earlier version.   Is there some kind of enumeration of
> > what events would cause sqlite3ResetInternalSchema to be called in
> version
> > 3.7.5 vs 3.7.6?
> >
>
> I agree that the fact that the triggers are being reparsed is a big hint.
> But we still don't know why they are being reparsed.  And we still cannot
> reproduce the problem in a command-line shell - in only seems to occur
> within your Delphi app.
>
> Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out
> from whence it is being called so often?
>
>

Just did.  Most of the calls are bubbling up from _sqlite3_step which I
mentioned before.  There are no new changes to the schema immediately prior
to these calls.  Here is one example where step triggers a call to
sqlite3ResetInternalSchema() (apologies for using Delphi and our wrappers,
but I hope it's clear):

  itsSaveTabsheet :=
itsDatabase.prepareExec('INSERT INTO TABSHEETS VALUES (null,
:workbookType, '
  + ':manifestOid, :surveyOid, :formDefinitionId, :prefKey, '
  + ':name, :originalName, :sequence, :columnList)');
...

function TDataServices.saveTabsheet(const tabsheetId, aWorkbookType,
manifestOid, surveyOid,
  formDefinitionOid, aPrefKey, aName, aSeq, aColumnList: string): string;
begin
  itsSaveTabsheet := itsSaveTabsheet;
  itsSaveTabsheet.bind(':workbookType', aWorkbookType);
  itsSaveTabsheet.bind(':manifestOid', manifestOid);
  itsSaveTabsheet.bind(':surveyOid', surveyOid);
  itsSaveTabsheet.bind(':formDefinitionId', StrToIntDef(formDefinitionOid,
0));
  itsSaveTabsheet.bind(':prefKey', aPrefKey);
  itsSaveTabsheet.bind(':name', aName);
  itsSaveTabsheet.bind(':originalName', aName);
  itsSaveTabsheet.bind(':sequence', aSeq);
  itsSaveTabsheet.bind(':columnList', aColumnList);
  itsSaveTabsheet.execute;
  result := IntToStr(itsDatabase.lastInsertRowId);
end;
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread Richard Hipp 
On Tue, Jan 24, 2012 at 11:59 AM, John Elrick wrote:

> While reviewing our other thread, I noticed a piece of information
> concerning sqlite3ResetInternalSchema.  Based on the number of times that I
> reported it had been called, I believe that something was changed between
> 3.7.5 and 3.7.6 which causes sqlite3ResetInternalSchema to be called when
> it wasn't in the earlier version.   Is there some kind of enumeration of
> what events would cause sqlite3ResetInternalSchema to be called in version
> 3.7.5 vs 3.7.6?
>

I agree that the fact that the triggers are being reparsed is a big hint.
But we still don't know why they are being reparsed.  And we still cannot
reproduce the problem in a command-line shell - in only seems to occur
within your Delphi app.

Can you put a breakpoint on sqlite3ResetInternalSchema() and figure out
from whence it is being called so often?


> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
While reviewing our other thread, I noticed a piece of information
concerning sqlite3ResetInternalSchema.  Based on the number of times that I
reported it had been called, I believe that something was changed between
3.7.5 and 3.7.6 which causes sqlite3ResetInternalSchema to be called when
it wasn't in the earlier version.   Is there some kind of enumeration of
what events would cause sqlite3ResetInternalSchema to be called in version
3.7.5 vs 3.7.6?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Found it

2012-01-24 Thread John Elrick 
Addendum.  After looking further it appears that the reparsing is happening
independently of triggering.  In this case, the reparsing occurs 30,502
times for every view and trigger I have examined.  Only one trigger appears
in the 3.7.5 listing and it is parsed exactly 1 time.

3.7.5

CREATE TRIGGER error_warning_after_delete after delete on
local_errors_warnings begin   insert into local_error_warning_deletions
values( old.error_warning_oid);   update alerter_links
 set is_dirty = 1where sequence_instance_oid =
old.sequence_instance_oid  and response_definition_oid in(
select response_definition_oidfrom behavior_independents
where behavior_oid = old.behavior_oid); end
Times called:  1
Cumulative Allocated Memory:   5,960
Count of _malloc Calls:56
Cumulative Reallocated Memory: 0
Count of _realloc Calls:   0
Cumulative Reallocated Memory where nil:   0
Count of _realloc Calls where nil: 0
Count of _free Calls:  32
Cumulative _mallocs by size
<= 1kb:3,304 bytes (54 count; 61 avg)
1kb to 4kb:2,656 bytes (2 count; 1,328 avg)
4bk to 8kb:0 bytes (0 count; NAN avg)
8bk to 16kb:   0 bytes (0 count; NAN avg)
16bk to 32kb:  0 bytes (0 count; NAN avg)
32bk to 64kb:  0 bytes (0 count; NAN avg)
64bk to 128kb: 0 bytes (0 count; NAN avg)
128kb to 256kb:0 bytes (0 count; NAN avg)
256kb to 512kb:0 bytes (0 count; NAN avg)
512kb to 1024kb:   0 bytes (0 count; NAN avg)
> 1mb: 0 bytes (0 count; NAN avg)
No reallocs
No reallocs of nil pointers
---

3.7.6

CREATE TRIGGER error_warning_after_delete after delete on
local_errors_warnings begin   insert into local_error_warning_deletions
values( old.error_warning_oid);   update alerter_links
 set is_dirty = 1where sequence_instance_oid =
old.sequence_instance_oid  and response_definition_oid in(
select response_definition_oidfrom behavior_independents
where behavior_oid = old.behavior_oid); end
Times called:  30502
Cumulative Allocated Memory:   180,327,824
Count of _malloc Calls:1,708,112
Cumulative Reallocated Memory: 0
Count of _realloc Calls:   0
Cumulative Reallocated Memory where nil:   0
Count of _realloc Calls where nil: 0
Count of _free Calls:  976,064
Cumulative _mallocs by size
<= 1kb:99,314,512 bytes (1,647,108
count; 60 avg)
1kb to 4kb:81,013,312 bytes (61,004 count;
1,328 avg)
4bk to 8kb:0 bytes (0 count; NAN avg)
8bk to 16kb:   0 bytes (0 count; NAN avg)
16bk to 32kb:  0 bytes (0 count; NAN avg)
32bk to 64kb:  0 bytes (0 count; NAN avg)
64bk to 128kb: 0 bytes (0 count; NAN avg)
128kb to 256kb:0 bytes (0 count; NAN avg)
256kb to 512kb:0 bytes (0 count; NAN avg)
512kb to 1024kb:   0 bytes (0 count; NAN avg)
> 1mb: 0 bytes (0 count; NAN avg)
No reallocs
No reallocs of nil pointers
---
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] FOUND IT!!! What is the PDO last_insert_id method???`

2008-05-16 Thread Skip Evans 
Hey all,

Found the PHP site page with all the PDO 
functions... gadzoooks!

Skip

Skip Evans wrote:
> Hey all,
> 
> I'm new to SQLite and having a heck of a time 
> finding the equivalent to MySQL's
> 
> mysql_insert_id().
> 
> I'm using SQLite with  PDO in PHP5.

-- 
Skip Evans
Big Sky Penguin, LLC
503 S Baldwin St, #1
Madison, WI 53703
608-250-2720
http://bigskypenguin.com
=-=-=-=-=-=-=-=-=-=
Check out PHPenguin, a lightweight and versatile
PHP/MySQL, AJAX & DHTML development framework.
http://phpenguin.bigskypenguin.com/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users