[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Keith Medcalf]

I hate correcting myself.  You do not need the --export-all-symbols
Using it will cause all non-static symbols to be exported (even without a 
specific dllexport) just like using -shared on unices.

---
Theory is when you know everything but nothing works.  Practice is when 
everything works but no one knows why.  Sometimes theory and practice are 
combined:  nothing works and no one knows why.


>-Original Message-
>From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-
>bounces at mailinglists.sqlite.org] On Behalf Of Keith Medcalf
>Sent: Thursday, 2 April, 2015 17:51
>To: General Discussion of SQLite Database
>Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck -
>NXCheck - DBCheck
>
>
>BTW, I have verified that these options all work as described and the
>options are recognized and processed properly by Windows, and that
>BinScope is happy:
>
>Failed checks
>d:\source\sqlite\sqlite3.dll - SafeSEHCheck ( FAIL )
>
>Passed checks
>d:\source\sqlite\sqlite3.dll - NXCheck ( PASS )
>d:\source\sqlite\sqlite3.dll - DBCheck ( PASS )
>
>(Note, position independent code (PIC) is by definition loadable at any
>base.  Microsoft is just several decades behind in generating position
>independent code.)
>
>---
>Theory is when you know everything but nothing works.  Practice is when
>everything works but no one knows why.  Sometimes theory and practice are
>combined:  nothing works and no one knows why.
>
>
>>-Original Message-
>>From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-
>>bounces at mailinglists.sqlite.org] On Behalf Of Keith Medcalf
>>Sent: Thursday, 2 April, 2015 17:43
>>To: General Discussion of SQLite Database
>>Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck -
>>NXCheck - DBCheck
>>
>>
>>add the following linker options with MinGW:
>>
>>-static-libgcc -Wl,-Bstatic,--nxcompat,--dynamicbase,--export-all-
>symbols
>>
>>You may or may not need -static-libgcc or the linker -Bstatic options
>>unless you are also enabling things that require MingW DLL runtime
>>support (such as using the -mthreads option to make the DLL thread
>>exception safe).
>>
>>-Wl,--nxcompat,--dynamicbase,--export-all-symbols,--large-address-aware
>>
>>Sets the NXCOMPAT and DYNAMICBASE (ASLR) flags on the output of the
>>linker (so you can add them to the shell tool and other pre-built
>>executables as well).  Files also need to export the symbol table (hence
>>the --export-all-symbols).  --large-address-aware sets the (you guessed
>>it) large address aware flag which basically means that you are using
>>unsigned pointers and not signed pointers (that is, you understand that
>>neither disk space nor memory comes in negative sizes) and don't diddle
>>with your pointers (eg, assigning magical meaning to bits in the
>>pointer).  This allows use of more than 2GB of virtual address space per
>>process on systems that support it (ie, 32-bit userland on a 64-bit
>>Windows -- on 64-bit Windows windows does not co-habitate with 32-bit
>>processes, instead a wee trampoline is installed in the top 100MB or so
>>of the virtual space to thunk back and forth to the OS -- making it
>>impossible to insert code into the OS from a 32-bit process, while 64-
>bit
>>processes on 64
>> -bit windows have full and complete access to the OS as it resides in
>>each processes address space) or if the /3GB boot option is set on 32-
>bit
>>OSes (so the OS is forced to shoehorn into the top 1GB of each process).
>>
>>The only thing this does not fix is the Exception Handlers.  But that
>>particular protection is not implemented by gcc (not needed because of
>>non-ill-conceived design?).  However, you might still want to add -
>>mthreads and the static linking options to the build so that the GCC
>>threadsafe exception handlers are used.
>>
>>---
>>Theory is when you know everything but nothing works.  Practice is when
>>everything works but no one knows why.  Sometimes theory and practice
>are
>>combined:  nothing works and no one knows why.
>>
>>>-Original Message-
>>>From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-
>users-
>>>bounces at mailinglists.sqlite.org] On Behalf Of Richard Hipp
>>>Sent: Thursday, 2 April, 2015 14:25
>>>To: General Discussion of SQLite Database
>>>Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck
>-
>>>NXCheck - DBCheck
>>>
>>>On 4/2/15, Random Coder  wrote:

 I'd recommend the SQLite team turn them on for the version of the DLL
 they distribute, but I'm honestly not sure if there are negative side
 effects to doing so.
>>>
>>>That's not possible, unfortunately,   We compile the published DLL
>>>(the 32-bit DLL at least) with MinGW so that they will work on older
>>>versions of Windows.  If we used a recent MSVC then the resulting DLL
>>>will not work on WinXP, I am told.
>>>
>>>If the published DLL is not to your liking, it is simple to make your
>>>own.  I suggest you do 

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[maarten.most...@stakepoint.com]

This are basically compiler switches as far as I understand.

I am not a C programmer, so also quite difficult for me to link the library 
statically.

This is what I can read about it here:

[ 
https://msdn.microsoft.com/en-us/library/windows/apps/jj657973.aspx#binscope_2 
]( 
https://msdn.microsoft.com/en-us/library/windows/apps/jj657973.aspx#binscope_2 )



/SafeSEH Exception Handling Protection
Windows App Certification Kit error message: SafeSEHCheck Test failed
An exception handler runs when the app encounters an exceptional condition, 
such as a divide-by-zero error. Because the address of the exception handler is 
stored on the stack when a function is called, it could be vulnerable to a 
buffer overflow attacker if some malicious software were to overwrite the stack.
What to do if your app fails this test
Enable the /SAFESEH option in the linker command when you build your app. This 
option is on by default in the Release configurations of Visual Studio. Verify 
this option is enabled in the build instructions for all executable modules in 
your app.
Remarks
The test is not performed on 64-bit binaries or ARM chipset binaries because 
they don't store exception handler addresses on the stack.
Data Execution Prevention
Windows App Certification Kit error message: NXCheck Test failed
This test verifies that an app doesn't run code that is stored in a data 
segment.
What to do if your app fails this test
Enable the /NXCOMPAT option in the linker command when you build your app. This 
option is on by default in linker versions that support Data Execution 
Prevention (DEP).
Remarks
We recommend that you test your apps on a DEP-capable CPU and fix any failures 
you find that result from DEP.
Address Space Layout Randomization
Windows App Certification Kit error message: DBCheck Test failed
Address Space Layout Randomization (ASLR) loads executable images into 
unpredictable locations in memory, which makes it harder for malicious software 
that expects a program to be loaded at a certain virtual address to operate 
predictably. Your app and all components that your app uses must support ASLR.
What to do if your app fails this test
Enable the /DYNAMICBASE option in the linker command when you build your app. 
Verify that all modules that your app uses also use this linker option.
Remarks
Normally, ASLR doesn't affect performance. But in some scenarios there is a 
slight performance improvement on 32-bit systems. It is possible that 
performance could degrade in a highly congested system that have many images 
loaded in many different memory locations.
This test is performed on only apps written in managed code, such as by using 
C# or .NET Framework.


Best regards,

Maarten MOSTERT,

> "Richard Hipp"  |



> On 4/2/15, maarten.mostert at stakepoint.com
>  wrote:
> >
> > WARNING
> > Binary analyzer
> > Warning: The binary analyzer test detected the following errors:
> > File C:\Program Files\StakePoint\sqlite3.dll has failed the NXCheck check.
> > File C:\Program Files\StakePoint\sqlite3.dll has failed the DBCheck check.
> > File C:\Program Files\StakePoint\sqlite3.dll has failed the SafeSEHCheck
> > check.
> >
> 
> I have no idea what the above means.
> 
> Can you work around it by statically linking SQLite into your
> application, rather than using a separate DLL?
> --
> D. Richard Hipp
> drh at sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> 

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[maarten.most...@stakepoint.com]

Please find here under the test results.

Best regards,


Maarten MOSTERT



28 Av Alphonse Denis
83400 Hy?res, France
+33 676411296 
[ http://stakepoint.com/ ]( http://stakepoint.com/ )



==
Windows App Certification Kit - Test Results
Windows App Certification Kit - Test Results
App name:StakePoint ?Technical-Preview?
App publisher:Maarten MOSTERT
App version:15.03.0019
OS Version:Microsoft Windows 8.1 (6.3.9600.0)
Kit Version:3.4
Report time:31/03/2015 15:03:53

Overall result: PASSED WITH WARNINGS
It is highly recommended that you investigate and root cause warnings to ensure 
that users of your app aren?t negatively impacted.




Windows security features test 



WARNING
Binary analyzer
Warning: The binary analyzer test detected the following errors:
File C:\Program Files\StakePoint\sqlite3.dll has failed the NXCheck check.
File C:\Program Files\StakePoint\sqlite3.dll has failed the DBCheck check.
File C:\Program Files\StakePoint\sqlite3.dll has failed the SafeSEHCheck check.

Impact if not fixed: If the app doesn?t use the available Windows protections, 
it can increase the vulnerability of the customer's computer to malware.

How to fix: Apply the SAFESEH, DYNAMICBASE, and NXCOMPAT options when you link 
the app. See link below for more information:
[ Fixing Binary Analyzer Errors ]( 
http://go.microsoft.com/fwlink/?LinkId=247433 )


> "Richard Hipp"  |



> On 4/2/15, maarten.mostert at stakepoint.com
>  wrote:
> >
> > Hi Everyone,
> >
> > I am new here, and have a question.
> >
> > For my application that relies on Sqlite I am seeking Windows 8
> > certification in accordance with the following article.
> >
> > [ https://msdn.microsoft.com/en-us/windows/desktop/hh749939.aspx ](
> > https://msdn.microsoft.com/en-us/windows/desktop/hh749939.aspx )
> >
> > To verify its compliance I run the installer against The Windows
> > Certification KIT 3.4.
> >
> > This indicates that the SQLite dll I use fails the above mentioned test.
> 
> The links you provide mention dozens of different tests. Which ones
> are failing?
> 
> 
> >
> > Is there any SQLite binary out there that complies with the security mesures
> > MS imposes for desktop app certification.
> >
> > Best regards,
> >
> > Maarten MOSTERT
> >
> >
> >
> > 28 Av Alphonse Denis
> > 83400 Hy?res, France
> > +33 676411296
> > [ http://stakepoint.com/ ]( http://stakepoint.com/ )
> >
> > ___
> > sqlite-users mailing list
> > sqlite-users at mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
> 
> 
> --
> D. Richard Hipp
> drh at sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> 

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Igor Korot]

HI,
If I may...

On Thu, Apr 2, 2015 at 4:25 PM, Richard Hipp  wrote:
> On 4/2/15, Random Coder  wrote:
>>
>> I'd recommend the SQLite team turn them on for the version of the DLL
>> they distribute, but I'm honestly not sure if there are negative side
>> effects to doing so.
>
> That's not possible, unfortunately,   We compile the published DLL
> (the 32-bit DLL at least) with MinGW so that they will work on older
> versions of Windows.  If we used a recent MSVC then the resulting DLL
> will not work on WinXP, I am told.

That's not true.
Recent version of MSVC has the option which allows it to build the binary
that will be compatible with the older version of Windows. And this is
for MSVC 2012/13.
MSVC 2010 still able to build WinXP binary just fine.

Unfortunately I don't have the reference at hand. Sorry.

>
> If the published DLL is not to your liking, it is simple to make your
> own.  I suggest you do so. You can start with the amalgamated source
> code file "sqlite3.c".  All you need is to compile that one file into
> a DLL that the security checker approves of.  How hard can that be?

Or even better - include this file in the project solution and build the binary
as statically linked with all required options.
That's what statical linking means.

>
> --
> D. Richard Hipp
> drh at sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Keith Medcalf]

BTW, I have verified that these options all work as described and the options 
are recognized and processed properly by Windows, and that BinScope is happy:

Failed checks
d:\source\sqlite\sqlite3.dll - SafeSEHCheck ( FAIL ) 

Passed checks
d:\source\sqlite\sqlite3.dll - NXCheck ( PASS ) 
d:\source\sqlite\sqlite3.dll - DBCheck ( PASS ) 

(Note, position independent code (PIC) is by definition loadable at any base.  
Microsoft is just several decades behind in generating position independent 
code.)

---
Theory is when you know everything but nothing works.  Practice is when 
everything works but no one knows why.  Sometimes theory and practice are 
combined:  nothing works and no one knows why.


>-Original Message-
>From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-
>bounces at mailinglists.sqlite.org] On Behalf Of Keith Medcalf
>Sent: Thursday, 2 April, 2015 17:43
>To: General Discussion of SQLite Database
>Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck -
>NXCheck - DBCheck
>
>
>add the following linker options with MinGW:
>
>-static-libgcc -Wl,-Bstatic,--nxcompat,--dynamicbase,--export-all-symbols
>
>You may or may not need -static-libgcc or the linker -Bstatic options
>unless you are also enabling things that require MingW DLL runtime
>support (such as using the -mthreads option to make the DLL thread
>exception safe).
>
>-Wl,--nxcompat,--dynamicbase,--export-all-symbols,--large-address-aware
>
>Sets the NXCOMPAT and DYNAMICBASE (ASLR) flags on the output of the
>linker (so you can add them to the shell tool and other pre-built
>executables as well).  Files also need to export the symbol table (hence
>the --export-all-symbols).  --large-address-aware sets the (you guessed
>it) large address aware flag which basically means that you are using
>unsigned pointers and not signed pointers (that is, you understand that
>neither disk space nor memory comes in negative sizes) and don't diddle
>with your pointers (eg, assigning magical meaning to bits in the
>pointer).  This allows use of more than 2GB of virtual address space per
>process on systems that support it (ie, 32-bit userland on a 64-bit
>Windows -- on 64-bit Windows windows does not co-habitate with 32-bit
>processes, instead a wee trampoline is installed in the top 100MB or so
>of the virtual space to thunk back and forth to the OS -- making it
>impossible to insert code into the OS from a 32-bit process, while 64-bit
>processes on 64
> -bit windows have full and complete access to the OS as it resides in
>each processes address space) or if the /3GB boot option is set on 32-bit
>OSes (so the OS is forced to shoehorn into the top 1GB of each process).
>
>The only thing this does not fix is the Exception Handlers.  But that
>particular protection is not implemented by gcc (not needed because of
>non-ill-conceived design?).  However, you might still want to add -
>mthreads and the static linking options to the build so that the GCC
>threadsafe exception handlers are used.
>
>---
>Theory is when you know everything but nothing works.  Practice is when
>everything works but no one knows why.  Sometimes theory and practice are
>combined:  nothing works and no one knows why.
>
>>-Original Message-
>>From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-
>>bounces at mailinglists.sqlite.org] On Behalf Of Richard Hipp
>>Sent: Thursday, 2 April, 2015 14:25
>>To: General Discussion of SQLite Database
>>Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck -
>>NXCheck - DBCheck
>>
>>On 4/2/15, Random Coder  wrote:
>>>
>>> I'd recommend the SQLite team turn them on for the version of the DLL
>>> they distribute, but I'm honestly not sure if there are negative side
>>> effects to doing so.
>>
>>That's not possible, unfortunately,   We compile the published DLL
>>(the 32-bit DLL at least) with MinGW so that they will work on older
>>versions of Windows.  If we used a recent MSVC then the resulting DLL
>>will not work on WinXP, I am told.
>>
>>If the published DLL is not to your liking, it is simple to make your
>>own.  I suggest you do so. You can start with the amalgamated source
>>code file "sqlite3.c".  All you need is to compile that one file into
>>a DLL that the security checker approves of.  How hard can that be?
>>
>>--
>>D. Richard Hipp
>>drh at sqlite.org
>>___
>>sqlite-users mailing list
>>sqlite-users at mailinglists.sqlite.org
>>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
>___
>sqlite-users mailing list
>sqlite-users at mailinglists.sqlite.org
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Keith Medcalf]

add the following linker options with MinGW:

-static-libgcc -Wl,-Bstatic,--nxcompat,--dynamicbase,--export-all-symbols

You may or may not need -static-libgcc or the linker -Bstatic options unless 
you are also enabling things that require MingW DLL runtime support (such as 
using the -mthreads option to make the DLL thread exception safe).

-Wl,--nxcompat,--dynamicbase,--export-all-symbols,--large-address-aware

Sets the NXCOMPAT and DYNAMICBASE (ASLR) flags on the output of the linker (so 
you can add them to the shell tool and other pre-built executables as well).  
Files also need to export the symbol table (hence the --export-all-symbols).  
--large-address-aware sets the (you guessed it) large address aware flag which 
basically means that you are using unsigned pointers and not signed pointers 
(that is, you understand that neither disk space nor memory comes in negative 
sizes) and don't diddle with your pointers (eg, assigning magical meaning to 
bits in the pointer).  This allows use of more than 2GB of virtual address 
space per process on systems that support it (ie, 32-bit userland on a 64-bit 
Windows -- on 64-bit Windows windows does not co-habitate with 32-bit 
processes, instead a wee trampoline is installed in the top 100MB or so of the 
virtual space to thunk back and forth to the OS -- making it impossible to 
insert code into the OS from a 32-bit process, while 64-bit processes on 64-bit 
windows have full and complete access to the OS as it resides in each processes 
address space) or if the /3GB boot option is set on 32-bit OSes (so the OS is 
forced to shoehorn into the top 1GB of each process).

The only thing this does not fix is the Exception Handlers.  But that 
particular protection is not implemented by gcc (not needed because of 
non-ill-conceived design?).  However, you might still want to add -mthreads and 
the static linking options to the build so that the GCC threadsafe exception 
handlers are used.

---
Theory is when you know everything but nothing works.  Practice is when 
everything works but no one knows why.  Sometimes theory and practice are 
combined:  nothing works and no one knows why.

>-Original Message-
>From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-
>bounces at mailinglists.sqlite.org] On Behalf Of Richard Hipp
>Sent: Thursday, 2 April, 2015 14:25
>To: General Discussion of SQLite Database
>Subject: Re: [sqlite] Windows 8.x security requirements / SafeSEHCheck -
>NXCheck - DBCheck
>
>On 4/2/15, Random Coder  wrote:
>>
>> I'd recommend the SQLite team turn them on for the version of the DLL
>> they distribute, but I'm honestly not sure if there are negative side
>> effects to doing so.
>
>That's not possible, unfortunately,   We compile the published DLL
>(the 32-bit DLL at least) with MinGW so that they will work on older
>versions of Windows.  If we used a recent MSVC then the resulting DLL
>will not work on WinXP, I am told.
>
>If the published DLL is not to your liking, it is simple to make your
>own.  I suggest you do so. You can start with the amalgamated source
>code file "sqlite3.c".  All you need is to compile that one file into
>a DLL that the security checker approves of.  How hard can that be?
>
>--
>D. Richard Hipp
>drh at sqlite.org
>___
>sqlite-users mailing list
>sqlite-users at mailinglists.sqlite.org
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[maarten.most...@stakepoint.com]

Hi Everyone,

I am new here, and have a question.

For my application that relies on Sqlite I am seeking Windows 8 certification 
in accordance with the following article.

[ https://msdn.microsoft.com/en-us/windows/desktop/hh749939.aspx ]( 
https://msdn.microsoft.com/en-us/windows/desktop/hh749939.aspx )

To verify its compliance I run the installer against The Windows Certification 
KIT 3.4.

This indicates that the SQLite dll I use fails the above mentioned test.

Is there any SQLite binary out there that complies with the security mesures MS 
imposes for desktop app certification.

Best regards,

Maarten MOSTERT



28 Av Alphonse Denis
83400 Hy?res, France
+33 676411296 
[ http://stakepoint.com/ ]( http://stakepoint.com/ )

[sqlite] Version 3.8.9 is in testing

[Jan Nijtmans]

2015-04-02 15:15 GMT+02:00 Richard Hipp :
> A draft of the 3.8.9 website is found at http://www.sqlite.org/draft/
> and in particular the change log can be seen at
> https://www.sqlite.org/draft/releaselog/3_8_9.html

I would have expected to find this change somewhere in the
release log:



Regards,
Jan Nijtmans

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Richard Hipp]

On 4/2/15, Random Coder  wrote:
>
> I'd recommend the SQLite team turn them on for the version of the DLL
> they distribute, but I'm honestly not sure if there are negative side
> effects to doing so.

That's not possible, unfortunately,   We compile the published DLL
(the 32-bit DLL at least) with MinGW so that they will work on older
versions of Windows.  If we used a recent MSVC then the resulting DLL
will not work on WinXP, I am told.

If the published DLL is not to your liking, it is simple to make your
own.  I suggest you do so. You can start with the amalgamated source
code file "sqlite3.c".  All you need is to compile that one file into
a DLL that the security checker approves of.  How hard can that be?

-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Did A Recent Windows & Update Affect SQLite???

[Kees Nuyt]

On Wed, 1 Apr 2015 14:26:15 -0700, Mark Romero  wrote:

> Thank you all for all your help.
>
> I have been soliciting advice from all sorts of Lightroom forums / adobe
> forums / Dell computer forums and other forums more specific to hardware
> and Lightroom, and they all ended up in a dead end.
>
> But seeing as though it DOESN'T look like a recent Win 7 update has
> affected SQLite performance, I guess I have reached a dead end, too. (You
> were my last remaining hope, guys!!!)
>
> Anyway, thanks all for all your help and suggestions. Looks like I might
> just have to stick with Adobe Camera Raw and Photoshop for now until I can
> get a new computer.

Other things you can try:

* Reduce the number of files in directories (folders) 
  with images  that are under control of Lightroom

* If possible, disable automatic (re-)cataloging in
  Lightroom

* Exclude directories with sqlite databases / 
  Lightroom catalogs from all viruscanners. 
  If that is not possible, (temporarily) disable
  all real-time virusscanners


-- 
Regards,

Kees Nuyt

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Richard Hipp]

On 4/2/15, maarten.mostert at stakepoint.com
 wrote:
>
> WARNING
> Binary analyzer
> Warning: The binary analyzer test detected the following errors:
> File C:\Program Files\StakePoint\sqlite3.dll has failed the NXCheck check.
> File C:\Program Files\StakePoint\sqlite3.dll has failed the DBCheck check.
> File C:\Program Files\StakePoint\sqlite3.dll has failed the SafeSEHCheck
> check.
>

I have no idea what the above means.

Can you work around it by statically linking SQLite into your
application, rather than using a separate DLL?
-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Random Coder]

On Thu, Apr 2, 2015 at 10:58 AM,   wrote:
>
> This are basically compiler switches as far as I understand.
>
> I am not a C programmer, so also quite difficult for me to link the library 
> statically.

These tests basically map to the /SAFESEH, /DYNAMICBASE, and /NXCOMPAT
linker flags.  Though, as I understand it, passing these tests is
recommended, and not required for Windows 8 certification.

I have a dump of a build of sqlite from a recent internal build.  I'm
not connected with the Sqlite project, I just use it, and happen to
turn on these flags in my build environment for other reasons.

https://bucket-1f8c9d86.s3.amazonaws.com/sqlite3.zip

You can either use my DLL, or build your own version of the DLL with
these options enabled.

I'd recommend the SQLite team turn them on for the version of the DLL
they distribute, but I'm honestly not sure if there are negative side
effects to doing so.

[sqlite] Interrupt SQL query in SHELL

[Gert Van Assche]

Ryan, Simon, thank you both for the help.
The script works fine, and I now also understand how to use this bail
mechanism. It is great!

once more: THANK YOU!

gert

2015-04-01 23:17 GMT+02:00 Simon Slavin :

>
> On 1 Apr 2015, at 8:09pm, Gert Van Assche  wrote:
>
> > But this is definitely not the right way to do it. If I understand the
> doc
> > correctly, it should be an expression, but I don't see how I can do
> this...
>
> You can trigger a bail by causing any SQL error.  For example, inserting a
> duplicate value in a column which is declared UNIQUE, maybe a primary key.
> Or inserting a NULL value in a column defined as NOT NULL.  Or failing any
> other constraint check.  So you may be able to use CASE or a calculation to
> arrange that under the condition you're watching, one of these happens.
>
> You may be able to check the value returned by sqlite3 as it exits to see
> whether it bailed or exited normally.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] Windows 8.x security requirements / SafeSEHCheck - NXCheck - DBCheck

[Richard Hipp]

On 4/2/15, maarten.mostert at stakepoint.com
 wrote:
>
> Hi Everyone,
>
> I am new here, and have a question.
>
> For my application that relies on Sqlite I am seeking Windows 8
> certification in accordance with the following article.
>
> [ https://msdn.microsoft.com/en-us/windows/desktop/hh749939.aspx ](
> https://msdn.microsoft.com/en-us/windows/desktop/hh749939.aspx )
>
> To verify its compliance I run the installer against The Windows
> Certification KIT 3.4.
>
> This indicates that the SQLite dll I use fails the above mentioned test.

The links you provide mention dozens of different tests.  Which ones
are failing?


>
> Is there any SQLite binary out there that complies with the security mesures
> MS imposes for desktop app certification.
>
> Best regards,
>
> Maarten MOSTERT
>
>
>
> 28 Av Alphonse Denis
> 83400 Hy?res, France
> +33 676411296
> [ http://stakepoint.com/ ]( http://stakepoint.com/ )
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] windows network and wal mode

[Clemens Ladisch]

Zaumseil Ren? wrote:
> I have a separate table for each parameter with time stamp and value.
> The time stamp is used as "integer primary key asc".
> [...]
> It is also possible to go back in time and then start from there.
> Currently I remove all values from the tables after the specified time.
> New values can be added after this.
> With larger backtrack intervals I have performance problems.
>
> Can I speed up the deletion?

How exactly are you doing the deletion?
Are you using a single transaction?

> I will only read the data with a second program on another computer.
> I have read it is not possible to use sqlite in wal mode over a network.
> Is it so in my scenario?

If "on another computer" means that you are using a network file system
(and not copying the DB files to the other computer's local disk), then
WAL will not work correctly.

> Would it be possible if I can stop the writing for the duration of the
> read?

Updates of memory-mapped files are not guaranteed to be seen on remote
computers (this might corrupt data).  But if everything happens on
a single computer, it should work.

Using WAL mode appears not to be a good idea, but I guess you need the
read/write concurrency.  In that case, consider using a client/server
database.

> Diese Nachricht (inkl. Anhaenge) beinhaltet moeglicherweise vertrauliche ...

This e-mail contains public information intended for any subscriber of
this mailing list and for anybody else who bothers to read it; it will
be copied, disclosed and distributed to the public.  If you think you
are not the intended recipient, please commit suicide immediately.
These terms apply also to any e-mails quoted in, referenced from, or
answering this e-mail, and supersede any confidentiality notices in
those e-mails.  Additionally, confidentiality notices in those e-mails
will incur legal processing fees of $42 per line; you have agreed to
this by reading this confidentiality notice.


Regards,
Clemens

[sqlite] Did A Recent Windows & Update Affect SQLite???

[Mark Romero]

Thank you Kess for the suggestions:

Number of photos per directory: I tried this but it seems to have had no
benefit. i went from 200 photos in one directory down to 50 photos in four
directories, and there was no change.

I will try your other suggestions (disable automatic re-catalogging and
disable virus scanning of folders).

BTW: I have tried running lightroom in safe mode and the problem still
occurred.

On Thu, Apr 2, 2015 at 5:56 AM, Kees Nuyt  wrote:

> On Wed, 1 Apr 2015 14:26:15 -0700, Mark Romero 
> wrote:
>
> > Thank you all for all your help.
> >
> > I have been soliciting advice from all sorts of Lightroom forums / adobe
> > forums / Dell computer forums and other forums more specific to hardware
> > and Lightroom, and they all ended up in a dead end.
> >
> > But seeing as though it DOESN'T look like a recent Win 7 update has
> > affected SQLite performance, I guess I have reached a dead end, too. (You
> > were my last remaining hope, guys!!!)
> >
> > Anyway, thanks all for all your help and suggestions. Looks like I might
> > just have to stick with Adobe Camera Raw and Photoshop for now until I
> can
> > get a new computer.
>
> Other things you can try:
>
> * Reduce the number of files in directories (folders)
>   with images  that are under control of Lightroom
>
> * If possible, disable automatic (re-)cataloging in
>   Lightroom
>
> * Exclude directories with sqlite databases /
>   Lightroom catalogs from all viruscanners.
>   If that is not possible, (temporarily) disable
>   all real-time virusscanners
>
>
> --
> Regards,
>
> Kees Nuyt
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] Version 3.8.9 is in testing

[Richard Hipp]

SQLite version 3.8.9 is now in testing.  When the status board at
https://www.sqlite.org/checklists/3080900/index goes all green, we
will cut the release.  If you have any concerns about the current
code, please speak up soon!

A draft of the 3.8.9 website is found at http://www.sqlite.org/draft/
and in particular the change log can be seen at
https://www.sqlite.org/draft/releaselog/3_8_9.html

-- 
D. Richard Hipp
drh at sqlite.org

[sqlite] windows network and wal mode

[Hick Gunter]

Ad 1)

You may be able to speed up deletion if you can partition your logging tables 
by time, e.g. each table holds the changes within a certain time frame or a 
fixed number of changes.
When you jump back in time, dropping the tables created after the target 
timestamp is faster than deleting the individual records.
This will, however, require that your application handle reading data from a 
sequence of files.

Ad 2)

Network filesystem locking is notoriously broken. SQLite depends on file locks 
being reliable.

-Urspr?ngliche Nachricht-
Von: Zaumseil Ren? [mailto:RZaumseil at kkg.ch]
Gesendet: Donnerstag, 02. April 2015 07:44
An: 'sqlite-users at mailinglists.sqlite.org'
Betreff: [sqlite] windows network and wal mode

Hi there,



I have used sqlite to store parameter changes with a time stamp in a simulation 
environment.



Because of the number of parameters (2000+) and the max. change rate up to 50ms 
I have a separate table for each parameter with time stamp and value.

The time stamp is used as "integer primary key asc".

So is no need of an internal rowid.



The database is opened from one process in wal mode with the following settings:

PRAGMA synchronous = 1

PRAGMA journal_mode = WAL

PRAGMA wal_autocheckpoint = 1

PRAGMA page_size = 4096

PRAGMA cache_size = 5000



Only one process will write to the database.



It is also possible to go back in time and then start from there.

Currently I remove all values from the tables after the specified time.

New values can be added after this.

With larger backtrack intervals I have performance problems.



Question 1:



Can I speed up the deletion?



Question 2:



I will only read the data with a second program on another computer.

I have read it is not possible to use sqlite in wal mode over a network.

Is it so in my scenario?

Would it be possible if I can stop the writing for the duration of the read?

Is there another solution?





Thank you

Rene


Kernkraftwerk Goesgen-Daeniken AG
CH-4658 Daeniken, Switzerland

Diese Nachricht (inkl. Anhaenge) beinhaltet moeglicherweise vertrauliche oder 
gesetzlich geschuetzte Daten oder Informationen. Zum Empfang derselben ist 
(sind) ausschliesslich die genannte(n) Person(en) bestimmt. Falls Sie diese 
Nachricht irrtuemlicherweise erreicht hat, sind Sie hoeflich gebeten, diese 
unter Ausschluss jeder Reproduktion zu vernichten und den Absender umgehend zu 
benachrichtigen. Besten Dank.
___
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: hick at scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.

[sqlite] Did A Recent Windows & Update Affect SQLite???

[Michael Stephenson]

Some suggestions:

1) Check your temp folder and remove chaff.

2) Download CCleaner and check your registry integrity.

3) Review updates that have been applied and try removing them one at a time
or in a targeted fashion.  You should be able to see what updates were
applied around the time you think Lightroom went bad.  (At the start menu
type Update in the search box and select "View installed updates").

4) Have a look at Lightroom using Process Monitor and perhaps Performance
Monitor.  Maybe the process has a high number of threads, or files in use,
or handles or something like that.

4) Install Windows Performance Toolkit and try xperf or Windows Performance
Analyzer to profile Lightroom and try to see what is taking up time.

5) Turn off Avast shields for a while and see if that makes a difference.

6) Stop/Disable the Windows Search service temporarily and see if that makes
a difference.

-Original Message-
From: sqlite-users-boun...@mailinglists.sqlite.org
[mailto:sqlite-users-bounces at mailinglists.sqlite.org] On Behalf Of Mark
Romero
Sent: Wednesday, April 1, 2015 3:17 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Did A Recent Windows & Update Affect SQLite???

Hello everyone and thanks for your replies (I am new to using a mailing list
so I hope that everyone gets this response).

- Viruses and Antivirus software:

I use Avast in real time and I have scanned my system several times and it
came up clean.

I installed malwarebytes shortly after Lightroom began having problems, and
the scans came up clean, too. (I have since uninstalled malware bytes.)

Have not tried any of the native Microsoft virus tools.

- Software Updates:

I am using the latest version of Lightroom (5.7.1)

I am using Win 7 Pro 64-bit Service Pack 1 and I am pretty sure it is up to
date since I do automatic updates with it.

- CPU Usage:

In resource monitor and task manager, it is ONLY lightroom that is using up
all the CPU and is "spiking" the cpu (it repeatedly and rhythmically goes
from 50% use to 100% use over and over again. It spikes approximately every
5 seconds or so.

No other program or process is using an abnormally large percentage of CPU
(or memory)

Please also note that when Lightroom IS spiking the CPU, I am still able to
open other programs (like Photoshop, which is something of a resource hog),
and work in those programs, WHILE Lightroom is spiking the CPU. To rephrase
this, when lightroom stops responding I can go ahead and open photoshop and
work on a different photo, with only minimal lag compared to when Lightroom
is not spiking the CPU.

- Hard Drive (and other hardware):

I have used the Dell diagnostic tests and the Western Digital tests and they
have all been fine for the hard drive as well as ALL other hardware.

- Disk Performance / File corruption

Yes, I have seen NTFS errors in the event manager. But they only seem to
happen when Lightroom has stopped responding and I chose to do a
ctrl-alt-del instead of doing a ctrl-q and waiting for the program to
eventually close. (I know that ctrl-alt-delete is bad... will try to avoid
again!)

I have done another chkdsk and it came out fine, and I did a chkntfs and it
came out saying C: is not dirty.

- Tech Support from Adobe:

I have had THREE separate Adobe technicians login remotely to my computer
and make changes to different settings, and unfortunately it has not made a
difference.

Because the program stops responding sporadically, they will make a few
changes and then ask me to try it. If the problem doesn't happen within five
minutes, they end the chat and say, "It's probably ok, let  us know if you
have problems in the future."

Then five minutes after they end their support chat and log out of my
system, it will stop responding again.

The changes they have made include increasing the cache, giving adobe
programs "resource priority" . they also tried to update my drivers but I
don't know if they were able to update or not.

- Reinstallation:

I have deleted and reinstalled Lightroom 5.7.1 once already after the
problem started a few months back, and unfortunately it did not make a
difference.

- Lightroom versus Adobe Camera Raw (ACR)

Someone pointed out that Lightroom is different than ACR, and that is true.
However, I spend most of my time in the Development module of Lightroom,
which is (as far as I understand) extremely similar to ACR.

Just so you know, I can use ACR ALL DAY LONG without any problems.

My understanding is that Lightroom saves info to its SQlite database, while
ACR saves info to a separate file.

One Other Fun Fact:

A lot of people are dismissive of my problem because I have an older
computer (core 2 duo E8400 with 8 gigs DDR3 RAM).

"Just buy an i7" they like to chant.

However, Lightroom worked fine for over 8 months until just about two months
ago, when the problems started, and...

My sons computer has the same problem, which developed at the same time.
Again, worked fine for 8 months, 

[sqlite] windows network and wal mode

[Zaumseil René]

Hi there,



I have used sqlite to store parameter changes with a time stamp in a simulation 
environment.



Because of the number of parameters (2000+) and the max. change rate up to 50ms 
I have a separate table for each parameter with time stamp and value.

The time stamp is used as "integer primary key asc".

So is no need of an internal rowid.



The database is opened from one process in wal mode with the following settings:

PRAGMA synchronous = 1

PRAGMA journal_mode = WAL

PRAGMA wal_autocheckpoint = 1

PRAGMA page_size = 4096

PRAGMA cache_size = 5000



Only one process will write to the database.



It is also possible to go back in time and then start from there.

Currently I remove all values from the tables after the specified time.

New values can be added after this.

With larger backtrack intervals I have performance problems.



Question 1:



Can I speed up the deletion?



Question 2:



I will only read the data with a second program on another computer.

I have read it is not possible to use sqlite in wal mode over a network.

Is it so in my scenario?

Would it be possible if I can stop the writing for the duration of the read?

Is there another solution?





Thank you

Rene


Kernkraftwerk Goesgen-Daeniken AG
CH-4658 Daeniken, Switzerland

Diese Nachricht (inkl. Anhaenge) beinhaltet moeglicherweise vertrauliche oder 
gesetzlich geschuetzte Daten oder Informationen. Zum Empfang derselben ist 
(sind) ausschliesslich die genannte(n) Person(en) bestimmt. Falls Sie diese 
Nachricht irrtuemlicherweise erreicht hat, sind Sie hoeflich gebeten, diese 
unter Ausschluss jeder Reproduktion zu vernichten und den Absender umgehend zu 
benachrichtigen. Besten Dank.