Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-22 Thread Keith Medcalf
ber, 2019 12:46 >To: SQLite mailing list >Subject: Re: [sqlite] Theoretical write performance for low-throughput >devices > >>and only one thread accesses that connection at a time. The target >>lacks mmap(), posix file locks, and multiple address spaces, so this >>se

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-22 Thread Keith Medcalf
>and only one thread accesses that connection at a time. The target >lacks mmap(), posix file locks, and multiple address spaces, so this >seemed like the right settings to use. So what happens to the shm file? Is in not normally just an ordinary file that is mmap'ped? What would be the

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-22 Thread Jonathan Brandmeyer
On Mon, Oct 21, 2019 at 5:47 PM Richard Hipp wrote: > > On 10/21/19, Jonathan Brandmeyer wrote: > > > > No significant change. The target filesystem only caches non-aligned > > writes, so there usually isn't anything for it to do on fsync anyway. > > > > OK. I don't have any more ideas at the

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-22 Thread Jonathan Brandmeyer
On Tue, Oct 22, 2019 at 2:03 AM Wout Mertens wrote: > This probably won't change a thing, but I wonder why you wouldn't set the > sqlite page size to 2KB? Hopefully this means lots of aligned writes. At one point, the row blobs were exactly 1024 bytes. This isn't great for 4kB pages, but it is

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-22 Thread Wout Mertens
On Mon, Oct 21, 2019 at 5:28 PM Jonathan Brandmeyer < jbrandme...@planetiq.com> wrote: > I'm working on an embedded system that uses a log-structured > filesystem on raw NAND flash. This is not your typical workstation's > managed flash (SATA/NVMe), or portable managed flash (SD/USB). It's a >

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-22 Thread Rowan Worth
On Mon, 21 Oct 2019 at 23:28, Jonathan Brandmeyer wrote: > Or, how many times is each page written by SQLite for an insert-heavy > test? The answer appears to be "4", but I can only account for two of > those four. > > I'm working on an embedded system that uses a log-structured > filesystem on

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-21 Thread Richard Hipp
On 10/21/19, Jonathan Brandmeyer wrote: > > No significant change. The target filesystem only caches non-aligned > writes, so there usually isn't anything for it to do on fsync anyway. > OK. I don't have any more ideas at the moment, and without access to your code, and your platform, I can't

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-21 Thread Jonathan Brandmeyer
On Mon, Oct 21, 2019 at 5:00 PM Richard Hipp wrote: > > On 10/21/19, Jonathan Brandmeyer wrote: > > On Mon, Oct 21, 2019 at 10:31 AM Richard Hipp wrote: > >> > >> On 10/21/19, Jonathan Brandmeyer wrote: > >> > Or, how many times is each page written by SQLite for an insert-heavy > >> > test?

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-21 Thread Richard Hipp
On 10/21/19, Jonathan Brandmeyer wrote: > On Mon, Oct 21, 2019 at 10:31 AM Richard Hipp wrote: >> >> On 10/21/19, Jonathan Brandmeyer wrote: >> > Or, how many times is each page written by SQLite for an insert-heavy >> > test? The answer appears to be "4", but I can only account for two of >>

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-21 Thread Jonathan Brandmeyer
On Mon, Oct 21, 2019 at 10:31 AM Richard Hipp wrote: > > On 10/21/19, Jonathan Brandmeyer wrote: > > Or, how many times is each page written by SQLite for an insert-heavy > > test? The answer appears to be "4", but I can only account for two of > > those four. > > Are you measuring writes at

Re: [sqlite] Theoretical write performance for low-throughput devices

2019-10-21 Thread Richard Hipp
On 10/21/19, Jonathan Brandmeyer wrote: > Or, how many times is each page written by SQLite for an insert-heavy > test? The answer appears to be "4", but I can only account for two of > those four. Are you measuring writes at the OS-interface layer, or writes at the hardware layer? SQLite

[sqlite] Theoretical write performance for low-throughput devices

2019-10-21 Thread Jonathan Brandmeyer
Or, how many times is each page written by SQLite for an insert-heavy test? The answer appears to be "4", but I can only account for two of those four. I'm working on an embedded system that uses a log-structured filesystem on raw NAND flash. This is not your typical workstation's managed flash