Re: [sqlite] Open and query sqlite db in a buffer

[Lloyd]

>
> I see Richard already pointed out that you can create a connection to a
> strictly in-memory DB which doesn't touch a file, and then Deserialize to
> it.  Just to emphasize, if you can make that work, it is definitely worth
> it. From my experience, making the VFS will easily sink around 30 hours of
> your time vs. Deserialize and an in-memory DB which will likely take under
> 3 hours to implement.
>
>
Definitely it is the best suited solution as far as i understand.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[Lloyd]

>
>
> Do what you want, but remember you can always create a valid database
> connection using sqlite3_open(":memory:", ) and then do the
> sqlite3_deserialize().
>
>
Thank you Richard, I didn't know that it can be done this way.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[R Smith]

On 2018/05/18 6:27 PM, Lloyd wrote:

Only https://www.sqlite.org/draft/c3ref/deserialize.html comes to mind,
and that's brand new and assumes the buffer comes from the companion
serialize. Not sure it's release in 3.24 yet. Google finds it in the Draft
only. --DD


Deserialize cannot solve my problem as it needs a valid db connection. As
suggested by Ryan vfs can solve my problem. Thank you DD


I see Richard already pointed out that you can create a connection to a 
strictly in-memory DB which doesn't touch a file, and then Deserialize 
to it.  Just to emphasize, if you can make that work, it is definitely 
worth it. From my experience, making the VFS will easily sink around 30 
hours of your time vs. Deserialize and an in-memory DB which will likely 
take under 3 hours to implement.


Or maybe I'm just slow.

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[Richard Hipp]

On 5/18/18, Lloyd  wrote:
>
> Deserialize cannot solve my problem as it needs a valid db connection. As
> suggested by Ryan vfs can solve my problem. Thank you DD

Do what you want, but remember you can always create a valid database
connection using sqlite3_open(":memory:", ) and then do the
sqlite3_deserialize().

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[Lloyd]

>
> Only https://www.sqlite.org/draft/c3ref/deserialize.html comes to mind,
> and that's brand new and assumes the buffer comes from the companion
> serialize. Not sure it's release in 3.24 yet. Google finds it in the Draft
> only. --DD


Deserialize cannot solve my problem as it needs a valid db connection. As
suggested by Ryan vfs can solve my problem. Thank you DD
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[Lloyd]

>
>
> But I imagine you had a more specific question in mind - please ask
> exactly what you would like to know, and make sure to include all the
> details what we need to know to be able to give answers or advice that
> somewhat resemble the truth.
>
> (Is the buffer a queried cursor? a part of the file?, the entire file?
> Does it also exist on a disk? If so, why not read the file? is it embedded
> in another file? Can we put the file to disk or is the disk a no-go zone?
> Are you a DB admin, or a programmer? i.e. Are you stuck with someone else's
> data or are you able to program a custom file-system interface (see vfs
> documentation)?  There are many solutions but we don't know which one is
> best, or if perhaps none of them will work, since we do not have all the
> information.)
>
>
The buffer contains the entire file. It doesn't exist on disk as file. It
is inside another file. Not stored continuously. So my application read
these spread data and can make a valid buffer containing the full database.
The database is small. We cannot put the file on to disk. I am a programmer
so talking about using sqlite API or something similar to use from my
application.

Thank you Ryan, I think sqlite VFS can solve my problem.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[Dominique Devienne]

On Fri, May 18, 2018 at 12:02 PM R Smith  wrote:

> On 2018/05/18 11:50 AM, Lloyd wrote:
> > I have a "buffer" containing data read from a file-based sqlite database.
> > Is there any possibility for processing this "buffer" to query the data?
>
> But I imagine you had a more specific question in mind - please ask
> exactly what you would like to know, and make sure to include all the
> details what we need to know to be able to give answers or advice that
> somewhat resemble the truth.
>

I concur.


> (Is the buffer a queried cursor? a part of the file?, the entire file?
> Does it also exist on a disk? If so, why not read the file? is it
> embedded in another file? Can we put the file to disk or is the disk a
> no-go zone? Are you a DB admin, or a programmer? i.e. Are you stuck with
> someone else's data or are you able to program a custom file-system
> interface (see vfs documentation)?  There are many solutions but we
> don't know which one is best, or if perhaps none of them will work,
> since we do not have all the information.)
>

Indeed, I didn't realize the "buffer" could be arbitrary data not the
"SQLite format".
Then I guess you might want to look into virtual tables:
https://www.sqlite.org/vtab.html

Many of us expose in-memory C/C++/etc... data structures as SQLite vtables,
which can then be queried. The learning curve is a little steep, there are
plenty
of examples in the SQLite repo. Maybe that's what you had in mind?

Otherwise, just "insert" the data into an in-memory DB, and query it.
Much simpler than implementing vtable(s), assuming your data does
not change "live", in which case vtables is a better choice IMHO. --DD
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[R Smith]

On 2018/05/18 11:50 AM, Lloyd wrote:

Hi,

I have a "buffer" containing data read from a file-based sqlite database.
Is there any possibility for processing this "buffer" to query the data?


Yes.

But I imagine you had a more specific question in mind - please ask 
exactly what you would like to know, and make sure to include all the 
details what we need to know to be able to give answers or advice that 
somewhat resemble the truth.


(Is the buffer a queried cursor? a part of the file?, the entire file?  
Does it also exist on a disk? If so, why not read the file? is it 
embedded in another file? Can we put the file to disk or is the disk a 
no-go zone? Are you a DB admin, or a programmer? i.e. Are you stuck with 
someone else's data or are you able to program a custom file-system 
interface (see vfs documentation)?  There are many solutions but we 
don't know which one is best, or if perhaps none of them will work, 
since we do not have all the information.)


Cheers,
Ryan.

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Open and query sqlite db in a buffer

[Dominique Devienne]

On Fri, May 18, 2018 at 11:50 AM Lloyd  wrote:

> I have a "buffer" containing data read from a file-based sqlite database.
> Is there any possibility for processing this "buffer" to query the data?
>

Only https://www.sqlite.org/draft/c3ref/deserialize.html comes to mind,
and that's brand new and assumes the buffer comes from the companion
serialize. Not sure it's release in 3.24 yet. Google finds it in the Draft
only. --DD
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Open and query sqlite db in a buffer

[Lloyd]

Hi,

I have a "buffer" containing data read from a file-based sqlite database.
Is there any possibility for processing this "buffer" to query the data?

Thanks,
  Lloyd
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users