[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: If you are ok with the proposed API in GH-30680, I'll mark it as ready for review. -- ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: PR without mapping protocol and context manager submitted: GH-30680. I've also changed the open API to reflect that of apsw. -- ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Change by Erlend E. Aasland : -- pull_requests: +28878 pull_request: https://github.com/python/cpython/pull/30680 ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: (the bpo bot is obviously confused by the News entry path, sigh) -- ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: Slimmed PR diff (excluding clinic), without context manager and mapping protocol: $ git diff main Modules/_sqlite/*.[ch] Lib Doc Misc PC* setup.py Doc/includes/sqlite3/blob.py | 12 +

[issue24905] Allow incremental I/O to blobs in sqlite3

Change by Erlend E. Aasland : -- Removed message: https://bugs.python.org/msg409590 ___ Python tracker ___ ___ Python-bugs-list

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: Slimmed PR diff (excluding clinic), without context manager and mapping protocol: $ git diff main Modules/_sqlite/*.[ch] Lib Doc Misc PC* setup.py Doc/includes/sqlite3/blob.py | 12 +

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: The diff is pretty heavy, here's the diffstat: 17 files changed, 1362 insertions(+), 7 deletions(-) It will be hard to find reviewers for such a large PR. I suggest to remove mapping support and context manager support for now, and then add those

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: I've submitted my changes as a separate PR. I believe we should consider duplicating the apsw API, for users convenience. Pr. now, they are very similar, except for the "open blob" API: apsw uses `blobopen`, we currently use `open_blob`. I suggest

[issue24905] Allow incremental I/O to blobs in sqlite3

Change by Erlend E. Aasland : -- pull_requests: +28570 pull_request: https://github.com/python/cpython/pull/30356 ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Erlend E. Aasland added the comment: FYI, I'm maintaining a rebased version on my fork: https://github.com/erlend-aasland/cpython/tree/sqlite-blob Here's the changes I've applied upon PR 271: - Use Argument Clinic - Convert to heap types - Adopt new CPython C APIs (Py_NewRef, Py_Is*, new

[issue24905] Allow incremental I/O to blobs in sqlite3

Ryan Mast (nightlark) added the comment: It looks like this has a PR that just needs rebasing, then it will be ready for another review. -- nosy: +rmast ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Change by Erlend Egeberg Aasland : -- nosy: +erlendaasland ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue24905] Allow incremental I/O to blobs in sqlite3

Emmanuel Arias added the comment: Hi everybody, I let a comment on github. I was asking about if are there some plans with this PR? Was open 3 years ago and there are not updating Cheers -- nosy: +eamanu ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Change by Serhiy Storchaka : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue24905] Allow incremental I/O to blobs in sqlite3

Holger Waldmann added the comment: I am looking forward for this to be included. My main use case is on restricted IoT devices, where I need to handle BLOBs of 2MB size. As this is larger than my available RAM, I store need to store them in the filesystem today. This is ugly because it is

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: As I wrote in the PR: I think that the contains operation should not be supported for blobs. As blobs can be very large, looking for a subset of bytes inside them will be a very inefficient process in memory or in compute. I believe that

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Pinging. As I mentioned in the PR I need a little help with __contains__. -- ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Just to make sure when you say "sequence protocol" you thin about the doing blob[4:6] = "ab"? I actually think this is a nice feature. The Blob and the mmap object has a lot in common so I think that making the same API will be best. I think that adding the

[issue24905] Allow incremental I/O to blobs in sqlite3

Serhiy Storchaka added the comment: Small discussion is started at pull request [1]. There are doubts about the usefulness of incremental I/O API. SQLite is rarely used for storing blobs of the size of hundreds of megabytes. For smaller blobs it may be enough to read or write all data at

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: I am not a DB expert but from a quick search I couldn't find a similar API. I did find a few API's in other programming languages to sqlite blob: 1. https://jgallagher.github.io/rusqlite/rusqlite/blob/index.html 2.

[issue24905] Allow incremental I/O to blobs in sqlite3

Serhiy Storchaka added the comment: Is there similar API in other databases? -- ___ Python tracker ___ ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: I opened a PR in github. I tagged some other developers if that will not start a discussion on the API I will post in the python-ideas. -- ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Changes by Aviv Palivoda : -- pull_requests: +244 ___ Python tracker ___ ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Uploading patch after fixes from berker CR. The `blob_open` API can can have the following options: 1. The table, column and row must be mandatory parameters. 2. The read/write permissions can have the following options: a. No default (mandatory parameter).

[issue24905] Allow incremental I/O to blobs in sqlite3

Serhiy Storchaka added the comment: apsw has different order of arguments for blobopen(), all arguments are mandatory, and the parameter for read/write mode is called "writeable" rather than "readonly". This part of API needs to be discussed. Ask for discussion on mailing lists: Python-Ideas

[issue24905] Allow incremental I/O to blobs in sqlite3

Changes by Berker Peksag : -- nosy: +berker.peksag ___ Python tracker ___ ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Thanks for the CR Serhiy. Attached is a new patch after the fixes from the CR. What other developers should I ask? The interface is file like and is the same as apsw. -- Added file: http://bugs.python.org/file46531/blob3.patch

[issue24905] Allow incremental I/O to blobs in sqlite3

Serhiy Storchaka added the comment: Sorry Aviv, I just forget about this issue. Added new comments on Rietveld. Many lines in sqlite3.rst still are too long. It would be worth to ask other developers about wanted interface. -- stage: -> patch review versions: +Python 3.7 -Python 3.6

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Pinging again. I think this would be a great enhancement to the sqlite module. -- ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Thanks for the review Serhiy. Attached is the updated patch after the changes. -- Added file: http://bugs.python.org/file44142/blob2.patch ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Serhiy Storchaka added the comment: Added comments on Rietveld. The patch contains typos and violates PEP 7 and PEP 12. And there are questions about API. -- nosy: +serhiy.storchaka ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: Pinging as mentioned in the devguide. -- ___ Python tracker ___ ___

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: I did the needed changes for the pull request at pysqlite for porting to python3. I will continue updating the patch if there will be changes in pysqlite and vice versa. -- keywords: +patch Added file: http://bugs.python.org/file41798/blob.patch

[issue24905] Allow incremental I/O to blobs in sqlite3

Aviv Palivoda added the comment: I opened a pull request for blob support in the pysqlite github repository: https://github.com/ghaering/pysqlite/pull/93 I will do the needed changes for python3 and will post a patch soon. -- ___ Python tracker

[issue24905] Allow incremental I/O to blobs in sqlite3

Changes by Aviv Palivoda : -- nosy: +palaviv ___ Python tracker ___ ___ Python-bugs-list

[issue24905] Allow incremental I/O to blobs in sqlite3

Changes by SilentGhost : -- components: +Extension Modules nosy: +ghaering ___ Python tracker ___

[issue24905] Allow incremental I/O to blobs in sqlite3

New submission from Jim Minter: SQLite supports incremental I/O to blobs, i.e. the capability to stream reads and writes to blobs without having to load the entire blob into memory first. See https://www.sqlite.org/c3ref/blob_open.html for more details on the C API. It'd be nice if it were