[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

After discussion with Victor by using DM, I decided to provide high-level API 
instead of low-level APIs.

- gdbm.open(filename, snapshots=(foo, bar)) 

will do everything at once.

Regards,
Dong-hee

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Gregory P. Smith]

Gregory P. Smith  added the comment:

The gdbm module's purpose is effectively one of just exposing the underlying C 
library APIs to Python as you said.  

Consider this a +1 in favor of exposing the new APIs in the Python gdbm module.

I'm not concerned about anyone wanting these in older Python versions. It 
really requires a combination of modern software in order to use.  Running on a 
recent kernel version, using a non-default fancy filesystem, and linking 
against a recent gdbm library.  So being a new feature in 3.11 makes sense.  
Anyone satisfying all of the above is highly likely to already also use a 
recent CPython.

If anyone _really_ wants it for older things, they can maintain backport on 
PyPI.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

Wow, long discussion than I expected, I wish that you don't feel uncomfortable 
with my opinion first :)

> The main concern is that it is not clear how to use this feature, and if it 
> is not clear

IMHO, this feature is similar usage level with gdbm.reorganize() API for 
end-user. 

https://docs.python.org/3/library/dbm.html?highlight=gdbm#dbm.gnu.gdbm.reorganize

I already show you how end-user will use this API in msg404065.
So I don't want to explain the usage again.

> For example, can it be used to implement transactions? 

AFAIk this feature is only used for left snapshot files storing
if the user wants to recover when the user needed.(for example, disk is too old 
so it can cause the system is down, or any disaster situation, or unexpected 
system fault)
And snapshot files can be stored anywhere(separated secondary disk, 
remote-mounted disk.)

So if you can ask snapshot is important? From my side *yes*, it can guarantee 
that we can recover the file when we want to.

IMHO using this API is up to the end user's purpose.

> I am not even sure that it is Pythonic,

Hmm, you mean API signature? Python has a long tradition of being a thin 
wrapper to C functions. (gdbm.reorganize() is a good example)

Since gdbm module is the most accessible python client that today Python users 
can use, I think we have to provide this feature since gdbm authors write this 
feature for end-user usage.
if not authors may not expose those APIs through `gdbmtool`.

FYI, gdbmtool is a CLI tool that you can execute basic gdbms operations.
If you installed gdbm 1.21 on your local machine, you can use crash tolerance 
features simply though gdbmtool

The essential of this feature looks simple.
* If you want to left snapshot files for gdbm, please create the gdbm file 
extension format(X flag) and then execute gdbm_failure_atomic.

If you don't feel the same way, I would like to suggest sending a mailing list 
and I may requests this to gdbms author since they request this issue to me 
through a mail and they are also more expert about gdbm more than me.

Thanks for reading.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The main concern is that it is not clear how to use this feature, and if it is 
not clear, it will not be used. I am not even sure that it is Pythonic, because 
I do not know how to use it. For example, can it be used to implement 
transactions? How it works with multithreading and multiprocessing if works at 
all? Does it restore after failure automatically or needs some user's action? 
And how do user can know that some actions are required?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

if you think that this feature is not usable, I will let gdbms maintainers 
write the idea to the Python-dev mailing list if they want to enable this 
feature.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

@serhiy
what is the main concern about this feature?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

So IMHO, those APIs are not that low-level API since they only need to create a 
file with 'x' flag and then calling gdbm_failure_atomic API.

and if the user failed to save the file due to several accidents.
They can easily restore the local file DB by using the latest valid snapshot.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

I've done my PoC in my local environment.

```
import dbm.gnu as dbm
db = dbm.open('x.db', 'nx')
db.gdbm_failure_atomic('even_snapshot.bin', 'odd_snapshot.bin')
for k, v in zip('abcdef', 'ghijkl'):
db[k] = v
db.sync()
db.close()
```

By doing this in local fs both snapshot files are created and it can be used 
for recovery x.db file.

gdbmtool> snapshot even_snapshot.bin odd_snapshot.bin
GDBM_SNAPSHOT_OK: Selected the most recent snapshot.
odd_snapshot.bin: 400 r 1634377177.462498326 6

you can use odd_snapshot.bin as the last successful snapshot file.

>>> import dbm.gnu as dbm
>>> db = dbm.open('odd_snapshot.bin', 'r')
>>> db.keys()
[b'c', b'f', b'a', b'd', b'b', b'e']

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Examples of using the new feature.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

> Please show examples.

Sorry, I don't understand, do you mean caching example or recovery example?
When we use rocksdb, we can get checkpoints files under local file system.
And we can recover the database by using them.
http://rocksdb.org/blog/2015/11/10/use-checkpoints-for-efficient-snapshots.html

If we provide gdbm for similar things, users can recover the database when the 
accident happens, I believe that it sometimes happens.(Power out, hardware 
fault etc..)
The recovering tool itself does not need to be Python, but python client needs 
to have a option to save snapshot, but currently not.

Please read how to recover the database if they have snapshots for gdbm.
https://www.gnu.org.ua/software/gdbm/manual/Crash-recovery.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Please show examples.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

Yeah, I agree with gdbm_latest_snapshot(), this API might be too low-level.
But as I am one of the rocksdb user who directly uses rocksdb API for the 
production platform as caching purposes.
so when I think about my and my team's rocksdb usage, gdbm_failure_atomic() 
itself can be useful if the user needs snapshots if they need to recover for 
the special situation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I am interesting how these gdbm_failure_atomic() and gdbm_latest_snapshot() can 
be used in user code. Some real world examples. They look very low-level and 
requiring an additional boilerplate code to be useful if I understand it 
correctly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

dbm.gdbm_failure_atomic('snapshot_path0', 'snapshot_path1')

-> db.gdbm_failure_atomic('snapshot_path0', 'snapshot_path1')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

Sorry, I don't know my answer is enough.
Or do you have any ideas or the opposite ideas?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

> And what's next?

I may introduce following API which only available to use extension format.
(I am PoC with this)

> dbm.gdbm_failure_atomic('snapshot_path0', 'snapshot_path1')


About gdbm_latest_snapshot(), I am still curious that Python module should 
provide this API so I am still communicating with original authors about this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

And what's next?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

> How would it be used from Python? What are scenarios?

I am preparing PoC, I will share you once done.

>From gdbm 1.21, gdbm provides 2 kinds of format. (Standard format / Extended 
>format.)
To create a gdm format with the extended option, GDBM_NUMSYNC flag needs to be 
supported.
Without this, there is no way to create a database file with extension option 
from Python module.

I am thinking about the following usage.

import dbm.gnu as dbm
db = dbm.open('x.db', 'nx')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

How would it be used from Python? What are scenarios?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Change by Dong-hee Na :


--
keywords: +patch
pull_requests: +27230
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28942

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

See also issue22035.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Change by Dong-hee Na :


--
title: Support crash tolerance for gdbm module -> Support crash tolerance 
feature for gdbm module

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45452] Support crash tolerance feature for gdbm module

[Dong-hee Na]

Dong-hee Na  added the comment:

FYI, I got a mail about this feature from Terence Kelly who design these 
amazing things :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com