Re: Fwd: LMDB and text encoding

2015-02-02 Thread Timur Kristóf
DB names are purely internal to LMDB, so they bear no relation to OS filenames and none of this discussion matters to them. If we let the users treat db names as an MDB_val (essentially, an arbitrary byte array), then all bets are off: we can't even make the assumption that a db name is

Fwd: LMDB and text encoding

2015-02-02 Thread Timur Kristóf
On Mon, Feb 2, 2015 at 3:37 AM, Howard Chu h...@symas.com wrote: Hallvard Breien Furuseth wrote: On 02/02/15 00:40, Howard Chu wrote: It looks OK to me. No one raises any concerns I'll commit it in a few hours. Some sudden last thoughts: mdb_dump.c also has a check (memchr(key.mv_data,

Fwd: LMDB and text encoding

2015-02-02 Thread Timur Kristóf
I just had a look at how BDB handled this. As you can see they used a TO_TSTRING macro to convert incoming pathnames from UTF8 to UTF16. https://gitorious.org/berkeleydb/berkeleydb/source/347d239a1e44ed4f773ae9274c2a32cf2b8999c0:src/os_windows/os_open.c

Re: Fwd: LMDB and text encoding

2015-02-02 Thread Timur Kristóf
DB names are purely internal to LMDB, so they bear no relation to OS filenames and none of this discussion matters to them. They're exposed to the programmer and the program's users. Either may want them on command-line arguments, in config files, etc. It will be inconvenient if LMDB

Re: Fwd: LMDB and text encoding

2015-02-02 Thread Timur Kristóf
I suggest we wait to deal with DB names until we also have a way to deal with filenames. And this time test that it works is practice. And then I also suggest to try to make this mess simple to deal with for programmers and or users. I guess I should have separated that from the rest more

LMDB and text encoding

2015-01-27 Thread Timur Kristóf
regards, Timur Kristóf

Re: LMDB and text encoding

2015-01-29 Thread Timur Kristóf
mdb_dbi_open treats its name parameter as a C string. This means UTF-8 on unixes and ANSI on Windows, which is problematic for cross-platform applications. [...] Here is a patch that addresses this concern. If you like it, I'll move on to the other issue. From

Re: LMDB and text encoding

2015-01-29 Thread Timur Kristóf
Here is a fixed version of the patch. On Thu, Jan 29, 2015 at 10:29 AM, Timur Kristóf timur.kris...@gmail.com wrote: mdb_dbi_open treats its name parameter as a C string. This means UTF-8 on unixes and ANSI on Windows, which is problematic for cross-platform applications. [...] Here

Re: LMDB and text encoding

2015-01-29 Thread Timur Kristóf
I've had a brief chat with Hallvard on IRC. We came up with several possible solutions, although each of them has its drawbacks. Writing cross-platform code that supports unicode is always a messy business. I vote for option 4, but would like to hear everyone's opinions before starting to work on

Re: LMDB and text encoding

2015-02-01 Thread Timur Kristóf
Hi, I forgot to add an ENOMEM check. I added it now. I think this patch is ready for Howard and Hallvard to review. :) Timur On Thu, Jan 29, 2015 at 2:42 PM, Timur Kristóf timur.kris...@gmail.com wrote: Here is a fixed version of the patch. On Thu, Jan 29, 2015 at 10:29 AM, Timur Kristóf

Re: Fwd: LMDB and text encoding

2015-02-15 Thread Timur Kristóf
A path is always a Unicode string, while a DB name can be an arbitrary binary blob. On many POSIX platforms, a path is a blob which does not contain '\000'. These systems do not enforce Unicode encoding at all. My mistake. I was unaware. On those platforms, how do you type a path name

Re: Fwd: LMDB and text encoding

2015-02-15 Thread Timur Kristóf
A path is always a Unicode string, while a DB name can be an arbitrary binary blob. On many POSIX platforms, a path is a blob which does not contain '\000'. These systems do not enforce Unicode encoding at all. My mistake. I was unaware. On those platforms, how do you type a path name

LMDB and gitorious

2015-04-22 Thread Timur Kristóf
Hi, As you might've heard, Gitorious is shutting down on 15th May. Maybe we start should look for a new home for the projects at https://gitorious.org/mdb - do you guys have a plan for this? Cheers, Timur

Re: lmdb: is read operation permitted inside the write transaction?

2015-05-19 Thread Timur Kristóf
On Mon, 2015-05-18 at 23:03 +0200, Hallvard Breien Furuseth wrote: On 12/05/15 19:36, Howard Chu wrote: Dmytro Milinevskyy wrote: Basically I need to have an opportunity to call mdb_get while in write transaction. The sequence is smth like: - txn = mdb_txn_begin(flags=0)

Re: Compiling LMDB with Visual C++

2015-12-23 Thread Timur Kristóf
> > Made some code changes to compile in VC++. Does anyone want? > > We don't really support Microsoft tools. You're welcome to submit > code patches  > against the latest code in mdb.master. MSVC Project files and other > such stuff  > will not be accepted, we have no desire to maintain

Re: LMDB encryption support

2017-08-10 Thread Timur Kristóf
Hi, > I've recently added support for page-level encryption to LMDB 1.x > using user-supplied callbacks That does sound cool. :) > One question is whether we should actually make this pluggable like > this, or > we should just hardcode support for a specific algorithm and leave it > at that.

Re: Fwd: LMDB and text encoding

2017-06-07 Thread Timur Kristóf
Hi Everyone, I've just came accross this old thread and am wondering, is this still an issue? Does LMDB have a way to use non-ASCII path names with mdb_env_open in a cross-platform way? If not, would you guys accept patches to LMDB with this regard? Thanks, Timur