On 10/08/2020 22.56, sebb wrote:
On Mon, 10 Aug 2020 at 21:49, Daniel Gruno <[email protected]> wrote:
On 10/08/2020 22.46, sebb wrote:
On Mon, 10 Aug 2020 at 18:51, Daniel Gruno <[email protected]> wrote:
Hi folks,
with the incoming new UI and with ElasticSearch moving far away from
what Pony Mail supports currently, I think it's also time to get started
on a "next generation" of Pony Mail, that supports the new structures
laid out in ES 7.x and above, and while we're at it....I think we should
ditch Lua and use a pure python implementation instead, to increase the
number of potential contributors (and make use of Python's excellent
libraries).
SO, with all that said, I'll be setting up a new repository for this
'next generation' of Pony Mail, so as to not start mixing the old and
the new too much. I think this is justified, as it's a very major shift
from the old code-base, and the two would be internally incompatible
(except for the JSON API, I believe that should remain as is.)
My plan is to:
- create a new repository
- import the UIX code-base into this
- import the tools we have from pony v1, tweak those later on
- get started on a pure python back-end for the UI (I have some
semblance of a prototype that I'm hacking on currently)
- finally, write a migration script for migrating from the old DBs to
the new format.
Before one can even think about migrating a database, there needs to
be a full test suite.
In particular, there need to be exhaustive tests to show that the same
Permalinks will be generated.
I think perhaps you misunderstand me here :)
The database *contents* would be migrated verbatim, just with the new ES
structure, where each doctype in the old setup is its own index with
doctype _doc. The IDs stay the same as they were before.
I realise that.
However, I don't think this can be done without changing the backend
code which is responsible for generating the IDs going forward.
There have already been unplanned changes to the generators which
affect the output.
Such changes need to be caught before they cause compatibility issues.
There have not been changes to what's allowed as a document ID in ES.
'generating the IDs' does not play into this at all, I'm not talking
about re-importing, but copying the database content including document
IDs verbatim, just to a new "directory structure". If something has ID
1234foo in the old database, it would still be 1234foo in the new one,
but it would be present in the index ponymail-mbox instead of ponymail.
thus, the ponymail database (or whatever you have called it) would get
split into several databases instead:
- ponymail-mbox
- ponymail-attachments
- ponymail-sources
- ponymail-sessions
etc etc
With regards,
Daniel.
Comments/feedback is always welcome as usual :)
With regards,
Daniel.
