[incubator-ponymail-foal] branch master updated: store sources as sha3-256 of themselves, add a permalink reference to the digested doc.

[humbedooh]

This is an automated email from the ASF dual-hosted git repository.

humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git


The following commit(s) were added to refs/heads/master by this push:
     new 8a1baf1  store sources as sha3-256 of themselves, add a permalink 
reference to the digested doc.
8a1baf1 is described below

commit 8a1baf16c64933f1fb7a9d281df8ebec418cf771
Author: Daniel Gruno <humbed...@apache.org>
AuthorDate: Tue Aug 25 20:02:44 2020 +0200

    store sources as sha3-256 of themselves, add a permalink reference to the 
digested doc.
    
    Conform to what archiver.py does.
    
    This allows us to store multiple copies of the same digested email, if
    received more than once (and if the raw data differs).
---
 tools/import-mbox.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/tools/import-mbox.py b/tools/import-mbox.py
index 482a2b8..663e6da 100755
--- a/tools/import-mbox.py
+++ b/tools/import-mbox.py
@@ -38,6 +38,7 @@ from os.path import isdir, isfile, join
 from threading import Lock, Thread
 from urllib.request import urlopen
 
+
 import archiver
 from plugins.elastic import Elastic
 
@@ -208,6 +209,7 @@ class SlurpThread(Thread):
             for key in messages.iterkeys():
                 message = messages.get(key)
                 message_raw = messages.get_bytes(key)
+                sha3 = hashlib.sha3_256(message_raw).hexdigest()
                 # If --filter is set, discard any messages not matching by 
continuing to next email
                 if (
                     fromFilter
@@ -313,9 +315,8 @@ class SlurpThread(Thread):
                     try:  # temporary hack to try and find an encoding issue
                         # needs to be replaced by proper exception handling
                         json_source = {
-                            "mid": json[
-                                "mid"
-                            ],  # needed for bulk-insert only, not needed in 
database
+                            "permalink": json["mid"],
+                            "mid": sha3,
                             "message-id": json["message-id"],
                             "source": archiver.mbox_source(raw_msg),
                         }