Humbedooh commented on issue #516:
URL:
https://github.com/apache/incubator-ponymail/issues/516#issuecomment-671483191
This should be fixed now, sorry for the late reply.
This is an automated message from the Apache Git Serv
Humbedooh closed issue #516:
URL: https://github.com/apache/incubator-ponymail/issues/516
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Humbedooh commented on issue #511:
URL:
https://github.com/apache/incubator-ponymail/issues/511#issuecomment-671488125
None of those would be valid according to https://tools.ietf.org/html/rfc2919
However, we might as well be lenient in what we accept.
-
Humbedooh closed issue #511:
URL: https://github.com/apache/incubator-ponymail/issues/511
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
sbp opened a new pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517
This PR adds **DKIM style Ponymail ID generation**.
## Why?
There are a number of existing Ponymail ID generators, two of which are
currently recommended: `full` for a single node, a
sbp commented on a change in pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517#discussion_r468523637
##
File path: tools/generators.py
##
@@ -19,14 +19,137 @@
This file contains the various ID generators for Pony Mail's archivers.
"""
+import bas
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-671903245
Please note that this PR also adds an extra parameter passing the bytes of
the original message to the `compute_updates` and `generator` functions in
`archiver.py` and `ge
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672167623
Thanks very much. This looks very good.
However I think the hash won't be sufficiently unique, given that PonyMail
uses the hash both for the Permalink and for s
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672237119
Isn't that just trading one potential risk (if you will) for another?
If you keep *any* received headers in the permalink ID, you risk losing
reproducibility in pe
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672242055
To expand upon my previous comment:
Email A comes in. It gets "pibbled" to abcdefg1234. A SHA3 digest is
123412341234
Email B comes in, identical to A but w
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672252421
AFAICT, so long as one only takes into account the Received headers that
relate to the hops before arrival at ezmlm, all recipients of the email will be
able to genera
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672255008
They would have the same pibble, but different SHA3 if the SHA3 is done
using the full message source. What headers are/aren't in the source wouldn't
matter, as it w
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672257012
Email A: pibble is abcdefg1234, SHA3 of full message is 123412341234
Email B: pibble is still abcdefg1234, SHA3 of full message is 432143214321
Both have the sam
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672260461
Furthermore, blue-skying here, this could be made backwards compatible with
older databases easily.
For all new sources, store the source document with a pibble fi
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672849648
On closer examination, I see that the DKIM generator has several options as
to how the hash is generated.
This means that the generated hash will depend on which
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-67285
Color me stupid, but...you would manually have to go in and change those
settings to get a different result, would you not?
Whether there are options set or not wo
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672927852
Yes, you would have to change the options.
However if different instances have different settings, then their hashes
won't be the same.
I see the DKIM hash be
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672932799
The optional nonce makes things worse as there are effectively infinite
values it can take.
At least with boolean options it would be possible to generate all the
d
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-672954409
If different instance have different settings, then that is the problem of
the person that set that up, not us, not the generator's fault.
Having options make it e
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673011821
@sebbASF
An 80 bit truncated hash provides 80 bits of preimage resistance, but only
40 bits of collision resistance. In terms of Ponymail, preimage resistance
preve
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673298592
@sbp This sounds like we have two options here then:
1) pibble with 80 bits if nonce is set, 128 bits if no nonce?
2) always use 128 bits for pibbling?
As
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673405041
@sbp
In the case of the nonce, does the additional security rely on using a
variable nonce, or would a fixed nonce be sufficient?
==
Not all emails in
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673408683
AIUI, the destination list-id (not the one in the origin, which may not
exist) is appended in the generator with:
~~~python
headers.append([b"X-Archive-
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673462395
I mean that the Permalink should include the list id, as it does at present.
For example: aabbcc@
Thi
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673468896
@sebbASF
One nonce can be used for all messages archived by a host, but it must never
be disclosed. It is more accurately called a
[pepper](https://en.wikipedia.or
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673476509
I don't think the ID should include the list name by default, I like it
short and neat - makes life easier for people using links :)
It could perhaps be an option
Humbedooh edited a comment on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673476509
I don't think the ID should include the list name by default, I like it
short and neat - makes life easier for people using links :)
It could perhaps be an
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673483655
@Humbedooh
Collision forgery would require control over entire input messages, unless
the source identifier algorithm uses a subset. It also does not enable attacks
Humbedooh opened a new pull request #518:
URL: https://github.com/apache/incubator-ponymail/pull/518
Additionally, be more specific in our format=flowed search
This is an automated message from the Apache Git Service.
To resp
sebbASF edited a comment on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-673462395
I mean that the Permalink should include the list id, as it does at present.
For example: aabbcc@
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-674584890
If an externally provided list-id is included in the hash, then the has
will change if the lid changes.
Suppose there is an mbox to be imported.
If the individua
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-674709117
Arguably, if you use a custom list ID different from what's in the source,
you are going to potentially 404 your permalink in any case if you change it or
forget wha
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-674759668
"re-importing from gmail mbox sources, that would not work" - why not?
It would certainly work with the mod_mbox software, as that relies on an
intrinsic part of
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-675058525
gmail does some (nasty) normalization of header values, such as lower-casing
email addresses, which is not standard practice, so you cannot reliably
generate the sam
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-675075884
In which case maybe dkim should do the same normalisation as GMail to avoid
the issue?
This is an aut
sebbASF commented on pull request #518:
URL:
https://github.com/apache/incubator-ponymail/pull/518#issuecomment-675078245
-1, as this changes the source used for hash generation. This will break
most of the existing generators.
If it is desired to apply the reformatting, it needs t
Humbedooh commented on pull request #518:
URL:
https://github.com/apache/incubator-ponymail/pull/518#issuecomment-675122518
Would it be better to look into rolling our own converter? The codebase
looks very simple.
This is
Humbedooh commented on pull request #518:
URL:
https://github.com/apache/incubator-ponymail/pull/518#issuecomment-675124817
or perhaps the formatflowed.decode() option is better suited instead of
convertToWrapped...
This is
sebbASF commented on pull request #518:
URL:
https://github.com/apache/incubator-ponymail/pull/518#issuecomment-675139928
I think we need to gather some test cases.
There are quite a few instances of format=flowed in the unit test corpus
file users_httpd_apache_org_2020-07.mbox, how
sebbASF opened a new issue #519:
URL: https://github.com/apache/incubator-ponymail/issues/519
The code that parses boundary strings strips <>. This breaks parsing of some
messages, for example the unit test corpus file tomcat-ancient-boundary.mbox
which has the following boundary:
C
sebbASF opened a new issue #520:
URL: https://github.com/apache/incubator-ponymail/issues/520
The archiver supports the keyword parameter dump_dir but does nothing useful
with it.
It should either be dropped or implemented
---
sebbASF commented on issue #488:
URL:
https://github.com/apache/incubator-ponymail/issues/488#issuecomment-677725963
Not a PonyMail bug
This is an automated message from the Apache Git Service.
To respond to the message, ple
sebbASF closed issue #488:
URL: https://github.com/apache/incubator-ponymail/issues/488
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
sebbASF commented on issue #520:
URL:
https://github.com/apache/incubator-ponymail/issues/520#issuecomment-677944354
It looks like the connection between dump_dir and args.dump was lost in
commit 95beb51158b58bcb9fdb1371af7699b72598ac34
---
asfgit closed issue #520:
URL: https://github.com/apache/incubator-ponymail/issues/520
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
sebbASF commented on issue #463:
URL:
https://github.com/apache/incubator-ponymail/issues/463#issuecomment-678799880
I've just found a couple of messages that trigger this behaviour:
http://mailarchive-vm.apache.org/mod_mbox/httpd-dev/200709.mbox/raw/%3c14ee094e0709041258o68a6f771ma
sebbASF commented on issue #463:
URL:
https://github.com/apache/incubator-ponymail/issues/463#issuecomment-679028555
Note: the same MID will be generated each time if the same message is
processed repeatedly during the same test run.
It looks like whatever determines the ordering of Set
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679033012
I am not aware of an RFC with the changes that GMail employs - in other
words, I don't know what they do in addition to lowercasing the sender address,
so I can't re
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679033211
(furthermore, we shouldn't be beholden to proprietary changes outside RFCs)
This is an automated mes
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679033755
I am inclined to merge this PR with the fixes required to do so. We can then
add a longer version of the output ID for those that wish to include the list
ID in the
Humbedooh commented on issue #463:
URL:
https://github.com/apache/incubator-ponymail/issues/463#issuecomment-679047423
Currently, the decoding tries to use all possible character sets found in
the message, not necessarily ones pertaining to the body itself. It also
assumes that the defaul
sebbASF commented on issue #463:
URL:
https://github.com/apache/incubator-ponymail/issues/463#issuecomment-679061819
Note that decode is only invoked if type(body) is bytes, and a successful
decode will change body to a string.
So the decoding uses the first character set only. If that
sebbASF edited a comment on issue #463:
URL:
https://github.com/apache/incubator-ponymail/issues/463#issuecomment-679061819
Note that decode is only invoked if type(body) is bytes, and a successful
decode will change body to a string.
So the decoding uses the first character set only. I
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679067179
Sorry, but I don't think the discussion is yet complete.
We need more reviews by other interested parties.
And we need more tests using the same emails from multi
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679070399
I'm okay with giving it a few more days for others to review, but at some
point we should acknowledge that we have hit the number of reviewers we are
going to get.
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679085054
I've run 2124 tests (1062 different emails) on dkim using two different
sources, and it matches on every single one.
I've added it to the unit tests repo.
--
rbowen commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679135455
I, for one, love having the ID be shorter. The current permalinks are long,
cumbersome, and frequently get broken by linebreaks in mail clients, leading me
to use link
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679215118
I agree that the generator looks very promising.
However, I think it has some issues that need to be solved.
For example, the current implementation includes t
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679278984
If lists are renamed, then the permalinks would change on a re-import in any
case, so I consider the argument moot.
It's not the generator's job to secure against
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679282204
As for `Does the generator work well with mails that don't have list
headers?`, I see no indication that it would make any difference if a header is
present or not.
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679287598
@sbp what are your thoughts on the pibble length? Is it safe as is, does it
need to be longer?
This
rbowen commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679299947
So ... it sounds like the concerns have been considered, addressed, and
tested for?
Like I said, I really look forward to more manageable permalinks.
--
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679952122
@sebbASF
It is not yet documented why the command line list ID would need to be
present in the permalink. Am I right in thinking that the following is the only
use
sebbASF opened a new issue #521:
URL: https://github.com/apache/incubator-ponymail/issues/521
The archiver main() code currently adds an 'archived-at' header to mails
(unless they have one) before they are processed.
This means that generators which use the header in any way may gene
asfgit closed issue #521:
URL: https://github.com/apache/incubator-ponymail/issues/521
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
sebbASF commented on pull request #518:
URL:
https://github.com/apache/incubator-ponymail/pull/518#issuecomment-680227488
I found a few cases where the first convertToWrapped call actually worked.
So I think the solution is to leave the current code exactly as is so id
generation is not
asfgit closed pull request #518:
URL: https://github.com/apache/incubator-ponymail/pull/518
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
sebbASF commented on issue #462:
URL:
https://github.com/apache/incubator-ponymail/issues/462#issuecomment-682592711
Alternative approach is to leave the existing code as it is, and reformat
the body after it has been used by the id generator.
This will only change the displayed message
asfgit closed issue #462:
URL: https://github.com/apache/incubator-ponymail/issues/462
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
sebbASF commented on a change in pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517#discussion_r479368516
##
File path: tools/generators.py
##
@@ -19,14 +19,137 @@
This file contains the various ID generators for Pony Mail's archivers.
"""
+import
asfgit closed issue #244:
URL: https://github.com/apache/incubator-ponymail/issues/244
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
asfgit closed issue #463:
URL: https://github.com/apache/incubator-ponymail/issues/463
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
sebbASF commented on a change in pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517#discussion_r482187709
##
File path: tools/generators.py
##
@@ -19,14 +19,137 @@
This file contains the various ID generators for Pony Mail's archivers.
"""
+import
sebbASF opened a new issue #505:
URL: https://github.com/apache/incubator-ponymail/issues/505
The medium generator was originally committed in
https://gitbox.apache.org/repos/asf?p=incubator-ponymail.git;a=commitdiff;h=aa989610
This was used by lists.a.o (and possibly elsewhere) for
sebbASF closed issue #505:
URL: https://github.com/apache/incubator-ponymail/issues/505
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
sebbASF commented on issue #505:
URL:
https://github.com/apache/incubator-ponymail/issues/505#issuecomment-686614420
Does not appear useful to support the generator, because the mdatestring was
created using localtime.
This
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-686813208
This PR affects various different aspects of generation:
- the fields used to build the hash source
- how many bits of hash are generated
- how the hash is pres
sbp commented on a change in pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517#discussion_r483522248
##
File path: tools/generators.py
##
@@ -19,14 +19,137 @@
This file contains the various ID generators for Pony Mail's archivers.
"""
+import bas
sbp commented on a change in pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517#discussion_r483526359
##
File path: tools/generators.py
##
@@ -19,14 +19,137 @@
This file contains the various ID generators for Pony Mail's archivers.
"""
+import bas
sbp commented on a change in pull request #517:
URL: https://github.com/apache/incubator-ponymail/pull/517#discussion_r483526906
##
File path: tools/generators.py
##
@@ -19,14 +19,137 @@
This file contains the various ID generators for Pony Mail's archivers.
"""
+import bas
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687120076
The following test:
https://github.com/apache/incubator-ponymail-unit-tests/blob/master/yaml/gens-ponymail-dev-1079-1080.yaml
shows that currently the generator p
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687153531
Assuming the emails are the same with all the same elements according to the
DKIM rules, I disagree that it's essential with two IDs - I think they should
be the sam
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687210087
@sebbASF
The test that you linked to also produces equal hashes when modified to use
the current Ponymail recommended `cluster` generator. The modified test is
inc
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687236055
I do not personally recommend the cluster generator.
Its output depends on the parsed message and how attachments are processed.
This is not stable over time.
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687257317
@sebbASF
There are several improvements in the pipeline, including:
* highly modularised code
* simplified algorithms
* better RFC compliance
* a bet
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687288365
Thanks very much.
As already indicated, I think the approach is very good.
There are some areas that *may* need tweaking, for example which headers are
includ
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687645749
Commit dfd18eb contains the improvements [previously
mentioned](https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687257317).
The generator has been
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-687812671
I'm just working my way through the new documentation, which seems very
thorough.
There is a section in dkim_id_test.html which states that a mbox From line
may
sebbASF opened a new issue #522:
URL: https://github.com/apache/incubator-ponymail/issues/522
Body lines in mbox files must not start with the characters 'From ' as the
string identifies the start of the next message.
Unfortunately there is no single standard for how such lines are h
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691010564
We should consider canonicalising 'From ' lines in the message body.
This is because it's not possible to ensure that an email imported from an
mbox file will ha
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691187392
While I like the idea, I think that's moving beyond the scope of this PR -
it sounds like something you could add a flag for, potentially a second
generator that nor
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691190787
Perhaps we could accept this PR and make a dkim_mboxrd or some such
additional generator that addresses this mbox issue.
Leave the DKIM as is (I think it's strictl
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691349802
On further reflection I think it's not worth normalising the From lines
fully.
The most common case would be covered.
However, I think it's vital that we do n
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691460489
I think that would defeat the purpose of shortening the hash, I am not in
favor of it.
Perhaps a compromise could be to again have two generators, a dkim and a
dk
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691469821
Well of course the id should be as short as possible -- but no shorter.
The current dkim_id ignores the lid, so I think the generated hash is not
guaranteed uniq
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691482048
What if it only appends that *if a list override has been used* ?
So, normal use of List-ID in the email header would give you a short pibble,
but using a --lid ov
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691515959
Previous [arguments
against](https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679952122)
including manual List-IDs in generated permalink IDs include:
Humbedooh commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691187392
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
sbp commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691515959
Previous [arguments
against](https://github.com/apache/incubator-ponymail/pull/517#issuecomment-679952122)
including manual List-IDs in generated permalink IDs include:
sebbASF commented on pull request #517:
URL:
https://github.com/apache/incubator-ponymail/pull/517#issuecomment-691010564
This is an automated message from the Apache Git Service.
To respond to the message, please log on to G
1 - 100 of 628 matches
Mail list logo
