Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Seth Blank]

As Chair, I'm closing ticket #69 and recording consensus as remaining with
XML for report formatting.

This consensus may be revisited if
https://trac.ietf.org/trac/dmarc/ticket/42 opens a new reporting mechanism
for which JSON is optimal, but the group's current consensus is that even
in that case, XML will be sufficient.

On Thu, May 21, 2020 at 3:38 AM Alessandro Vesely  wrote:

> On Wed 20/May/2020 22:00:34 +0200 Hector Santos wrote:
> > On 5/20/2020 2:43 PM, Alessandro Vesely wrote:
> >
> >> I mean, what is the CSV format of the following report, that I sent
> yesterday
> >> for this list:
> >
> > Sorry, if I ignored it.
> >
> > Forgetting fact that you can your report easier to read for consumers,
> these
> > would be an example of the CSV field headers.
> >
> > CSV headers:
> >
> > report_metadata.org_name, report_metadata.email,
> report_metadata.report_id,
> > report_metadata.date_range.begin, report_metadata.date_range.end
> >
> > Policy_Published.domain, Policy_Published.adkim, Policy_Published.aspf,
> > Policy_Published.p
> >
> > record.row, record.row.source_ip.record.count.
> record.row.policy_evaluated,
> > record.row.policy_evaluated.disposition,
> > record.row,policy_evaluated.disposition,
> record.row.policy_evaluated.dkim,
> > record.row.policy_evaluated.spf
> >
> > Note: You don't have to stick to redundant "name space" field names.
>
>
> You didn't include auth_results.  That's tricky because you can have zero
> or
> more dkim and spf results.  That would bring on a variable number of
> columns,
> with no hint about which is which.
>
> As Freddie noted, repeating the headers for every record may sound a
> little bit
> wasteful, but gzip may come to rescue here too.
>
> As for readability, aggregate reports are designed to be machine-readable
> to
> the extreme detriment of any kind of human readability.  The best samples
> of
> such attitude are the date_range elements.  HTML provides for a much better
> human readability.  (Indeed, it is one of the formats you mentioned, as it
> is
> supported by Google Docs).  I attach the HTML equivalent of the data I sent
> yesterday.  Posters whose From: was rewritten can easily spot their row
> —readability meaning exactly that.
>
> Let me note that such htmlization is the result of a DMARC XSLT applied by
> a
> mail filter after rua messages authentication but before delivery.  That
> way, a
> readable format of the reports is ready in the mail folder whenever I'd
> care to
> look at it, *as if it had been written in HTML in the first place*.  By
> similar
> techniques one can obtain JSON, SQL, TXT, ….  In the face of such
> flexibility,
> I'm puzzled by your asking for more.
>
>
>  ...  Can we get back to work, please?
> >>>
> >>> Sorry, but I consider a rude, disrespectful and ignorant statement, to
> be
> >>> saying that.
> >>
> >>
> >> No personal attack intended.  I'm being rude because I have the
> impression that
> >> you are not defending a concrete, well defined need, but instead find
> new
> >> arguments opportunistically to pursue a vague sense of format fashion.
> >
> > That's a personal attack. If you don't understand the proposal, you
> should back
> > off or ask for clarification.
>
>
> I /am/ asking for clarifications.  Please excuse my tone if it hurts you.
> I'll
> try and keep calm, but please restrain from technically flaky arguments
> such as
> CSV.  I think you're perfectly aware of the capabilities I'm exemplifying
> here.
>  They rest on the fact that a report consumer knows in advance the format
> of
> the data inside the received gzip.  Hence, that flexibility would be
> destroyed
> by the introduction of multiple formats, as they would come randomly at the
> mercy of the report generators, any prf= notwithstanding.  Not a great
> loss,
> given your feeling about reporting?
>
>
> >> You shifted from an asserted necessity of producers to a possible desire
> >> of consumers.>
> > I did no such thing. I won't repeat it, but it appears you didn't
> understand
> > the proposal.
>
>
> For sure, I don't understand the proposal.  Let's start over:
>
> On Sat 16/May/2020 19:48:25 +0200 Hector Santos wrote:
> > Just consider, when the spec has XML-only, then for those who use a solid
> > JSON I/O system, they are now going to be required to add XML. So for
> them,
> > its additional development complexity.  Everything they probably do JSON
> > related. The exception would be DMARC using XML. This alone can delay or
> > push aside DMARC Reporting implementation.
>
>
> Does DMARC Reporting implementation mean generation or consumption?
>
>
> On Wed 20/May/2020 02:23:37 +0200 Hector Santos wrote:
> > I suggest that there is a new tag that provides a "Preferred Report
> Format"
> > or "prf=" tag using registered acronymns for long time "standard"
> formats.
> > For example:>
> > prf=cvs,json,xml,afrf,iodef
> >
> > [...]
> >
> > The verifier will do what can it offer. The publisher is providing a
> > preference, that it may not 

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Alessandro Vesely]

On Wed 20/May/2020 22:00:34 +0200 Hector Santos wrote:
> On 5/20/2020 2:43 PM, Alessandro Vesely wrote:
> 
>> I mean, what is the CSV format of the following report, that I sent yesterday
>> for this list:
> 
> Sorry, if I ignored it.
> 
> Forgetting fact that you can your report easier to read for consumers, these
> would be an example of the CSV field headers.
> 
> CSV headers:
> 
> report_metadata.org_name, report_metadata.email, report_metadata.report_id,
> report_metadata.date_range.begin, report_metadata.date_range.end
> 
> Policy_Published.domain, Policy_Published.adkim, Policy_Published.aspf,
> Policy_Published.p
> 
> record.row, record.row.source_ip.record.count. record.row.policy_evaluated,
> record.row.policy_evaluated.disposition,
> record.row,policy_evaluated.disposition, record.row.policy_evaluated.dkim,
> record.row.policy_evaluated.spf
> 
> Note: You don't have to stick to redundant "name space" field names.


You didn't include auth_results.  That's tricky because you can have zero or
more dkim and spf results.  That would bring on a variable number of columns,
with no hint about which is which.

As Freddie noted, repeating the headers for every record may sound a little bit
wasteful, but gzip may come to rescue here too.

As for readability, aggregate reports are designed to be machine-readable to
the extreme detriment of any kind of human readability.  The best samples of
such attitude are the date_range elements.  HTML provides for a much better
human readability.  (Indeed, it is one of the formats you mentioned, as it is
supported by Google Docs).  I attach the HTML equivalent of the data I sent
yesterday.  Posters whose From: was rewritten can easily spot their row
—readability meaning exactly that.

Let me note that such htmlization is the result of a DMARC XSLT applied by a
mail filter after rua messages authentication but before delivery.  That way, a
readable format of the reports is ready in the mail folder whenever I'd care to
look at it, *as if it had been written in HTML in the first place*.  By similar
techniques one can obtain JSON, SQL, TXT, ….  In the face of such flexibility,
I'm puzzled by your asking for more.


 ...  Can we get back to work, please?
>>>
>>> Sorry, but I consider a rude, disrespectful and ignorant statement, to be
>>> saying that.
>>
>>
>> No personal attack intended.  I'm being rude because I have the impression 
>> that
>> you are not defending a concrete, well defined need, but instead find new
>> arguments opportunistically to pursue a vague sense of format fashion.
> 
> That's a personal attack. If you don't understand the proposal, you should 
> back
> off or ask for clarification.


I /am/ asking for clarifications.  Please excuse my tone if it hurts you.  I'll
try and keep calm, but please restrain from technically flaky arguments such as
CSV.  I think you're perfectly aware of the capabilities I'm exemplifying here.
 They rest on the fact that a report consumer knows in advance the format of
the data inside the received gzip.  Hence, that flexibility would be destroyed
by the introduction of multiple formats, as they would come randomly at the
mercy of the report generators, any prf= notwithstanding.  Not a great loss,
given your feeling about reporting?


>> You shifted from an asserted necessity of producers to a possible desire
>> of consumers.>
> I did no such thing. I won't repeat it, but it appears you didn't understand
> the proposal.


For sure, I don't understand the proposal.  Let's start over:

On Sat 16/May/2020 19:48:25 +0200 Hector Santos wrote:
> Just consider, when the spec has XML-only, then for those who use a solid
> JSON I/O system, they are now going to be required to add XML. So for them,
> its additional development complexity.  Everything they probably do JSON
> related. The exception would be DMARC using XML. This alone can delay or
> push aside DMARC Reporting implementation.


Does DMARC Reporting implementation mean generation or consumption?


On Wed 20/May/2020 02:23:37 +0200 Hector Santos wrote:
> I suggest that there is a new tag that provides a "Preferred Report Format"
> or "prf=" tag using registered acronymns for long time "standard" formats.
> For example:>
> prf=cvs,json,xml,afrf,iodef
>
> [...]
> 
> The verifier will do what can it offer. The publisher is providing a
> preference, that it may not get.


That's 100% uncertainty of the result, isn't it?  How is flexibility improved?
 I cannot store a received report as-is in Google Docs anyway, can I?


Best
Ale
-- 



































Title: 
Feedback from
tana.it


Feedback from
tana.itId: eb74db90c32d432ea1652a0eb656fe5a; begin: 2020-05-19T00:00:00Z; end: 2020-05-20T00:00:00ZDomain: dmarc.ietf.org; DKIM: relaxed; SPF: relaxed; policy published: none  
			Relaying IP
		
			message count
		
			reason and disposition
		
			From header
			
			(opt. envelope)
		
			

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Freddie Leeman]

Let me get this straight? You want to repeat all the header values on each row 
for each record? How is that better than XML?

CSV is just not a format that we can use here, it just isn't. I bet that there 
are people that would like the reports in D64 or MP3 format, but for obvious 
reasons that is just as unrealistic. Personally I would prefer JSON over XML, 
but I do not see a benefit to having multiple formats, and since we already 
have XML, I suggest we keep it that way.

I don't think 'consumers' would want to read DMARC reports anyway. Reading 
DMARC reports is one thing, understanding them is something else. If a consumer 
is willing to read DMARC reports, and understands them, they would have no 
problem reading XML or converting them to something that fits their needs.

--- Freddie

-Oorspronkelijk bericht-
Van: Hector Santos [mailto:hsan...@isdg.net] 
Verzonden: woensdag 20 mei 2020 22:01
Aan: dmarc@ietf.org
Onderwerp: Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?


On 5/20/2020 2:43 PM, Alessandro Vesely wrote:

> Transaction?  I thought we were talking about aggregate reports.

So am I.

> There are no transactions there.

Really?

Each SMTP session can be considered a transaction. You are provided results on 
the these DMARC processed transaction.

> I mean, what is the CSV format of the following report, that I sent 
> yesterday for this list:

Sorry, if I ignored it.

Forgetting fact that you can your report easier to read for consumers, these 
would be an example of the CSV field headers.

CSV headers:

report_metadata.org_name, report_metadata.email, report_metadata.report_id, 
report_metadata.date_range.begin, report_metadata.date_range.end

Policy_Published.domain, Policy_Published.adkim, Policy_Published.aspf, 
Policy_Published.p

record.row, record.row.source_ip.record.count. 
record.row.policy_evaluated, record.row.policy_evaluated.disposition,
record.row,policy_evaluated.disposition,
record.row.policy_evaluated.dkim, record.row.policy_evaluated.spf

Note: You don't have to stick to redundant "name space" field names.

>>> ...  Can we get back to work, please?
>>
>> Sorry, but I consider a rude, disrespectful and ignorant statement, 
>> to be saying that.
>
>
> No personal attack intended.  I'm being rude because I have the 
> impression that you are not defending a concrete, well defined need, 
> but instead find new arguments opportunistically to pursue a vague sense of 
> format fashion.

That's a personal attack. If you don't understand the proposal, you should back 
off or ask for clarification.

> You
> shifted from an asserted necessity of producers to a possible desire 
> of consumers.

I did no such thing. I won't repeat it, but it appears you didn't understand 
the proposal.

>Now you introduce formats like CSV which make no sense in DMARC  
>context.

I disagree. See above.

> I hold that CSV cannot satisfy DMARC requirements.

Hold it all you want. You know it would not be true. See above.

? Please do contradict me by
> showing us how an aggregate report in CSV would look like, as well as 
> some ideas for defining the corresponding template, similar to what 
> Appendix C of RFC 7489 specifies for XML.

Ok, I won't but if you don't understand the proposal, you should ask for 
clarification.

CSV can work, so can JSON.  Limiting it and locking it down to XML 
only would be a limitation.   You can agree or not agree with that.

--
HLS




___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Seth Blank]

There appears to be broad consensus in the group that there is no need for
JSON output from aggregate reports, except for one argument that the
additional flexibility of allowing JSON would be good for those who work in
Javascript. But I haven't heard any group specifically identified that
would want to use this.

Can anyone raise their hand, or bring someone to the table, who has a
working DMARC implementation that would be improved by a standards track
documents which includes JSON support, or has not implemented DMARC solely
due to lack of JSON support?

Short this, I'll call rough consensus that DMARCbis need not support JSON.

Seth, as Chair


On Wed, May 20, 2020 at 1:06 PM Scott Kitterman 
wrote:

>
>
> On May 20, 2020 8:00:34 PM UTC, Hector Santos  40isdg@dmarc.ietf.org> wrote:
> >
> >On 5/20/2020 2:43 PM, Alessandro Vesely wrote:
> >
> >> Transaction?  I thought we were talking about aggregate reports.
> >
> >So am I.
> >
> >> There are no transactions there.
> >
> >Really?
> >
> >Each SMTP session can be considered a transaction. You are provided
> >results on the these DMARC processed transaction.
> >
> >> I mean, what is the CSV format of the following report, that I sent
> >yesterday
> >> for this list:
> >
> >Sorry, if I ignored it.
> >
> >Forgetting fact that you can your report easier to read for consumers,
> >these would be an example of the CSV field headers.
> >
> >CSV headers:
> >
> >report_metadata.org_name, report_metadata.email,
> >report_metadata.report_id, report_metadata.date_range.begin,
> >report_metadata.date_range.end
> >
> >Policy_Published.domain, Policy_Published.adkim,
> >Policy_Published.aspf, Policy_Published.p
> >
> >record.row, record.row.source_ip.record.count.
> >record.row.policy_evaluated, record.row.policy_evaluated.disposition,
> >record.row,policy_evaluated.disposition,
> >record.row.policy_evaluated.dkim, record.row.policy_evaluated.spf
> >
> >Note: You don't have to stick to redundant "name space" field names.
> >
>  ...  Can we get back to work, please?
> >>>
> >>> Sorry, but I consider a rude, disrespectful and ignorant statement,
> >to be
> >>> saying that.
> >>
> >>
> >> No personal attack intended.  I'm being rude because I have the
> >impression that
> >> you are not defending a concrete, well defined need, but instead find
> >new
> >> arguments opportunistically to pursue a vague sense of format
> >fashion.
> >
> >That's a personal attack. If you don't understand the proposal, you
> >should back off or ask for clarification.
> >
> >> You
> >> shifted from an asserted necessity of producers to a possible desire
> >of
> >> consumers.
> >
> >I did no such thing. I won't repeat it, but it appears you didn't
> >understand the proposal.
> >
> >>Now you introduce formats like CSV which make no sense in DMARC
> >> context.
> >
> >I disagree. See above.
> >
> >> I hold that CSV cannot satisfy DMARC requirements.
> >
> >Hold it all you want. You know it would not be true. See above.
> >
> >? Please do contradict me by
> >> showing us how an aggregate report in CSV would look like, as well as
> >some
> >> ideas for defining the corresponding template, similar to what
> >Appendix C of
> >> RFC 7489 specifies for XML.
> >
> >Ok, I won't but if you don't understand the proposal, you should ask
> >for clarification.
> >
> >CSV can work, so can JSON.  Limiting it and locking it down to XML
> >only would be a limitation.   You can agree or not agree with that.
>
> Not requiring RFC 1149 support is also a limitation, but I'm fine with
> that.
>
> Scott K
>
> ___
> dmarc mailing list
> dmarc@ietf.org
> https://www.ietf.org/mailman/listinfo/dmarc
>


-- 

*Seth Blank* | VP, Standards and New Technologies
*e:* s...@valimail.com
*p:* 415.273.8818



This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Scott Kitterman]

On May 20, 2020 8:00:34 PM UTC, Hector Santos 
 wrote:
>
>On 5/20/2020 2:43 PM, Alessandro Vesely wrote:
>
>> Transaction?  I thought we were talking about aggregate reports.
>
>So am I.
>
>> There are no transactions there.
>
>Really?
>
>Each SMTP session can be considered a transaction. You are provided
>results on the these DMARC processed transaction.
>
>> I mean, what is the CSV format of the following report, that I sent
>yesterday
>> for this list:
>
>Sorry, if I ignored it.
>
>Forgetting fact that you can your report easier to read for consumers, 
>these would be an example of the CSV field headers.
>
>CSV headers:
>
>report_metadata.org_name, report_metadata.email, 
>report_metadata.report_id, report_metadata.date_range.begin, 
>report_metadata.date_range.end
>
>Policy_Published.domain, Policy_Published.adkim, 
>Policy_Published.aspf, Policy_Published.p
>
>record.row, record.row.source_ip.record.count. 
>record.row.policy_evaluated, record.row.policy_evaluated.disposition, 
>record.row,policy_evaluated.disposition, 
>record.row.policy_evaluated.dkim, record.row.policy_evaluated.spf
>
>Note: You don't have to stick to redundant "name space" field names.
>
 ...  Can we get back to work, please?
>>>
>>> Sorry, but I consider a rude, disrespectful and ignorant statement,
>to be
>>> saying that.
>>
>>
>> No personal attack intended.  I'm being rude because I have the
>impression that
>> you are not defending a concrete, well defined need, but instead find
>new
>> arguments opportunistically to pursue a vague sense of format
>fashion.
>
>That's a personal attack. If you don't understand the proposal, you 
>should back off or ask for clarification.
>
>> You
>> shifted from an asserted necessity of producers to a possible desire
>of
>> consumers.
>
>I did no such thing. I won't repeat it, but it appears you didn't 
>understand the proposal.
>
>>Now you introduce formats like CSV which make no sense in DMARC
>> context.
>
>I disagree. See above.
>
>> I hold that CSV cannot satisfy DMARC requirements.
>
>Hold it all you want. You know it would not be true. See above.
>
>? Please do contradict me by
>> showing us how an aggregate report in CSV would look like, as well as
>some
>> ideas for defining the corresponding template, similar to what
>Appendix C of
>> RFC 7489 specifies for XML.
>
>Ok, I won't but if you don't understand the proposal, you should ask 
>for clarification.
>
>CSV can work, so can JSON.  Limiting it and locking it down to XML 
>only would be a limitation.   You can agree or not agree with that.

Not requiring RFC 1149 support is also a limitation, but I'm fine with that.

Scott K

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/20/2020 2:43 PM, Alessandro Vesely wrote:


Transaction?  I thought we were talking about aggregate reports.


So am I.


There are no transactions there.


Really?

Each SMTP session can be considered a transaction. You are provided
results on the these DMARC processed transaction.


I mean, what is the CSV format of the following report, that I sent yesterday
for this list:


Sorry, if I ignored it.

Forgetting fact that you can your report easier to read for consumers, 
these would be an example of the CSV field headers.


CSV headers:

report_metadata.org_name, report_metadata.email, 
report_metadata.report_id, report_metadata.date_range.begin, 
report_metadata.date_range.end


Policy_Published.domain, Policy_Published.adkim, 
Policy_Published.aspf, Policy_Published.p


record.row, record.row.source_ip.record.count. 
record.row.policy_evaluated, record.row.policy_evaluated.disposition, 
record.row,policy_evaluated.disposition, 
record.row.policy_evaluated.dkim, record.row.policy_evaluated.spf


Note: You don't have to stick to redundant "name space" field names.


...  Can we get back to work, please?


Sorry, but I consider a rude, disrespectful and ignorant statement, to be
saying that.



No personal attack intended.  I'm being rude because I have the impression that
you are not defending a concrete, well defined need, but instead find new
arguments opportunistically to pursue a vague sense of format fashion.


That's a personal attack. If you don't understand the proposal, you 
should back off or ask for clarification.



You
shifted from an asserted necessity of producers to a possible desire of
consumers.


I did no such thing. I won't repeat it, but it appears you didn't 
understand the proposal.



Now you introduce formats like CSV which make no sense in DMARC
context.


I disagree. See above.


I hold that CSV cannot satisfy DMARC requirements.


Hold it all you want. You know it would not be true. See above.

? Please do contradict me by

showing us how an aggregate report in CSV would look like, as well as some
ideas for defining the corresponding template, similar to what Appendix C of
RFC 7489 specifies for XML.


Ok, I won't but if you don't understand the proposal, you should ask 
for clarification.


CSV can work, so can JSON.  Limiting it and locking it down to XML 
only would be a limitation.   You can agree or not agree with that.


--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Alessandro Vesely]

On Wed 20/May/2020 18:51:32 +0200 Hector Santos wrote:
> On 5/20/2020 12:26 PM, Alessandro Vesely wrote:
>>
>> Hector, what are you talking about?
> 
> hhmmm, I don't known any other way of describing this simple concept.
> 
>> Can you show us how does a CSV report look like?
> 
> Like a standard format for CSV:
> 
> line #1, name of fields, separated by a comma
> line #2, transaction #1 values separated by comma
> ..
> ..
> ..
> Line #N, transaction #N-1 values separated by comma


Transaction?  I thought we were talking about aggregate reports.  There are no
transactions there.

I mean, what is the CSV format of the following report, that I sent yesterday
for this list:




tana.it
postmas...@noloop.tana.it
eb74db90c32d432ea1652a0eb656fe5a

1589846400
1589932800



dmarc.ietf.org
r
r
none



4.31.198.44
1

none
pass
pass



dmarc.ietf.org



ietf.org
ietf1
pass


ietf.org
ietf1
fail


akamai.com
jan2016.eng
fail


ietf.org
pass
mfrom





4.31.198.44
1

none
pass
pass



dmarc.ietf.org



ietf.org
ietf1
pass


ietf.org
ietf1
fail


cisco.com
iport
fail


ietf.org
pass
mfrom





4.31.198.44
2

none
pass
pass



dmarc.ietf.org



ietf.org
ietf1
pass


ietf.org
ietf1
fail


junc.eu
default
fail


ietf.org
pass
mfrom





4.31.198.44
1

none
pass
pass



dmarc.ietf.org



ietf.org
ietf1
pass


ietf.org
ietf1
fail


valimail.com
google2048
fail


ietf.org
pass
mfrom






> A standard csv layout. Most if not all table like "tools" like a spreadsheet 
> or
> SQL, support a CSV import, maybe a XML import and if you are lucky, the JSON
> format. But XML is older than JSON so you may see CSV and XML with older 
> tools.
> But relatively soon, JSON will be the more common comm I/O technique.
> 
>> XML, like JSON, support complex template schemata.  CSV templates require 
>> fixed
>> columns.  I doubt you're talking seriously if you make such claims.
> 
> It is possible to have XML and JSON represented in a table-like flat name 
> space
> with multiple columns.
> 
>>
>> Yes, and I am Catherine Deneuve...
>>
> 
> Who?
> 
>> ...  Can we get back to work, please?
> 
> Sorry, but I consider a rude, disrespectful and ignorant statement, to be
> saying that.


No personal attack intended.  I'm being rude because I have the impression that
you are not defending a concrete, well defined need, but instead find new
arguments opportunistically to pursue a vague sense of format fashion.  You
shifted from an asserted necessity of producers to a possible desire of
consumers.  Now you introduce formats like CSV which make no sense in DMARC
context.  Is that work?


> We are working. If you don't agree or lack the ability to support anything 
> [but
> XML] others can, you should note it and but also not push your limitations on
> others.


I don't think that the inability to support CSV is a limitation of mine.  I
hold that CSV cannot satisfy DMARC requirements.  Please do contradict me by
showing us how an aggregate report in CSV would look like, as well as some
ideas for defining the corresponding template, similar to what Appendix C of
RFC 

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/20/2020 12:26 PM, Alessandro Vesely wrote:


I agree we should consider ways to increase the report generators base, but
requiring support for multiple formats goes in exactly the opposite direction.
Although automatic converters exist, having to provide multiple formats can be
a showstopper.


I don't agree this would be show-stopper. Not after all the work that 
has to be done
to move from Draft Informational status RFC to DMARC proposed standard 
RFC.


Its not complicated at all.

The proposal is to offer a new optional "prf=" tag that provides a 
preferred reporting format where the verifier MAY implement (NOT MUST) 
and provide to consumers.  I would say SHOULD but MAY is fine.




Hector, what are you talking about?


hhmmm, I don't known any other way of describing this simple concept.


Can you show us how does a CSV report look like?


Like a standard format for CSV:

line #1, name of fields, separated by a comma
line #2, transaction #1 values separated by comma
..
..
..
Line #N, transaction #N-1 values separated by comma

A standard csv layout. Most if not all table like "tools" like a 
spreadsheet or SQL, support a CSV import, maybe a XML import and if 
you are lucky, the JSON format. But XML is older than JSON so you may 
see CSV and XML with older tools. But relatively soon, JSON will be 
the more common comm I/O technique.



XML, like JSON, support complex template schemata.  CSV templates require fixed
columns.  I doubt you're talking seriously if you make such claims.


It is possible to have XML and JSON represented in a table-like flat 
name space with multiple columns.




Yes, and I am Catherine Deneuve...



Who?


...  Can we get back to work, please?


Sorry, but I consider a rude, disrespectful and ignorant statement, to 
be saying that.


We are working. If you don't agree or lack the ability to support 
anything XML but others can, you should note it and but also not push 
your limitations on others.



--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Alessandro Vesely]

On Wed 20/May/2020 14:49:13 +0200 Hector Santos wrote:
> On 5/19/2020 8:46 PM, Scott Kitterman wrote:
>>
>> Please don't.  This is a large stack of protocol and implementation
>> complexity for little or no gain.
>>
> 
> Rhetorically, I feel the same about reporting.  What is gained from it?  But
> that is just my opinion since the proposed ideas for reporting came and it was
> mostly rejected with SSP. It's redundant overhead.  If reporting is going to 
> be
> optional feature to possibly implement, it might as well be useful for
> consumers and that is where the gain will be.


Other people here consider reporting the (only) useful innovation of DMARC.

I agree we should consider ways to increase the report generators base, but
requiring support for multiple formats goes in exactly the opposite direction.
 Although automatic converters exist, having to provide multiple formats can be
a showstopper.


> The proposal is more to IANA register the formats (the common ones mostly used
> in practice) and not to mandate implementers use them all.  XML would be the
> fall back. At a minimum, XML is what publisher (or report handler) will get
> despite the publisher stating a preferred reporting format, like:
> 
> prf=csv, json, xml.
> 
> That's asking the verifier, if you can handle a csv or json format, please 
> send
> us these reports.


Hector, what are you talking about?


> I can support any format. Easily done with reporting/output templates for CSV,
> JSON or XML -- single sourced output generator.


Can you show us how does a CSV report look like?  Let me recall that DMARC
reports consist of a header part and a series of record with a varying number
of columns.

XML, like JSON, support complex template schemata.  CSV templates require fixed
columns.  I doubt you're talking seriously if you make such claims.

DMARC reports are supposed to be machine-readable.  To define multiple formats,
we'd need to supply schemata for each.  We'll likely have to update the schema
version for PS DMARC.  Given that automatic translation /of schemata/ is going
to lack for some more years, how would we manage equivalence proofs of all
those formats?


> So, from a product design standpoint, for immediate consumer "pay off," at the
> very least, csv should be the default option, not XML.


Yes, and I am Catherine Deneuve...  Can we get back to work, please?


Best
Ale
-- 

































___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

Thanks Scoot,  I hope all is safe for you and family.


argg sorry about that, Scott.

--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/20/2020 11:49 AM, Dave Crocker wrote:

On 5/20/2020 8:29 AM, Kurt Andersen (b) wrote:

Agree 100%



This looks like it has a strong constituency against the proposal, a
much smaller constituency in favor of it, and little or no offered
benefit.  Yes?


Just like DKIM Policy had a "strong constituency" against DKIM Policy, 
yet, the wrong decision was made to abandon DKIM Policy, only to find 
out today (the last few years), it was the wrong decision.  Just look 
at the support we have today for DKIM policy.


Whatever.

In regards to the report format, I am proposing an easy 
non-complicated compromise. Consumer will instantly benefit by 
offering additional registered report formats such as:


csv
json
xml  as the fall back.

The key is the register the tags, not mandate that verifiers need to 
support csv or json. I am proposing an new tag "prf=" to be recognized 
by standard DMARC verifiers but optional implementation of the 
registered tags csv and json. XML will be registered and required 
reporting format IFF reporting is offered by the verifier.  Reporting 
is not a requirement.


Keep in mind DMARC specs need a major cleanup in this regard and many 
others. It mentions XML, but the "rf=" no longer defined.  It says the 
tag is registered, but no description describing its possible values.


We need to stop this lock down methods and expand the desperately 
needed fix up for DMARC policy tags.  We have discussed these issues 
but the "smaller" entities in this group are getting absolutely pushed 
out with no recognition. In the end, we get the usual, the "Follow the 
Chieftain" syndrome and other folks, who may agree, stay quiet or just 
agree with "chief" +1 -- no rocking of the proverbial boat.


Rough Consensus is good, but is not always correct.

--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Benny Pedersen]

On 2020-05-20 17:49, Dave Crocker wrote:


This looks like it has a strong constituency against the proposal, a
much smaller constituency in favor of it, and little or no offered
benefit.  Yes?


if the mouse have 2 holes to select from, it will be 50% fails in 50% 
reports, this is why postfix keep old dokumented bugs :)


bla i am just funny here, it can be solved, but main part is that 
software need to be updated then, and not all precompiled problems do 
this


same reason i dropped sid-milter, opendkim, openarc, opendmarc, to 
unmatined and still have unclear bugs not solved, no thanks to it


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/20/2020 10:01 AM, Scott Kitterman wrote:


So, from a product design standpoint, for immediate consumer "pay
off," at the very least, csv should be the default option, not XML.


You're several years too late on deciding a default.


DMARC is informational, not a proposed standard. My confused 
understanding of the RFC7489 informational status DMARC draft, it has 
removed the IANA registered "rf=" report format tag, continues to 
describe the afrf and iodef format. But the XML schema is described as 
the report format?  Sound there is a lot of clean up to do.   We 
haven't even discussed 3rd party policy support. DMARC PS can not 
ignore it.  The rewrite kludge is not a preferred and acceptable concept.



If I understand your proposal correctly, everyone would still need to
implement XML ("At a minimum, XML is what publisher (or report handler) will
get").  If there's no avoiding implementing XML (which is what I thought the
original point was), why would this need to be defined now?  It seems like it
could easily be added later if there was a significant demand signal.


But XML was added unofficially?  As I pointed out, Google doesn't even 
mention the "rf=" tag and just described reports are in XML format.


If that is the decision for the initial DMARC proposed standard, I am 
fine with the XML default, but it would be simple to add an optional 
"prf=" tag with additional report format IANA registered well known 
acronyms namely csv, json.  XML will be registered for the PS so I am 
suggesting to also register csv and json with optional implementation 
semantics.


This may give verifiers (new or PS updated), preparedness to recognize 
the "prf=" tag for future support.  You are correct. The "prf=" can 
alway be added but the format tags won't be registered. Another RFC to 
cover the registration.


Thanks Scoot,  I hope all is safe for you and family.

--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Dave Crocker]

On 5/20/2020 8:29 AM, Kurt Andersen (b) wrote:

Agree 100%



This looks like it has a strong constituency against the proposal, a 
much smaller constituency in favor of it, and little or no offered 
benefit.  Yes?



d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Kurt Andersen (b)]

On Fri, May 15, 2020 at 6:14 PM John Levine  wrote:

> In article  zmn0oxmgns0cspywu58hqlaw7wvvbh1442yu4e...@mail.gmail.com> you write:
> >-=-=-=-=-=-
> >Should we consider adding JSON formatting to DMARC?
>
> What Scott said, no. Report processors will always have to be able to
> decode the existing XML so adding more options just adds more
> complexity with no more function.
>

Agree 100%

--Kurt
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Scott Kitterman]

On Wednesday, May 20, 2020 8:49:13 AM EDT Hector Santos wrote:
> On 5/19/2020 8:46 PM, Scott Kitterman wrote:
> > Please don't.  This is a large stack of protocol and implementation
> > complexity for little or no gain.
> Rhetorically, I feel the same about reporting.  What is gained from
> it?  But that is just my opinion since the proposed ideas for
> reporting came and it was mostly rejected with SSP. It's redundant
> overhead.  If reporting is going to be optional feature to possibly
> implement, it might as well be useful for consumers and that is where
> the gain will be.
> 
> > Hector, you realize that for this to work reliably you would need to code
> > up support for everything so that you wouldn't have undeliverable
> > reports?
> The proposal is more to IANA register the formats (the common ones
> mostly used in practice) and not to mandate implementers use them all.
>   XML would be the fall back. At a minimum, XML is what publisher (or
> report handler) will get despite the publisher stating a preferred
> reporting format, like:
> 
> prf=csv, json, xml.
> 
> That's asking the verifier, if you can handle a csv or json format,
> please send us these reports.
> 
> I can see where certain 3rd party report handlers would only need XML
> and they generate readable reports for their DMARC customers. But we
> can't assume all publishers will be using a 3rd party report handler.
> 
> So its about being ready for the future and DMARC not be obsolete
> fixed with XML only. In my opinion, JSON is the direction most systems
> are heading too.
> 
> > Have you implemented the XML format already and you're willing to code up
> > the alternatives too?
> I can support any format. Easily done with reporting/output templates
> for CSV, JSON or XML -- single sourced output generator.
> 
> This benefits customers, consumers, publishers who will have tools
> already, today, that imports certain common formats already, and
> generally, it would begin with a CSV format.  Just consider, all
> spreadsheets apps, Microsoft, Google, etc, all support CSV today, but
> not JSON or XML. Most or not all, SQL packages, support CSV and
> probably XML and maybe JSON.  Let me double check with Google Docs
> ..  no.  It supports
> 
> csv, txt, tab, htm, html, xls, xlsx, xlsm, xlt, xltm, xltx, ods
> 
> I don't know if the list will strip attachments, but this link shows
> an image of the Google Docs popup I got:
> 
> https://secure.winserver.com/public/files/google-json-no-support.png
> 
> So, from a product design standpoint, for immediate consumer "pay
> off," at the very least, csv should be the default option, not XML.

You're several years too late on deciding a default.

If I understand your proposal correctly, everyone would still need to 
implement XML ("At a minimum, XML is what publisher (or report handler) will 
get").  If there's no avoiding implementing XML (which is what I thought the 
original point was), why would this need to be defined now?  It seems like it 
could easily be added later if there was a significant demand signal.

Scott K



___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/19/2020 8:46 PM, Scott Kitterman wrote:


Please don't.  This is a large stack of protocol and implementation complexity 
for little or no gain.



Rhetorically, I feel the same about reporting.  What is gained from 
it?  But that is just my opinion since the proposed ideas for 
reporting came and it was mostly rejected with SSP. It's redundant 
overhead.  If reporting is going to be optional feature to possibly 
implement, it might as well be useful for consumers and that is where 
the gain will be.



Hector, you realize that for this to work reliably you would need to code up 
support for everything so that you wouldn't have undeliverable reports?



The proposal is more to IANA register the formats (the common ones 
mostly used in practice) and not to mandate implementers use them all. 
 XML would be the fall back. At a minimum, XML is what publisher (or 
report handler) will get despite the publisher stating a preferred 
reporting format, like:


prf=csv, json, xml.

That's asking the verifier, if you can handle a csv or json format, 
please send us these reports.


I can see where certain 3rd party report handlers would only need XML 
and they generate readable reports for their DMARC customers. But we 
can't assume all publishers will be using a 3rd party report handler.


So its about being ready for the future and DMARC not be obsolete 
fixed with XML only. In my opinion, JSON is the direction most systems 
are heading too.



Have you implemented the XML format already and you're willing to code up the 
alternatives too?


I can support any format. Easily done with reporting/output templates 
for CSV, JSON or XML -- single sourced output generator.


This benefits customers, consumers, publishers who will have tools 
already, today, that imports certain common formats already, and 
generally, it would begin with a CSV format.  Just consider, all 
spreadsheets apps, Microsoft, Google, etc, all support CSV today, but 
not JSON or XML. Most or not all, SQL packages, support CSV and 
probably XML and maybe JSON.  Let me double check with Google Docs 
..  no.  It supports


csv, txt, tab, htm, html, xls, xlsx, xlsm, xlt, xltm, xltx, ods

I don't know if the list will strip attachments, but this link shows 
an image of the Google Docs popup I got:


https://secure.winserver.com/public/files/google-json-no-support.png

So, from a product design standpoint, for immediate consumer "pay 
off," at the very least, csv should be the default option, not XML.


--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Scott Kitterman]

On May 20, 2020 12:23:37 AM UTC, Hector Santos 
 wrote:
>On 5/18/2020 8:25 PM, Seth Blank wrote:
>> On Mon, May 18, 2020 at 1:31 PM John Levine > > wrote:
>>
>> There are vast numbers of sites producing and consuming XML
>reports.
>> They interoperate. It works. There is no problem to be solved
>here.
>> Can we stop now and work on something else?
>>
>>
>> The working group's consensus appears to be that aggregate reporting
>> does not need to support JSON output. Does anyone disagree?
>>
>> Seth, as Chair--
>
>Since the specs will already be outdated by the time it is completed, 
>may I suggest that there is a new tag that provides a "Preferred 
>Report Format" or "prf=" tag using registered acronymns for long time 
>"standard" formats. For example:
>
>prf=cvs,json,xml,afrf,iodef
>
>1st choice cvs,
>2nd choice json
>3rd choice xml   new current fall back
>4rd choice afrf  old fall back, obsolete? See Note 1
>3rd choice iodef old fall back, obsolete? See note 1
>
>The verifier will do what can it offer. The publisher is providing a 
>preference, that it may not get. The fall back could be the XML format.
>
>This does two things:
>
>1) Flexibility offers a consumer preference, verifiers can eventually, 
>as a "Product Update," provide additional report formats.
>
>2) It doesn't lock in just the "XML" format.
>
>By the time a PS DMARC appears and it is finally completed, it would 
>be outdated as the growth of DMARC is realized with a "better spec." 
>So flexibility here would be a plus.  Consider GitHub for its Web 
>Hooks, JSON only.
>
>The conversion tools availability is surely there, but we can't assume 
>the consumers have programming skills.
>
>Note #1, there are DMARC descriptions and record generation wizards, 
>who don't even offer the "rf=" tag in their wizards and just document 
>XML as the report format.  Case in point, google:
>
>https://support.google.com/a/answer/2466563?hl=en_topic=2759254
>
>So does this suggest a complete removal of "rf=" tag, no more afrf and 
>iodef, and xml only?
>
>If so, can we add the "Preferred Report Format" "prf=" tag?  I am 
>asking for the "prf=" tag and to register the common format acronyms, 
>allowing for advanced DMARC processors to add support over time.

Please don't.  This is a large stack of protocol and implementation complexity 
for little or no gain.

Hector, you realize that for this to work reliably you would need to code up 
support for everything so that you wouldn't have undeliverable reports?

Have you implemented the XML format already and you're willing to code up the 
alternatives too?

Scott K

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/18/2020 8:25 PM, Seth Blank wrote:

On Mon, May 18, 2020 at 1:31 PM John Levine mailto:jo...@taugh.com>> wrote:

There are vast numbers of sites producing and consuming XML reports.
They interoperate. It works. There is no problem to be solved here.
Can we stop now and work on something else?


The working group's consensus appears to be that aggregate reporting
does not need to support JSON output. Does anyone disagree?

Seth, as Chair--


Since the specs will already be outdated by the time it is completed, 
may I suggest that there is a new tag that provides a "Preferred 
Report Format" or "prf=" tag using registered acronymns for long time 
"standard" formats. For example:


prf=cvs,json,xml,afrf,iodef

1st choice cvs,
2nd choice json
3rd choice xml   new current fall back
4rd choice afrf  old fall back, obsolete? See Note 1
3rd choice iodef old fall back, obsolete? See note 1

The verifier will do what can it offer. The publisher is providing a 
preference, that it may not get. The fall back could be the XML format.


This does two things:

1) Flexibility offers a consumer preference, verifiers can eventually, 
as a "Product Update," provide additional report formats.


2) It doesn't lock in just the "XML" format.

By the time a PS DMARC appears and it is finally completed, it would 
be outdated as the growth of DMARC is realized with a "better spec." 
So flexibility here would be a plus.  Consider GitHub for its Web 
Hooks, JSON only.


The conversion tools availability is surely there, but we can't assume 
the consumers have programming skills.


Note #1, there are DMARC descriptions and record generation wizards, 
who don't even offer the "rf=" tag in their wizards and just document 
XML as the report format.  Case in point, google:


https://support.google.com/a/answer/2466563?hl=en_topic=2759254

So does this suggest a complete removal of "rf=" tag, no more afrf and 
iodef, and xml only?


If so, can we add the "Preferred Report Format" "prf=" tag?  I am 
asking for the "prf=" tag and to register the common format acronyms, 
allowing for advanced DMARC processors to add support over time.


thanks.

--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Seth Blank]

On Mon, May 18, 2020 at 1:31 PM John Levine  wrote:

> There are vast numbers of sites producing and consuming XML reports.
> They interoperate. It works. There is no problem to be solved here.
> Can we stop now and work on something else?
>

The working group's consensus appears to be that aggregate reporting does
not need to support JSON output. Does anyone disagree?

Seth, as Chair

On Mon, May 18, 2020 at 5:25 AM Benny Pedersen 
wrote:

> On 2020-05-18 10:27, Alessandro Vesely wrote:
>
> > Best
> > Ale
> > --
> >
>
> could you remove empty lines in your signature ?
>
> lets see dmarc fail, dkim invalid test in spamassassin
>
> ___
> dmarc mailing list
> dmarc@ietf.org
> https://www.ietf.org/mailman/listinfo/dmarc
>


-- 

*Seth Blank* | VP, Standards and New Technologies
*e:* s...@valimail.com
*p:* 415.273.8818



This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format? No.

[John Levine]

In article <492288f0-3fc3-f85e-26c0-eda418de6...@tana.it> you write:
>Since receivers have to verify report correctness and trustworthiness, /they/
>deserve the right to decide what language they want reports to be written in.

There is lots of freely available code to parse DMARC XML reports and
put the results into a database or otherwise interpret them. You can
find my scripts here:

https://www.taugh.com/rddmarc/

The part of my routine that parses the XML into internal form is about
three lines since it uses a generic XML parser.

There are vast numbers of sites producing and consuming XML reports.
They interoperate. It works. There is no problem to be solved here.
Can we stop now and work on something else?

R's,
John

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Benny Pedersen]

On 2020-05-18 10:27, Alessandro Vesely wrote:


Best
Ale
--



could you remove empty lines in your signature ?

lets see dmarc fail, dkim invalid test in spamassassin

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Alessandro Vesely]

On Sat 16/May/2020 20:45:43 +0200 Seth Blank wrote:
> On Fri, May 15, 2020 at 7:20 PM John Levine  wrote:
>> In article 
>>  you 
>> write:
>>> +1, unless there are known cases where the XML payload size is a problem
>>> that switching to JSON would solve.
>>
>> It's gzipped, do I doubt it would make much difference. Gzip is really
>> good at compressing the boilerplate strings that make XML bigger than
>> JSON.
>>
>> Also, FWIW, in seven years the largest report message I ever got was
>> under 300K, and if people really are concerned about big reports, I
>> would much rather revive https reporting which avoids base64 encoding
>> and doesn't have to buffer copies of the report for relay.
>>
> 
> Hatless, as a data point: at Valimail we see reports that are hundreds of
> megabytes in size, and sometimes push close to a gigabyte. These reports
> continue to increase in size month over month as sending volume, geographic
> footprint, deployed third party services, and fraudulent mail attempting to
> impersonate the domain continually increase


Another way to halve the size of the reports is to halve the reporting
interval.  I think this can be made clearer in DMARCbis, so I added ticket #71.


Best
Ale
-- 









































___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Steven M Jones]

On 05/15/2020 13:16, Scott Kitterman wrote:


When there is only one report format, there's no need to discover a common
format between report senders and report receivers.  As soon as there are two,
then to be interoperable, then either both report formats are required or
there has to be some kind of discovery mechanism, because we don't want report
senders sending reports in a format the receiver can't parse.


And every time we add another flag or k/v pair we're adding another way 
for the domain owner/manager to screw up and invalidate their published 
policy.


Something to bear in mind throughout these deliberations.

--S.

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Steven M Jones]

On 05/15/2020 11:26, Seth Blank wrote:

https://trac.ietf.org/trac/dmarc/ticket/69

Aggregate reports are only sent in XML format. There's been some 
discussion (that's also related to 
https://trac.ietf.org/trac/dmarc/ticket/42 about expanding reporting 
functionality beyond mailto:), of adding additional reporting formats, 
specifically JSON.


Should we consider adding JSON formatting to DMARC?


I don't see the benefit. Can anybody make that case?

More overhead to maintain, more opportunity for errors. Does adding JSON 
somehow eliminate the anecdotes about report generators who have issues 
with their XML output? Any reason to think that wouldn't also happen 
with JSON?


Does adding JSON reporting solve one of the other open tickets?

--S.

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[John R Levine]

Hatless, as a data point: at Valimail we see reports that are hundreds of
megabytes in size, and sometimes push close to a gigabyte. These reports
continue to increase in size month over month as sending volume, geographic
footprint, deployed third party services, and fraudulent mail attempting to
impersonate the domain continually increase


I believe you, but switching to JSON would not make any difference.  The 
reports are gzipped and the extra verbiage of XML is all squeezed out in 
the compression.  I have a 4.1 meg XML report that zipped to 136K, about 
3% of the original size, which I think is a typical ratio for reports of 
any size.


E-mail is a lousy way to send around gigabyte files, so we should make 
https work.  Semantically HTTP PUT is the right way to do it but HTTP POST 
like mta-sts does would be OK.  It uploads a binary application/gzip 
version of the report so it's about as efficient as it can be.


R's,
John

PS: Another think we might consider, although it would be significant 
work, is to add application/bzip2 as a compression type.  That 4.1M XML 
file crunches down to 51K as bzip2.  It has some issues, it's not well 
documented other than by the reference implementation, and it'd be yet 
another way for things not to interoperate so we'd have to add a signal in 
the DMARC record to say that reports can use it.




___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Seth Blank]

On Fri, May 15, 2020 at 7:20 PM John Levine  wrote:

> In article <
> cal0qlwaskzdyeqnjs-bj4ttfdgevfc8gaukmvnost0wj3ge...@mail.gmail.com> you
> write:
> >+1, unless there are known cases where the XML payload size is a problem
> >that switching to JSON would solve.
>
> It's gzipped, do I doubt it would make much difference. Gzip is really
> good at compressing the boilerplate strings that make XML bigger than
> JSON.
>
> Also, FWIW, in seven years the largest report message I ever got was
> under 300K, and if people really are concerned about big reports, I
> would much rather revive https reporting which avoids base64 encoding
> and doesn't have to buffer copies of the report for relay.
>

Hatless, as a data point: at Valimail we see reports that are hundreds of
megabytes in size, and sometimes push close to a gigabyte. These reports
continue to increase in size month over month as sending volume, geographic
footprint, deployed third party services, and fraudulent mail attempting to
impersonate the domain continually increase

Seth

-- 

*Seth Blank* | VP, Standards and New Technologies
*e:* s...@valimail.com
*p:* 415.273.8818



This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/16/2020 2:41 AM, Dave Crocker wrote:

On 5/15/2020 9:54 PM, Hector Santos wrote:

Protocols should be flexible. Adding it doesn't mean replace.


Explain the need.

Convince plenty of folk that it is necessary enough that they are
inclined to write the code, deploy it, and use it.  Adding things is
expensive.

Flexibility that does not have a clear need is especially expensive.


A rather large point to consider, with many 'Modern' developers today, 
the "expense" would to fallback to XML when in fact, they are using 
JSON regularity and increasingly for new protocols and I/O exchanges.


I personally prefer an easier ad-hoc reporting format for immediate 
reading by the laymen and not be dependent on XML or JSON readers.


--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Hector Santos]

On 5/16/2020 11:49 AM, Pete Resnick wrote:

On 15 May 2020, at 23:54, Hector Santos wrote:

On 5/15/2020 2:26 PM, Seth Blank wrote:

Should we consider adding JSON formatting to DMARC?

Protocols should be flexible. Adding it doesn't mean replace.

Flexible sometimes means less interoperable. You've already got to get
the XML right. Adding another format that you've got to get right
sounds like it increases the odds that an implementer is going to get
one of them wrong, and you've got to make sure that the XML and JSON
semantics match each other. As Dave said, unless there's a compelling
reason to add JSON, the potential for decreased interoperability says
to me this isn't a good idea.


On 5/16/2020 2:09 AM, Pete Resnick wrote:> On 15 May 2020, at 23:54, 
Hector Santos wrote:



On 5/15/2020 2:26 PM, Seth Blank wrote:


Should we consider adding JSON formatting to DMARC?


Protocols should be flexible. Adding it doesn't mean replace.


Flexible sometimes means less interoperable. You've already got to get
the XML right. Adding another format that you've got to get right
sounds like it increases the odds that an implementer is going to get
one of them wrong, and you've got to make sure that the XML and JSON
semantics match each other. As Dave said, unless there's a compelling
reason to add JSON, the potential for decreased interoperability says
to me this isn't a good idea.



One can suggest XML is also complex to get right. The direction has 
been with JSON for a number of years, especially with the JavaScript 
developers using established tools and platforms based JSON. The APIs 
are readily available.


Just consider, when the spec has XML-only, then for those who use a 
solid JSON I/O system, they are now going to be required to add XML. 
So for them, its additional development complexity.  Everything they 
probably do JSON related. The exception would be DMARC using XML. 
This alone can delay or push aside DMARC Reporting implementation.


There are other formats:

- CSV
- Flat Name spaces, i.e. A.B.C.D

Even the Authentication-Result: header(s) can be sent, the one the 
verifier would create.


Even an ad-hoc reporter that people can read immediately the email 
without any additional format readers would work very well.


Anyway, when it comes to product development, and this all being put 
together as a "product" by many key folks here, then "High Quality" 
flexibility (options) generally works to cover a wider spectrum of 
implementators.


We assume it will be done correctly.

--
HLS


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Pete Resnick]

On 15 May 2020, at 23:54, Hector Santos wrote:

On 5/15/2020 2:26 PM, Seth Blank wrote:

Should we consider adding JSON formatting to DMARC?

Protocols should be flexible. Adding it doesn't mean replace.

Flexible sometimes means less interoperable. You've already got to get 
the XML right. Adding another format that you've got to get right sounds 
like it increases the odds that an implementer is going to get one of 
them wrong, and you've got to make sure that the XML and JSON semantics 
match each other. As Dave said, unless there's a compelling reason to 
add JSON, the potential for decreased interoperability says to me this 
isn't a good idea.


pr
--
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Benny Pedersen]

On 2020-05-16 11:38, Alessandro Vesely wrote:

Please let's not even air such hypothesies.  There is still people who 
send zip

rather than gzip...


and still people that does not use sid-milter, opendkim, openarc, 
opendmarc, all of them wait for software updates or precompiled problems 
to be solved


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Alessandro Vesely]

On Fri 15/May/2020 20:26:32 +0200 Seth Blank wrote:
> https://trac.ietf.org/trac/dmarc/ticket/69
> 
> Aggregate reports are only sent in XML format. There's been some discussion
> (that's also related to https://trac.ietf.org/trac/dmarc/ticket/42 about
> expanding reporting functionality beyond mailto:), of adding additional
> reporting formats, specifically JSON.
> 
> Should we consider adding JSON formatting to DMARC?


Please let's not even air such hypothesies.  There is still people who send zip
rather than gzip...


Best
Ale
-- 


































___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Dave Crocker]

On 5/15/2020 9:54 PM, Hector Santos wrote:
Protocols should be flexible. Adding it doesn't mean replace. 


Explain the need.

Convince plenty of folk that it is necessary enough that they are 
inclined to write the code, deploy it, and use it.  Adding things is 
expensive.


Flexibility that does not have a clear need is especially expensive.


d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[John Levine]

In article  
you write:
>+1, unless there are known cases where the XML payload size is a problem
>that switching to JSON would solve.

It's gzipped, do I doubt it would make much difference. Gzip is really
good at compressing the boilerplate strings that make XML bigger than
JSON.

Also, FWIW, in seven years the largest report message I ever got was
under 300K, and if people really are concerned about big reports, I
would much rather revive https reporting which avoids base64 encoding
and doesn't have to buffer copies of the report for relay.

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[John Levine]

In article  
you write:
>-=-=-=-=-=-
>Should we consider adding JSON formatting to DMARC?

What Scott said, no. Report processors will always have to be able to
decode the existing XML so adding more options just adds more
complexity with no more function.

Apropos ticket #29 we might consider resuscitating the https: URI. It
seems to work OK for the one reporter who uses it for mta-sts.

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Dave Crocker]

On 5/15/2020 3:09 PM, Murray S. Kucherawy wrote:
+1, unless there are known cases where the XML payload size is a 
problem that switching to JSON would solve.



For any interesting specification, there are many ways to improve it.  
The question is whether the improvement is worth the effort.


So, when someone suggests a change, there should be an expectation of an 
accompanying explanation that makes the change compelling.


A particular trap is changes that are fashionable rather than 
important.  That is, they suit the tastes of the day, but they don't 
really fix a problem or makes things markedly better.


d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Murray S. Kucherawy]

On Fri, May 15, 2020 at 1:16 PM Scott Kitterman 
wrote:

> If this were a green field effort and we were trying to decide on a report
> format, I would probably think JSON was a great choice, but it's not.
> Trying
> to support multiple formats is going to be a lot of work for little to no
> real
> gain.
>
> I'd suggest no.
>

+1, unless there are known cases where the XML payload size is a problem
that switching to JSON would solve.

-MSK, once again devoid of hat
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Scott Kitterman]

On Friday, May 15, 2020 2:26:32 PM EDT Seth Blank wrote:
> https://trac.ietf.org/trac/dmarc/ticket/69
> 
> Aggregate reports are only sent in XML format. There's been some discussion
> (that's also related to https://trac.ietf.org/trac/dmarc/ticket/42 about
> expanding reporting functionality beyond mailto:), of adding additional
> reporting formats, specifically JSON.
> 
> Should we consider adding JSON formatting to DMARC?

When there is only one report format, there's no need to discover a common 
format between report senders and report receivers.  As soon as there are two, 
then to be interoperable, then either both report formats are required or 
there has to be some kind of discovery mechanism, because we don't want report 
senders sending reports in a format the receiver can't parse.

Each report format has to have a defined structure so that receivers can 
consistently parse them.  Maintaining two report formats represents additional 
complexity and increased work to be done.

If this were a green field effort and we were trying to decide on a report 
format, I would probably think JSON was a great choice, but it's not.  Trying 
to support multiple formats is going to be a lot of work for little to no real 
gain.

I'd suggest no.

Scott K


___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

[dmarc-ietf] DMARC bis: ticket 69: add JSON reporting format?

[Seth Blank]

https://trac.ietf.org/trac/dmarc/ticket/69

Aggregate reports are only sent in XML format. There's been some discussion
(that's also related to https://trac.ietf.org/trac/dmarc/ticket/42 about
expanding reporting functionality beyond mailto:), of adding additional
reporting formats, specifically JSON.

Should we consider adding JSON formatting to DMARC?

-- 

*Seth Blank* | VP, Standards and New Technologies
*e:* s...@valimail.com
*p:* 415.273.8818



This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc