[PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes

[Andrew Donnellan]

To avoid triggering spam filters due to failed signature validation, many
mailing lists mangle the From header to change the From address to be the
address of the list, typically where the sender's domain has a strict DMARC
policy enabled.

In this case, we should try to unmangle the From header.

Add support for using the X-Original-From or Reply-To headers, as used by
Google Groups and Mailman respectively, to unmangle the From header when
necessary and associate the patch with the correct submitter based on the
unmangled email address.

When downloading mboxes, rewrite the From header using the unmangled
address, and preserve the original header as X-Patchwork-Original-From in
case someone needs it for some reason. The original From header will still
be stored in the database and exposed via the API, as we want to keep
messages as close to the original received format as possible.

Closes: #64 ("Incorrect submitter when using googlegroups")
Reported-by: Alexandre Belloni 
Reported-by: Stephen Rothwell 
Signed-off-by: Andrew Donnellan 

---

v1->v2:
- use X-Original-From rather than X-Original-Sender
- unmangle From header when downloading mbox

rewrite from header

Signed-off-by: Andrew Donnellan 

use x original from

Signed-off-by: Andrew Donnellan 
---
 patchwork/parser.py   | 75 +++
 patchwork/tests/test_mboxviews.py | 21 +
 patchwork/tests/test_parser.py| 68 ++--
 patchwork/views/utils.py  | 12 +
 4 files changed, 163 insertions(+), 13 deletions(-)

diff --git a/patchwork/parser.py b/patchwork/parser.py
index 7dc66bc05a5b..be1e51652dd3 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -321,12 +321,7 @@ def find_series(project, mail, author):
 return _find_series_by_markers(project, mail, author)
 
 
-def get_or_create_author(mail):
-from_header = clean_header(mail.get('From'))
-
-if not from_header:
-raise ValueError("Invalid 'From' header")
-
+def split_from_header(from_header):
 name, email = (None, None)
 
 # tuple of (regex, fn)
@@ -355,6 +350,65 @@ def get_or_create_author(mail):
 (name, email) = fn(match.groups())
 break
 
+return (name, email)
+
+
+# Unmangle From addresses that have been mangled for DMARC purposes.
+#
+# To avoid triggering spam filters due to failed signature validation, many
+# mailing lists mangle the From header to change the From address to be the
+# address of the list, typically where the sender's domain has a strict
+# DMARC policy enabled.
+#
+# Unfortunately, there's no standardised way of preserving the original
+# From address.
+#
+# Google Groups adds an X-Original-From header. If present, we use that.
+#
+# Mailman preserves the original address by adding a Reply-To, except in the
+# case where the list is set to either reply to list, or reply to a specific
+# address, in which case the original From is added to Cc instead. These corner
+# cases are dumb, but we try and handle things as sensibly as possible by
+# looking for a name in Reply-To/Cc that matches From. It's inexact but should
+# be good enough for our purposes.
+def get_original_sender(mail, name, email):
+if name and ' via ' in name:
+# Mailman uses the format " via "
+# Google Groups uses "'' via "
+stripped_name = name[:name.rfind(' via ')].strip().strip("'")
+
+original_from = clean_header(mail.get('X-Original-From', ''))
+if original_from:
+new_email = split_from_header(original_from)[1].strip()[:255]
+return (stripped_name, new_email)
+
+addrs = []
+reply_to_headers = mail.get_all('Reply-To') or []
+cc_headers = mail.get_all('Cc') or []
+for header in reply_to_headers + cc_headers:
+header = clean_header(header)
+addrs = header.split(",")
+for addr in addrs:
+new_name, new_email = split_from_header(addr)
+if new_name:
+new_name = new_name.strip()[:255]
+if new_email:
+new_email = new_email.strip()[:255]
+if new_name == stripped_name:
+return (stripped_name, new_email)
+
+# If we can't figure out the original sender, just keep it as is
+return (name, email)
+
+
+def get_or_create_author(mail, project=None):
+from_header = clean_header(mail.get('From'))
+
+if not from_header:
+raise ValueError("Invalid 'From' header")
+
+name, email = split_from_header(from_header)
+
 if not email:
 raise ValueError("Invalid 'From' header")
 
@@ -362,6 +416,9 @@ def get_or_create_author(mail):
 if name is not None:
 name = name.strip()[:255]
 
+if project and email.lower() == project.listemail.lower():
+name, email = get_original_sender(mail, name, email)
+
 # this correctly handles the case where we lose the race to create
 # the person and another process beats us to it. (If the record
 # does 

Re: [PATCH 0/2] Add writeable 'bundles' API

[Daniel Axtens]

Stephen Finucane  writes:

> On Sun, 2019-09-08 at 23:31 +0100, Stephen Finucane wrote:
>> Hopefully self-explanatory. This closes another gap with the web UI and
>> REST API.
>
> Assuming no one disagrees, I'm going to merge this series before the
> end of the week. It has tests and documentation, suggesting it's low
> enough risk, and is the sole remaining blocker for v2.2.0.

Oh, sorry, had meant to test it by now. Not sure I will be able to get
that done in time, so I will trust the tests.

Could we squeeze the DMARC fix into 2.2? (And probably the fix for
https://github.com/getpatchwork/patchwork/issues/318 as well.)

Apart from that I think I'm OK with what we've got so far. I had hoped
we could get the patch relations stuff in but without a new spin of the
series I guess it will have to wait.

Regards,
Daniel


>
> Stephen
>
> ___
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Drew DeVault]

On Tue Oct 15, 2019 at 7:44 PM Laurent Pinchart wrote:
> I'm not very confident that perfect transparent e-mail bridges could be
> developed, and the culprit here it e-mail. From forge to e-mail there's
> no real issue, we have structure data, and we write it out in text form.
> The other way around, though, involves recovering structure from text.
> If the MTAs, MDAs, MUAs and, quite importanttly, users, behave
> correctly, that's doable. We can handle some of the "features" of common
> M*A, but if someone decides to throw the formatting through the window
> completely, we'll be screwed.

Well, no, it can never be perfect. But I'm not trying to accomodate for,
as an example, someone who's deliberately trying to confusing the
system. It will probably always be based on heuristics in some degrees.

However, I think that social conformance is a force to be reckoned with.
Most people don't have to be taught how code review works on mailing
lists, they just watch others do it and naturally fall in line. Without
any formal structure at all, we've already more or less settled on a
shared set of patterns for email-driven code review. However, I think
that the following:

> An idea I was toying with in the past was to create a structured format
> for review, that would match what we use now in e-mail very closely, but
> with a clearly defined syntax and grammar. The format would appear as
> just plain e-mails to users, and a MUA that behaves reasonably would
> likely produce valid structured data as e-mail replies. Over time we
> could develop clients or teach some MUAs about the structured format,
> removing the risk that they, or the end users, would mess up a reply.
> The migration would be mostly transparent as it wouldn't really impact
> existing users of e-mail, and could thus be rolled out over a long
> period of time.

Is very reasonable. If we can come up with a standard which feels like
what we're already doing, but is written down, then tools have something
to conform to and users have some expectations for how to behave to make
their tools happy.

I would be up for putting up some kind of simple spec for bikeshedding
purposes, and teaching it to the thread parser that was built for
lists.sr.ht:

https://git.sr.ht/~emersion/python-emailthreads
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Tue, 15 Oct 2019 19:37:04 +0300
Laurent Pinchart  wrote:

> But how far could we push this reasoning ? I've worked for a company
> whose corporate policy was that all source code had to be stored in SVN,
> not exception. I didn't reach the community to move kernel development
> away from git. I've also worked with people whose corporate policy was
> that they had to do Linux kernel development on Windows, without using
> any virtual machine. There are all kind of crazy corporate policies, and
> if we don't push the pain it inflicts on all of us back to the people
> who enact those crazy policies, we'll always lose.

I understand your sentiment. But most of your examples are one-off
"crazy corporate policies". The "sucky email server" and "you must use
this sucky email server" is rather standard. Not saying that we don't
want to pressure them to change, but I'm hearing from people (like Dave
Miller), that new developers are moving away from email for
development.

I thought part of this thread was talking about having other ways
besides email to submit a patch. Something that can help people
correctly send a patch (at least their formatting) by making sure they
fill out the proper fields and have a tool that helps them do so. I
brought up the corporate email servers just as an example of having
this help out there too.

I plan on continuing to develop mostly in email (I still send my patch
series via quilt!). But I'm not going to enforce everyone to continue
to use email if we can come up with a better way. I also want to make
sure that whatever we do come up with will still support email.

-- Steve
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Greg KH]

On Tue, Oct 15, 2019 at 06:41:03AM +0200, Dmitry Vyukov wrote:
> On Mon, Oct 14, 2019 at 5:17 PM Greg KH  wrote:
> > > As one data point, I cannot send emails with git send-email anymore.
> > > It used to work, then broke and I don't know how to fix it. Now it says:
> > >
> > > 5.7.8 Username and Password not accepted. Learn more at
> > > 5.7.8  https://support.google.com/mail/?p=BadCredentials
> > > s10sm8376885wrr.5 - gsmtp
> > >
> > > I suspect it has something to do with two factor auth.
> > > So that's it: it cannot contribute to kernel right now.
> >
> > That is because your employer changed how it manages imap.  So yes, this
> > configuration is now broken, you can not contribute to the kernel this
> > way.  They know about it, and there's an "opt-out" list you can sign up
> > for if you want to fix it.  Nothing the community can do about something
> > crazy like this.
> 
> I used my private gmail account to send emails. My private email
> account stopped working.

I think you need a better email provider :)
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] Allow ordering events by date

[Jeremy Cline]

By default, the events API orders events by date in descending order
(newest first). However, it's useful to be able to order the events by
oldest events first. For example, when a client is polling the events
API for new events since a given date and wishes to process them in
chronological order.

Signed-off-by: Jeremy Cline 
---
 patchwork/api/event.py |  2 +-
 patchwork/tests/api/test_event.py  | 18 ++
 ...-order-events-by-date-7484164761c5231b.yaml |  5 +
 3 files changed, 24 insertions(+), 1 deletion(-)
 create mode 100644 
releasenotes/notes/api-order-events-by-date-7484164761c5231b.yaml

diff --git a/patchwork/api/event.py b/patchwork/api/event.py
index c0d973d..e6d467d 100644
--- a/patchwork/api/event.py
+++ b/patchwork/api/event.py
@@ -77,7 +77,7 @@ class EventList(ListAPIView):
 serializer_class = EventSerializer
 filter_class = filterset_class = EventFilterSet
 page_size_query_param = None  # fixed page size
-ordering_fields = ()
+ordering_fields = ('date',)
 ordering = '-date'
 
 def get_queryset(self):
diff --git a/patchwork/tests/api/test_event.py 
b/patchwork/tests/api/test_event.py
index 8816538..bff8f40 100644
--- a/patchwork/tests/api/test_event.py
+++ b/patchwork/tests/api/test_event.py
@@ -149,6 +149,24 @@ class TestEventAPI(utils.APITestCase):
 resp = self.client.get(self.api_url(), {'series': 99})
 self.assertEqual(0, len(resp.data))
 
+def test_order_by_date_default(self):
+"""Assert the default ordering is by date descending."""
+self._create_events()
+
+resp = self.client.get(self.api_url())
+events = Event.objects.order_by("-date").all()
+for api_event, event in zip(resp.data, events):
+self.assertEqual(api_event["id"], event.id)
+
+def test_order_by_date_ascending(self):
+"""Assert the default ordering is by date descending."""
+self._create_events()
+
+resp = self.client.get(self.api_url(), {'order': 'date'})
+events = Event.objects.order_by("date").all()
+for api_event, event in zip(resp.data, events):
+self.assertEqual(api_event["id"], event.id)
+
 def test_create(self):
 """Ensure creates aren't allowed"""
 user = create_maintainer()
diff --git a/releasenotes/notes/api-order-events-by-date-7484164761c5231b.yaml 
b/releasenotes/notes/api-order-events-by-date-7484164761c5231b.yaml
new file mode 100644
index 000..5d5328d
--- /dev/null
+++ b/releasenotes/notes/api-order-events-by-date-7484164761c5231b.yaml
@@ -0,0 +1,5 @@
+---
+features:
+  - |
+Allow ordering events from the events API by date. This can be done by
+adding ``order=date`` or ``order=-date`` (the default) parameters.
-- 
2.21.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 07:32:41PM +0300, Laurent Pinchart wrote:

Well, this is largely what GitGitGadget does
(https://gitgitgadget.github.io), and we could go that route, sure. I'm
reluctant only because, quoth:

  GitGitGadget itself is a GitHub App that is backed by an Azure
  Function written in pure Javascript which in turn triggers an Azure
  Pipeline written in Typescript (which is really easy to understand and
  write for everybody who knows even just a little Javascript),
  maintained at https://github.com/gitgitgadget/gitgitgadget.

I have zero familiarity with any of the above. That said, we do have a
bunch of CI engineers working at the LF, and I can probably avail myself
of their expertise if we decide to set this up.


I certainly wouldn't recommend a solution based on a proprietary
closed-source stack :-) But as we're talking about performing new
development for patchwork, I wanted to point out that we could also
consider a different technical approach that would involve new
development for a different open-source project. For instance, is the
above idea something that could be developed on top of gitolite ? Or
possibly even as a tiny standalone git server ?


I wouldn't do it on top of gitolite, because the administrative overhead 
would be too large. It certainly can be done as a separate service -- 
after all, any relation between this tool and patchwork is pretty 
tenuous.


That said, I'm leaning towards shelving this idea for the moment, at 
least as an official service provided by kernel.org -- it is easier to 
limit the scope at first and target maintainer tools and communication 
frameworks. It seems that sr.ht folks are already making something like 
this available, so we can just sit on our hands for a bit and see where 
this takes us.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Drew,

On Tue, Oct 15, 2019 at 12:34:57PM -0400, Drew DeVault wrote:
> On Tue Oct 15, 2019 at 7:32 PM Laurent Pinchart wrote:
> > I certainly wouldn't recommend a solution based on a proprietary
> > closed-source stack :-) But as we're talking about performing new
> > development for patchwork, I wanted to point out that we could also
> > consider a different technical approach that would involve new
> > development for a different open-source project. For instance, is the
> > above idea something that could be developed on top of gitolite ? Or
> > possibly even as a tiny standalone git server ?
> 
> Some of the work I'm doing with code review on sourcehut can be
> generalized, to support writing up arbitrary mailing lists with
> arbitrary review tools with a bidirectional workflow. I intend to make a
> reality of this for at least lists.sr.ht <-> gitlab and lists.sr.ht <->
> github, so that patchsets and the resulting discussion gets turned into
> git{hub,lab} code reviews, and vise versa. I think it's possible to make
> a system which presents both sides with an experience that is idiomatic
> for each context, so that you can't really tell whose using which tool
> because they're both speaking each other's conventional language.

First of all, please let me thank you for all the effort you're putting
into this.

I'm not very confident that perfect transparent e-mail bridges could be
developed, and the culprit here it e-mail. From forge to e-mail there's
no real issue, we have structure data, and we write it out in text form.
The other way around, though, involves recovering structure from text.
If the MTAs, MDAs, MUAs and, quite importanttly, users, behave
correctly, that's doable. We can handle some of the "features" of common
M*A, but if someone decides to throw the formatting through the window
completely, we'll be screwed.

An idea I was toying with in the past was to create a structured format
for review, that would match what we use now in e-mail very closely, but
with a clearly defined syntax and grammar. The format would appear as
just plain e-mails to users, and a MUA that behaves reasonably would
likely produce valid structured data as e-mail replies. Over time we
could develop clients or teach some MUAs about the structured format,
removing the risk that they, or the end users, would mess up a reply.
The migration would be mostly transparent as it wouldn't really impact
existing users of e-mail, and could thus be rolled out over a long
period of time.

-- 
Regards,

Laurent Pinchart
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Steven,

On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:
> On Mon, 14 Oct 2019 16:56:58 -0400 "Theodore Y. Ts'o" wrote:
> 
> > So it's an extreme case, and is an example of corp e-mail insanity.
> > The good news is that for companies that are really serious about
> > working with upstream, they will roll out solutions for their own
> > customers.  For example, with IBM, since Lotus notes was maximally
> > unfriendly to OSS development practices, they created a separate mail
> > system with l...@linux.ibm.com addresses be cause it was simply
> > impossible to engage with upstream development using ibm.com
> > addresses.  So there are work arounds, and I encourage you to reach
> > out to other Google kernel developers --- and if you are unhappy,
> > please file internal bugs against the corp infrastructure.
> 
> Note, one issue is that some corporations do not want to support two
> email services. It's hard enough securing one, opening up another one
> is just opening another door to be cracked.
> 
> We do file bugs when there's issues, but that doesn't mean they will be
> resolved. I have special permission to use my personal account, but
> most people at my company do not have that permission, and this is a
> heart ache for those that need to send email via the corporate email
> servers.

But how far could we push this reasoning ? I've worked for a company
whose corporate policy was that all source code had to be stored in SVN,
not exception. I didn't reach the community to move kernel development
away from git. I've also worked with people whose corporate policy was
that they had to do Linux kernel development on Windows, without using
any virtual machine. There are all kind of crazy corporate policies, and
if we don't push the pain it inflicts on all of us back to the people
who enact those crazy policies, we'll always lose.

> Now, if we had a way to send and receive upstream patches via a web
> site, that would actually make things easier.

-- 
Regards,

Laurent Pinchart
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Konstantin,

On Tue, Oct 15, 2019 at 11:40:10AM -0400, Konstantin Ryabitsev wrote:
> On Mon, Oct 14, 2019 at 06:33:33PM +0300, Laurent Pinchart wrote:
> > If the goal is to work around SMTP-related technical issues, is a web UI
> > really the best way to go ? Wouldn't it be better to do the same through
> > a git push ? We could setup a git server that requires authentication,
> > and implement a push-to-email bridge. The information that would need to
> > be entered in a web UI could be put in a tag message, and we could have
> > a CLI to create the tag from a list of questions.
> 
> Well, this is largely what GitGitGadget does 
> (https://gitgitgadget.github.io), and we could go that route, sure. I'm 
> reluctant only because, quoth:
> 
>   GitGitGadget itself is a GitHub App that is backed by an Azure 
>   Function written in pure Javascript which in turn triggers an Azure 
>   Pipeline written in Typescript (which is really easy to understand and 
>   write for everybody who knows even just a little Javascript), 
>   maintained at https://github.com/gitgitgadget/gitgitgadget.
> 
> I have zero familiarity with any of the above. That said, we do have a 
> bunch of CI engineers working at the LF, and I can probably avail myself 
> of their expertise if we decide to set this up.

I certainly wouldn't recommend a solution based on a proprietary
closed-source stack :-) But as we're talking about performing new
development for patchwork, I wanted to point out that we could also
consider a different technical approach that would involve new
development for a different open-source project. For instance, is the
above idea something that could be developed on top of gitolite ? Or
possibly even as a tiny standalone git server ?

-- 
Regards,

Laurent Pinchart
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

Hi Dmitry,

On Tue, Oct 15, 2019 at 06:49:27AM +0200, Dmitry Vyukov wrote:
> On Mon, Oct 14, 2019 at 5:12 PM Laurent Pinchart wrote:
> > On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote:
> >> On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote:
> >>> On 10/11/19 2:57 AM, Greg KH wrote:
>  On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote:
> > Hi, all:
> >
> > I would like to propose a new (large) feature to patchwork with the 
> > goal to
> > make the process of submitting a patch easier for newbies and people
> > generally less familiar with patch-based development. This was discussed
> > previously on the workflows list:
> > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/
> >
> > How I envision this would work:
> >
> > - user creates an account (which requires a mail confirmation) >> - 
> > they choose a "submit patch" option from the menu
> > - the patch submission screen has a succession of screens:
> >
> >   1. a screen with a single field allowing a user to paste a URL to 
> > their
> > fork of the git repository. Once submitted, patchwork does a "git
> > ls-remote" to attempt to get a list of refs and to verify that this 
> > is
> > indeed a valid git repository
> 
>  s/valid git repository/valid git repository based on the kernel git tree/
> 
>  Otherwise you might be sending out lots of emails for other projects :)
> 
> >   2. next screen asks the user to select the ref to work from using the
> > list obtained from the remote. Once submitted, patchwork performs a 
> > `git
> > clone --reference` to clone the repository locally using a local 
> > fork of
> > the same repo to minimize object transfer. This part requires that:
> >a. patchwork project is configured with a path to a local fork,
> > if this feature is enabled for a project
> >b. that fork is kept current via some mechanism outside of
> > patchwork (e.g. with grokmirror)
> >c. there is some sanity-checking during the clone process to
> > avoid abuse (e.g. a sane timeout, a tmpdir with limited size,  
> > etc
> > -- other suggestions welcome)
> >
> >   3. next screen asks the user to pick a starting commit from the log.
> > Once submitted, patchwork generates the patch from the commit 
> > provided
> > to the tip of the branch selected by the user earlier,
> >  using git format-patch.
> >
> >   4. next screen asks the user to review the patch to make sure this is
> > what they want to submit. Once confirmed, patchwork performs two
> > admin-defined optional hooks:
> >
> >a. a hook to generate a list of cc's (e.g. get_maintainer.pl)
> >b. a sanity check hook (e.g. checkpatch.pl)
> 
>  I will note that many "first patch" submissions are checkpatch.pl
>  cleanups for staging.  When doing that, I require that they do "one
>  logical change per patch", which means that many of the individual
>  patches themselves will not be checkpatch.pl clean, because many lines
>  have multiple issues with them (tabs, spaces, format, length, etc.)
> 
>  So other than that minor thing, sounds interesting.  It's hard to
>  determine just how difficult the whole "set up git and send a patch out"
>  process is for people these days given the _huge_ numbers of new
>  contributions we keep getting, and the numerous good tutorials we have
>  created that spell out exactly how to do this.
> 
>  So you might be "solving" a problem that we don't really have.  It's
>  hard to tell :(
> >>>
> >>> I agree with this. I don't think this a problem that is worth solving.
> >>> When a new developer wants to send a patch, they don't need to create
> >>> any accounts. They setup their email client and send patch.
> >>>
> >>> We have several resources that walk them through setting up email
> >>> clients and sending patches. checkpatch.pl can be automated with
> >>> git hooks.
> >>>
> > I know this is a pretty big RFE, and I would like to hear your thoughts
> > about this. If there is general agreement that this is doable/good 
> > idea, I
> > may be able to come up with funding for this development as part of the
> > overall tooling improvement proposal.
> 
>  The workflow seems sane, and matches what most people do today, with the
>  exception that it "solves" the git send-email issue, right?  Is that our
>  biggest barrier?
> 
>  I would recommend interviewing some of the recent kernel mentor project
>  and outreachy applicants first, to try to determine exactly what their
>  problems, if any, were with our development process.  If they say that
>  this type of tool/workflow would have saved them hours of time 

Re: RFE: use patchwork to submit a patch

[Laurent Pinchart]

On Tue, Oct 15, 2019 at 11:11:57AM +0200, Dmitry Vyukov wrote:
> On Tue, Oct 15, 2019 at 10:57 AM Eric Wong wrote:
> > Dmitry Vyukov wrote:
> >> As one data point, I cannot send emails with git send-email anymore.
> >> It used to work, then broke and I don't know how to fix it. Now it says:
> >>
> >> 5.7.8 Username and Password not accepted. Learn more at
> >> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> >> s10sm8376885wrr.5 - gsmtp
> >>
> >> I suspect it has something to do with two factor auth.
> >> So that's it: it cannot contribute to kernel right now.
> >> I will not consider time spent fixing it as useful time investment.
> >
> > I'm sorry you feel that way about time investments...
> > But I've always assumed that's also the sentiment for time spent
> > learning ANY new tools or workflow changes that come along.
> 
> This is true. But the fact that there is a learning curve to anything
> does not justify any learning curve for everything. Some parts of
> technology may be isolated completely and one does not need to learn
> anything about that part. For example, today to compile a high-level
> language one generally does not need to learn anything about machine
> instructions.

That's only true to some extent. Yes, you can develop in java without
knowing what a CPU is, but to develop efficient and safe software, a
knowledge of the whole stack is very useful, when not required.

> So the question is: is SMTP/IMAP is something that
> inherently needs to be learned for contribution to kernel or it can be
> hidden/not required/made simpler? And looking at github/facebook I
> would assume that contributors do not have to be exposed to that.

Could we develop a patch submission and review based on facebook if we
wanted to ? Most likely yes. Would we consider that as a good idea ?
Most likely no. There are always pros and cons in any workflow, and
while I could personally move away from e-mail, I would want a solution
that brings me most of the benefits of e-mail, in particular
decentralisation. git**b creates both a central point of failure and a
central point of trust, so it's a big no-go as far as I'm concerned. On
the other hand, creating a solution that enables optional use of forges
for contributors would prefer using them doesn't bother me at all, quite
the contrary.

> >> Any kernel documentation that I can find for gmail, mentions config
> >> that I am already using and that is not working:
> >> https://www.kernel.org/doc/html/latest/search.html?q=gmail_keywords=yes=default#
> >> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail
> >
> > Fwiw, git-send-email(1) manpage also has a special section for gmail:
> >
> >   https://kernel.org/pub/software/scm/git/docs/git-send-email.html
> >
> > and a link for app-specific passwords:
> >
> >   https://security.google.com/settings/security/apppasswords
> >
> > Perhaps that helps?
> 
> For me that page says "The setting you are looking for is not
> available for your account".
> I suspect app passwords work if 2-factor auth is enabled, but what
> enabled on my account is "Use your phone to sign in", which is
> different from 2-factor auth setting.

Did git-send-email start failing when enabling phone as a mean to sign
in, or was it unrelated ?

-- 
Regards,

Laurent Pinchart
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Sat, Oct 12, 2019 at 09:19:11AM +0200, Greg KH wrote:
This is basically why SMTP sucks in my view -- and it's worthless 
trying to

pick fights with IT departments, because they are told to do so by lawyers.
So, I want to take SMTP out of the equation:

1. provide a way for someone to submit a patch using a web interface   (but
still in a way that From: is their corporate ID)


If you do this, what happens when a maintainer/reviewer responds to that
patch and says "looks good, but can you change X and resend it?"

How will they get that message if it didn't go through their email
system?  How will they be able to respond to it?


The magic of git and email headers. Only the initial submission will go 
out of this web service -- the follow-up is expected to arrive into 
their mailbox.



2. use individual git feeds as a way to send out patches instead of   always
being secondary to SMTP


Sending patches that way is one thing, the interaction based on those
patches is another.

Everyone needs to remember that only 1/3 of the patches submitted are
applied.  The "normal" path of development is at least a review/resend
cycle for submissions (2/3 of patches).  So that 2/3 can't be ignored as
the "new/drive-by submissions" are probably more in that category than
not.


Right, the idea is that the imaginary tool that is backed by 
public-inbox will use multiple sources of content:


- the mailing list repository
- individual developer feeds
- the person's imap folder

Anything that shows up in the individual feeds should have a 
corresponding entry in the mailing list repository, but the latter is 
also cryptographically signed and therefore end-to-end attestable.  
Mailing lists and SMTP continue to be the fallback delivery method for 
the vast majority of the content.


I expect to lay this out in more detail as I prepare the "kthul MVP" 
roadplan.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Konstantin Ryabitsev]

On Tue, Oct 15, 2019 at 09:35:27AM -0400, Theodore Y. Ts'o wrote:

On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:

Now, if we had a way to send and receive upstream patches via a web
site, that would actually make things easier.


Alas, for those corporations who choose to enable DMARC in hard-fail
mode (which includes google.com BTW, although thankfully, not
gmail.com) sending (from DMARC's perspective, "forging") e-mail from
someone at that particular domain won't work unless it comes from an
authorized e-mail server.


This is not a huge deal for web services mailing patches, because they 
just need to specify the `From:` header in the body of the patch. From 
`man git-am`:


 "From: " and "Subject: " lines starting the body override the 
 respective commit author name and title values taken from the headers.


So, a web service emailing patches can have:

 From j...@webserviceaddr.example.org
 From: J.Doe 
 Reply-to: J.Doe 
 Cc: j...@corpaddress.example.com
 Subject: [PATCH] Fix foo in bar

 From: J.Doe 

 Signed-off-by: J.Doe 
 ---

This *should* be kosher with legal requirements in most companies:

- the email traverses the corporate server when the cc is received, so 
 there is a log of it in their legally required auto-cc inbox

- all replies will be sent to the corporate server
- when the patch is applied, the commit will be properly listing the 
 corporate address



Of course, we do have a workaround for that, which is to use a
kernel.org address.


Only a very small subset of people have these, and it may still be not 
allowed by individual company's legal departments even when this is 
available.


-K
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Theodore Y. Ts'o]

On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote:
> Now, if we had a way to send and receive upstream patches via a web
> site, that would actually make things easier.

Alas, for those corporations who choose to enable DMARC in hard-fail
mode (which includes google.com BTW, although thankfully, not
gmail.com) sending (from DMARC's perspective, "forging") e-mail from
someone at that particular domain won't work unless it comes from an
authorized e-mail server.

Of course, we do have a workaround for that, which is to use a
kernel.org address.

- Ted

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Steven Rostedt]

On Mon, 14 Oct 2019 16:56:58 -0400
"Theodore Y. Ts'o"  wrote:

> So it's an extreme case, and is an example of corp e-mail insanity.
> The good news is that for companies that are really serious about
> working with upstream, they will roll out solutions for their own
> customers.  For example, with IBM, since Lotus notes was maximally
> unfriendly to OSS development practices, they created a separate mail
> system with l...@linux.ibm.com addresses be cause it was simply
> impossible to engage with upstream development using ibm.com
> addresses.  So there are work arounds, and I encourage you to reach
> out to other Google kernel developers --- and if you are unhappy,
> please file internal bugs against the corp infrastructure.

Note, one issue is that some corporations do not want to support two
email services. It's hard enough securing one, opening up another one
is just opening another door to be cracked.

We do file bugs when there's issues, but that doesn't mean they will be
resolved. I have special permission to use my personal account, but
most people at my company do not have that permission, and this is a
heart ache for those that need to send email via the corporate email
servers.

Now, if we had a way to send and receive upstream patches via a web
site, that would actually make things easier.

-- Steve
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: RFE: use patchwork to submit a patch

[Eric Wong]

Dmitry Vyukov  wrote:
> As one data point, I cannot send emails with git send-email anymore.
> It used to work, then broke and I don't know how to fix it. Now it says:
> 
> 5.7.8 Username and Password not accepted. Learn more at
> 5.7.8  https://support.google.com/mail/?p=BadCredentials
> s10sm8376885wrr.5 - gsmtp
> 
> I suspect it has something to do with two factor auth.
> So that's it: it cannot contribute to kernel right now.
> I will not consider time spent fixing it as useful time investment.

I'm sorry you feel that way about time investments...
But I've always assumed that's also the sentiment for time spent
learning ANY new tools or workflow changes that come along.

> Any kernel documentation that I can find for gmail, mentions config
> that I am already using and that is not working:
> https://www.kernel.org/doc/html/latest/search.html?q=gmail_keywords=yes=default#
> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail

Fwiw, git-send-email(1) manpage also has a special section for gmail:

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

and a link for app-specific passwords:

  https://security.google.com/settings/security/apppasswords

Perhaps that helps?
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 0/2] Add writeable 'bundles' API

[Stephen Finucane]

On Sun, 2019-09-08 at 23:31 +0100, Stephen Finucane wrote:
> Hopefully self-explanatory. This closes another gap with the web UI and
> REST API.

Assuming no one disagrees, I'm going to merge this series before the
end of the week. It has tests and documentation, suggesting it's low
enough risk, and is the sole remaining blocker for v2.2.0.

Stephen

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork