Re: RFE: use patchwork to submit a patch
On Mon, Nov 11, 2019 at 10:35:10AM +0100, Dmitry Vyukov wrote: > On Sat, Nov 9, 2019 at 5:31 AM Theodore Y. Ts'o wrote: > > Essentially format=flowed is only supposed to be used when it's > > considered OK for the receiver to reflow (or not) the lines any darned > > way it wants. > But for me the point re email still stands: why are we even spending > time discussing this? Why are there such extensions with MAY status? > If one doesn't care if the received with recover the original message > or not, why caring adding/specifying these "format=flowed; > delsp=yes"?... > Obviously somebody in the middle got confused about these flowed/delsp > as well and either assumed that they are MUST or assumed that > preserving precise text for emails is just never important... Not sure if you're looking for a serious answer here or not and it's not really your point but the use case for format=flowed is for plain text mails. The sending MUA should flow the mail into 80 columns so it will render OK as text but if something wants to reflow (eg, to fit within a window) like it would for a paragraph in HTML mail then it can. It's optional so that things where the formatting is important don't get disrupted. > Regarding another mail agent: again this only proves the point for me: > this is what tool developers are forced to be spending their resources > on, rather then working on adding more useful features... > I don't even know where to start re switching mail transport; how much > the switch will cost? what are other transport costs in the long term > maintenance? what are their problems? We're not going to get away from interoperability problems no matter what we use, especially if we are mixing things intended for human and machine consumption. signature.asc Description: PGP signature ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Nov 08, 2019 at 03:25:00PM +0100, Dmitry Vyukov wrote: > > Would you like a mail sending account so you can use mail.kernel.org > > as > > your outgoing smtp? We promise we won't do anything that nasty. :) > > > > -K > > You know that we also serve Fuchsia, Akaros, FreeBSD, NetBSD, OpenBSD > and potentially Windows and Darwin in the future, right? :) > > How does that work? Currently we just call mail.Send: > https://cloud.google.com/appengine/docs/standard/go/mail/reference > and it magically does the thing. I can't even setup my own email > client, so don't expect lots from me :) I didn't realize you were using appengine, so I have no idea if using an external SMTP submission server is even feasible. I'm sure it can be done, but since you're serving all those other communities as well, that makes the offer of acting as a submission point a bit more tricky. :) Perhaps we can revisit this idea once you start publishing a feed? -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, Nov 11, 2019 at 10:35:10AM +0100, Dmitry Vyukov wrote: > > Regarding another mail agent: again this only proves the point for me: > this is what tool developers are forced to be spending their resources > on, rather then working on adding more useful features... > I don't even know where to start re switching mail transport; how much > the switch will cost? what are other transport costs in the long term > maintenance? what are their problems? This is a GMail bug/problem. Your reply to me was sent without format=flowed, so it's certainly possible that GMail not use format=flowed. There are plenty of open source mail transport agents where sending this without format=flowed would be trivial - Ted ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Nov 08, 2019 at 03:25:00PM +0100, Dmitry Vyukov wrote: > > It has the format=flowed; delsp=yes part: > > Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes > > Which according to https://tools.ietf.org/html/rfc3676 seems to mean > that that line is not wrapped: new line and 1 space needs to be > removed as part of decoding it. Actually if you read RFC 3676 carefully, the receiver MAY decode as you have described above. It's not even a SHOULD or a MUST. Essentially format=flowed is only supposed to be used when it's considered OK for the receiver to reflow (or not) the lines any darned way it wants. If you use format=flowed in a context where (a) there might be whitespace at the end of lines, and random reflowing is a really bad idea (e.g., C/C++/Python/Java source files, or patches), or (b) where you really care about whether line breaks when text is quoted and used in replies (e.g., syzbot's parser which cares about line breaks), you MUST NOT use format=flowed, or things will only end in tears. If the problem here is that syzbot mail, and the responses to its mails, can't be broken in random places, or syzbot parser will be unhappy. In that case, you shouldn't have sent it using format=flowed in the first place. There are other MIME text formats that are appropriate in that particular case, but format=flowed is not one of them. And if GMail doesn't give you that option, then I suggest that you either make the Syzbot parser more forgiving, or you find another mail transport agent. - Ted ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Ted, On Fri, Nov 08, 2019 at 09:02:49AM -0500, Theodore Y. Ts'o wrote: > On Fri, Nov 08, 2019 at 10:44:25AM +0100, Dmitry Vyukov wrote: > > syzbot asks to provide fixing commit in a particular format (and on 1 > > line, otherwise not possible to parse back). > > > > It sent this yesterday: > > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/xAkxm6H4EQAJ > > This is Shuah's reply, not the syzbot original message. Looks like > you cut and pasted the same URL twice? > > The correct URL is: > > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/IZZUu-BobtI/wQdqUSHuEQAJ > > .. and it looks like what Syzbot sent was already line-wrapped: > > > If the result looks correct, please mark the bug fixed by replying with: > > > >#syz fix: usb: usbip: Fix BUG: KASAN: slab-out-of-bounds in > >vhci_hub_control() > > So I don't think you can blame this on how Shuah replied, or how > Mozilla handled things. It looks like either Syzbot or more likely, > GMail on its outgoing path "added value" by line-wrapping the message, > probably because of the following MIME setting: > > > Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes > > Flowing text lines and deleting spaces is always, ALWAYS, a bad idea > when sending plain text. In this case, it wasn't just a patch that > got corrupted, but the instructions from Syzbot, which it appears > Shuah followed faithfully. :-) > > IBM handled this problem by standing up an open source e-mail system > because Lotus Notes couldn't be fixed. If we can't fix GMail, then > David Miller may be right, and e-mail may be doomed. (I don't like > that answer myself, since so far all of the alternatives seem to be > categorically worse for our use case, but...) Since when did we decide to let Google dictate what tools were are or are not allowed to use ? :-) If gmail isn't suitable as an e-mail provider, we should be vocal about it. -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Nov 08, 2019 at 03:12:51PM +0100, Dmitry Vyukov wrote: > Just to clarify: syzbot does not do this on purpose. It generates and > hands off unwrapped lines. The "format=flowed; delsp=yes" part was > done by somebody else. Looks like vger received it already mangled: https://lore.kernel.org/linux-usb/c460630596c1d...@google.com/raw Would you like a mail sending account so you can use mail.kernel.org as your outgoing smtp? We promise we won't do anything that nasty. :) -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, Oct 24, 2019 at 3:58 PM Steven Rostedt wrote: > > On Thu, 24 Oct 2019 15:33:04 +0200 > Dmitry Vyukov wrote: > > > On Thu, Oct 24, 2019 at 3:15 PM Steven Rostedt wrote: > > > > > > On Mon, 21 Oct 2019 18:39:18 +0300 > > > Laurent Pinchart wrote: > > > > Purely theoretically let's consider that the changes do not improve > > _your_ efficiency, but they significantly improve overall project > > efficiency by positively affecting people who did not develop a > > workflow over the past decades (maybe there were not around 2 decades > > ago) and positively affecting various tooling that _you_ may be > > directly interested in, but otherwise they are important for the > > project overall. So for you it's no change in efficiency except that > > you now need to do things differently. What do you think about such > > changes? Are you ready to force yourself? :) > > I think it's quite cornerstone question here. All (?) major figures in > > the kernel (who are ~~98% of decision making, but ~~2% of kernel > > developers overall) have developed workflows over the past decades > > that work reasonably well for them. If they veto all proposed changes > > based on the criteria you described, every new contributor will need > > decades to develop own workflows to become an efficient contributor > > and lots of tooling will be painfully hard to do. > > > > The above sound like a one size fits all approach, which I would caste > a veto to. I would like a solution that works for multiple workflows. > One where mine and others still work too. > > Please, lets work on a infrastructure that is robust and flexible, that > is split into back and front ends. That way, we have a single "back > end" and multiple front ends that suite everyone's needs. > > -- Steve I agree that the system must not lock in into a single UI. However, (1) somebody will still need to glue all old frontends with the new backend. (2) some workflows may still break regardless of a perfect backend/frontend split. Consider, previously one could say in arbitrary plain English that a bug X is fixed by commit Y, but if we want to build anything on top of that can't be plain English anymore and will require changes in workflows. FWIW, Gerrit satisfies most of the criteria. It provides a nicely formatted database that you can easily git fetch and take with you offline. In this sense it is a robust and flexible backend. However, one won't get all of hundreds of existing scripts and systems work on top of it out-of-the-box. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, 24 Oct 2019 15:33:04 +0200 Dmitry Vyukov wrote: > On Thu, Oct 24, 2019 at 3:15 PM Steven Rostedt wrote: > > > > On Mon, 21 Oct 2019 18:39:18 +0300 > > Laurent Pinchart wrote: > > Purely theoretically let's consider that the changes do not improve > _your_ efficiency, but they significantly improve overall project > efficiency by positively affecting people who did not develop a > workflow over the past decades (maybe there were not around 2 decades > ago) and positively affecting various tooling that _you_ may be > directly interested in, but otherwise they are important for the > project overall. So for you it's no change in efficiency except that > you now need to do things differently. What do you think about such > changes? Are you ready to force yourself? :) > I think it's quite cornerstone question here. All (?) major figures in > the kernel (who are ~~98% of decision making, but ~~2% of kernel > developers overall) have developed workflows over the past decades > that work reasonably well for them. If they veto all proposed changes > based on the criteria you described, every new contributor will need > decades to develop own workflows to become an efficient contributor > and lots of tooling will be painfully hard to do. > The above sound like a one size fits all approach, which I would caste a veto to. I would like a solution that works for multiple workflows. One where mine and others still work too. Please, lets work on a infrastructure that is robust and flexible, that is split into back and front ends. That way, we have a single "back end" and multiple front ends that suite everyone's needs. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, Oct 24, 2019 at 3:15 PM Steven Rostedt wrote: > > On Mon, 21 Oct 2019 18:39:18 +0300 > Laurent Pinchart wrote: > > > > I plan on continuing to develop mostly in email (I still send my patch > > > series via quilt!). But I'm not going to enforce everyone to continue > > > to use email if we can come up with a better way. I also want to make > > > sure that whatever we do come up with will still support email. > > > > Hypothetically speaking, if there was a service that allowed sending a > > patch series through a git push with a tag containing a cover letter in > > its message, and that service would send out e-mails to mailing lists > > (with the option to self-host it if you want), would you consider using > > it ? > > Only if I was forced to ;-) > > Seriously, it's hard to teach an old dog new tricks. I wont change > until it becomes necessary to do so. As the old saying goes: "Don't fix > what ain't broke!" > > I've developed a workflow over the past decade or so, that I've > optimized to fit my needs. Why would I want to change that? Unless > there's something that changes that can make an impact on my > efficiency. Purely theoretically let's consider that the changes do not improve _your_ efficiency, but they significantly improve overall project efficiency by positively affecting people who did not develop a workflow over the past decades (maybe there were not around 2 decades ago) and positively affecting various tooling that _you_ may be directly interested in, but otherwise they are important for the project overall. So for you it's no change in efficiency except that you now need to do things differently. What do you think about such changes? Are you ready to force yourself? :) I think it's quite cornerstone question here. All (?) major figures in the kernel (who are ~~98% of decision making, but ~~2% of kernel developers overall) have developed workflows over the past decades that work reasonably well for them. If they veto all proposed changes based on the criteria you described, every new contributor will need decades to develop own workflows to become an efficient contributor and lots of tooling will be painfully hard to do. >I've created a bunch of scripts that do exactly what I > expect them to. To change, I'm going back into the world of the unknown, > and will need to take up time (which I'm struggling to come up with), > to learn something new that may or may not make it better for me. > > The tool I would like is simply something where all the patches I > receive are stored in a nice formatted database that I can process > offline somewhere (like when I'm flying to Europe). > > Right now I hacked an old version of patchwork (when I upgraded, > everything broke, and I had to revert it), along with procmail, that > creates a local patchwork queue I can access locally. Unfortunately, > this doesn't work when I'm away from home without an internet > connection. > > I would love to be able to take this database with me and process the > patches while traveling without connectivity. > > -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, 21 Oct 2019 18:39:18 +0300 Laurent Pinchart wrote: > > I plan on continuing to develop mostly in email (I still send my patch > > series via quilt!). But I'm not going to enforce everyone to continue > > to use email if we can come up with a better way. I also want to make > > sure that whatever we do come up with will still support email. > > Hypothetically speaking, if there was a service that allowed sending a > patch series through a git push with a tag containing a cover letter in > its message, and that service would send out e-mails to mailing lists > (with the option to self-host it if you want), would you consider using > it ? Only if I was forced to ;-) Seriously, it's hard to teach an old dog new tricks. I wont change until it becomes necessary to do so. As the old saying goes: "Don't fix what ain't broke!" I've developed a workflow over the past decade or so, that I've optimized to fit my needs. Why would I want to change that? Unless there's something that changes that can make an impact on my efficiency. I've created a bunch of scripts that do exactly what I expect them to. To change, I'm going back into the world of the unknown, and will need to take up time (which I'm struggling to come up with), to learn something new that may or may not make it better for me. The tool I would like is simply something where all the patches I receive are stored in a nice formatted database that I can process offline somewhere (like when I'm flying to Europe). Right now I hacked an old version of patchwork (when I upgraded, everything broke, and I had to revert it), along with procmail, that creates a local patchwork queue I can access locally. Unfortunately, this doesn't work when I'm away from home without an internet connection. I would love to be able to take this database with me and process the patches while traveling without connectivity. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Steven, On Tue, Oct 15, 2019 at 12:47:49PM -0400, Steven Rostedt wrote: > On Tue, 15 Oct 2019 19:37:04 +0300 Laurent Pinchart wrote: > > > But how far could we push this reasoning ? I've worked for a company > > whose corporate policy was that all source code had to be stored in SVN, > > not exception. I didn't reach the community to move kernel development > > away from git. I've also worked with people whose corporate policy was > > that they had to do Linux kernel development on Windows, without using > > any virtual machine. There are all kind of crazy corporate policies, and > > if we don't push the pain it inflicts on all of us back to the people > > who enact those crazy policies, we'll always lose. > > I understand your sentiment. But most of your examples are one-off > "crazy corporate policies". The "sucky email server" and "you must use > this sucky email server" is rather standard. Not saying that we don't > want to pressure them to change, but I'm hearing from people (like Dave > Miller), that new developers are moving away from email for > development. I think that's two different issues though, which certainly overlap (but I'm not sure to what extent). I'm also not sure if we can say that new developers are moving away from e-mail, or simply start development with a different, non e-mail-based process. It makes a bit of a difference, because some new developers may not have any issue with e-mail, they may just not have considered it. This of course doesn't mean that nobody has issues with e-mail, that nobody is moving away from e-mail, or that nobody would have trouble moving to e-mail. > I thought part of this thread was talking about having other ways > besides email to submit a patch. Something that can help people > correctly send a patch (at least their formatting) by making sure they > fill out the proper fields and have a tool that helps them do so. I > brought up the corporate email servers just as an example of having > this help out there too. I'm very biased here, but for me a CLI tool that asks me questions and generates the patch series (or directly sends it) would be easier to use than a web UI. That's because I would stay on the CLI that I'm already using for git. This of course assumes a working git-send-email configuration, which we all no is not a given. I thus wonder if it wouldn't be better to have a CLI helper to create the patches, and provide some sort of https-to-smtp service (possibly in the form of pushing a tag with a message for instance). There are many options possible to work around broken e-mail workflows, so instead of focussing on a web UI because it was the first idea that was proposed, let's make sure we consider other ones. And maybe we'll decide that a web UI is the best option. > I plan on continuing to develop mostly in email (I still send my patch > series via quilt!). But I'm not going to enforce everyone to continue > to use email if we can come up with a better way. I also want to make > sure that whatever we do come up with will still support email. Hypothetically speaking, if there was a service that allowed sending a patch series through a git push with a tag containing a cover letter in its message, and that service would send out e-mails to mailing lists (with the option to self-host it if you want), would you consider using it ? -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Tue Oct 15, 2019 at 7:44 PM Laurent Pinchart wrote: > I'm not very confident that perfect transparent e-mail bridges could be > developed, and the culprit here it e-mail. From forge to e-mail there's > no real issue, we have structure data, and we write it out in text form. > The other way around, though, involves recovering structure from text. > If the MTAs, MDAs, MUAs and, quite importanttly, users, behave > correctly, that's doable. We can handle some of the "features" of common > M*A, but if someone decides to throw the formatting through the window > completely, we'll be screwed. Well, no, it can never be perfect. But I'm not trying to accomodate for, as an example, someone who's deliberately trying to confusing the system. It will probably always be based on heuristics in some degrees. However, I think that social conformance is a force to be reckoned with. Most people don't have to be taught how code review works on mailing lists, they just watch others do it and naturally fall in line. Without any formal structure at all, we've already more or less settled on a shared set of patterns for email-driven code review. However, I think that the following: > An idea I was toying with in the past was to create a structured format > for review, that would match what we use now in e-mail very closely, but > with a clearly defined syntax and grammar. The format would appear as > just plain e-mails to users, and a MUA that behaves reasonably would > likely produce valid structured data as e-mail replies. Over time we > could develop clients or teach some MUAs about the structured format, > removing the risk that they, or the end users, would mess up a reply. > The migration would be mostly transparent as it wouldn't really impact > existing users of e-mail, and could thus be rolled out over a long > period of time. Is very reasonable. If we can come up with a standard which feels like what we're already doing, but is written down, then tools have something to conform to and users have some expectations for how to behave to make their tools happy. I would be up for putting up some kind of simple spec for bikeshedding purposes, and teaching it to the thread parser that was built for lists.sr.ht: https://git.sr.ht/~emersion/python-emailthreads ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Tue, 15 Oct 2019 19:37:04 +0300 Laurent Pinchart wrote: > But how far could we push this reasoning ? I've worked for a company > whose corporate policy was that all source code had to be stored in SVN, > not exception. I didn't reach the community to move kernel development > away from git. I've also worked with people whose corporate policy was > that they had to do Linux kernel development on Windows, without using > any virtual machine. There are all kind of crazy corporate policies, and > if we don't push the pain it inflicts on all of us back to the people > who enact those crazy policies, we'll always lose. I understand your sentiment. But most of your examples are one-off "crazy corporate policies". The "sucky email server" and "you must use this sucky email server" is rather standard. Not saying that we don't want to pressure them to change, but I'm hearing from people (like Dave Miller), that new developers are moving away from email for development. I thought part of this thread was talking about having other ways besides email to submit a patch. Something that can help people correctly send a patch (at least their formatting) by making sure they fill out the proper fields and have a tool that helps them do so. I brought up the corporate email servers just as an example of having this help out there too. I plan on continuing to develop mostly in email (I still send my patch series via quilt!). But I'm not going to enforce everyone to continue to use email if we can come up with a better way. I also want to make sure that whatever we do come up with will still support email. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Tue, Oct 15, 2019 at 06:41:03AM +0200, Dmitry Vyukov wrote: > On Mon, Oct 14, 2019 at 5:17 PM Greg KH wrote: > > > As one data point, I cannot send emails with git send-email anymore. > > > It used to work, then broke and I don't know how to fix it. Now it says: > > > > > > 5.7.8 Username and Password not accepted. Learn more at > > > 5.7.8 https://support.google.com/mail/?p=BadCredentials > > > s10sm8376885wrr.5 - gsmtp > > > > > > I suspect it has something to do with two factor auth. > > > So that's it: it cannot contribute to kernel right now. > > > > That is because your employer changed how it manages imap. So yes, this > > configuration is now broken, you can not contribute to the kernel this > > way. They know about it, and there's an "opt-out" list you can sign up > > for if you want to fix it. Nothing the community can do about something > > crazy like this. > > I used my private gmail account to send emails. My private email > account stopped working. I think you need a better email provider :) ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Tue, Oct 15, 2019 at 07:32:41PM +0300, Laurent Pinchart wrote: Well, this is largely what GitGitGadget does (https://gitgitgadget.github.io), and we could go that route, sure. I'm reluctant only because, quoth: GitGitGadget itself is a GitHub App that is backed by an Azure Function written in pure Javascript which in turn triggers an Azure Pipeline written in Typescript (which is really easy to understand and write for everybody who knows even just a little Javascript), maintained at https://github.com/gitgitgadget/gitgitgadget. I have zero familiarity with any of the above. That said, we do have a bunch of CI engineers working at the LF, and I can probably avail myself of their expertise if we decide to set this up. I certainly wouldn't recommend a solution based on a proprietary closed-source stack :-) But as we're talking about performing new development for patchwork, I wanted to point out that we could also consider a different technical approach that would involve new development for a different open-source project. For instance, is the above idea something that could be developed on top of gitolite ? Or possibly even as a tiny standalone git server ? I wouldn't do it on top of gitolite, because the administrative overhead would be too large. It certainly can be done as a separate service -- after all, any relation between this tool and patchwork is pretty tenuous. That said, I'm leaning towards shelving this idea for the moment, at least as an official service provided by kernel.org -- it is easier to limit the scope at first and target maintainer tools and communication frameworks. It seems that sr.ht folks are already making something like this available, so we can just sit on our hands for a bit and see where this takes us. -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Drew, On Tue, Oct 15, 2019 at 12:34:57PM -0400, Drew DeVault wrote: > On Tue Oct 15, 2019 at 7:32 PM Laurent Pinchart wrote: > > I certainly wouldn't recommend a solution based on a proprietary > > closed-source stack :-) But as we're talking about performing new > > development for patchwork, I wanted to point out that we could also > > consider a different technical approach that would involve new > > development for a different open-source project. For instance, is the > > above idea something that could be developed on top of gitolite ? Or > > possibly even as a tiny standalone git server ? > > Some of the work I'm doing with code review on sourcehut can be > generalized, to support writing up arbitrary mailing lists with > arbitrary review tools with a bidirectional workflow. I intend to make a > reality of this for at least lists.sr.ht <-> gitlab and lists.sr.ht <-> > github, so that patchsets and the resulting discussion gets turned into > git{hub,lab} code reviews, and vise versa. I think it's possible to make > a system which presents both sides with an experience that is idiomatic > for each context, so that you can't really tell whose using which tool > because they're both speaking each other's conventional language. First of all, please let me thank you for all the effort you're putting into this. I'm not very confident that perfect transparent e-mail bridges could be developed, and the culprit here it e-mail. From forge to e-mail there's no real issue, we have structure data, and we write it out in text form. The other way around, though, involves recovering structure from text. If the MTAs, MDAs, MUAs and, quite importanttly, users, behave correctly, that's doable. We can handle some of the "features" of common M*A, but if someone decides to throw the formatting through the window completely, we'll be screwed. An idea I was toying with in the past was to create a structured format for review, that would match what we use now in e-mail very closely, but with a clearly defined syntax and grammar. The format would appear as just plain e-mails to users, and a MUA that behaves reasonably would likely produce valid structured data as e-mail replies. Over time we could develop clients or teach some MUAs about the structured format, removing the risk that they, or the end users, would mess up a reply. The migration would be mostly transparent as it wouldn't really impact existing users of e-mail, and could thus be rolled out over a long period of time. -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Steven, On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote: > On Mon, 14 Oct 2019 16:56:58 -0400 "Theodore Y. Ts'o" wrote: > > > So it's an extreme case, and is an example of corp e-mail insanity. > > The good news is that for companies that are really serious about > > working with upstream, they will roll out solutions for their own > > customers. For example, with IBM, since Lotus notes was maximally > > unfriendly to OSS development practices, they created a separate mail > > system with l...@linux.ibm.com addresses be cause it was simply > > impossible to engage with upstream development using ibm.com > > addresses. So there are work arounds, and I encourage you to reach > > out to other Google kernel developers --- and if you are unhappy, > > please file internal bugs against the corp infrastructure. > > Note, one issue is that some corporations do not want to support two > email services. It's hard enough securing one, opening up another one > is just opening another door to be cracked. > > We do file bugs when there's issues, but that doesn't mean they will be > resolved. I have special permission to use my personal account, but > most people at my company do not have that permission, and this is a > heart ache for those that need to send email via the corporate email > servers. But how far could we push this reasoning ? I've worked for a company whose corporate policy was that all source code had to be stored in SVN, not exception. I didn't reach the community to move kernel development away from git. I've also worked with people whose corporate policy was that they had to do Linux kernel development on Windows, without using any virtual machine. There are all kind of crazy corporate policies, and if we don't push the pain it inflicts on all of us back to the people who enact those crazy policies, we'll always lose. > Now, if we had a way to send and receive upstream patches via a web > site, that would actually make things easier. -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Konstantin, On Tue, Oct 15, 2019 at 11:40:10AM -0400, Konstantin Ryabitsev wrote: > On Mon, Oct 14, 2019 at 06:33:33PM +0300, Laurent Pinchart wrote: > > If the goal is to work around SMTP-related technical issues, is a web UI > > really the best way to go ? Wouldn't it be better to do the same through > > a git push ? We could setup a git server that requires authentication, > > and implement a push-to-email bridge. The information that would need to > > be entered in a web UI could be put in a tag message, and we could have > > a CLI to create the tag from a list of questions. > > Well, this is largely what GitGitGadget does > (https://gitgitgadget.github.io), and we could go that route, sure. I'm > reluctant only because, quoth: > > GitGitGadget itself is a GitHub App that is backed by an Azure > Function written in pure Javascript which in turn triggers an Azure > Pipeline written in Typescript (which is really easy to understand and > write for everybody who knows even just a little Javascript), > maintained at https://github.com/gitgitgadget/gitgitgadget. > > I have zero familiarity with any of the above. That said, we do have a > bunch of CI engineers working at the LF, and I can probably avail myself > of their expertise if we decide to set this up. I certainly wouldn't recommend a solution based on a proprietary closed-source stack :-) But as we're talking about performing new development for patchwork, I wanted to point out that we could also consider a different technical approach that would involve new development for a different open-source project. For instance, is the above idea something that could be developed on top of gitolite ? Or possibly even as a tiny standalone git server ? -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Dmitry, On Tue, Oct 15, 2019 at 06:49:27AM +0200, Dmitry Vyukov wrote: > On Mon, Oct 14, 2019 at 5:12 PM Laurent Pinchart wrote: > > On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote: > >> On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote: > >>> On 10/11/19 2:57 AM, Greg KH wrote: > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote: > > Hi, all: > > > > I would like to propose a new (large) feature to patchwork with the > > goal to > > make the process of submitting a patch easier for newbies and people > > generally less familiar with patch-based development. This was discussed > > previously on the workflows list: > > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > > > > How I envision this would work: > > > > - user creates an account (which requires a mail confirmation) >> - > > they choose a "submit patch" option from the menu > > - the patch submission screen has a succession of screens: > > > > 1. a screen with a single field allowing a user to paste a URL to > > their > > fork of the git repository. Once submitted, patchwork does a "git > > ls-remote" to attempt to get a list of refs and to verify that this > > is > > indeed a valid git repository > > s/valid git repository/valid git repository based on the kernel git tree/ > > Otherwise you might be sending out lots of emails for other projects :) > > > 2. next screen asks the user to select the ref to work from using the > > list obtained from the remote. Once submitted, patchwork performs a > > `git > > clone --reference` to clone the repository locally using a local > > fork of > > the same repo to minimize object transfer. This part requires that: > >a. patchwork project is configured with a path to a local fork, > > if this feature is enabled for a project > >b. that fork is kept current via some mechanism outside of > > patchwork (e.g. with grokmirror) > >c. there is some sanity-checking during the clone process to > > avoid abuse (e.g. a sane timeout, a tmpdir with limited size, > > etc > > -- other suggestions welcome) > > > > 3. next screen asks the user to pick a starting commit from the log. > > Once submitted, patchwork generates the patch from the commit > > provided > > to the tip of the branch selected by the user earlier, > > using git format-patch. > > > > 4. next screen asks the user to review the patch to make sure this is > > what they want to submit. Once confirmed, patchwork performs two > > admin-defined optional hooks: > > > >a. a hook to generate a list of cc's (e.g. get_maintainer.pl) > >b. a sanity check hook (e.g. checkpatch.pl) > > I will note that many "first patch" submissions are checkpatch.pl > cleanups for staging. When doing that, I require that they do "one > logical change per patch", which means that many of the individual > patches themselves will not be checkpatch.pl clean, because many lines > have multiple issues with them (tabs, spaces, format, length, etc.) > > So other than that minor thing, sounds interesting. It's hard to > determine just how difficult the whole "set up git and send a patch out" > process is for people these days given the _huge_ numbers of new > contributions we keep getting, and the numerous good tutorials we have > created that spell out exactly how to do this. > > So you might be "solving" a problem that we don't really have. It's > hard to tell :( > >>> > >>> I agree with this. I don't think this a problem that is worth solving. > >>> When a new developer wants to send a patch, they don't need to create > >>> any accounts. They setup their email client and send patch. > >>> > >>> We have several resources that walk them through setting up email > >>> clients and sending patches. checkpatch.pl can be automated with > >>> git hooks. > >>> > > I know this is a pretty big RFE, and I would like to hear your thoughts > > about this. If there is general agreement that this is doable/good > > idea, I > > may be able to come up with funding for this development as part of the > > overall tooling improvement proposal. > > The workflow seems sane, and matches what most people do today, with the > exception that it "solves" the git send-email issue, right? Is that our > biggest barrier? > > I would recommend interviewing some of the recent kernel mentor project > and outreachy applicants first, to try to determine exactly what their > problems, if any, were with our development process. If they say that > this type of tool/workflow would have saved them hours of time
Re: RFE: use patchwork to submit a patch
On Tue, Oct 15, 2019 at 11:11:57AM +0200, Dmitry Vyukov wrote: > On Tue, Oct 15, 2019 at 10:57 AM Eric Wong wrote: > > Dmitry Vyukov wrote: > >> As one data point, I cannot send emails with git send-email anymore. > >> It used to work, then broke and I don't know how to fix it. Now it says: > >> > >> 5.7.8 Username and Password not accepted. Learn more at > >> 5.7.8 https://support.google.com/mail/?p=BadCredentials > >> s10sm8376885wrr.5 - gsmtp > >> > >> I suspect it has something to do with two factor auth. > >> So that's it: it cannot contribute to kernel right now. > >> I will not consider time spent fixing it as useful time investment. > > > > I'm sorry you feel that way about time investments... > > But I've always assumed that's also the sentiment for time spent > > learning ANY new tools or workflow changes that come along. > > This is true. But the fact that there is a learning curve to anything > does not justify any learning curve for everything. Some parts of > technology may be isolated completely and one does not need to learn > anything about that part. For example, today to compile a high-level > language one generally does not need to learn anything about machine > instructions. That's only true to some extent. Yes, you can develop in java without knowing what a CPU is, but to develop efficient and safe software, a knowledge of the whole stack is very useful, when not required. > So the question is: is SMTP/IMAP is something that > inherently needs to be learned for contribution to kernel or it can be > hidden/not required/made simpler? And looking at github/facebook I > would assume that contributors do not have to be exposed to that. Could we develop a patch submission and review based on facebook if we wanted to ? Most likely yes. Would we consider that as a good idea ? Most likely no. There are always pros and cons in any workflow, and while I could personally move away from e-mail, I would want a solution that brings me most of the benefits of e-mail, in particular decentralisation. git**b creates both a central point of failure and a central point of trust, so it's a big no-go as far as I'm concerned. On the other hand, creating a solution that enables optional use of forges for contributors would prefer using them doesn't bother me at all, quite the contrary. > >> Any kernel documentation that I can find for gmail, mentions config > >> that I am already using and that is not working: > >> https://www.kernel.org/doc/html/latest/search.html?q=gmail_keywords=yes=default# > >> https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail > > > > Fwiw, git-send-email(1) manpage also has a special section for gmail: > > > > https://kernel.org/pub/software/scm/git/docs/git-send-email.html > > > > and a link for app-specific passwords: > > > > https://security.google.com/settings/security/apppasswords > > > > Perhaps that helps? > > For me that page says "The setting you are looking for is not > available for your account". > I suspect app passwords work if 2-factor auth is enabled, but what > enabled on my account is "Use your phone to sign in", which is > different from 2-factor auth setting. Did git-send-email start failing when enabling phone as a mean to sign in, or was it unrelated ? -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Sat, Oct 12, 2019 at 09:19:11AM +0200, Greg KH wrote: This is basically why SMTP sucks in my view -- and it's worthless trying to pick fights with IT departments, because they are told to do so by lawyers. So, I want to take SMTP out of the equation: 1. provide a way for someone to submit a patch using a web interface (but still in a way that From: is their corporate ID) If you do this, what happens when a maintainer/reviewer responds to that patch and says "looks good, but can you change X and resend it?" How will they get that message if it didn't go through their email system? How will they be able to respond to it? The magic of git and email headers. Only the initial submission will go out of this web service -- the follow-up is expected to arrive into their mailbox. 2. use individual git feeds as a way to send out patches instead of always being secondary to SMTP Sending patches that way is one thing, the interaction based on those patches is another. Everyone needs to remember that only 1/3 of the patches submitted are applied. The "normal" path of development is at least a review/resend cycle for submissions (2/3 of patches). So that 2/3 can't be ignored as the "new/drive-by submissions" are probably more in that category than not. Right, the idea is that the imaginary tool that is backed by public-inbox will use multiple sources of content: - the mailing list repository - individual developer feeds - the person's imap folder Anything that shows up in the individual feeds should have a corresponding entry in the mailing list repository, but the latter is also cryptographically signed and therefore end-to-end attestable. Mailing lists and SMTP continue to be the fallback delivery method for the vast majority of the content. I expect to lay this out in more detail as I prepare the "kthul MVP" roadplan. -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Tue, Oct 15, 2019 at 09:35:27AM -0400, Theodore Y. Ts'o wrote: On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote: Now, if we had a way to send and receive upstream patches via a web site, that would actually make things easier. Alas, for those corporations who choose to enable DMARC in hard-fail mode (which includes google.com BTW, although thankfully, not gmail.com) sending (from DMARC's perspective, "forging") e-mail from someone at that particular domain won't work unless it comes from an authorized e-mail server. This is not a huge deal for web services mailing patches, because they just need to specify the `From:` header in the body of the patch. From `man git-am`: "From: " and "Subject: " lines starting the body override the respective commit author name and title values taken from the headers. So, a web service emailing patches can have: From j...@webserviceaddr.example.org From: J.Doe Reply-to: J.Doe Cc: j...@corpaddress.example.com Subject: [PATCH] Fix foo in bar From: J.Doe Signed-off-by: J.Doe --- This *should* be kosher with legal requirements in most companies: - the email traverses the corporate server when the cc is received, so there is a log of it in their legally required auto-cc inbox - all replies will be sent to the corporate server - when the patch is applied, the commit will be properly listing the corporate address Of course, we do have a workaround for that, which is to use a kernel.org address. Only a very small subset of people have these, and it may still be not allowed by individual company's legal departments even when this is available. -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Tue, Oct 15, 2019 at 08:37:41AM -0400, Steven Rostedt wrote: > Now, if we had a way to send and receive upstream patches via a web > site, that would actually make things easier. Alas, for those corporations who choose to enable DMARC in hard-fail mode (which includes google.com BTW, although thankfully, not gmail.com) sending (from DMARC's perspective, "forging") e-mail from someone at that particular domain won't work unless it comes from an authorized e-mail server. Of course, we do have a workaround for that, which is to use a kernel.org address. - Ted ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, 14 Oct 2019 16:56:58 -0400 "Theodore Y. Ts'o" wrote: > So it's an extreme case, and is an example of corp e-mail insanity. > The good news is that for companies that are really serious about > working with upstream, they will roll out solutions for their own > customers. For example, with IBM, since Lotus notes was maximally > unfriendly to OSS development practices, they created a separate mail > system with l...@linux.ibm.com addresses be cause it was simply > impossible to engage with upstream development using ibm.com > addresses. So there are work arounds, and I encourage you to reach > out to other Google kernel developers --- and if you are unhappy, > please file internal bugs against the corp infrastructure. Note, one issue is that some corporations do not want to support two email services. It's hard enough securing one, opening up another one is just opening another door to be cracked. We do file bugs when there's issues, but that doesn't mean they will be resolved. I have special permission to use my personal account, but most people at my company do not have that permission, and this is a heart ache for those that need to send email via the corporate email servers. Now, if we had a way to send and receive upstream patches via a web site, that would actually make things easier. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Dmitry Vyukov wrote: > As one data point, I cannot send emails with git send-email anymore. > It used to work, then broke and I don't know how to fix it. Now it says: > > 5.7.8 Username and Password not accepted. Learn more at > 5.7.8 https://support.google.com/mail/?p=BadCredentials > s10sm8376885wrr.5 - gsmtp > > I suspect it has something to do with two factor auth. > So that's it: it cannot contribute to kernel right now. > I will not consider time spent fixing it as useful time investment. I'm sorry you feel that way about time investments... But I've always assumed that's also the sentiment for time spent learning ANY new tools or workflow changes that come along. > Any kernel documentation that I can find for gmail, mentions config > that I am already using and that is not working: > https://www.kernel.org/doc/html/latest/search.html?q=gmail_keywords=yes=default# > https://www.kernel.org/doc/html/latest/process/email-clients.html?highlight=gmail Fwiw, git-send-email(1) manpage also has a special section for gmail: https://kernel.org/pub/software/scm/git/docs/git-send-email.html and a link for app-specific passwords: https://security.google.com/settings/security/apppasswords Perhaps that helps? ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, Oct 14, 2019 at 11:28:59AM -0300, Mauro Carvalho Chehab wrote: Yeah, our current security model is based at the maintainer for him to do his duties, properly reviewing the patch. Yet, at the example that Daniel gave: Instead of: if ((permissions == allowed) && other_stuff) { do_things(); } do_more_stuff(permissions); Patch was maliciously modified to: if ((permission == allowed) && other_stuff) { do_things(); } do_more_stuff(permissions); I suspect that a change like that might sleep though the maintainer's review. I submitted a proposal to the git list that would address this, but it saw little uptake in the discussion: https://public-inbox.org/git/20190910121324.GA6867@pure.paranoia.local/t/#u To summarize it in brief, I suggested using minisign signatures and git-tracked TOFU (trust on first use) databases. TOFU is not perfect, but it's better than nothing at all, which is what we have right now. -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, 14 Oct 2019 10:38:51 +1100 Daniel Axtens wrote: > > It does bring up that any new workflow has to have security protocol > > and threat model as part of its design. > > This is actually something that worries me about the patchwork > workflow. Maintainers seem to trust the patchwork version of a patch > without much (or any) verification that it matches what was sent to the > list. I tend to download the patch from patchwork, and review that. When I find something to comment on, I go back to my INBOX and reply to the original patch where I want to comment on. Thus, what's in patchwork is what I have verified anyway. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Em Mon, 14 Oct 2019 08:26:37 -0400 "Theodore Y. Ts'o" escreveu: > On Mon, Oct 14, 2019 at 12:42:36PM +0200, Toke Høiland-Jørgensen wrote: > > It should be detectable, though, right? > > > > Say you have two independently administered patchwork instances (or even > > better, two different software packages entirely) that both subscribe to > > the mailing lists, and compare patch content with each other. They > > should at least be able to detect mismatches. Especially if you add a > > sanity check before discarding duplicate message-ids. > > They don't even need to compare against each other; patchwork is about > to add a feature where you can look up patches via message-id, right? > That means it's easy enough to write a program which fetches patches > from patchwork, and compares it to the patches found in > lore.kernel.org. If they don't match, then an alarm can be sounded. > > Individuals who are reviewing patches can also compare the copy in > their inbox with the copy from lore or some other public inbox. And > maintainers can compare copies from lore.kernel.org and patchwork > before they apply a patch. (99% of the time, I actually use the patch > from my inbox, anyway.) It can still have man-in-the-middle (MITM) attacks between the sender and vger.kernel.org. Please notice that using https and adding the patch via a web interface can also be subject to MITM, as companies and even some Countries with strong policy enforcement may have some gateway on their infra that will prevent end-to-end encryption[1], blocking direct client-server https tunnels. [1] They add an internal certificate to the browsers, so that the client will see the connection as trustful, but the infra will actually do two separate HTTPS encryption: client ---> Gateway Gateway ---> Server While unlikely, nothing prevents that the patch would be maliciously altered at the Gateway. From security PoV, the only way to ensure that the patch was not altered is to have it signed by the one who wrote it. > > > This way you'd need to compromise multiple machines to achieve the kind > > of compromise you're worried about. And you can add more independent > > machines until you're satisfied that the risk is low enough :) > > Yep, exactly. This is basically the theory behind Certificate > Transparency[1], applied to patches. For example, here's the > certificate transparency report for kernel.org: > > > https://transparencyreport.google.com/https/certificates?cert_search=domain:kernel.org > > - Ted > > [1] http://www.certificate-transparency.org/what-is-ct That won't prevent companies/governments to require the manual installation of the gateway's certificate on their browers, in order to be able to navigate using https. Thanks, Mauro ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Em Mon, 14 Oct 2019 09:53:58 -0400 "Theodore Y. Ts'o" escreveu: > On Mon, Oct 14, 2019 at 10:41:32AM -0300, Mauro Carvalho Chehab wrote: > > It can still have man-in-the-middle (MITM) attacks between the sender and > > vger.kernel.org. Please notice that using https and adding the patch > > via a web interface can also be subject to MITM, as companies and even some > > Countries with strong policy enforcement may have some gateway on their > > infra that will prevent end-to-end encryption[1], blocking direct > > client-server https tunnels. > > > > [1] They add an internal certificate to the browsers, so that the client > > will see the connection as trustful, but the infra will actually do two > > separate HTTPS encryption: > > > > client ---> Gateway > > Gateway ---> Server > > > > While unlikely, nothing prevents that the patch would be maliciously > > altered at the Gateway. > > > > From security PoV, the only way to ensure that the patch was not > > altered is to have it signed by the one who wrote it. > > Well, sure, but the maintainer should be reviewing the patch looking > for problems anyway. There is the risk that people might slap a > "Reviewed-by:" tag on a patch without sufficiently careful review if > it's from a prominent kernel contributor, but we've always had that > problem. And so nothing, not even a digitally signed patch from a > reviewer should absolve the maintainer from doing their own review. Yeah, our current security model is based at the maintainer for him to do his duties, properly reviewing the patch. Yet, at the example that Daniel gave: Instead of: if ((permissions == allowed) && other_stuff) { do_things(); } do_more_stuff(permissions); Patch was maliciously modified to: if ((permission == allowed) && other_stuff) { do_things(); } do_more_stuff(permissions); I suspect that a change like that might sleep though the maintainer's review. > > Now, one might argue that if there is a forged patch from "famous > kernel developer A", followed up with a forged patch from "famous > kernel developer B", that might cause a maintainer to happily take the > patch without doing their own, independent review, for scaling > reasons. But that's a "vulernability" we've lived with for a long > time, since today neither patches or "Reviewed-by" messages are > usually signed. > > And at least (as far as we know) no one has managed to sneak a > malicious patch with a zero-day hidden with malice aforethought. And > perhaps that shouldn't be surprising. We seem to be quite capable of > introducing our own security vulererabilities without "help", so > perhaps most malicious attackers wouldn't want to do something which > could be so easily detected, when they can just pay money to a black > hat hacker. True, but, once we discover a patch introduced with a malicious code, some action should be taken to eliminate the source of the bad code. I mean, one thing is if "famous kernel developer A" maliciously wrote a patch to violate security. The other thing is that the infra used by "famous kernel developer A" is not safe, and has some hidden back hat hacker in the middle of it. The only way to identify that is by using signed patches/PRs. Thanks, Mauro ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Eric, On Mon, Oct 14, 2019 at 2:12 AM Eric Wong wrote: > Konstantin Ryabitsev wrote: > > On Fri, Oct 11, 2019 at 09:23:08PM +, Eric Wong wrote: > > > > (This is the same reason I generally disagree with Eric Wong about > > > > preserving SMTP as the primary transmission protocol -- I've heard lots > > > > of > > > > complaints both from kernel developers and especially from people > > > > trying to > > > > contribute to CAF about corporate policies actually making it > > > > impossible to > > > > submit patches -- and no, using a different mail server is not a > > > > possibility > > > > for them because it can be a firing offense under their IT AUP rules.) > > > > > > I'm not opposed to a webmail interface tailored to kernel hacking > > > which does stuff like checkpatch.pl and get_maintainer.pl before > > > sending (similar to your patchwork proposal and > > > gitgadgetgadget). That would get around security appliances > > > but SMTP would still be used in the background. > > > > > > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any > > > other webmail provider + checkpatch + get_maintainer helpers. > > > > Well, this is the bit where I say that it may not be allowed by corporate > > rules. I see this all the time in CAF/Android world where companies > > *require* that all email goes through their SMTP server so that it can be > > properly logged (often for legal reasons). And it is often equally required > > that any code submissions come from per...@corporate.com and not > > per...@free-email-provider.com for License/CLA reasons, so setting up a > > webmail server is not a solution either. > > Aren't they still allowed to submit stuff via forges the same way > they'd use a potential hacker-oriented webmail/SMTP/IMAP solution? > > Sometimes I see @username_COMPANY-type names on forges, but > AFAIK it's not very common. > > > This is basically why SMTP sucks in my view -- and it's worthless trying to > > pick fights with IT departments, because they are told to do so by lawyers. > > So, I want to take SMTP out of the equation: > > If the open source community can fight to get GPL accepted, I > don't see why we can't fight or subvert dumb corporate policies. > > > 1. provide a way for someone to submit a patch using a web interface (but > > still in a way that From: is their corporate ID) > > 2. use individual git feeds as a way to send out patches instead of always > > being secondary to SMTP > > username-comp...@users.kernel.org could probably work if they're > required to use @username_COMPANY on forges. username+foo is the standard way. > We can also find creative ways to subvert corporate policies: > For example; if their policy specifically prevents outgoing SMTP, > "git imap-send" could be used. IMAP may be blocked, too? Bascially the only thing you can rely on is HTTP(S), through a proxy, possibly with HTTPS inspection through a company-specific trusted certificate that allows MITM. > If their policy forbids using external "email" services, we'd > name it "Kernel Hackers' Messaging System" or something of that > sort and say we use an email bridge :> Anything named "Hacker" may be blocked, too ;-) Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Sat, Oct 12, 2019 at 09:19:11AM +0200, Greg KH wrote: > On Fri, Oct 11, 2019 at 05:35:53PM -0400, Konstantin Ryabitsev wrote: > > 1. provide a way for someone to submit a patch using a web interface (but > > still in a way that From: is their corporate ID) > If you do this, what happens when a maintainer/reviewer responds to that > patch and says "looks good, but can you change X and resend it?" > How will they get that message if it didn't go through their email > system? How will they be able to respond to it? If they're using their work e-mail address like Konstantin says they'll get the reply there. Probably any response from them will be some top posted thing with HTML but this happens often enough currently - people often use git send-email or something to send but continue using a mail setup that's not really what's expected for kernel work. Usually it's not the end of the world even if people get a bit grumpy at them. signature.asc Description: PGP signature ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Geert Uytterhoeven wrote: > On Mon, Oct 14, 2019 at 2:12 AM Eric Wong wrote: > > We can also find creative ways to subvert corporate policies: > > For example; if their policy specifically prevents outgoing SMTP, > > "git imap-send" could be used. > > IMAP may be blocked, too? Yes, was just pointing to "git imap-send" which already exists; but anything can go over HTTPS POST + Transfer-Encoding:chunked. > Bascially the only thing you can rely on is HTTP(S), through a proxy, > possibly with HTTPS inspection through a company-specific trusted > certificate that allows MITM. Right, I was tunneling arbitrary data over HTTP/1.1 via Transfer-Encoding:chunked on both requests/responses over a decade ago. Probably won't work with nginx because of input buffering, but public-inbox-httpd can be made to support it w/o buffering, too (it already does HTTPS + chunk parsing). I got something working on the server-side for git:// using Ruby back in 2009: https://public-inbox.org/git/20090702085440.gc11...@dcvr.yhbt.net/ Client-side needed some work, though... > > If their policy forbids using external "email" services, we'd > > name it "Kernel Hackers' Messaging System" or something of that > > sort and say we use an email bridge :> > > Anything named "Hacker" may be blocked, too ;-) Unpxref, then :> ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, Oct 14, 2019 at 10:41:32AM -0300, Mauro Carvalho Chehab wrote: > It can still have man-in-the-middle (MITM) attacks between the sender and > vger.kernel.org. Please notice that using https and adding the patch > via a web interface can also be subject to MITM, as companies and even some > Countries with strong policy enforcement may have some gateway on their > infra that will prevent end-to-end encryption[1], blocking direct > client-server https tunnels. > > [1] They add an internal certificate to the browsers, so that the client > will see the connection as trustful, but the infra will actually do two > separate HTTPS encryption: > > client ---> Gateway > Gateway ---> Server > > While unlikely, nothing prevents that the patch would be maliciously > altered at the Gateway. > > From security PoV, the only way to ensure that the patch was not > altered is to have it signed by the one who wrote it. Well, sure, but the maintainer should be reviewing the patch looking for problems anyway. There is the risk that people might slap a "Reviewed-by:" tag on a patch without sufficiently careful review if it's from a prominent kernel contributor, but we've always had that problem. And so nothing, not even a digitally signed patch from a reviewer should absolve the maintainer from doing their own review. Now, one might argue that if there is a forged patch from "famous kernel developer A", followed up with a forged patch from "famous kernel developer B", that might cause a maintainer to happily take the patch without doing their own, independent review, for scaling reasons. But that's a "vulernability" we've lived with for a long time, since today neither patches or "Reviewed-by" messages are usually signed. And at least (as far as we know) no one has managed to sneak a malicious patch with a zero-day hidden with malice aforethought. And perhaps that shouldn't be surprising. We seem to be quite capable of introducing our own security vulererabilities without "help", so perhaps most malicious attackers wouldn't want to do something which could be so easily detected, when they can just pay money to a black hat hacker. - Ted ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
"Theodore Y. Ts'o" writes: > On Mon, Oct 14, 2019 at 12:42:36PM +0200, Toke Høiland-Jørgensen wrote: >> It should be detectable, though, right? >> >> Say you have two independently administered patchwork instances (or even >> better, two different software packages entirely) that both subscribe to >> the mailing lists, and compare patch content with each other. They >> should at least be able to detect mismatches. Especially if you add a >> sanity check before discarding duplicate message-ids. > > They don't even need to compare against each other; patchwork is about > to add a feature where you can look up patches via message-id, right? > That means it's easy enough to write a program which fetches patches > from patchwork, and compares it to the patches found in > lore.kernel.org. If they don't match, then an alarm can be sounded. Yeah. I guess what is needed is to go from "can be" to "will be" (as Daniel pointed out in his simultaneous reply). >> This way you'd need to compromise multiple machines to achieve the kind >> of compromise you're worried about. And you can add more independent >> machines until you're satisfied that the risk is low enough :) > > Yep, exactly. This is basically the theory behind Certificate > Transparency[1], applied to patches. Indeed I'm familiar with certificate transparency, so this was certainly not an idea conceived in a vacuum ;) -Toke ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, Oct 14, 2019 at 12:42:36PM +0200, Toke Høiland-Jørgensen wrote: > It should be detectable, though, right? > > Say you have two independently administered patchwork instances (or even > better, two different software packages entirely) that both subscribe to > the mailing lists, and compare patch content with each other. They > should at least be able to detect mismatches. Especially if you add a > sanity check before discarding duplicate message-ids. They don't even need to compare against each other; patchwork is about to add a feature where you can look up patches via message-id, right? That means it's easy enough to write a program which fetches patches from patchwork, and compares it to the patches found in lore.kernel.org. If they don't match, then an alarm can be sounded. Individuals who are reviewing patches can also compare the copy in their inbox with the copy from lore or some other public inbox. And maintainers can compare copies from lore.kernel.org and patchwork before they apply a patch. (99% of the time, I actually use the patch from my inbox, anyway.) > This way you'd need to compromise multiple machines to achieve the kind > of compromise you're worried about. And you can add more independent > machines until you're satisfied that the risk is low enough :) Yep, exactly. This is basically the theory behind Certificate Transparency[1], applied to patches. For example, here's the certificate transparency report for kernel.org: https://transparencyreport.google.com/https/certificates?cert_search=domain:kernel.org - Ted [1] http://www.certificate-transparency.org/what-is-ct ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Daniel Axtens writes: >> It does bring up that any new workflow has to have security protocol >> and threat model as part of its design. > > This is actually something that worries me about the patchwork > workflow. Maintainers seem to trust the patchwork version of a patch > without much (or any) verification that it matches what was sent to the > list. > > Say Alice emails a patch that says something like: > > if ((permissions == allowed) && other_stuff) { > do_things(); > } > do_more_stuff(permissions); > > List members read that email, and many review it in the email client. > > However, say that the version in Patchwork actually reads like this: > > if ((permission = allowed) && other_stuff) { > do_things(); > } > do_more_stuff(permissions); > > (You could get this with a malicious/compromised patchwork admin, or a > sufficiently advanced network adversary - patchwork takes the first mail > for a given message-id + project and later ones are discarded, so > there's a race you can win.) > > If the maintainer or someone else happens to apply the patch from > patchwork and then review it, or if tests happen to catch the relevant > case, we'll catch this. Otherwise... > > It's not the easiest or lowest risk way to get malicious code into the > kernel, but nonetheless, I worry about it. > > I can't think of a sensible way to fix this, unless we want to move to a > world where patch submissions are GPG signed, and teach patchwork to > verify sigs. It should be detectable, though, right? Say you have two independently administered patchwork instances (or even better, two different software packages entirely) that both subscribe to the mailing lists, and compare patch content with each other. They should at least be able to detect mismatches. Especially if you add a sanity check before discarding duplicate message-ids. This way you'd need to compromise multiple machines to achieve the kind of compromise you're worried about. And you can add more independent machines until you're satisfied that the risk is low enough :) -Toke ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Konstantin Ryabitsev wrote: > On Fri, Oct 11, 2019 at 09:23:08PM +, Eric Wong wrote: > > > (This is the same reason I generally disagree with Eric Wong about > > > preserving SMTP as the primary transmission protocol -- I've heard lots of > > > complaints both from kernel developers and especially from people trying > > > to > > > contribute to CAF about corporate policies actually making it impossible > > > to > > > submit patches -- and no, using a different mail server is not a > > > possibility > > > for them because it can be a firing offense under their IT AUP rules.) > > > > I'm not opposed to a webmail interface tailored to kernel hacking > > which does stuff like checkpatch.pl and get_maintainer.pl before > > sending (similar to your patchwork proposal and > > gitgadgetgadget). That would get around security appliances > > but SMTP would still be used in the background. > > > > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any > > other webmail provider + checkpatch + get_maintainer helpers. > > Well, this is the bit where I say that it may not be allowed by corporate > rules. I see this all the time in CAF/Android world where companies > *require* that all email goes through their SMTP server so that it can be > properly logged (often for legal reasons). And it is often equally required > that any code submissions come from per...@corporate.com and not > per...@free-email-provider.com for License/CLA reasons, so setting up a > webmail server is not a solution either. Aren't they still allowed to submit stuff via forges the same way they'd use a potential hacker-oriented webmail/SMTP/IMAP solution? Sometimes I see @username_COMPANY-type names on forges, but AFAIK it's not very common. > This is basically why SMTP sucks in my view -- and it's worthless trying to > pick fights with IT departments, because they are told to do so by lawyers. > So, I want to take SMTP out of the equation: If the open source community can fight to get GPL accepted, I don't see why we can't fight or subvert dumb corporate policies. > 1. provide a way for someone to submit a patch using a web interface (but > still in a way that From: is their corporate ID) > 2. use individual git feeds as a way to send out patches instead of always > being secondary to SMTP username-comp...@users.kernel.org could probably work if they're required to use @username_COMPANY on forges. We can also find creative ways to subvert corporate policies: For example; if their policy specifically prevents outgoing SMTP, "git imap-send" could be used. If their policy forbids using external "email" services, we'd name it "Kernel Hackers' Messaging System" or something of that sort and say we use an email bridge :> ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote: > As one data point, I cannot send emails with git send-email anymore. > It used to work, then broke and I don't know how to fix it. Now it says: > > 5.7.8 Username and Password not accepted. Learn more at > 5.7.8 https://support.google.com/mail/?p=BadCredentials > s10sm8376885wrr.5 - gsmtp > > I suspect it has something to do with two factor auth. > So that's it: it cannot contribute to kernel right now. > I will not consider time spent fixing it as useful time investment. Note that this is a Googler-specific problem, as it's due to corp-security requirements that other GMail users and most GSuites customers won't have. (For example, you could use another SMTP server and bypass the corp. mail submission server, but for google.com e-mail addresses, the DMARC settings foreclose that particular workaround.) So it's an extreme case, and is an example of corp e-mail insanity. The good news is that for companies that are really serious about working with upstream, they will roll out solutions for their own customers. For example, with IBM, since Lotus notes was maximally unfriendly to OSS development practices, they created a separate mail system with l...@linux.ibm.com addresses be cause it was simply impossible to engage with upstream development using ibm.com addresses. So there are work arounds, and I encourage you to reach out to other Google kernel developers --- and if you are unhappy, please file internal bugs against the corp infrastructure. Personally, my solution to this problem is to use a non-google address (ty...@mit.edu) for upstream work, because life is too short to fight with GMail. (Although it is doable; as many people can attest.) > I would say that we need to aim at a process that does not require a > friendly experienced person to answer any of your questions in the > common case. Lots of people will simply not ask any questions. If you work inside a corporation with "unique" e-mail settings, you may need to ask a friendly experienced person inside your company, or look for the "new employee" (Noogler) web documentation for working with the kernel. - Ted ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Konstantin, On Fri, Oct 11, 2019 at 05:35:53PM -0400, Konstantin Ryabitsev wrote: > On Fri, Oct 11, 2019 at 09:23:08PM +, Eric Wong wrote: > >> (This is the same reason I generally disagree with Eric Wong about > >> preserving SMTP as the primary transmission protocol -- I've heard lots of > >> complaints both from kernel developers and especially from people trying to > >> contribute to CAF about corporate policies actually making it impossible to > >> submit patches -- and no, using a different mail server is not a > >> possibility > >> for them because it can be a firing offense under their IT AUP rules.) > > > > I'm not opposed to a webmail interface tailored to kernel hacking > > which does stuff like checkpatch.pl and get_maintainer.pl before > > sending (similar to your patchwork proposal and > > gitgadgetgadget). That would get around security appliances > > but SMTP would still be used in the background. > > > > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any > > other webmail provider + checkpatch + get_maintainer helpers. > > Well, this is the bit where I say that it may not be allowed by > corporate rules. I see this all the time in CAF/Android world where > companies *require* that all email goes through their SMTP server so > that it can be properly logged (often for legal reasons). And it is > often equally required that any code submissions come from > per...@corporate.com and not per...@free-email-provider.com for > License/CLA reasons, so setting up a webmail server is not a solution > either. > > This is basically why SMTP sucks in my view -- and it's worthless trying > to pick fights with IT departments, because they are told to do so by > lawyers. So, I want to take SMTP out of the equation: > > 1. provide a way for someone to submit a patch using a web interface >(but still in a way that From: is their corporate ID) > 2. use individual git feeds as a way to send out patches instead of >always being secondary to SMTP If the goal is to work around SMTP-related technical issues, is a web UI really the best way to go ? Wouldn't it be better to do the same through a git push ? We could setup a git server that requires authentication, and implement a push-to-email bridge. The information that would need to be entered in a web UI could be put in a tag message, and we could have a CLI to create the tag from a list of questions. -- Regards, Laurent Pinchart ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Hi Greg, On Mon, Oct 14, 2019 at 05:17:11PM +0200, Greg KH wrote: > On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote: > > On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote: > > > On 10/11/19 2:57 AM, Greg KH wrote: > > > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote: > > > >> Hi, all: > > > >> > > > >> I would like to propose a new (large) feature to patchwork with the > > > >> goal to > > > >> make the process of submitting a patch easier for newbies and people > > > >> generally less familiar with patch-based development. This was > > > >> discussed > > > >> previously on the workflows list: > > > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > > > >> > > > >> How I envision this would work: > > > >> > > > >> - user creates an account (which requires a mail confirmation) >> - > > > >> they choose a "submit patch" option from the menu > > > >> - the patch submission screen has a succession of screens: > > > >> > > > >> 1. a screen with a single field allowing a user to paste a URL to > > > >> their > > > >> fork of the git repository. Once submitted, patchwork does a "git > > > >> ls-remote" to attempt to get a list of refs and to verify that > > > >> this is > > > >> indeed a valid git repository > > > > > > > > s/valid git repository/valid git repository based on the kernel git > > > > tree/ > > > > > > > > Otherwise you might be sending out lots of emails for other projects :) > > > > > > > >> > > > >> 2. next screen asks the user to select the ref to work from using the > > > >> list obtained from the remote. Once submitted, patchwork performs a > > > >> `git > > > >> clone --reference` to clone the repository locally using a local > > > >> fork of > > > >> the same repo to minimize object transfer. This part requires that: > > > >>a. patchwork project is configured with a path to a local fork, > > > >> if this feature is enabled for a project > > > >>b. that fork is kept current via some mechanism outside of > > > >> patchwork (e.g. with grokmirror) > > > >>c. there is some sanity-checking during the clone process to > > > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size, > > > >> etc > > > >> -- other suggestions welcome) > > > >> > > > >> 3. next screen asks the user to pick a starting commit from the log. > > > >> Once submitted, patchwork generates the patch from the commit > > > >> provided > > > >> to the tip of the branch selected by the user earlier, > > > >> using git format-patch. > > > >> > > > >> 4. next screen asks the user to review the patch to make sure this is > > > >> what they want to submit. Once confirmed, patchwork performs two > > > >> admin-defined optional hooks: > > > >> > > > >>a. a hook to generate a list of cc's (e.g. get_maintainer.pl) > > > >>b. a sanity check hook (e.g. checkpatch.pl) > > > > > > > > I will note that many "first patch" submissions are checkpatch.pl > > > > cleanups for staging. When doing that, I require that they do "one > > > > logical change per patch", which means that many of the individual > > > > patches themselves will not be checkpatch.pl clean, because many lines > > > > have multiple issues with them (tabs, spaces, format, length, etc.) > > > > > > > > So other than that minor thing, sounds interesting. It's hard to > > > > determine just how difficult the whole "set up git and send a patch out" > > > > process is for people these days given the _huge_ numbers of new > > > > contributions we keep getting, and the numerous good tutorials we have > > > > created that spell out exactly how to do this. > > > > > > > > So you might be "solving" a problem that we don't really have. It's > > > > hard to tell :( > > > > > > > > > > I agree with this. I don't think this a problem that is worth solving. > > > When a new developer wants to send a patch, they don't need to create > > > any accounts. They setup their email client and send patch. > > > > > > We have several resources that walk them through setting up email > > > clients and sending patches. checkpatch.pl can be automated with > > > git hooks. > > > > > > >> I know this is a pretty big RFE, and I would like to hear your thoughts > > > >> about this. If there is general agreement that this is doable/good > > > >> idea, I > > > >> may be able to come up with funding for this development as part of the > > > >> overall tooling improvement proposal. > > > > > > > > The workflow seems sane, and matches what most people do today, with the > > > > exception that it "solves" the git send-email issue, right? Is that our > > > > biggest barrier? > > > > > > > > I would recommend interviewing some of the recent kernel mentor project > > > > and outreachy applicants first, to try to determine exactly what their > > > > problems, if any, were with our development process. If they say that > > > > this type of
Re: RFE: use patchwork to submit a patch
Hi Dmitry, On Mon, Oct 14, 2019 at 04:58:17PM +0200, Dmitry Vyukov wrote: > On Fri, Oct 11, 2019 at 7:20 PM Shuah Khan wrote: > > On 10/11/19 2:57 AM, Greg KH wrote: > >> On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote: > >>> Hi, all: > >>> > >>> I would like to propose a new (large) feature to patchwork with the goal > >>> to > >>> make the process of submitting a patch easier for newbies and people > >>> generally less familiar with patch-based development. This was discussed > >>> previously on the workflows list: > >>> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > >>> > >>> How I envision this would work: > >>> > >>> - user creates an account (which requires a mail confirmation) >> - they > >>> choose a "submit patch" option from the menu > >>> - the patch submission screen has a succession of screens: > >>> > >>> 1. a screen with a single field allowing a user to paste a URL to > >>> their > >>> fork of the git repository. Once submitted, patchwork does a "git > >>> ls-remote" to attempt to get a list of refs and to verify that this is > >>> indeed a valid git repository > >> > >> s/valid git repository/valid git repository based on the kernel git tree/ > >> > >> Otherwise you might be sending out lots of emails for other projects :) > >> > >>> 2. next screen asks the user to select the ref to work from using the > >>> list obtained from the remote. Once submitted, patchwork performs a > >>> `git > >>> clone --reference` to clone the repository locally using a local fork > >>> of > >>> the same repo to minimize object transfer. This part requires that: > >>>a. patchwork project is configured with a path to a local fork, > >>> if this feature is enabled for a project > >>>b. that fork is kept current via some mechanism outside of > >>> patchwork (e.g. with grokmirror) > >>>c. there is some sanity-checking during the clone process to > >>> avoid abuse (e.g. a sane timeout, a tmpdir with limited size, etc > >>> -- other suggestions welcome) > >>> > >>> 3. next screen asks the user to pick a starting commit from the log. > >>> Once submitted, patchwork generates the patch from the commit provided > >>> to the tip of the branch selected by the user earlier, > >>> using git format-patch. > >>> > >>> 4. next screen asks the user to review the patch to make sure this is > >>> what they want to submit. Once confirmed, patchwork performs two > >>> admin-defined optional hooks: > >>> > >>>a. a hook to generate a list of cc's (e.g. get_maintainer.pl) > >>>b. a sanity check hook (e.g. checkpatch.pl) > >> > >> I will note that many "first patch" submissions are checkpatch.pl > >> cleanups for staging. When doing that, I require that they do "one > >> logical change per patch", which means that many of the individual > >> patches themselves will not be checkpatch.pl clean, because many lines > >> have multiple issues with them (tabs, spaces, format, length, etc.) > >> > >> So other than that minor thing, sounds interesting. It's hard to > >> determine just how difficult the whole "set up git and send a patch out" > >> process is for people these days given the _huge_ numbers of new > >> contributions we keep getting, and the numerous good tutorials we have > >> created that spell out exactly how to do this. > >> > >> So you might be "solving" a problem that we don't really have. It's > >> hard to tell :( > > > > I agree with this. I don't think this a problem that is worth solving. > > When a new developer wants to send a patch, they don't need to create > > any accounts. They setup their email client and send patch. > > > > We have several resources that walk them through setting up email > > clients and sending patches. checkpatch.pl can be automated with > > git hooks. > > > >>> I know this is a pretty big RFE, and I would like to hear your thoughts > >>> about this. If there is general agreement that this is doable/good idea, I > >>> may be able to come up with funding for this development as part of the > >>> overall tooling improvement proposal. > >> > >> The workflow seems sane, and matches what most people do today, with the > >> exception that it "solves" the git send-email issue, right? Is that our > >> biggest barrier? > >> > >> I would recommend interviewing some of the recent kernel mentor project > >> and outreachy applicants first, to try to determine exactly what their > >> problems, if any, were with our development process. If they say that > >> this type of tool/workflow would have saved them hours of time and > >> energy, then that's a great indication that we should try to do this. > > > > I would say considering the number of applicants to mentorship program > > and new developers it will be lot overhead to require them to create > > patchwork accounts, and it might even be hard overtime. A lot of them > > start out and drop out in the
Re: RFE: use patchwork to submit a patch
> It does bring up that any new workflow has to have security protocol > and threat model as part of its design. This is actually something that worries me about the patchwork workflow. Maintainers seem to trust the patchwork version of a patch without much (or any) verification that it matches what was sent to the list. Say Alice emails a patch that says something like: if ((permissions == allowed) && other_stuff) { do_things(); } do_more_stuff(permissions); List members read that email, and many review it in the email client. However, say that the version in Patchwork actually reads like this: if ((permission = allowed) && other_stuff) { do_things(); } do_more_stuff(permissions); (You could get this with a malicious/compromised patchwork admin, or a sufficiently advanced network adversary - patchwork takes the first mail for a given message-id + project and later ones are discarded, so there's a race you can win.) If the maintainer or someone else happens to apply the patch from patchwork and then review it, or if tests happen to catch the relevant case, we'll catch this. Otherwise... It's not the easiest or lowest risk way to get malicious code into the kernel, but nonetheless, I worry about it. I can't think of a sensible way to fix this, unless we want to move to a world where patch submissions are GPG signed, and teach patchwork to verify sigs. Regards, Daniel ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Sat, 2019-10-12 at 14:16 +0100, Stephen Finucane wrote: > On Thu, 2019-10-10 at 10:41 -0400, Konstantin Ryabitsev wrote: > > Hi, all: > > > > I would like to propose a new (large) feature to patchwork with the goal > > to make the process of submitting a patch easier for newbies and people > > generally less familiar with patch-based development. This was discussed > > previously on the workflows list: > > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ [top posting, sort of] After writing this, I reopened my email client and noticed a lot of replies that hadn't been there before. Stupid client/email provider (I'm not sure who's to blame). It _seems_ like I'm not the only one questioning the need for a point and click patch submission format, but that doesn't mean there isn't some work we can do here. Previously, Patchwork had the stated goal of aquiring the ability to reply to patches via the web UI a lá Google Groups. If email is really an issue, what about if we added that functionality along with the ability to POST patches via the REST API? That would mean people would be able to submit patches via e.g. 'git pw series submit' and then reply on the web UI. I haven't really thought it through fully, but this _should_ do most of what you're looking for, right? Stephen > I'll echo Daniel's sentiments that I like the idea of a git-to-email > bridge, but that I'm not sure if that should belong in Patchwork core. > I too am open to being convinced but before we get to anything like a > solution though, I'd like to identify the problem(s) you're trying to > solve. From reading this thread, it seems there are three separate > issues issues intertwined here. > > * Corporate email is often broken, meaning people have to jump through >hoops to simply submit a patch, if it's even possible. [1] > * Encoding metadata in emails has to be done in an ad-hoc, freeform >fashion, through special headers, tags in the subject or specific >annotations in the body. This requires reading large contributors >guides before sending anything. > * Using CLI tools in general is hard for newbies (?) > > Asssuming those are correct, I'd like to challenge some of them :) > There isn't a whole lot that can be done about broken email, at least > until JMAP becomes a thing (it's done over HTTP so that could help. Or > not. idk), but I'm not sure about the other two. I don't know what kind > of metadata would be needed to submit a patch to a random subtree of > the kernel, but I assume the metadata exists for a reason? If so, is > this actually something we can tackle via a UI or CLI without producing > custom workflow tools for every single project and if not, can we > actually solve this particular issue? Personal, I would consider > reading the contributor guide a minimum barrier to entry, as without > this you're simply transferring work from contributors to (often > overworked) maintainers but I do acknowledge that it's easy for me to > say that since I know how to do this stuff already. The same idea > applies to the idea of not using CLIs. Is there an statistically > significant amount of people that would be able to submit useful > changes but can't use a CLI tool? I know you can get away without using > a terminal for traditional Windows development or web development, but > surely terminal knowledge is a prerequisite for almost everything else? > > What I'm trying to get at here is figure out if (a) this is something > that can really benefit from living in Patchwork, (b) this is something > that needs a UI, (c) this is something's that necessary full stop (vs. > just waiting for projects to switch to GitLab or Gerrit or whatever new > cool ends up being). If we can figure that out, we can look into how we > can go about implementing stuff. > > Stephen > > [1] I've multiple personal examples of this, from having to ask IT > during my Intel days to remove automatic legal signatures to > outlook.com refusing to respect message-id's, resulting in broken > threading at a minimum. Thankfully I don't have to jump through those > hoops at Red Hat but yeah, eew. > > ___ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Oct 11, 2019 at 04:02:28PM -0400, Konstantin Ryabitsev wrote: > On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote: > > So other than that minor thing, sounds interesting. It's hard to > > determine just how difficult the whole "set up git and send a patch out" > > process is for people these days given the _huge_ numbers of new > > contributions we keep getting, and the numerous good tutorials we have > > created that spell out exactly how to do this. > > > > So you might be "solving" a problem that we don't really have. It's > > hard to tell :( > > It is interesting that there are split views on this. The main reason why I > was thinking about it was because the topic came up a few times already. For > example, in a conversation last year on ksummit-discuss: > > https://lore.kernel.org/ksummit-discuss/ecadff3fd767c149ad96a924e7ea6eaf7c1ea...@usculxmsg01.am.sony.com/ > > Tim Bird mentioned that Sony developers couldn't send/receive patches > because their corporate mail server rewrote all links to go through some > kind of security appliance verification. If you read that thread, what we > are discussing now is what I suggested we did then -- a web tool that could > take corporate SMTP servers out of the equation. Yes, compamy SMTP servers are a huge problem, which is why we have so many contributors from gmail accounts :) But will your "send a patch" tool work for someone behind a corporate firewall? If we are needing/wanting corporate contributions that bad, by providing an end-run around their infrastructure is not going to be seen as a "good" thing, once those corporate security teams realize what is going on... And again, how will the feedback loop look like here? If joe@sony sends me a patch, how can they respond to my response of "please fix the change up to do X instead"? I don't want this to be a "fire and forget" type of system where companies submit patches through this interface and never respond. We already have enough examples of companies that do that today. My last example was a company that send out a 4 patch series, the response was "great work, but can you resend just the last 2 so we can take them now, and do X and Y on patches 1 and 2 so we can take them?" That was a year ago, with no response, when internally the developer said "I sent it upstream, but it was rejected!". > (This is the same reason I generally disagree with Eric Wong about > preserving SMTP as the primary transmission protocol -- I've heard lots of > complaints both from kernel developers and especially from people trying to > contribute to CAF about corporate policies actually making it impossible to > submit patches -- and no, using a different mail server is not a possibility > for them because it can be a firing offense under their IT AUP rules.) So if we create yet-another-patch-submission-path, how will that circumvent those rules? CAF is crazy, I've been in the middle of that many times. I doubt the "well I didn't send it in an email, I used this tool over here instead." will play well with their lawyers/managers to keep them from being fired. It's not our job to solve the "my managers are not letting me send patches out" problem here. To solve that requires working with the company at a totally different level. > > > I know this is a pretty big RFE, and I would like to hear your thoughts > > > about this. If there is general agreement that this is doable/good idea, I > > > may be able to come up with funding for this development as part of the > > > overall tooling improvement proposal. > > > > The workflow seems sane, and matches what most people do today, with the > > exception that it "solves" the git send-email issue, right? Is that our > > biggest barrier? > > Well, I can't really speak from my extensive experience as a kernel > developer, but I *have* submitted patches to Documentation/* before. This > happens infrequently enough that I basically have to relearn the whole > process from scratch, and it *is* a lot of steps. I can't fault people who > are only familiar with the GitHub way of doing things when > they complain that this process is a challenge for them. > > Not everyone submitting changes to the kernel are going to be highly skilled > and comfortable with the terminal and command-line tools. They may be > submitting a documentation fix, or it can be a driver developer who never > leaves Visual Studio submitting a small bugfix so their driver works better > in Linux. Simple drive-by patches for this is fine, if this type of tool will help, as long as the feedback loop is there. > > I would recommend interviewing some of the recent kernel mentor project > > and outreachy applicants first, to try to determine exactly what their > > problems, if any, were with our development process. If they say that > > this type of tool/workflow would have saved them hours of time and > > energy, then that's a great indication that we should try to do this. > > I
Re: RFE: use patchwork to submit a patch
On Fri, Oct 11, 2019 at 05:35:53PM -0400, Konstantin Ryabitsev wrote: > On Fri, Oct 11, 2019 at 09:23:08PM +, Eric Wong wrote: > > > (This is the same reason I generally disagree with Eric Wong about > > > preserving SMTP as the primary transmission protocol -- I've heard lots of > > > complaints both from kernel developers and especially from people trying > > > to > > > contribute to CAF about corporate policies actually making it impossible > > > to > > > submit patches -- and no, using a different mail server is not a > > > possibility > > > for them because it can be a firing offense under their IT AUP rules.) > > > > I'm not opposed to a webmail interface tailored to kernel hacking > > which does stuff like checkpatch.pl and get_maintainer.pl before > > sending (similar to your patchwork proposal and > > gitgadgetgadget). That would get around security appliances > > but SMTP would still be used in the background. > > > > Or offer full-blown HTTPS webmail + IMAP + SMTP access like any > > other webmail provider + checkpatch + get_maintainer helpers. > > Well, this is the bit where I say that it may not be allowed by corporate > rules. I see this all the time in CAF/Android world where companies > *require* that all email goes through their SMTP server so that it can be > properly logged (often for legal reasons). And it is often equally required > that any code submissions come from per...@corporate.com and not > per...@free-email-provider.com for License/CLA reasons, so setting up a > webmail server is not a solution either. > > This is basically why SMTP sucks in my view -- and it's worthless trying to > pick fights with IT departments, because they are told to do so by lawyers. > So, I want to take SMTP out of the equation: > > 1. provide a way for someone to submit a patch using a web interface (but > still in a way that From: is their corporate ID) If you do this, what happens when a maintainer/reviewer responds to that patch and says "looks good, but can you change X and resend it?" How will they get that message if it didn't go through their email system? How will they be able to respond to it? > 2. use individual git feeds as a way to send out patches instead of always > being secondary to SMTP Sending patches that way is one thing, the interaction based on those patches is another. Everyone needs to remember that only 1/3 of the patches submitted are applied. The "normal" path of development is at least a review/resend cycle for submissions (2/3 of patches). So that 2/3 can't be ignored as the "new/drive-by submissions" are probably more in that category than not. thanks, greg k-h ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, 11 Oct 2019 17:08:39 -0700 Stephen Hemminger wrote: > Yes, my life is too short to fight with corporate stuff. Heh, me too. > There is a security first mindset which is good but gets in the way. > The current generation of Internet standard mail protocols does not have > enough security to be allowed. Wish we could come up with a "secured email" RFC. > > It does bring up that any new workflow has to have security protocol > and threat model as part of its design. True. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Sat, 12 Oct 2019 08:54:22 +1000 Dave Airlie wrote: > On Sat, 12 Oct 2019 at 08:34, Steven Rostedt wrote: > > > > On Fri, 11 Oct 2019 14:19:09 -0700 > > Stephen Hemminger wrote: > > > > > As someone who has to deal with a corporate busted mail server. > > > I agree mailing patches is a barrier to entry for many people today. > > > > Hmm, and I wonder what company fosters so many issues with doing that > > today? :-/ > > > > /me notices that Stephen uses his personal email account ;-) > > Your current employer also defaults to a crappy link replacer on most emails > :-) Yes, because they use Stephen's employer's crappy email servers. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
> I don't disagree and Shuah's comments are very valuable here. However, I > would argue that these folks don't necessarily represent the target > audience for this tool. They may be newbies, but they join these > initiatives with the goal of spending significant time with the kernel > and its code, so they don't mind the effort of learning the proper way > of submitting patches. > It's two different group, a) people who aren't currently employed to work on Linux and want to be will jump through a lot more hoops with a mentor, they also are unlikely to been working at a company with horrible emails. b) people who are employed to work on Linux or at least Linux ecosystem tasks and want to send one or two patches upstream they found while developing their products, but can't use personal email and corp email sucks, and who use github for most of their lives elsewhere. I'm guessing we've got a lot of (b) we don't really see represented in mentorship programmes. Dave. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
I happen to have been working on something like this today: https://sr.ht/x71F.png This doesn't require any JavaScript. I've also been working on the other side, but it's less shiny - I'm teaching pygit2 about pluggable object db backends so that I can teach my mailing lists to talk to my git repos, and do things like offer expandable diffs, check mergability of patches, submit CI jobs, etc. Some more screenshots from my work today: https://sr.ht/3grl.png https://sr.ht/rcsA.png https://sr.ht/CLHm.png ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Sat, 12 Oct 2019 at 08:34, Steven Rostedt wrote: > > On Fri, 11 Oct 2019 14:19:09 -0700 > Stephen Hemminger wrote: > > > As someone who has to deal with a corporate busted mail server. > > I agree mailing patches is a barrier to entry for many people today. > > Hmm, and I wonder what company fosters so many issues with doing that > today? :-/ > > /me notices that Stephen uses his personal email account ;-) Your current employer also defaults to a crappy link replacer on most emails :-) Dave. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Em Fri, 11 Oct 2019 11:32:54 -0700 (PDT) David Miller escreveu: > From: Steven Rostedt > Date: Fri, 11 Oct 2019 14:01:08 -0400 > > > Thus, if we want people to send us their fixes, we better keep just > > an email with a patch the lowest bar for entry. > > I argue that for people coming into the software engineering world > today, a PR is the lowest bar for entry. And email is the exact > opposite, _especially_ our way of doing email. > > Because it IS NOT just an email with a patch. > > It is an email with a specific Subject line format, with various > fields. Some field are optional and some are mandatory, and it also > depends upon which tree you are targetting. > > Then there is the commit message which has content and formatting > requirements. > > And then there are the tags, of various types and flavors, and the > context (which is sometimes subtle) determines which of those tags are > relevant and how they should be filled in. > > The various other standard email fields like TO: and CC: have to > be set a certain way otherwise the patch won't even be looked at. I can't see how just adding a web interface like github/gitlab would solve any of the above. I mean, the submitter will still need to write a proper subject/description before submitting the patch to the web interface, add the tags, etc. The only thing that would be different if we write a a Kernel-specific web GUI would be to automate some sanity checks on it, like starting a CI build for the patch series, run checkpatch and get_maintainers, but that could also happen if we write a script/submit_patches tool, that would internally call checkpatch, get_maintainers and git send e-mail (or use some other interface like patchwork REST to submit patches). Thanks, Mauro ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, 11 Oct 2019 14:19:09 -0700 Stephen Hemminger wrote: > As someone who has to deal with a corporate busted mail server. > I agree mailing patches is a barrier to entry for many people today. Hmm, and I wonder what company fosters so many issues with doing that today? :-/ /me notices that Stephen uses his personal email account ;-) -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Em Fri, 11 Oct 2019 14:44:01 -0400 Steven Rostedt escreveu: > On Fri, 11 Oct 2019 11:32:54 -0700 (PDT) > David Miller wrote: > > > From: Steven Rostedt > > Date: Fri, 11 Oct 2019 14:01:08 -0400 > > > > > Thus, if we want people to send us their fixes, we better keep just > > > an email with a patch the lowest bar for entry. > > > > I argue that for people coming into the software engineering world > > today, a PR is the lowest bar for entry. And email is the exact > > opposite, _especially_ our way of doing email. > > > > Because it IS NOT just an email with a patch. > > That is if the maintainer wants to be anal about the submission. I've > taken patches where I only asked the person to give me a signed off by. > I also ask: > > "Hi, do you intend on being a contributor, or do you only want to get > this fix upstream? If the latter, I will handle the change log and > other formatting for you, all I need is the signed-off-by. Otherwise, I > will help you submit a proper patch." > > Again, this is for one offs, where someone found something like an off > by one error or other trivial bug to fix. I just want the fix in, but > will let the submitter decide how strict I will be to get it in. I've > had people say "I don't care, just get it fixed", and I do the > formatting and all the grudge work, but still give the submitter the > credit. > > Not to mention, there's several times I get a patch where the solution > is totally wrong, and I need to make the fix anyway. A simple > Reported-by is what the submitter gets. Also, perfectly done by email. > About the same here: if needed, I change patch descriptions, and for trivial patches submitted by newbies, I even correct bad whitespacing, if needed. All it takes for a patch to be handled is to send it in a way that patchwork will recognize as a patch - e. g. it should carry a properly formatted diff. > > > > You are not helping casual contributors with this "simple" email based > > submission method. It is understood and easy us, but nobody else. > > I'm not saying that email is the only way, this entire thread is about > getting another tool to help. But I will scream very loudly if we > eliminate email totally. Yes, e-mail should stay. Thanks, Mauro ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Em Thu, 10 Oct 2019 15:53:35 -0400 Konstantin Ryabitsev escreveu: > On Thu, Oct 10, 2019 at 03:07:29PM -0300, Mauro Carvalho Chehab wrote: > >> - the patch submission screen has a succession of screens: > >> > >> 1. a screen with a single field allowing a user to paste a URL to > >> their fork of the git repository. > > > >This will raise the bar, as it will force all developers to have a > >public site to host the tree. I guess only a fraction of the 4k kernel > >devs have it... In special, the ones that just want to send us a patch > >fixing a bug may have serious troubles implementing that. > > I don't think this will raise the bar, as Github/Gitlab allow for very > easy forking of https://github.com/torvalds/linux. This is also not at > all aimed at "all developers" -- only those that don't want to use the > current CLI workflow and are more comfortable with web tools like > Github. I guess people have different views about what's the goal. If the goal is to attract more developers, the focus should be on making something that would be simpler than what we current have. What we currently have here is that just adding this to .git/config: [sendemail] smtpserver = smtp.gmail.com smtpserverport = 587 smtpencryption = tls smtpuser = from = Is usually enough for the vast majority of the newcomers. Using github/gitlab for Kernel development takes a lot more time and a lot more steps than writing the above at .git/config, even if the user already have an account there. Also, instead of just doing: git send-email origin.. Your proposal will require to do: 1) git push github_clone 2) open the browser 3) fill the forms, pointing to "github_clone" URL 4) click at the submission button That is adding a lot additional complexity. I fail to see any gain with that. > >> 2. next screen asks the user to select the ref to work from using > >> the > >> list obtained from the remote. Once submitted, patchwork performs a > >> `git clone --reference` to clone the repository locally using a > >> local fork of the same repo to minimize object transfer. This part > >> requires that: > >>a. patchwork project is configured with a path to a local fork, > >> if this feature is enabled for a project > >>b. that fork is kept current via some mechanism outside of > >> patchwork (e.g. with grokmirror) > >>c. there is some sanity-checking during the clone process to > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size, > >> etc -- other suggestions welcome) > > > >That would require a high bandwidth at the machine with as patchwork. > >Also, doesn't sound a good idea to me, as the server may end by having > >tons of open sections, most waiting for tens of minutes, in order to > >complete git clone. > > This is actually really fast if you already have a local copy of the > repository with most objects. Try this yourself if you have > torvalds/linux.git locally: > > git clone --bare -s torvalds/linux.git test > cd test > git remote add arm-soc > https://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc > git fetch arm-soc for-next > > The whole process takes a second or so and the resulting repo is 328K in > size. > > Of course, this assumes that the remote repository isn't trying to do > something nasty, which is why I suggest anti-abuse precautions. Well, one could, for example, send a pull request, let's say, from a DRM development-based tree, into media (or vice versa), with would require to sync the "alien" patches, just to get the ones that are useful. It can even be worse (still valid): the tree to be pulled could be based on linux-next. Here, I receive bull requests that are sometimes based on non-media trees: it may take a few mins to get the patches. - Except if the idea is to have this only at kernel.org, and add an alternates for every single other tree, even a non-nasty PR would take a while, as not all kernel trees are hosted there. Also, as you said, one could do something really nasty, like sending a PR from a huge non-kernel repository into the Kernel. Not sure how easy/hard would be to avoid that. This could even happen by mistake, as, at least on some places (like linuxtv.org) non-kernel trees are also hosted. Btw, on media, our patchwork instance is also used by VDR, whose project is even hosted elsewhere. > > >> I know this is a pretty big RFE, and I would like to hear your > >> thoughts > >> about this. If there is general agreement that this is doable/good idea, > >> I may be able to come up with funding for this development as part of > >> the overall tooling improvement proposal. > > > >The procedure itself is not bad, but, if implemented, IMHO, it should, > >instead, be something that will run at the machine of the one submitting > >the patch. For
Re: RFE: use patchwork to submit a patch
Em Fri, 11 Oct 2019 11:20:17 -0600 Shuah Khan escreveu: > On 10/11/19 2:57 AM, Greg KH wrote: > > On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote: > >> Hi, all: > >> > >> I would like to propose a new (large) feature to patchwork with the goal to > >> make the process of submitting a patch easier for newbies and people > >> generally less familiar with patch-based development. This was discussed > >> previously on the workflows list: > >> https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > >> > >> How I envision this would work: > >> > >> - user creates an account (which requires a mail confirmation) >> - they > >> choose a "submit patch" option from the menu > >> - the patch submission screen has a succession of screens: > >> > >> 1. a screen with a single field allowing a user to paste a URL to > >> their > >> fork of the git repository. Once submitted, patchwork does a "git > >> ls-remote" to attempt to get a list of refs and to verify that this is > >> indeed a valid git repository > > > > s/valid git repository/valid git repository based on the kernel git tree/ > > > > Otherwise you might be sending out lots of emails for other projects :) > > > >> > >> 2. next screen asks the user to select the ref to work from using the > >> list obtained from the remote. Once submitted, patchwork performs a > >> `git > >> clone --reference` to clone the repository locally using a local fork > >> of > >> the same repo to minimize object transfer. This part requires that: > >>a. patchwork project is configured with a path to a local fork, > >> if this feature is enabled for a project > >>b. that fork is kept current via some mechanism outside of > >> patchwork (e.g. with grokmirror) > >>c. there is some sanity-checking during the clone process to > >> avoid abuse (e.g. a sane timeout, a tmpdir with limited size, etc > >> -- other suggestions welcome) > >> > >> 3. next screen asks the user to pick a starting commit from the log. > >> Once submitted, patchwork generates the patch from the commit provided > >> to the tip of the branch selected by the user earlier, > >> using git format-patch. > >> > >> 4. next screen asks the user to review the patch to make sure this is > >> what they want to submit. Once confirmed, patchwork performs two > >> admin-defined optional hooks: > >> > >>a. a hook to generate a list of cc's (e.g. get_maintainer.pl) > >>b. a sanity check hook (e.g. checkpatch.pl) > > > > I will note that many "first patch" submissions are checkpatch.pl > > cleanups for staging. When doing that, I require that they do "one > > logical change per patch", which means that many of the individual > > patches themselves will not be checkpatch.pl clean, because many lines > > have multiple issues with them (tabs, spaces, format, length, etc.) > > > > So other than that minor thing, sounds interesting. It's hard to > > determine just how difficult the whole "set up git and send a patch out" > > process is for people these days given the _huge_ numbers of new > > contributions we keep getting, and the numerous good tutorials we have > > created that spell out exactly how to do this. > > > > So you might be "solving" a problem that we don't really have. It's > > hard to tell :( > > > > I agree with this. I don't think this a problem that is worth solving. > When a new developer wants to send a patch, they don't need to create > any accounts. They setup their email client and send patch. > > We have several resources that walk them through setting up email > clients and sending patches. checkpatch.pl can be automated with > git hooks. > > >> I know this is a pretty big RFE, and I would like to hear your thoughts > >> about this. If there is general agreement that this is doable/good idea, I > >> may be able to come up with funding for this development as part of the > >> overall tooling improvement proposal. > > > > The workflow seems sane, and matches what most people do today, with the > > exception that it "solves" the git send-email issue, right? Is that our > > biggest barrier? > > > > I would recommend interviewing some of the recent kernel mentor project > > and outreachy applicants first, to try to determine exactly what their > > problems, if any, were with our development process. If they say that > > this type of tool/workflow would have saved them hours of time and > > energy, then that's a great indication that we should try to do this. > > > > I would say considering the number of applicants to mentorship program > and new developers it will be lot overhead to require them to create > patchwork accounts, and it might even be hard overtime. A lot of them > start out and drop out in the middle. With the current setup, nothing > to cleanup. > > Setting up email clients and git hooks is one time task. It is the > easiest of the
Re: RFE: use patchwork to submit a patch
On 10/11/19 2:57 AM, Greg KH wrote: On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote: Hi, all: I would like to propose a new (large) feature to patchwork with the goal to make the process of submitting a patch easier for newbies and people generally less familiar with patch-based development. This was discussed previously on the workflows list: https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ How I envision this would work: - user creates an account (which requires a mail confirmation) >> - they choose a "submit patch" option from the menu - the patch submission screen has a succession of screens: 1. a screen with a single field allowing a user to paste a URL to their fork of the git repository. Once submitted, patchwork does a "git ls-remote" to attempt to get a list of refs and to verify that this is indeed a valid git repository s/valid git repository/valid git repository based on the kernel git tree/ Otherwise you might be sending out lots of emails for other projects :) 2. next screen asks the user to select the ref to work from using the list obtained from the remote. Once submitted, patchwork performs a `git clone --reference` to clone the repository locally using a local fork of the same repo to minimize object transfer. This part requires that: a. patchwork project is configured with a path to a local fork, if this feature is enabled for a project b. that fork is kept current via some mechanism outside of patchwork (e.g. with grokmirror) c. there is some sanity-checking during the clone process to avoid abuse (e.g. a sane timeout, a tmpdir with limited size, etc -- other suggestions welcome) 3. next screen asks the user to pick a starting commit from the log. Once submitted, patchwork generates the patch from the commit provided to the tip of the branch selected by the user earlier, using git format-patch. 4. next screen asks the user to review the patch to make sure this is what they want to submit. Once confirmed, patchwork performs two admin-defined optional hooks: a. a hook to generate a list of cc's (e.g. get_maintainer.pl) b. a sanity check hook (e.g. checkpatch.pl) I will note that many "first patch" submissions are checkpatch.pl cleanups for staging. When doing that, I require that they do "one logical change per patch", which means that many of the individual patches themselves will not be checkpatch.pl clean, because many lines have multiple issues with them (tabs, spaces, format, length, etc.) So other than that minor thing, sounds interesting. It's hard to determine just how difficult the whole "set up git and send a patch out" process is for people these days given the _huge_ numbers of new contributions we keep getting, and the numerous good tutorials we have created that spell out exactly how to do this. So you might be "solving" a problem that we don't really have. It's hard to tell :( I agree with this. I don't think this a problem that is worth solving. When a new developer wants to send a patch, they don't need to create any accounts. They setup their email client and send patch. We have several resources that walk them through setting up email clients and sending patches. checkpatch.pl can be automated with git hooks. I know this is a pretty big RFE, and I would like to hear your thoughts about this. If there is general agreement that this is doable/good idea, I may be able to come up with funding for this development as part of the overall tooling improvement proposal. The workflow seems sane, and matches what most people do today, with the exception that it "solves" the git send-email issue, right? Is that our biggest barrier? I would recommend interviewing some of the recent kernel mentor project and outreachy applicants first, to try to determine exactly what their problems, if any, were with our development process. If they say that this type of tool/workflow would have saved them hours of time and energy, then that's a great indication that we should try to do this. I would say considering the number of applicants to mentorship program and new developers it will be lot overhead to require them to create patchwork accounts, and it might even be hard overtime. A lot of them start out and drop out in the middle. With the current setup, nothing to cleanup. Setting up email clients and git hooks is one time task. It is the easiest of the learning curve for many new developers. New developers struggle with getting the change logs right, coding styles right, and responding to review comments and acting on them. These aren't something that can be automated and they just have to learn through experience of sending patches. My opinion based on contact with new developers as well running the mentorship program, I would sat this isn't something that needs solving. thanks, -- Shuah
Re: RFE: use patchwork to submit a patch
Konstantin Ryabitsev wrote: > On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote: > > So other than that minor thing, sounds interesting. It's hard to > > determine just how difficult the whole "set up git and send a patch out" > > process is for people these days given the _huge_ numbers of new > > contributions we keep getting, and the numerous good tutorials we have > > created that spell out exactly how to do this. > > > > So you might be "solving" a problem that we don't really have. It's > > hard to tell :( > > It is interesting that there are split views on this. The main reason why I > was thinking about it was because the topic came up a few times already. For > example, in a conversation last year on ksummit-discuss: > > https://lore.kernel.org/ksummit-discuss/ecadff3fd767c149ad96a924e7ea6eaf7c1ea...@usculxmsg01.am.sony.com/ > > Tim Bird mentioned that Sony developers couldn't send/receive patches > because their corporate mail server rewrote all links to go through some > kind of security appliance verification. If you read that thread, what we > are discussing now is what I suggested we did then -- a web tool that could > take corporate SMTP servers out of the equation. > > (This is the same reason I generally disagree with Eric Wong about > preserving SMTP as the primary transmission protocol -- I've heard lots of > complaints both from kernel developers and especially from people trying to > contribute to CAF about corporate policies actually making it impossible to > submit patches -- and no, using a different mail server is not a possibility > for them because it can be a firing offense under their IT AUP rules.) I'm not opposed to a webmail interface tailored to kernel hacking which does stuff like checkpatch.pl and get_maintainer.pl before sending (similar to your patchwork proposal and gitgadgetgadget). That would get around security appliances but SMTP would still be used in the background. Or offer full-blown HTTPS webmail + IMAP + SMTP access like any other webmail provider + checkpatch + get_maintainer helpers. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, 11 Oct 2019 11:32:54 -0700 (PDT) David Miller wrote: > From: Steven Rostedt > Date: Fri, 11 Oct 2019 14:01:08 -0400 > > > Thus, if we want people to send us their fixes, we better keep just > > an email with a patch the lowest bar for entry. > > I argue that for people coming into the software engineering world > today, a PR is the lowest bar for entry. And email is the exact > opposite, _especially_ our way of doing email. > > Because it IS NOT just an email with a patch. That is if the maintainer wants to be anal about the submission. I've taken patches where I only asked the person to give me a signed off by. I also ask: "Hi, do you intend on being a contributor, or do you only want to get this fix upstream? If the latter, I will handle the change log and other formatting for you, all I need is the signed-off-by. Otherwise, I will help you submit a proper patch." Again, this is for one offs, where someone found something like an off by one error or other trivial bug to fix. I just want the fix in, but will let the submitter decide how strict I will be to get it in. I've had people say "I don't care, just get it fixed", and I do the formatting and all the grudge work, but still give the submitter the credit. Not to mention, there's several times I get a patch where the solution is totally wrong, and I need to make the fix anyway. A simple Reported-by is what the submitter gets. Also, perfectly done by email. > > You are not helping casual contributors with this "simple" email based > submission method. It is understood and easy us, but nobody else. I'm not saying that email is the only way, this entire thread is about getting another tool to help. But I will scream very loudly if we eliminate email totally. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, 11 Oct 2019 14:37:52 -0300 Mauro Carvalho Chehab wrote: > I agree with Shuah. From time to time, we have one time only patch > submission from people that just want to fix something that affects > his machine. I like to stress this point. There's been projects that I have written a patch for to fix or update a feature for that project for my own personal machine. Then when looking at upstreaming it, if the procedure is more than simply sending a patch to a mailing list, I usually stop right there, and just maintain the fix locally. (Note, this is for simple updates for my personal machine, if this is for something for work, I spend the time to get it upstream). Thus, if we want people to send us their fixes, we better keep just an email with a patch the lowest bar for entry. -- Steve ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, 2019-10-10 at 10:41 -0400, Konstantin Ryabitsev wrote: > Hi, all: > > I would like to propose a new (large) feature to patchwork with the goal > to make the process of submitting a patch easier for newbies and people > generally less familiar with patch-based development. This was discussed > previously on the workflows list: > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ I'll echo Daniel's sentiments that I like the idea of a git-to-email bridge, but that I'm not sure if that should belong in Patchwork core. I too am open to being convinced but before we get to anything like a solution though, I'd like to identify the problem(s) you're trying to solve. From reading this thread, it seems there are three separate issues issues intertwined here. * Corporate email is often broken, meaning people have to jump through hoops to simply submit a patch, if it's even possible. [1] * Encoding metadata in emails has to be done in an ad-hoc, freeform fashion, through special headers, tags in the subject or specific annotations in the body. This requires reading large contributors guides before sending anything. * Using CLI tools in general is hard for newbies (?) Asssuming those are correct, I'd like to challenge some of them :) There isn't a whole lot that can be done about broken email, at least until JMAP becomes a thing (it's done over HTTP so that could help. Or not. idk), but I'm not sure about the other two. I don't know what kind of metadata would be needed to submit a patch to a random subtree of the kernel, but I assume the metadata exists for a reason? If so, is this actually something we can tackle via a UI or CLI without producing custom workflow tools for every single project and if not, can we actually solve this particular issue? Personal, I would consider reading the contributor guide a minimum barrier to entry, as without this you're simply transferring work from contributors to (often overworked) maintainers but I do acknowledge that it's easy for me to say that since I know how to do this stuff already. The same idea applies to the idea of not using CLIs. Is there an statistically significant amount of people that would be able to submit useful changes but can't use a CLI tool? I know you can get away without using a terminal for traditional Windows development or web development, but surely terminal knowledge is a prerequisite for almost everything else? What I'm trying to get at here is figure out if (a) this is something that can really benefit from living in Patchwork, (b) this is something that needs a UI, (c) this is something's that necessary full stop (vs. just waiting for projects to switch to GitLab or Gerrit or whatever new cool ends up being). If we can figure that out, we can look into how we can go about implementing stuff. Stephen [1] I've multiple personal examples of this, from having to ask IT during my Intel days to remove automatic legal signatures to outlook.com refusing to respect message-id's, resulting in broken threading at a minimum. Thankfully I don't have to jump through those hoops at Red Hat but yeah, eew. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, 11 Oct 2019 19:00:09 -0400 Steven Rostedt wrote: > > > > As someone who has to deal with a corporate busted mail server. > > > > I agree mailing patches is a barrier to entry for many people today. > > > > > > Hmm, and I wonder what company fosters so many issues with doing that > > > today? :-/ > > > > > > /me notices that Stephen uses his personal email account ;-) > > > > Your current employer also defaults to a crappy link replacer on most > > emails :-) > > Yes, because they use Stephen's employer's crappy email servers. Yes, my life is too short to fight with corporate stuff. There is a security first mindset which is good but gets in the way. The current generation of Internet standard mail protocols does not have enough security to be allowed. It does bring up that any new workflow has to have security protocol and threat model as part of its design. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Oct 11, 2019 at 09:23:08PM +, Eric Wong wrote: (This is the same reason I generally disagree with Eric Wong about preserving SMTP as the primary transmission protocol -- I've heard lots of complaints both from kernel developers and especially from people trying to contribute to CAF about corporate policies actually making it impossible to submit patches -- and no, using a different mail server is not a possibility for them because it can be a firing offense under their IT AUP rules.) I'm not opposed to a webmail interface tailored to kernel hacking which does stuff like checkpatch.pl and get_maintainer.pl before sending (similar to your patchwork proposal and gitgadgetgadget). That would get around security appliances but SMTP would still be used in the background. Or offer full-blown HTTPS webmail + IMAP + SMTP access like any other webmail provider + checkpatch + get_maintainer helpers. Well, this is the bit where I say that it may not be allowed by corporate rules. I see this all the time in CAF/Android world where companies *require* that all email goes through their SMTP server so that it can be properly logged (often for legal reasons). And it is often equally required that any code submissions come from per...@corporate.com and not per...@free-email-provider.com for License/CLA reasons, so setting up a webmail server is not a solution either. This is basically why SMTP sucks in my view -- and it's worthless trying to pick fights with IT departments, because they are told to do so by lawyers. So, I want to take SMTP out of the equation: 1. provide a way for someone to submit a patch using a web interface (but still in a way that From: is their corporate ID) 2. use individual git feeds as a way to send out patches instead of always being secondary to SMTP -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, 11 Oct 2019 12:11:53 -0700 (PDT) David Miller wrote: > From: Mauro Carvalho Chehab > Date: Fri, 11 Oct 2019 15:59:49 -0300 > > > I can't see how just adding a web interface like github/gitlab > > would solve any of the above. I mean, the submitter will still > > need to write a proper subject/description before submitting the > > patch to the web interface, add the tags, etc. > > You can actually codify the various fields into a web form with > help texts etc. > > Things that are mandatory can even be marked required and people > can't even submit a PR without Signed-off-by: for example. As someone who has to deal with a corporate busted mail server. I agree mailing patches is a barrier to entry for many people today. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Oct 11, 2019 at 10:57:02AM +0200, Greg KH wrote: So other than that minor thing, sounds interesting. It's hard to determine just how difficult the whole "set up git and send a patch out" process is for people these days given the _huge_ numbers of new contributions we keep getting, and the numerous good tutorials we have created that spell out exactly how to do this. So you might be "solving" a problem that we don't really have. It's hard to tell :( It is interesting that there are split views on this. The main reason why I was thinking about it was because the topic came up a few times already. For example, in a conversation last year on ksummit-discuss: https://lore.kernel.org/ksummit-discuss/ecadff3fd767c149ad96a924e7ea6eaf7c1ea...@usculxmsg01.am.sony.com/ Tim Bird mentioned that Sony developers couldn't send/receive patches because their corporate mail server rewrote all links to go through some kind of security appliance verification. If you read that thread, what we are discussing now is what I suggested we did then -- a web tool that could take corporate SMTP servers out of the equation. (This is the same reason I generally disagree with Eric Wong about preserving SMTP as the primary transmission protocol -- I've heard lots of complaints both from kernel developers and especially from people trying to contribute to CAF about corporate policies actually making it impossible to submit patches -- and no, using a different mail server is not a possibility for them because it can be a firing offense under their IT AUP rules.) I know this is a pretty big RFE, and I would like to hear your thoughts about this. If there is general agreement that this is doable/good idea, I may be able to come up with funding for this development as part of the overall tooling improvement proposal. The workflow seems sane, and matches what most people do today, with the exception that it "solves" the git send-email issue, right? Is that our biggest barrier? Well, I can't really speak from my extensive experience as a kernel developer, but I *have* submitted patches to Documentation/* before. This happens infrequently enough that I basically have to relearn the whole process from scratch, and it *is* a lot of steps. I can't fault people who are only familiar with the GitHub way of doing things when they complain that this process is a challenge for them. Not everyone submitting changes to the kernel are going to be highly skilled and comfortable with the terminal and command-line tools. They may be submitting a documentation fix, or it can be a driver developer who never leaves Visual Studio submitting a small bugfix so their driver works better in Linux. I would recommend interviewing some of the recent kernel mentor project and outreachy applicants first, to try to determine exactly what their problems, if any, were with our development process. If they say that this type of tool/workflow would have saved them hours of time and energy, then that's a great indication that we should try to do this. I don't disagree and Shuah's comments are very valuable here. However, I would argue that these folks don't necessarily represent the target audience for this tool. They may be newbies, but they join these initiatives with the goal of spending significant time with the kernel and its code, so they don't mind the effort of learning the proper way of submitting patches. I'm thinking of someone who needs to submit an occasional contribution once every six months and to whom this document is both long and daunting: https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html. -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
From: Mauro Carvalho Chehab Date: Fri, 11 Oct 2019 15:59:49 -0300 > I can't see how just adding a web interface like github/gitlab > would solve any of the above. I mean, the submitter will still > need to write a proper subject/description before submitting the > patch to the web interface, add the tags, etc. You can actually codify the various fields into a web form with help texts etc. Things that are mandatory can even be marked required and people can't even submit a PR without Signed-off-by: for example. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
From: Steven Rostedt Date: Fri, 11 Oct 2019 14:01:08 -0400 > Thus, if we want people to send us their fixes, we better keep just > an email with a patch the lowest bar for entry. I argue that for people coming into the software engineering world today, a PR is the lowest bar for entry. And email is the exact opposite, _especially_ our way of doing email. Because it IS NOT just an email with a patch. It is an email with a specific Subject line format, with various fields. Some field are optional and some are mandatory, and it also depends upon which tree you are targetting. Then there is the commit message which has content and formatting requirements. And then there are the tags, of various types and flavors, and the context (which is sometimes subtle) determines which of those tags are relevant and how they should be filled in. The various other standard email fields like TO: and CC: have to be set a certain way otherwise the patch won't even be looked at. If you think all of this stuff is a low barrier to entry you are frankly kidding yourself. It is ONLY a low barrier to entry for an experienced Linux kernel developer. Full stop. You are not helping casual contributors with this "simple" email based submission method. It is understood and easy us, but nobody else. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, Oct 10, 2019 at 10:41:50AM -0400, Konstantin Ryabitsev wrote: > Hi, all: > > I would like to propose a new (large) feature to patchwork with the goal to > make the process of submitting a patch easier for newbies and people > generally less familiar with patch-based development. This was discussed > previously on the workflows list: > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > > How I envision this would work: > > - user creates an account (which requires a mail confirmation) > - they choose a "submit patch" option from the menu > - the patch submission screen has a succession of screens: > > 1. a screen with a single field allowing a user to paste a URL to their > fork of the git repository. Once submitted, patchwork does a "git > ls-remote" to attempt to get a list of refs and to verify that this is > indeed a valid git repository s/valid git repository/valid git repository based on the kernel git tree/ Otherwise you might be sending out lots of emails for other projects :) > > 2. next screen asks the user to select the ref to work from using the > list obtained from the remote. Once submitted, patchwork performs a `git > clone --reference` to clone the repository locally using a local fork of > the same repo to minimize object transfer. This part requires that: > a. patchwork project is configured with a path to a local fork, > if this feature is enabled for a project > b. that fork is kept current via some mechanism outside of > patchwork (e.g. with grokmirror) > c. there is some sanity-checking during the clone process to > avoid abuse (e.g. a sane timeout, a tmpdir with limited size, etc > -- other suggestions welcome) > > 3. next screen asks the user to pick a starting commit from the log. > Once submitted, patchwork generates the patch from the commit provided > to the tip of the branch selected by the user earlier, > using git format-patch. > > 4. next screen asks the user to review the patch to make sure this is > what they want to submit. Once confirmed, patchwork performs two > admin-defined optional hooks: > > a. a hook to generate a list of cc's (e.g. get_maintainer.pl) > b. a sanity check hook (e.g. checkpatch.pl) I will note that many "first patch" submissions are checkpatch.pl cleanups for staging. When doing that, I require that they do "one logical change per patch", which means that many of the individual patches themselves will not be checkpatch.pl clean, because many lines have multiple issues with them (tabs, spaces, format, length, etc.) So other than that minor thing, sounds interesting. It's hard to determine just how difficult the whole "set up git and send a patch out" process is for people these days given the _huge_ numbers of new contributions we keep getting, and the numerous good tutorials we have created that spell out exactly how to do this. So you might be "solving" a problem that we don't really have. It's hard to tell :( > I know this is a pretty big RFE, and I would like to hear your thoughts > about this. If there is general agreement that this is doable/good idea, I > may be able to come up with funding for this development as part of the > overall tooling improvement proposal. The workflow seems sane, and matches what most people do today, with the exception that it "solves" the git send-email issue, right? Is that our biggest barrier? I would recommend interviewing some of the recent kernel mentor project and outreachy applicants first, to try to determine exactly what their problems, if any, were with our development process. If they say that this type of tool/workflow would have saved them hours of time and energy, then that's a great indication that we should try to do this. thanks, greg k-h ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Jonathan Nieder wrote: > Eric Wong wrote: > > Konstantin Ryabitsev wrote: > > >> This is actually really fast if you already have a local copy of the > >> repository with most objects. Try this yourself if you have > >> torvalds/linux.git locally: > >> > >> git clone --bare -s torvalds/linux.git test > > > > Yep, -s (--shared) makes cloning really cheap. One of my goals is to get > > > > git clone -s https://example.com/torvalds/linux.git > > > > and > > > > git clone -s https://example.com/torvalds/linux.git/clone.bundle > > > > working. That would make it easier for new contributors to > > setup lightweight clones and pull in history on an as-needed > > basis w/o hacks like shallow cloning. > > Does "git clone --filter=blob:none" do what you're looking for? Oops, haven't seen that new feature :x And haven't tried, pu @ 8d9027fa59b943db96a8a9090ec31d7f0f935596 is broken due to conflicts with hashmap (probably won't have time for a bit to look at it). What I'm hoping to do with "git clone -s" would be client-only and compatible with existing HTTP servers that run "git update-server-info". ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
Konstantin Ryabitsev wrote: > This is actually really fast if you already have a local copy of the > repository with most objects. Try this yourself if you have > torvalds/linux.git locally: > > git clone --bare -s torvalds/linux.git test Yep, -s (--shared) makes cloning really cheap. One of my goals is to get git clone -s https://example.com/torvalds/linux.git and git clone -s https://example.com/torvalds/linux.git/clone.bundle working. That would make it easier for new contributors to setup lightweight clones and pull in history on an as-needed basis w/o hacks like shallow cloning. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, 10 Oct 2019 15:07:29 -0300 Mauro Carvalho Chehab wrote: > Em Thu, 10 Oct 2019 10:41:50 -0400 > Konstantin Ryabitsev escreveu: > > > Hi, all: > > > > I would like to propose a new (large) feature to patchwork with the goal > > to make the process of submitting a patch easier for newbies and people > > generally less familiar with patch-based development. This was discussed > > previously on the workflows list: > > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > > > > How I envision this would work: > > > > - user creates an account (which requires a mail confirmation) > > - they choose a "submit patch" option from the menu > > - the patch submission screen has a succession of screens: > > > > 1. a screen with a single field allowing a user to paste a URL to > > their fork of the git repository. > > This will raise the bar, as it will force all developers to have a > public site to host the tree. I guess only a fraction of the 4k kernel > devs have it... In special, the ones that just want to send us a patch > fixing a bug may have serious troubles implementing that. A lot of people are going the way of gitlab/github. I don't think this raises the bar very high. Anyone can trivially get a public git repository somewhere. We can still have email coming in for those die hards, as long as we have someone to proxy it in. Note, I'm closer to one of the die hards, but I see where the world is heading. > > > Once submitted, patchwork does a > > "git ls-remote" to attempt to get a list of refs and to verify that > > this is indeed a valid git repository > > > > 2. next screen asks the user to select the ref to work from using the > > list obtained from the remote. Once submitted, patchwork performs a > > `git clone --reference` to clone the repository locally using a > > local fork of the same repo to minimize object transfer. This part > > requires that: > >a. patchwork project is configured with a path to a local fork, > > if this feature is enabled for a project > >b. that fork is kept current via some mechanism outside of > > patchwork (e.g. with grokmirror) > >c. there is some sanity-checking during the clone process to > > avoid abuse (e.g. a sane timeout, a tmpdir with limited size, > > etc -- other suggestions welcome) > > That would require a high bandwidth at the machine with as patchwork. Which most solutions may not be able to avoid this. > > Also, doesn't sound a good idea to me, as the server may end by having > tons of open sections, most waiting for tens of minutes, in order to > complete git clone. > > > > > 3. next screen asks the user to pick a starting commit from the log. > > Once submitted, patchwork generates the patch from the commit > > provided to the tip of the branch selected by the user earlier, > > using git format-patch. > > > > 4. next screen asks the user to review the patch to make sure this is > > what they want to submit. Once confirmed, patchwork performs two > > admin-defined optional hooks: > > > >a. a hook to generate a list of cc's (e.g. get_maintainer.pl) > >b. a sanity check hook (e.g. checkpatch.pl) > > > > 5. if sanity checking is defined, next screen shows the output of the > > sanity check hook, asking confirmation to proceed. > > > > 6. next screen shows the user three fields: > > > >a. title of the patch/series > >b. cover letter for the patch/series > >c. message-id of the previous patch revision (can be picked from > > the list of previously submitted series by this user -- > > patchwork should have them already) > >d. number of the revision (can be auto-filled if previous field > > is provided) and other tags to include in [] > > > > 7. next screen shows final review of what would be sent out to the > > list (and cc's, if the hook to get cc's is defined and returned any > > results). Once submitted, patchwork sends the patch/series using > > patchwork's envelope-from and the user's own email in the From: > > header. > > > > 8. once sent successfully, cleanups are performed (also needs to be > > done as part of the regular cron job, for any aborted attempts) > > > > I know this is a pretty big RFE, and I would like to hear your thoughts > > about this. If there is general agreement that this is doable/good idea, > > I may be able to come up with funding for this development as part of > > the overall tooling improvement proposal. > > The procedure itself is not bad, but, if implemented, IMHO, it should, > instead, be something that will run at the machine of the one submitting > the patch. For instance, this could be a perl or python script inside > Kernel's ./script directory that would handle everything locally,
Re: RFE: use patchwork to submit a patch
Em Thu, 10 Oct 2019 10:41:50 -0400 Konstantin Ryabitsev escreveu: > Hi, all: > > I would like to propose a new (large) feature to patchwork with the goal > to make the process of submitting a patch easier for newbies and people > generally less familiar with patch-based development. This was discussed > previously on the workflows list: > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > > How I envision this would work: > > - user creates an account (which requires a mail confirmation) > - they choose a "submit patch" option from the menu > - the patch submission screen has a succession of screens: > > 1. a screen with a single field allowing a user to paste a URL to > their fork of the git repository. This will raise the bar, as it will force all developers to have a public site to host the tree. I guess only a fraction of the 4k kernel devs have it... In special, the ones that just want to send us a patch fixing a bug may have serious troubles implementing that. > Once submitted, patchwork does a > "git ls-remote" to attempt to get a list of refs and to verify that > this is indeed a valid git repository > > 2. next screen asks the user to select the ref to work from using the > list obtained from the remote. Once submitted, patchwork performs a > `git clone --reference` to clone the repository locally using a > local fork of the same repo to minimize object transfer. This part > requires that: >a. patchwork project is configured with a path to a local fork, > if this feature is enabled for a project >b. that fork is kept current via some mechanism outside of > patchwork (e.g. with grokmirror) >c. there is some sanity-checking during the clone process to > avoid abuse (e.g. a sane timeout, a tmpdir with limited size, > etc -- other suggestions welcome) That would require a high bandwidth at the machine with as patchwork. Also, doesn't sound a good idea to me, as the server may end by having tons of open sections, most waiting for tens of minutes, in order to complete git clone. > > 3. next screen asks the user to pick a starting commit from the log. > Once submitted, patchwork generates the patch from the commit > provided to the tip of the branch selected by the user earlier, > using git format-patch. > > 4. next screen asks the user to review the patch to make sure this is > what they want to submit. Once confirmed, patchwork performs two > admin-defined optional hooks: > >a. a hook to generate a list of cc's (e.g. get_maintainer.pl) >b. a sanity check hook (e.g. checkpatch.pl) > > 5. if sanity checking is defined, next screen shows the output of the > sanity check hook, asking confirmation to proceed. > > 6. next screen shows the user three fields: > >a. title of the patch/series >b. cover letter for the patch/series >c. message-id of the previous patch revision (can be picked from > the list of previously submitted series by this user -- > patchwork should have them already) >d. number of the revision (can be auto-filled if previous field > is provided) and other tags to include in [] > > 7. next screen shows final review of what would be sent out to the > list (and cc's, if the hook to get cc's is defined and returned any > results). Once submitted, patchwork sends the patch/series using > patchwork's envelope-from and the user's own email in the From: > header. > > 8. once sent successfully, cleanups are performed (also needs to be > done as part of the regular cron job, for any aborted attempts) > > I know this is a pretty big RFE, and I would like to hear your thoughts > about this. If there is general agreement that this is doable/good idea, > I may be able to come up with funding for this development as part of > the overall tooling improvement proposal. The procedure itself is not bad, but, if implemented, IMHO, it should, instead, be something that will run at the machine of the one submitting the patch. For instance, this could be a perl or python script inside Kernel's ./script directory that would handle everything locally, and then submit the patch via patchwork's REST API. By using the REST API, it would avoid the need of having to do special e-mail setups for the casual developers. Thanks, Mauro ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Fri, Oct 11, 2019 at 08:38:49AM +1100, Daniel Axtens wrote: Hi Konstantin, tl;dr: I think a git-to-email bridge is a good step. I'm not sure why patchwork would be the thing to build it on top of, and I'm worried that it would slow us both down. I'm very open to being convinced though. In very broad terms, I chose patchwork because: 1. in people's minds patchwork.kernel.org already deals with patches and mailing lists, so it seems to be a logical place for such tool to live 2. it's already built on top of a powerful web system (django), and we already have it up and running, so this wouldn't require setting up Yet Another Web Framework Service I will readily admit that both of these assertions are pretty tenuous. If I understand correctly, you're using Patchwork as: - user creates an account (which requires a mail confirmation) a) an identity provider, and Well, rather as a tool that already has account management which includes email confirmation at some point. b) a way to integrate with existing concepts of a project and keep metadata about them in 1 place Yes. c) a handy tool for getting previous series by a given user. Yes, it's convenient to already have that user's previously submitted series readily available. d) a 'trusted' source of email. Well, this part isn't really that important. Rather, it's a tool where, to have an account, one must confirm email delivery. Is that right? I just ask because this idea seems a long way from what Patchwork traditionally does. That's not necessarily bad, I just want to make sure I understand, and that if you get funding you're not tying yourself to a platform that doesn't suit your needs. Well, in my mind patchwork: 1. already deals with patches and series, including knowing how to do diff highlights and all the fancy stuff 2. will hopefully gain ability to do interdiffs in the future, so if someone submits a series revision, they can see what actually changed before their previous submission and their new attempt 3. already has a lot of knowledge around git, mboxes, formats, etc. This, of course, is not to say that patchwork is where this *must* happen, but I think it would be *nice* if this is where this happens. :) I'm particularly curious about Patchwork as (a) an identity provider. You wrote: - user creates an account (which requires a mail confirmation) This seems like "optional centralising" on Patchwork - it becomes a central identity provider but it's optional in that you can just send email directly if you prefer. Right, this is a tool to help people allergic to CLI (or who do this infrequently enough that they can never remember all the steps, and oh my god, why isn't there a web tool to hold my hand?). If you're going to do 'lightweight' centralisation like that, why not do it on a platform that already understands git? It's really easy to extract the information you describe in (c) just by querying the patchwork API. You don't need to actually integrate into patchwork, or be logged in, to do that. You lose the ability to load any git remote, but if you have a git remote that isn't github or gitlab, you probably already have a good email flow (e.g. if you repo is on kernel.org). If you really want to use Patchwork as an identity provider, rather than a forge, could we just teach Patchwork how to be an identity broker, and then build things separately, authenticating through Patchwork to confirm a user's identity? That means you could build in whatever language you like and, critically, run on whatever deployment schedule you want. You could also get much better isolation that way, which would be good - I don't want an RCE in the git library to allow someone to wipe out all patchwork data, for example. Well, ve hawe vays of prewenting that (e.g. by transitioning git calls into their own selinux domain which cannot talk to databases). I know this is a pretty big RFE, and I would like to hear your thoughts about this. If there is general agreement that this is doable/good idea, I may be able to come up with funding for this development as part of the overall tooling improvement proposal. As I said up top, I'm not opposed to this per se. I think a git-to-email bridge is a good step. I'm just confused as to why patchwork would be the thing to build it on top of, and I'm worried about how you'd deploy and update this extended Patchwork. I'm very open to being convinced though. Generally, I think patchwork, as a web application that already deals with patches and series is a convenient place for this tool to live, that's basically the extent of my thinking. For sure, it can exist as a separate tool, but then I'd have to set up and maintain that separate tool in addition to patchwork, as opposed to just patchwork. Best, -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org
Re: RFE: use patchwork to submit a patch
Hi Konstantin, tl;dr: I think a git-to-email bridge is a good step. I'm not sure why patchwork would be the thing to build it on top of, and I'm worried that it would slow us both down. I'm very open to being convinced though. > I would like to propose a new (large) feature to patchwork with the goal > to make the process of submitting a patch easier for newbies and people > generally less familiar with patch-based development. This was discussed > previously on the workflows list: > https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ > > How I envision this would work: > If I understand correctly, you're using Patchwork as: > - user creates an account (which requires a mail confirmation) a) an identity provider, and > - they choose a "submit patch" option from the menu > - the patch submission screen has a succession of screens: > > 1. a screen with a single field allowing a user to paste a URL to > their fork of the git repository. Once submitted, patchwork does a > "git ls-remote" to attempt to get a list of refs and to verify that > this is indeed a valid git repository > > 2. next screen asks the user to select the ref to work from using the > list obtained from the remote. Once submitted, patchwork performs a > `git clone --reference` to clone the repository locally using a > local fork of the same repo to minimize object transfer. This part > requires that: >a. patchwork project is configured with a path to a local fork, > if this feature is enabled for a project b) a way to integrate with existing concepts of a project and keep metadata about them in 1 place >b. that fork is kept current via some mechanism outside of > patchwork (e.g. with grokmirror) >c. there is some sanity-checking during the clone process to > avoid abuse (e.g. a sane timeout, a tmpdir with limited size, > etc -- other suggestions welcome) > > 3. next screen asks the user to pick a starting commit from the log. > Once submitted, patchwork generates the patch from the commit > provided to the tip of the branch selected by the user earlier, > using git format-patch. > > 4. next screen asks the user to review the patch to make sure this is > what they want to submit. Once confirmed, patchwork performs two > admin-defined optional hooks: > >a. a hook to generate a list of cc's (e.g. get_maintainer.pl) >b. a sanity check hook (e.g. checkpatch.pl) > > 5. if sanity checking is defined, next screen shows the output of the > sanity check hook, asking confirmation to proceed. > > 6. next screen shows the user three fields: > >a. title of the patch/series >b. cover letter for the patch/series >c. message-id of the previous patch revision (can be picked from > the list of previously submitted series by this user -- > patchwork should have them already) c) a handy tool for getting previous series by a given user. >d. number of the revision (can be auto-filled if previous field > is provided) and other tags to include in [] > > 7. next screen shows final review of what would be sent out to the > list (and cc's, if the hook to get cc's is defined and returned any > results). Once submitted, patchwork sends the patch/series using > patchwork's envelope-from and the user's own email in the From: > header. d) a 'trusted' source of email. Is that right? I just ask because this idea seems a long way from what Patchwork traditionally does. That's not necessarily bad, I just want to make sure I understand, and that if you get funding you're not tying yourself to a platform that doesn't suit your needs. I'm particularly curious about Patchwork as (a) an identity provider. You wrote: > - user creates an account (which requires a mail confirmation) This seems like "optional centralising" on Patchwork - it becomes a central identity provider but it's optional in that you can just send email directly if you prefer. If you're going to do 'lightweight' centralisation like that, why not do it on a platform that already understands git? It's really easy to extract the information you describe in (c) just by querying the patchwork API. You don't need to actually integrate into patchwork, or be logged in, to do that. You lose the ability to load any git remote, but if you have a git remote that isn't github or gitlab, you probably already have a good email flow (e.g. if you repo is on kernel.org). If you really want to use Patchwork as an identity provider, rather than a forge, could we just teach Patchwork how to be an identity broker, and then build things separately, authenticating through Patchwork to confirm a user's identity? That means you could build in whatever language you like and, critically, run on whatever deployment schedule you
Re: RFE: use patchwork to submit a patch
Eric Wong wrote: > Konstantin Ryabitsev wrote: >> This is actually really fast if you already have a local copy of the >> repository with most objects. Try this yourself if you have >> torvalds/linux.git locally: >> >> git clone --bare -s torvalds/linux.git test > > Yep, -s (--shared) makes cloning really cheap. One of my goals is to get > > git clone -s https://example.com/torvalds/linux.git > > and > > git clone -s https://example.com/torvalds/linux.git/clone.bundle > > working. That would make it easier for new contributors to > setup lightweight clones and pull in history on an as-needed > basis w/o hacks like shallow cloning. Does "git clone --filter=blob:none" do what you're looking for? Thanks, Jonathan ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
(+cc: git) Hi, Konstantin Ryabitsev wrote[1]: > How I envision this would work: > > - user creates an account (which requires a mail confirmation) > - they choose a "submit patch" option from the menu > - the patch submission screen has a succession of screens: Interesting! This reminds me a bit of https://gitgitgadget.github.io (except using patchwork instead of github pull requests as substrate). That leads me to wonder: should these kinds of tools share some code? Are there any subtleties that web-to-mail patch submission interfaces have in common or can learn from each other? Thanks, Jonathan [1] https://lore.kernel.org/workflows/20191010144150.hqiosvwolm3lmzp5@chatter.i7.local/ ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: RFE: use patchwork to submit a patch
On Thu, Oct 10, 2019 at 03:07:29PM -0300, Mauro Carvalho Chehab wrote: - the patch submission screen has a succession of screens: 1. a screen with a single field allowing a user to paste a URL to their fork of the git repository. This will raise the bar, as it will force all developers to have a public site to host the tree. I guess only a fraction of the 4k kernel devs have it... In special, the ones that just want to send us a patch fixing a bug may have serious troubles implementing that. I don't think this will raise the bar, as Github/Gitlab allow for very easy forking of https://github.com/torvalds/linux. This is also not at all aimed at "all developers" -- only those that don't want to use the current CLI workflow and are more comfortable with web tools like Github. 2. next screen asks the user to select the ref to work from using the list obtained from the remote. Once submitted, patchwork performs a `git clone --reference` to clone the repository locally using a local fork of the same repo to minimize object transfer. This part requires that: a. patchwork project is configured with a path to a local fork, if this feature is enabled for a project b. that fork is kept current via some mechanism outside of patchwork (e.g. with grokmirror) c. there is some sanity-checking during the clone process to avoid abuse (e.g. a sane timeout, a tmpdir with limited size, etc -- other suggestions welcome) That would require a high bandwidth at the machine with as patchwork. Also, doesn't sound a good idea to me, as the server may end by having tons of open sections, most waiting for tens of minutes, in order to complete git clone. This is actually really fast if you already have a local copy of the repository with most objects. Try this yourself if you have torvalds/linux.git locally: git clone --bare -s torvalds/linux.git test cd test git remote add arm-soc https://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc git fetch arm-soc for-next The whole process takes a second or so and the resulting repo is 328K in size. Of course, this assumes that the remote repository isn't trying to do something nasty, which is why I suggest anti-abuse precautions. I know this is a pretty big RFE, and I would like to hear your thoughts about this. If there is general agreement that this is doable/good idea, I may be able to come up with funding for this development as part of the overall tooling improvement proposal. The procedure itself is not bad, but, if implemented, IMHO, it should, instead, be something that will run at the machine of the one submitting the patch. For instance, this could be a perl or python script inside Kernel's ./script directory that would handle everything locally, and then submit the patch via patchwork's REST API. I think I didn't make clear that this isn't supposed to go straight to patchwork. Patchwork is merely a convenient tool where this happens -- the resulting patch gets mailed out to the mailing list just as the user would have done. -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
RFE: use patchwork to submit a patch
Hi, all: I would like to propose a new (large) feature to patchwork with the goal to make the process of submitting a patch easier for newbies and people generally less familiar with patch-based development. This was discussed previously on the workflows list: https://lore.kernel.org/workflows/20190930202451.GA14403@pure.paranoia.local/ How I envision this would work: - user creates an account (which requires a mail confirmation) - they choose a "submit patch" option from the menu - the patch submission screen has a succession of screens: 1. a screen with a single field allowing a user to paste a URL to their fork of the git repository. Once submitted, patchwork does a "git ls-remote" to attempt to get a list of refs and to verify that this is indeed a valid git repository 2. next screen asks the user to select the ref to work from using the list obtained from the remote. Once submitted, patchwork performs a `git clone --reference` to clone the repository locally using a local fork of the same repo to minimize object transfer. This part requires that: a. patchwork project is configured with a path to a local fork, if this feature is enabled for a project b. that fork is kept current via some mechanism outside of patchwork (e.g. with grokmirror) c. there is some sanity-checking during the clone process to avoid abuse (e.g. a sane timeout, a tmpdir with limited size, etc -- other suggestions welcome) 3. next screen asks the user to pick a starting commit from the log. Once submitted, patchwork generates the patch from the commit provided to the tip of the branch selected by the user earlier, using git format-patch. 4. next screen asks the user to review the patch to make sure this is what they want to submit. Once confirmed, patchwork performs two admin-defined optional hooks: a. a hook to generate a list of cc's (e.g. get_maintainer.pl) b. a sanity check hook (e.g. checkpatch.pl) 5. if sanity checking is defined, next screen shows the output of the sanity check hook, asking confirmation to proceed. 6. next screen shows the user three fields: a. title of the patch/series b. cover letter for the patch/series c. message-id of the previous patch revision (can be picked from the list of previously submitted series by this user -- patchwork should have them already) d. number of the revision (can be auto-filled if previous field is provided) and other tags to include in [] 7. next screen shows final review of what would be sent out to the list (and cc's, if the hook to get cc's is defined and returned any results). Once submitted, patchwork sends the patch/series using patchwork's envelope-from and the user's own email in the From: header. 8. once sent successfully, cleanups are performed (also needs to be done as part of the regular cron job, for any aborted attempts) I know this is a pretty big RFE, and I would like to hear your thoughts about this. If there is general agreement that this is doable/good idea, I may be able to come up with funding for this development as part of the overall tooling improvement proposal. Best regards, -K ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork