Re: qpsmtpd-dev
On Jun 29, 2012, at 1:53 PM, David Nicol wrote: >> Calling all forks. >> >> If you have a fork of QP and would like to become much less forked, or >> perhaps even be able to use QP without being forked at all, this would be a >> great time to bring out your fork. > > Should one add plugins? Perhaps as placeholder files referring to > where to get the genuine article, to reduce the possibility of naming > collisions? Using placeholders is one way to avoid collisions. Another is to have a plugin registry, like this one: https://github.com/qpsmtpd-dev/qpsmtpd-dev/blob/master/plugins/registry.txt The aliases field in the plugin registry provides historical clues as to what happened to older plugins, or what they evolved into. For example, the current 'badmailfrom' pattern came from merging the check_badmailfrom and check_badmailfrom_patterns plugins. A user looking for check_badmailfrom could see that plugin is now badmailfrom, and could read the POD for more details. That's one use of the registry. The reason I added the registry is so that my log parsing plugins could recognize the types of log entries, and bark loudly when they encountered entries they don't recognize. Another way to avoid collisions is to include all the plugins. It costs very, very little to include a plugin. Even a less than great one. Then every plugin has a canonical name, a home, version tracking, and plugin discoverability is easy. For plugins that have very little "general" appeal, we could create a rummage bin within the repo, so they're out of the way, but still easy to find, reference, and use. The current plugin situation, with a half dozen incomplete plugin lists, and plugins hosted all over, and many URLs suffering from bit rot, is a poor solution. Having all the plugins in one place would spare developers from reinventing wheels, users from STFW to find them, and Big Picture document authors a lot of research. Having all the plugins in one place is great for users. We've seen this already with Wordpress, App Stores, and CPAN. Get 'em all rounded up in one corral, and when they get unwieldy, we can work out a solution for managing them. Matt
Re: qpsmtpd-dev
> Calling all forks. > > If you have a fork of QP and would like to become much less forked, or > perhaps even be able to use QP without being forked at all, this would be a > great time to bring out your fork. Should one add plugins? Perhaps as placeholder files referring to where to get the genuine article, to reduce the possibility of naming collisions? -- Run it up the flagpole and see who salutes it
Re: qpsmtpd-dev
On Jun 29, 2012, at 12:11 PM, Jared Johnson wrote: >>> At any rate, if a branch like this exists and someone else had the time >>> and motivation to do the leg work of applying enhancements from the >>> various forks to the core branch, I would most definitely ask my employers >>> to put most of our codebase at your disposal to do this. Even if the >>> result wasn't something we could use, and we had to remain forked... at >>> least it would be useful to _someone_ :) >> >> https://github.com/qpsmtpd-dev/qpsmtpd-dev > > OK, as promised, I've asked my employers to put most of our codebase at > your disposal to do this :) I'm waiting on a response -- my boss is a > very busy guy, but he's also very friendly to FOSS. Hopefully I'll be > able to make most or all of our code available via public read-only SVN > soon. > > -Jared Calling all forks. If you have a fork of QP and would like to become much less forked, or perhaps even be able to use QP without being forked at all, this would be a great time to bring out your fork. My biggest goal for the forked dev branch is to push QP forward to the point where the main branch is usable for everyone. Ideally, by twiddling a few bits in config/*, everyone can get exactly the behavior they want. Yes, I know it's probably not possible, but I believe it's a worthy goal. I'm taking greater liberties with QP core than the main branch does. For example, I've added methods in Plugin.pm, which have me with resistance (passive or active, doesn't matter). I don't care if they exist there or in some other class I can inherit. They just need to have a place, and I'm not going to waste time guessing where Ask or Robert might approve. If many forks come forward, I expect to do more of this. With any luck, the -dev branch will be much better for it, and the results can be merged back into the main QP branch, in whatever fashion the bit keepers see fit. Matt
Re: qpsmtpd-dev
>> At any rate, if a branch like this exists and someone else had the time >> and motivation to do the leg work of applying enhancements from the >> various forks to the core branch, I would most definitely ask my >> employers >> to put most of our codebase at your disposal to do this. Even if the >> result wasn't something we could use, and we had to remain forked... at >> least it would be useful to _someone_ :) > > https://github.com/qpsmtpd-dev/qpsmtpd-dev OK, as promised, I've asked my employers to put most of our codebase at your disposal to do this :) I'm waiting on a response -- my boss is a very busy guy, but he's also very friendly to FOSS. Hopefully I'll be able to make most or all of our code available via public read-only SVN soon. -Jared
Re: qpsmtpd-dev
On Jun 25, 2012, at 12:27 PM, Jared Johnson wrote: >> I am to the point where I need to deploy. I was hoping that I could get my >> changes merged into the main branch, but that appears impossible. I hate >> the prospect of forking, but I see few alternatives. >> >> I hereby propose the creation of qpsmtpd-dev. >> >> The primary aim is merging the forks maintained by myself, Steve, Chris, >> Jared, and others. I am willing to do much of the work to accomplish this. >> I believe there is much to be gained by doing so, not the least of which >> is saving others the pain of learning they are wasting their time trying >> to improve QP in any way that doesn't fit with the unstated goals and >> vision of the project. A secondary goal is to provide a development branch >> of QP in which ideas can be rapidly tested, deployed, and when desired, >> backported to the main branch. > > I like the idea, although I'm not absolutely sure that it would accomplish > everything it would mean to. Not trying is guaranteed not to. > A primary example to me would be the > divergent methods used by various forks for supporting "per-recipient > configuration". That thread demonstrates how each of our various methods > made some sort of sense to the others, but were also completely > unacceptable to the others for various reasons. Simply because each entire enchilada is insufficient/unusable for every other persons doesn't mean they can't each benefit from each other. Perhaps there isn't one per-recipient solution to rule them all, but I'm willing to bet that if each fork was properly coded in paragraph sized methods, that there would be a significant amount of overlap that could and should be abstracted into a shared class. Then every fork would have less custom code to manage. A classic example: every plugin that uses DNS loads its own resolver, fetches results itself, and then does whatever processing it does with the results. Having a shared DNS class that lets plugins do all but the local processing is low hanging fruit. > The only solutions I can > think of would be (1) maintain more than one 'aggressive development' > fork, perhaps eventually letting the forks die that don't get used or > hacked on much; I think that's a perfect use for branches. :-) > (2) somehow support multiple options for how to accomplish > the same thing, selected within the config; (3) pick a single option that > is "blessed" by upstream -- forcing some to remain quite forked if the > single option is not useful for them. We could, for example, have a 3 branches within -dev that each implement per-recpient handling in one fashion or another. Then it would be easy to see how each is implemented and compare. Creating a "per-recipient" base branch, that the other 3 share from could immediately reduce the divergence. When someone wants to implement one of the 3 types, they can clone the dev-fork and just merge in the branch that does what they want. This is easy enough that we can even document the process. > At any rate, if a branch like this exists and someone else had the time > and motivation to do the leg work of applying enhancements from the > various forks to the core branch, I would most definitely ask my employers > to put most of our codebase at your disposal to do this. Even if the > result wasn't something we could use, and we had to remain forked... at > least it would be useful to _someone_ :) https://github.com/qpsmtpd-dev/qpsmtpd-dev It has been created. You have commit, as do Ask, Robert, and a couple others. The policy for gaining write access is liberal. The tolerance for change is great. Please use branches. :-) My goal is to get you (and others) running qpsmtpd-dev. It doesn't look like your fork on GitHub is up-to-date. If you get it updated, I should be able to diff against it and 0.84, and get a pretty good idea what you've changed, and then perhaps introduce changes to qpsmtpd-dev that minimize the distance between your fork and -dev. Small enough that you can reliably "git merge my-recipient-hacks" against master, and be off and running. As to the state of qpsmtpd-dev. All the pending pull requests awaiting "review" to the main repo are there. Many more logging tweaks have been made. I have greatly simplified several of the plugins that did deferred rejection, in favor of naughty. Most all the plugins use is_immune(). Between karma, naughty, and is_immune(), the -dev version is way more efficient than 0.84. Having the DNS plugins process to completion immediately, so that subsequent plugins can skip processing further improves efficiency. Disabling user validity tests for naughty connections also enhances security, as it reduces spammers ability to enumerate valid email addresses. Add in a trained dspam and the FPs that sneak past spamassassin are nabbed. Besides writing unit tests, writing the log processors has done the most to gain a level of consistency and
Re: qpsmtpd-dev
> I am to the point where I need to deploy. I was hoping that I could get my > changes merged into the main branch, but that appears impossible. I hate > the prospect of forking, but I see few alternatives. > > I hereby propose the creation of qpsmtpd-dev. > > The primary aim is merging the forks maintained by myself, Steve, Chris, > Jared, and others. I am willing to do much of the work to accomplish this. > I believe there is much to be gained by doing so, not the least of which > is saving others the pain of learning they are wasting their time trying > to improve QP in any way that doesn't fit with the unstated goals and > vision of the project. A secondary goal is to provide a development branch > of QP in which ideas can be rapidly tested, deployed, and when desired, > backported to the main branch. I like the idea, although I'm not absolutely sure that it would accomplish everything it would mean to. A primary example to me would be the divergent methods used by various forks for supporting "per-recipient configuration". That thread demonstrates how each of our various methods made some sort of sense to the others, but were also completely unacceptable to the others for various reasons. The only solutions I can think of would be (1) maintain more than one 'aggressive development' fork, perhaps eventually letting the forks die that don't get used or hacked on much; (2) somehow support multiple options for how to accomplish the same thing, selected within the config; (3) pick a single option that is "blessed" by upstream -- forcing some to remain quite forked if the single option is not useful for them. At any rate, if a branch like this exists and someone else had the time and motivation to do the leg work of applying enhancements from the various forks to the core branch, I would most definitely ask my employers to put most of our codebase at your disposal to do this. Even if the result wasn't something we could use, and we had to remain forked... at least it would be useful to _someone_ :) -Jared
Re: qpsmtpd-dev
I totally agree with you. A qpsmtpd-dev branch will speed tests allot. José Borges Ferreira PS: the log parser looks really good. On Fri, Jun 22, 2012 at 10:14 AM, Matt Simerson wrote: > > http://www.nntp.perl.org/group/perl.qpsmtpd/2009/01/msg8376.html > > This thread almost precisely describes my vision for QP. To whit: > > a) It describes the need for a standardized method for handling > rejections. > > I have implemented this in pull request #23: > https://github.com/smtpd/qpsmtpd/pull/23 > > It breaks nothing in qpsmtpd, facilitates consistency in plugins, > simplifies each plugin, and yet has met resistance (passive or intentional) > in getting integrated. > > b) The thread describes a consistent manner of deferring rejections among > plugins (wait for DATA). > > A number of plugins do this on their own, at the expense of plugin > complexity. Rather than each plugin implementing delayed deferral, I wrote > the naughty plugin, which greatly simplifies the process and allows > deferred rejection in all plugins with one configuration setting. Albeit at > the expense of adding a plugin. The advantages are huge. Documenting how to > handle rejections across any plugin is now possible. Plugins are simpler. > But the main reason I wrote naughty was resource efficiency. Having a > standard way to make a connection as junk allows all subsequent plugins to > skip processing after the connection is marked as naughty. Regardless of > when the connection is rejected (from, to, data, data_post). > > Additional benefits of consistent rejection handling and message deferral > align with the logging goals I outlined in the May 1st email, "uniform > plugin logging." I described my desire to improve the human readability of > log messages, use consistent (skip, pass, fail) syntax to describe plugin > results, have each plugin typically log a single message per connection, > and allow the ability to increase/decrease loglevel for individual plugins. > > I have completed all of these goals. In addition, I have written > log_watch.pl, a CLI log watcher, that parses the much more uniform > qpsmtpd logs and conveniently summarizes each connection into a single line > of of output. I have some example output here: > http://www.tnpi.net/internet/mail/qp/logwatch.html > > I have written a CLI script named log2sql.pl, which also parses the now > mostly consistent log entries and feeds a summary of them into a SQL > database. A companion CGI app presents a very similar view of the SMTP > transaction history as shown by log_watch.pl. The data is well normalized > and 100,000 messages of history consume only 100MB of disk. If one opts not > to store the OS, hostname, helo string, from, and to data in the tables, > the disk consumption is about 95% less. > > I wrote these log processing tools because all the log processing tools I > found were written for older versions of qpsmtpd and no longer worked. They > also made rare attempts to log plugin actions, which made them of little > value to a user or author who wished to monitor or measure the effects of > twiddling settings or tweaking code. > > In doing all of the above, I have posted regularly to the list, soliciting > the input of the community, and often making changes to address issues > raised. > > The responses, particularly from the keepers of the commit bits, have been > deafening. > > I am to the point where I need to deploy. I was hoping that I could get my > changes merged into the main branch, but that appears impossible. I hate > the prospect of forking, but I see few alternatives. > > I hereby propose the creation of qpsmtpd-dev. > > The primary aim is merging the forks maintained by myself, Steve, Chris, > Jared, and others. I am willing to do much of the work to accomplish this. > I believe there is much to be gained by doing so, not the least of which is > saving others the pain of learning they are wasting their time trying to > improve QP in any way that doesn't fit with the unstated goals and vision > of the project. A secondary goal is to provide a development branch of QP > in which ideas can be rapidly tested, deployed, and when desired, > backported to the main branch. > > Matt
