Hi everyone, The crowdmatch mechanism code is still a work in progress, and I want to talk about what I've done so far to get some feedback. This email has become rather long! Feel free to skim it to find parts that sound interesting. This was written for my benefit as much as yours, and in the hope that I get some feedback. Anything at all, like "Well hurry up and do it already", is very welcome.
It is also written for posterity and for your amusement. :) The new code is on a branch called split-mechanism. A giant comparison can be viewed, but it's longer than this email(!) and I'm going to call out specific items instead. This will be a technical discussion, but not too nutty. I will talk about how I structured the crowdmatch mechanism code, how I've designed it to be used, how I've designed it to be testable, and how some quirks popped up during my headlong crash through coding it. Please reply with feedback ABOUT ANY PART OF THIS! :D ### Table of Contents: 1. A separate module for the crowdmatch mechanism 2. API for the library 2.a. Client code requirements 2.b. Usage example 2.c. Exported actions 3. Database management 3.a. Migrations 3.b. The test database 3.c. run-persist library 4. Implementation to allow property testing 5. Implementation to allow more-than-one Stripe 6. Difference between MechAction and StripeI 7. What's left to do Appendix A: Questions ### 1. A separate module for the crowdmatch mechanism Although there is only one foreseeable client to this code, namely the Snowdrift website, I went ahead and structured it as a standalone library. It is in a directory called 'crowdmatch'. I did this so it can be tested independently. It will be a lot more stable than the website eventually, too, which is a good decider for whether something should be independent or not. ### 2. API for the library The crowdmatch library puts certain constraints on client code because it requires a database. Then, it does what any API does: provides actions and data types. ### 2.a. Client code requirements A client of this library must be using Postgres via Persistent. The only way to be agnostic about databases would be to completely reimplement a database. That's nuts. The library will just use Postgres, and thus requires client code to use it as well. The library also needs to do IO to talk to Stripe. Rather than rely on monad transformers or mtl-style classes to manage these different layers, I use plain old functions. Each API method needs to be passed a handler for running database actions. Will this be a pain in the ass? Probably not. It's also the easiest style to change FROM, should that become necessary. The library maintains its own notion of what a "Patron" is. To interface with the website, I created ToMechPatron and FromMechPatron type classes. These are easy to define for the User type. Thus, it's easy to create a relationship between "website Users" and "crowdmatch Patrons". ### 2.b. Usage example Here's a usage example that covers these parts so far. Given: fetchPatron :: (ToMechPatron usr, MonadIO io, MonadIO env) => SqlRunner io env -> usr -> env Patron You can run: someHandler = do Entity uid user <- requireAuth patron <- Crowdmatch.fetchPatron runDB uid ... See how 'runDB' is passed in as the database runner. ### 2.c. Exported actions There are about ten operations that will be exported, as well as a handful of data types. I am starting with fetching, storing, and deleting two items: payment-method tokens and pledges. The payment tokens are what Stripe gives us in lieu of credit card info. Pledges are what you think they are: the record of a donor pledged to a project. Next in the API are two super important methods (prototyped but not implemented yet): 'runCrowdmatch' and 'processPayments'. The first looks at all pledged patrons and calculates an outstanding donation balance. After a crowdmatch event, each pledged patron will then "owe" a certain amount to the project. That all happens in the database in a single transaction. Later, at our leisure, we can run 'processPayments'. That method inspects outstanding balances and sends payment commands via Stripe. This is where fee limits take effect. The website won't trigger the crowdmatch event or payment processing. Instead I'll have two simple utilities that do that stuff. I'll just run them manually at first. This section was longer than I intended, but that's probably good. It's one of the more important. Feedback on the API is highly welcome. API to date: https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/src/Crowdmatch.hs#L24 ### 3. Database management Since this library requires a database, it requires database management. Like I said, it uses Persistent, and has its own 'Model' module that works like the website's Model, where the database schema is defined. ### 3.a. Migrations The library exports a Persistent-generated migration action that clients need to ensure gets run. This handles "safe" migrations. I will also be adding manual migrations for the cases that Persistent can't handle. I am leaning towards using the 'drift' library, but this is actually an open question. Does anyone have experience with manual migration libraries? So far I've looked at drift, postgresql-simple-migrations, and dbmigrations. I haven't decided which to use yet. ### 3.b. The test database I had to set up a test database, since there's no point in running tests without one. I wrote code that creates a tablespace in memory, so the tests are relatively quick. (400 iterations of 0-100 actions in sixteen seconds. Yes, "relatively quick" is definitely *relative*.) ### 3.c. run-persist library Along the way, I wrote a whole 'nother library called run-persist. I got tired of playing Monad Jenga just to run a SqlPersistT action in IO. Right now this library is included in the project, but I plan on splitting it out once it is good enough. The run-persist library is pretty short: https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/run-persist/RunPersist.hs ### 4. Implementation to allow property testing If you look at the API for the crowdmatch library linked above, you'll see a thin wrapper over 'runMech' applied to values of type 'MechAction ret'. Why this design? I wanted to be able to create a long, random list of actions to blast at the library, so I could inspect invariants. This should catch any weird combination of actions that could lead to edge case behavior. I have two tests that use it: 'prop_pledgeHist' and 'prop_pledgeCapability'. https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/test/main.hs#L157 Something I struggled with here is the type variable in 'MechAction ret'. The 'ret' represents the return value of the action. MechAction is a GADT, and each constructor can specify its unique return type concretely. It took a while to figure out how to run a bunch of actions with these heterogeneous return types. That's why right now all the constructors all have type 'MechAction ()'. :) I know how to change that now, which I will have to do to handle Stripe errors. But fixing this in a satisfying way is an open question. I'll be "fixing" it using a hack that is possible because the properties being tested are already monadic. In summary, the GADT is nice but creates some complications. This is my first time using this design, so feedback is welcome. ### 5. Implementation to allow more-than-one Stripe Going one level deeper, I parameterized over the method of calling out to Stripe. This is primarily for testability. I don't want to actually hit Stripe hundreds of times when running the property tests discussed above. Once again I am using plain old functions, side stepping any long-winded discussion about "dependency injection". The Haskell library for Stripe actually already does this! But I wrapped over it anyway, to ignore the dozens of Stripe commands that I don't use, and to allow some flexibility in munging return types. The wrapper is another GADT named StripeI. The name will be explained in the next section. There is a dummy Stripe runner for tests: https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/test/main.hs#L48 Its implementation will be explained in the next section, as well. ### 6. Differences between MechAction and StripeI Both of these GADTs are used to define an API independent of how they are implemented. They only look different because I started out with different goals, and didn't realize I was solving the same problem. For a good intro to GADTs, read here: http://www.haskellforall.com/2012/06/gadts.html I wrote MechAction because I knew I would be creating an Arbitrary list of actions for tests. I used a GADT just so I could have different, but concrete, return types. Amusingly, that didn't work so well, since I still needed to monomorphize them some how when running tests. I think I will stick with it for another benefit, though: The GADT is like an OO interface, describing what the module can do without actually doing it. I wrote StripeI because I knew I wanted more than one way of running the actions. I used the 'operational' package, which is designed for exactly that. StripeI corresponds to the way instruction-types are built (and named!) in operational's documentation. This also explains dummyStripe's implementation, which follows the operational pattern. After I wrote both of these, however, I realized they were two versions of the same thing. Both describe the available commands without actually running them. My abstraction over the API's internals, and my abstraction over Stripe, could take on the exact same shape. At the very least, they should be named similarly (FoobarAction versus FoobarI?!). This is my second time using GADTs in anger. Super convenient, except when they are not. ### 7. What's left to do I am going to continue working on the mechanism until there is enough in place to let people make an actual pledge. Roughly, that entails the following: - Write more tests - Add Stripe return codes to MechAction results - Write a StripeI runner that actually calls out to Stripe - Choose a migrations library, and use it to add a constraint that pledges can only be made if a user has their payment info on file (Persistent can't handle this) - Update the website to use the library instead of the prototypal code - Pass -Wall -Werror everywhere Then I will switch to coordinating the work required for launching at SeaGL. There is work to do on the website and in operations. On the production website, we need the banner with instructions for verifying email addresses. Jason (JazzyEagle) has begun working on that, and could use more help. In operations, I need to put everything in place to use our Stripe keys, and I need to prepare for the user database migration. At the same time, the mechanism needs to implement the crowdmatch and payment-processing functions! I will get to that after the project is ready for launch, unless someone beats me to it. HINT HINT. :) ### Appendix A: Questions This is just a list of questions extracted/inferred from the body of the email, repeated (with a bit of context) for your convenience. 1. Any experience with manual migration libraries? drift, postgresql-simple-migrations, dbmigrations, ... 2. Thoughts on requiring client code to pass in database- and Stripe-runners to each API methods? I think it has a rewarding simplicity. Currying the API methods should make them really convenient. 3. Any good ideas on creating an Arbitrary list of heterogeneous MechActions to blast at the library for testing? I'm going to be using a low-sophistication method, so SOME solution exists, but I hope something better exists. Something something HList? But HList makes my spidey sense tingle in a bad way, and there are a lot of ways to mimic such a thing. I was directed to consider extensible-effects or possibly "reappropriating vinyl machinery". 4. Anybody want to help write mechanism methods? Both runCrowdmatch and processPayments are interesting and available. WHEW! If you're still here, wow. Thanks. It has been good to write this description and rationalization for the library's design. Hopefully it has been enlightening and inspiring. Good night. -Bryan aka chreekat --------------- : For reference, the entire list of changes (including diffs) is at: https://git.snowdrift.coop/sd/snowdrift/compare/master...split-mechanism
Description: Digital signature
_______________________________________________ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev