On 10/27/2016 06:55 PM, Bryan Richter wrote:
> Hi everyone,
> The crowdmatch mechanism code is still a work in progress, and I want
> to talk about what I've done so far to get some feedback. This email
> has become rather long! Feel free to skim it to find parts that sound
> interesting. This was written for my benefit as much as yours, and in
> the hope that I get some feedback. Anything at all, like "Well hurry up
> and do it already", is very welcome.
> It is also written for posterity and for your amusement. :)
> The new code is on a branch called split-mechanism. A giant comparison
> can be viewed, but it's longer than this email(!)[1] and I'm going to
> call out specific items instead. This will be a technical discussion,
> but not too nutty. I will talk about how I structured the crowdmatch
> mechanism code, how I've designed it to be used, how I've designed it
> to be testable, and how some quirks popped up during my headlong crash
> through coding it. Please reply with feedback ABOUT ANY PART OF THIS! :D
> ### Table of Contents:
>     1.   A separate module for the crowdmatch mechanism
>     2.   API for the library
>     2.a.   Client code requirements
>     2.b.   Usage example
>     2.c.   Exported actions
>     3.   Database management
>     3.a.   Migrations
>     3.b.   The test database
>     3.c.   run-persist library
>     4.   Implementation to allow property testing
>     5.   Implementation to allow more-than-one Stripe
>     6.   Difference between MechAction and StripeI
>     7.   What's left to do
>     Appendix A: Questions
> ### 1. A separate module for the crowdmatch mechanism
> Although there is only one foreseeable client to this code, namely the
> Snowdrift website, I went ahead and structured it as a standalone
> library. It is in a directory called 'crowdmatch'. I did this so it can
> be tested independently. It will be a lot more stable than the website
> eventually, too, which is a good decider for whether something should
> be independent or not.
> ### 2. API for the library
> The crowdmatch library puts certain constraints on client code because
> it requires a database. Then, it does what any API does: provides
> actions and data types.
> ### 2.a. Client code requirements
> A client of this library must be using Postgres via Persistent.
> The only way to be agnostic about databases would be to completely
> reimplement a database. That's nuts. The library will just use Postgres,
> and thus requires client code to use it as well.
> The library also needs to do IO to talk to Stripe. Rather than rely
> on monad transformers or mtl-style classes to manage these different
> layers, I use plain old functions. Each API method needs to be passed
> a handler for running database actions. Will this be a pain in the
> ass? Probably not. It's also the easiest style to change FROM, should
> that become necessary.
> The library maintains its own notion of what a "Patron" is. To
> interface with the website, I created ToMechPatron and FromMechPatron
> type classes. These are easy to define for the User type. Thus, it's
> easy to create a relationship between "website Users" and "crowdmatch
> Patrons".
> ### 2.b. Usage example
> Here's a usage example that covers these parts so far.
> Given:
>     fetchPatron
>         :: (ToMechPatron usr, MonadIO io, MonadIO env)
>         => SqlRunner io env
>         -> usr
>         -> env Patron
> You can run:
>     someHandler = do
>         Entity uid user <- requireAuth
>         patron <- Crowdmatch.fetchPatron runDB uid
>         ...
> See how 'runDB' is passed in as the database runner.
> ### 2.c. Exported actions
> There are about ten operations that will be exported, as well as a
> handful of data types. I am starting with fetching, storing, and
> deleting two items: payment-method tokens and pledges. The payment
> tokens are what Stripe gives us in lieu of credit card info. Pledges are
> what you think they are: the record of a donor pledged to a project.
> Next in the API are two super important methods (prototyped but not
> implemented yet): 'runCrowdmatch' and 'processPayments'.
> The first looks at all pledged patrons and calculates an outstanding
> donation balance. After a crowdmatch event, each pledged patron will
> then "owe" a certain amount to the project. That all happens in the
> database in a single transaction.
> Later, at our leisure, we can run 'processPayments'. That method
> inspects outstanding balances and sends payment commands via
> Stripe. This is where fee limits take effect.
> The website won't trigger the crowdmatch event or payment processing.
> Instead I'll have two simple utilities that do that stuff. I'll just
> run them manually at first.
> This section was longer than I intended, but that's probably good.
> It's one of the more important. Feedback on the API is highly welcome.
> API to date:
> https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/src/Crowdmatch.hs#L24
> ### 3. Database management
> Since this library requires a database, it requires database
> management. Like I said, it uses Persistent, and has its own 'Model'
> module that works like the website's Model, where the database schema
> is defined.
> ### 3.a. Migrations
> The library exports a Persistent-generated migration action that clients
> need to ensure gets run. This handles "safe" migrations. I will also be
> adding manual migrations for the cases that Persistent can't handle. I
> am leaning towards using the 'drift' library, but this is actually an
> open question.
> Does anyone have experience with manual migration libraries?
> So far I've looked at drift, postgresql-simple-migrations, and
> dbmigrations. I haven't decided which to use yet.
> ### 3.b. The test database
> I had to set up a test database, since there's no point in running tests
> without one. I wrote code that creates a tablespace in memory, so the
> tests are relatively quick.  (400 iterations of 0-100 actions in sixteen
> seconds. Yes, "relatively quick" is definitely *relative*.)
> ### 3.c. run-persist library
> Along the way, I wrote a whole 'nother library called run-persist. I
> got tired of playing Monad Jenga just to run a SqlPersistT action in
> IO. Right now this library is included in the project, but I plan on
> splitting it out once it is good enough.
> The run-persist library is pretty short:
> https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/run-persist/RunPersist.hs
> ### 4. Implementation to allow property testing
> If you look at the API for the crowdmatch library linked above, you'll
> see a thin wrapper over 'runMech' applied to values of type 'MechAction
> ret'. Why this design? I wanted to be able to create a long, random list
> of actions to blast at the library, so I could inspect invariants. This
> should catch any weird combination of actions that could lead to edge
> case behavior. I have two tests that use it: 'prop_pledgeHist' and
> 'prop_pledgeCapability'.
> https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/test/main.hs#L157
> Something I struggled with here is the type variable in 'MechAction
> ret'. The 'ret' represents the return value of the action. MechAction
> is a GADT, and each constructor can specify its unique return type
> concretely. It took a while to figure out how to run a bunch of actions
> with these heterogeneous return types. That's why right now all the
> constructors all have type 'MechAction ()'. :) I know how to change that
> now, which I will have to do to handle Stripe errors. But fixing this
> in a satisfying way is an open question. I'll be "fixing" it using a
> hack that is possible because the properties being tested are already
> monadic.
> In summary, the GADT is nice but creates some complications. This is
> my first time using this design, so feedback is welcome.
> ### 5. Implementation to allow more-than-one Stripe
> Going one level deeper, I parameterized over the method of calling out
> to Stripe. This is primarily for testability. I don't want to actually
> hit Stripe hundreds of times when running the property tests discussed
> above. Once again I am using plain old functions, side stepping any
> long-winded discussion about "dependency injection".
> The Haskell library for Stripe actually already does this! But I wrapped
> over it anyway, to ignore the dozens of Stripe commands that I don't
> use, and to allow some flexibility in munging return types.
> The wrapper is another GADT named StripeI. The name will be explained in
> the next section.
> There is a dummy Stripe runner for tests:
> https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/test/main.hs#L48
> Its implementation will be explained in the next section, as well.
> ### 6. Differences between MechAction and StripeI
> Both of these GADTs are used to define an API independent of how they
> are implemented. They only look different because I started out with
> different goals, and didn't realize I was solving the same problem.
> For a good intro to GADTs, read here:
> http://www.haskellforall.com/2012/06/gadts.html
> I wrote MechAction because I knew I would be creating an Arbitrary list
> of actions for tests. I used a GADT just so I could have different, but
> concrete, return types. Amusingly, that didn't work so well, since I
> still needed to monomorphize them some how when running tests. I think I
> will stick with it for another benefit, though: The GADT is like an OO
> interface, describing what the module can do without actually doing it.
> I wrote StripeI because I knew I wanted more than one way of running the
> actions. I used the 'operational' package, which is designed for exactly
> that. StripeI corresponds to the way instruction-types are built (and
> named!) in operational's documentation. This also explains dummyStripe's
> implementation, which follows the operational pattern.
> After I wrote both of these, however, I realized they were two versions
> of the same thing. Both describe the available commands without
> actually running them. My abstraction over the API's internals, and my
> abstraction over Stripe, could take on the exact same shape. At the very
> least, they should be named similarly (FoobarAction versus FoobarI?!).
> This is my second time using GADTs in anger. Super convenient, except
> when they are not.
> ### 7. What's left to do
> I am going to continue working on the mechanism until there is enough
> in place to let people make an actual pledge. Roughly, that entails
> the following:
> - Write more tests
> - Add Stripe return codes to MechAction results
> - Write a StripeI runner that actually calls out to Stripe
> - Choose a migrations library, and use it to add a constraint that
>   pledges can only be made if a user has their payment info on file
>   (Persistent can't handle this)
> - Update the website to use the library instead of the prototypal code
> - Pass -Wall -Werror everywhere
> Then I will switch to coordinating the work required for launching at
> SeaGL. There is work to do on the website and in operations. On the
> production website, we need the banner with instructions for verifying
> email addresses. Jason (JazzyEagle) has begun working on that, and could
> use more help.
> In operations, I need to put everything in place to use our Stripe
> keys, and I need to prepare for the user database migration.
> At the same time, the mechanism needs to implement the crowdmatch and
> payment-processing functions! I will get to that after the project is
> ready for launch, unless someone beats me to it. HINT HINT. :)
> ### Appendix A: Questions
> This is just a list of questions extracted/inferred from the body of the
> email, repeated (with a bit of context) for your convenience.
> 1. Any experience with manual migration libraries? drift,
> postgresql-simple-migrations, dbmigrations, ...
> 2. Thoughts on requiring client code to pass in database- and
> Stripe-runners to each API methods? I think it has a rewarding
> simplicity. Currying the API methods should make them really
> convenient.
> 3. Any good ideas on creating an Arbitrary list of heterogeneous
> MechActions to blast at the library for testing? I'm going to be using a
> low-sophistication method, so SOME solution exists, but I hope something
> better exists. Something something HList? But HList makes my spidey
> sense tingle in a bad way, and there are a lot of ways to mimic such
> a thing. I was directed to consider extensible-effects or possibly
> "reappropriating vinyl machinery".
> 4. Anybody want to help write mechanism methods? Both runCrowdmatch
> and processPayments are interesting and available.
> WHEW! If you're still here, wow. Thanks. It has been good to write
> this description and rationalization for the library's design.
> Hopefully it has been enlightening and inspiring. Good night.
> -Bryan
> aka chreekat
> ---------------
> [1]: For reference, the entire list of changes (including diffs) is at:
> https://git.snowdrift.coop/sd/snowdrift/compare/master...split-mechanism
> _______________________________________________
> Dev mailing list
> Dev@lists.snowdrift.coop
> https://lists.snowdrift.coop/mailman/listinfo/dev

Thanks, finally read through this. No feedback besides "keep it up!"

I hope others will weigh in.

I urge you to consider reposting this sort of broad summary to Haskell
Cafe or subreddits or other such things to invite wider feedback and
interest (but details left to your judgment)

Attachment: signature.asc
Description: OpenPGP digital signature

Dev mailing list

Reply via email to