Re: [Dev] Concerns regarding the mechanism
On Sat, Aug 01, 2015 at 03:08:47AM +0300, Nikita Karetnikov wrote: Oh, wow. Thanks for the tip. I will probably do that. Even after spending a few hours on this esqueleto, it will still save me time to switch. Just gotta figure out how to share database connection information... Is it really that bad? If it's a major slowdown, just go ahead, but paste what you've tried. Also, I'd suggest moving a subselect into a separate Haskell function, even with strings. Here's what is happening so far: -- | Find underfunded patrons. -- This function is used after something may have broken the invariant that -- all funded pledges are covered by the patron's account balance. It returns -- pertinent information about each underfunded pledge. -- underfundedUsers :: (MonadIO m, Functor m) = ProjectId - SqlPersistT m [Underfunded] underfundedUsers projId = do -- the inner query totalOutlays - fmap processOutlayRows $ select $ from $ \(pl `InnerJoin` pr `InnerJoin` u `InnerJoin` ac) - do on_ $ ac ^. AccountId ==. u ^. UserAccount on_ $ u ^. UserId ==. pl ^. PledgeUser on_ $ pr ^. ProjectId ==. pl ^. PledgeProject where_ $ pl ^. PledgeFundedShares . val 0 groupBy $ (u ^. UserId) return (u ^. UserId ,sum_ (($*.) (pr ^. ProjectShareValue) (pl ^. PledgeFundedShares)) ) let process' = processIntermediateRows totalOutlays intemediate - fmap process' $ select $ from $ \(pl `InnerJoin` u `InnerJoin` ac `InnerJoin` pr) - do on_ $ pr ^. ProjectId ==. pl ^. PledgeProject on_ $ ac ^. AccountId ==. u ^. UserAccount on_ $ u ^. UserId ==. pl ^. PledgeUser return (u ^. UserId ,ac ^. AccountBalance ,pl ^. PledgeFundedShares ,pr ^. ProjectId ,pr ^. ProjectShareValue ,ac ^. AccountId) where -- | Just cleans up types and removes rows that returned Nothing -- for the sum. (There actually shouldn't be any because of the inner -- join, but whatevs.) processOutlayRows :: [(Value UserId, Value (Maybe Milray))] - Map UserId Milray processOutlayRows = M.fromList . catMaybes . map sequence -- 'lift' the Maybe . unwrapValues -- | Primarily calculates the available balance for each pledge. -- That is defined, for each pledge, as the difference between the -- patron's total balance and the sum of the outlay going to the -- patron's *other* pledges. processIntermediateRows :: Map UserId Milray - [(Value UserId ,Value Milray ,Value Int64 ,Value ProjectId ,Value Milray ,Value AccountId)] - [Underfunded] processIntermediateRows totalOutlays = map ( (\(u, a, p, v) - Underfunded u a p v)) . map getAvailable . unwrapValues ) where getAvailable (UserId, Milray, Int64, ProjectId, Milray, AccountId) = -- XXX TODO undefined Note that with esqueleto, if you change the schema, you'll get a compile time warning. While with strings, it'll happen at runtime. True, but I have to accept that risk right now. Tests will help. signature.asc Description: Digital signature ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev
Re: [Dev] Concerns regarding the mechanism
Did Bryan reply offlist? What is that a reply to? On Fri, Jul 31, 2015, at 05:08 PM, Nikita Karetnikov wrote: Oh, wow. Thanks for the tip. I will probably do that. Even after spending a few hours on this esqueleto, it will still save me time to switch. Just gotta figure out how to share database connection information... Is it really that bad? If it's a major slowdown, just go ahead, but paste what you've tried. Also, I'd suggest moving a subselect into a separate Haskell function, even with strings. Note that with esqueleto, if you change the schema, you'll get a compile time warning. While with strings, it'll happen at runtime. ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev
Re: [Dev] Concerns regarding the mechanism
On 07/31/2015 03:17 PM, Bryan Richter wrote: On Thu, Jul 30, 2015 at 05:18:00PM +0300, Nikita Karetnikov wrote: I have to admit the current mechanism implementation makes me uneasy. Everything is mutable and some functions, like underfundedPatrons, do too much, making it hard to test and even follow. Incidentally, where's what I would like underfundedPatrons to look like: select * from ( select pl.user , balance - (outlay.tot - (funded_shares * share_value)) as avail , pr.id as project , pr.share_value This isn't certain to be back-end accurate terms: I don't object strongly, but share_value still seems opaque. It isn't a value that is ever specified by itself, it just is a variable that is a synonym of ( num-active-patrons * min-pledge-level ), right? I think I'm okay with keeping it if it isn't confusing anyone else, but we need the definition to be really easy to find at least… , pl.funded_shares , ac.id account from pledge pl join user u on pl.user = u.id join account ac on ac.id = u.account join project pr on pl.project = pr.id join ( select u.id as user, sum(funded_shares * share_value) as tot from pledge pl join project pr on pl.project = pr.id join user u on pl.user = u.id join account ac on u.account = ac.id group by u.id ) outlay on outlay.user = u.id ) as q where avail share_value * funded_shares order by q.user This is a pure function of the underlying tables (except perhaps for the use of share_value, which is strictly speaking non-normalized and thus spooky). It could be easily tested in isolation. The subquery q could be turned into a database view and thus encapsulated. The problem is I don't know how to interface this query with the Haskell code, yet. I would very much like to. Thanks to years of experience, I can write queries like this in a matter of minutes, but it still takes me hours to hammer them into the persistent/esqueleto mold. I also want compile time guarantees (types). There's a un(i)typed multiplication in there between funded_shares and share_value that should be a compile-time error. The sum itself is another potential type error, since in general it could return null (Nothing) — although I think it never will in this case. I should also be able to ensure that every use of this query expects the correct result types. So... any thoughts? For now I think I'll just crash through some more persistent and esqueleto to end up with the equivalent data. If anyone wants to look into this, though, I'm pretty sure you'd get good support from upstream. -*-*-*- holy crap -*-*-*- This is what I want: https://github.com/tomjaguarpaw/haskell-opaleye Using Opaleye would probably require some deep restructuring of the Snowdrift Model code, so I'll hold off for now. I do hope we can move towards using it, though. Not qualified to really weigh in, but Opaleye sounds great on the surface. Seems reasonable to me to keep it in mind and start using it / replacing things with it if it's better. As long as we keep all the Haskell functional / typed goodness, I'm all for whatever is smoothest. ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev -- Aaron Wolf Snowdrift.coop https://snowdrift.coop ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev
Re: [Dev] Concerns regarding the mechanism
On Fri, Jul 31, 2015 at 02:41:41PM -0700, Curtis Gagliardi wrote: I'd probably drop down to postgres-simple and run that query as is. I don't think the lack of typing needs to be a deal breaker if it's going to be easier to write and easier to test. Oh, wow. Thanks for the tip. I will probably do that. Even after spending a few hours on this esqueleto, it will still save me time to switch. Just gotta figure out how to share database connection information... signature.asc Description: Digital signature ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev
Re: [Dev] Concerns regarding the mechanism
On 2015-07-30 Nikita Karetnikov nik...@karetnikov.org wrote: I have to admit the current mechanism implementation makes me uneasy. Everything is mutable and some functions, like underfundedPatrons, do too much, making it hard to test and even follow. When I start a new Haskell program, I usually try to express the core idea with types, making illegal states unrepresentable. Then, I write pure functions connecting the said types together, which compute the result. IO is only used on the edges (to receive and send the data). This is a very common technique. Can't we use it for the mechanism? How do you feel about this? I think the first step should be the rigorous spec of the things we care about, i.e., anything touching money. Then we follow the process described above. Testing becomes trivial because the core stuff is pure while IO actions are just reads/writes. Sounds good to me (but I'm probably not the person to ask). I just wanted to express a little thought: One of the strengths of Haskell is the advanced type system and all the static verification it can give you. It means critical things can be verified statically instead of relying on tests to catch them (Yesod in general seems to take this approach seriously). All the m0ney related stuff is probably at least somewhat critical, so perhaps more work with types could replace some tests. Just a random idea, I don't know how the code actually works right now :P --- fr33domlover http://www.rel4tion.org/people/fr33domlover GPG key ID: 63E5E57D (size: 4096) GPG key fingerprint: 6FEE C222 7323 EF85 A49D 5487 5252 C5C8 63E5 E57D signature.asc Description: PGP signature ___ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev
