Re: [Dev] Concerns regarding the mechanism

[Curtis Gagliardi]

Did Bryan reply offlist?  What is that a reply to?

On Fri, Jul 31, 2015, at 05:08 PM, Nikita Karetnikov wrote:
> > Oh, wow. Thanks for the tip. I will probably do that. Even after
> > spending a few hours on this esqueleto, it will still save me time to
> > switch.
> >
> > Just gotta figure out how to share database connection information...
> 
> Is it really that bad?  If it's a major slowdown, just go ahead, but
> paste what you've tried.  Also, I'd suggest moving a subselect into a
> separate Haskell function, even with strings.
> 
> Note that with esqueleto, if you change the schema, you'll get a compile
> time warning.  While with strings, it'll happen at runtime.
> ___
> Dev mailing list
> Dev@lists.snowdrift.coop
> https://lists.snowdrift.coop/mailman/listinfo/dev
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Nikita Karetnikov]

> Oh, wow. Thanks for the tip. I will probably do that. Even after
> spending a few hours on this esqueleto, it will still save me time to
> switch.
>
> Just gotta figure out how to share database connection information...

Is it really that bad?  If it's a major slowdown, just go ahead, but
paste what you've tried.  Also, I'd suggest moving a subselect into a
separate Haskell function, even with strings.

Note that with esqueleto, if you change the schema, you'll get a compile
time warning.  While with strings, it'll happen at runtime.
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Curtis Gagliardi]

I assumed it'd be easy but I've been poking around looking for a way to
do that, the best thing I've seen is to get 
the pgConnStr from the apps persistConfig and use that with
postgres-simple's "connectPostgreSQL" function, circumventing the pool. 
Creating the connection every time probably isn't actually a big deal
though.

something like: 

app <- getYesod 
let connStr = pgConnStr . persistConfig $ app
conn <- liftIO $ connectPostgreSQL
... etc

let me know if you find something better
   

On Fri, Jul 31, 2015, at 03:59 PM, Bryan Richter wrote:
> On Fri, Jul 31, 2015 at 02:41:41PM -0700, Curtis Gagliardi wrote:
> > I'd probably drop down to postgres-simple and run that query as is.  I
> > don't think the lack of typing needs to be a deal breaker if it's going
> > to be easier to write and easier to test.  
> 
> Oh, wow. Thanks for the tip. I will probably do that. Even after
> spending a few hours on this esqueleto, it will still save me time to
> switch.
> 
> Just gotta figure out how to share database connection information...
> ___
> Dev mailing list
> Dev@lists.snowdrift.coop
> https://lists.snowdrift.coop/mailman/listinfo/dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Curtis Gagliardi]

I'd probably drop down to postgres-simple and run that query as is.  I
don't think the lack of typing needs to be a deal breaker if it's going
to be easier to write and easier to test.  

Opaleye does seem really nice, I've been looking forward to playing with
it

On Fri, Jul 31, 2015, at 12:17 PM, Bryan Richter wrote:
> On Thu, Jul 30, 2015 at 05:18:00PM +0300, Nikita Karetnikov wrote:
> > I have to admit the current mechanism implementation makes me uneasy.
> > Everything is mutable and some functions, like underfundedPatrons, do
> > too much, making it hard to test and even follow.
> 
> Incidentally, where's what I would like underfundedPatrons to "look"
> like:
> 
> select * from (
> select pl.user
> , balance - (outlay.tot - (funded_shares * share_value)) as
> avail
> , pr.id as project
> , pr.share_value
> , pl.funded_shares
> , ac.id account
> from pledge pl
> join "user" u on pl.user = u.id
> join account ac on ac.id = u.account
> join project pr on pl.project = pr.id
> join (
> select u.id as user, sum(funded_shares * share_value) as tot
> from pledge pl
> join project pr on pl.project = pr.id
> join "user" u on pl.user = u.id
> join account ac on u.account = ac.id
> group by u.id
> ) outlay on outlay.user = u.id
> ) as q
> where avail < share_value * funded_shares
> order by q.user
> 
> This is a "pure" function of the underlying tables (except perhaps for
> the use of share_value, which is strictly speaking non-normalized and
> thus spooky). It could be easily tested in isolation. The subquery "q"
> could be turned into a database view and thus encapsulated.
> 
> The problem is I don't know how to interface this query with the
> Haskell code, yet. I would very much like to. Thanks to years of
> experience, I can write queries like this in a matter of minutes, but
> it still takes me hours to hammer them into the persistent/esqueleto
> mold.
> 
> I also want compile time guarantees (types). There's a un(i)typed
> multiplication in there between funded_shares and share_value that
> should be a compile-time error. The sum itself is another potential
> type error, since in general it could return null (Nothing) — although
> I think it never will in this case. I should also be able to ensure
> that every use of this query expects the correct result types.
> 
> So... any thoughts? For now I think I'll just crash through some more
> persistent and esqueleto to end up with the equivalent data. If anyone
> wants to look into this, though, I'm pretty sure you'd get good
> support from upstream.
> 
> -*-*-*- holy crap -*-*-*-
> 
> This is what I want:
> 
> https://github.com/tomjaguarpaw/haskell-opaleye
> 
> Using Opaleye would probably require some deep restructuring of the
> Snowdrift Model code, so I'll hold off for now. I do hope we can move
> towards using it, though.
> ___
> Dev mailing list
> Dev@lists.snowdrift.coop
> https://lists.snowdrift.coop/mailman/listinfo/dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Bryan Richter]

On Fri, Jul 31, 2015 at 02:41:41PM -0700, Curtis Gagliardi wrote:
> I'd probably drop down to postgres-simple and run that query as is.  I
> don't think the lack of typing needs to be a deal breaker if it's going
> to be easier to write and easier to test.  

Oh, wow. Thanks for the tip. I will probably do that. Even after
spending a few hours on this esqueleto, it will still save me time to
switch.

Just gotta figure out how to share database connection information...


signature.asc
Description: Digital signature
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Aaron Wolf]

On 07/31/2015 03:17 PM, Bryan Richter wrote:
> On Thu, Jul 30, 2015 at 05:18:00PM +0300, Nikita Karetnikov wrote:
>> I have to admit the current mechanism implementation makes me uneasy.
>> Everything is mutable and some functions, like underfundedPatrons, do
>> too much, making it hard to test and even follow.
> 
> Incidentally, where's what I would like underfundedPatrons to "look"
> like:
> 
> select * from (
> select pl.user
> , balance - (outlay.tot - (funded_shares * share_value)) as avail
> , pr.id as project
> , pr.share_value

This isn't certain to be back-end accurate terms: I don't object
strongly, but "share_value" still seems opaque. It isn't a value that is
ever specified by itself, it just is a variable that is a synonym of (
num-active-patrons * min-pledge-level ), right? I think I'm okay with
keeping it if it isn't confusing anyone else, but we need the definition
to be really easy to find at least…

> , pl.funded_shares
> , ac.id account
> from pledge pl
> join "user" u on pl.user = u.id
> join account ac on ac.id = u.account
> join project pr on pl.project = pr.id
> join (
> select u.id as user, sum(funded_shares * share_value) as tot
> from pledge pl
> join project pr on pl.project = pr.id
> join "user" u on pl.user = u.id
> join account ac on u.account = ac.id
> group by u.id
> ) outlay on outlay.user = u.id
> ) as q
> where avail < share_value * funded_shares
> order by q.user
> 
> This is a "pure" function of the underlying tables (except perhaps for
> the use of share_value, which is strictly speaking non-normalized and
> thus spooky). It could be easily tested in isolation. The subquery "q"
> could be turned into a database view and thus encapsulated.
> 
> The problem is I don't know how to interface this query with the
> Haskell code, yet. I would very much like to. Thanks to years of
> experience, I can write queries like this in a matter of minutes, but
> it still takes me hours to hammer them into the persistent/esqueleto
> mold.
> 
> I also want compile time guarantees (types). There's a un(i)typed
> multiplication in there between funded_shares and share_value that
> should be a compile-time error. The sum itself is another potential
> type error, since in general it could return null (Nothing) — although
> I think it never will in this case. I should also be able to ensure
> that every use of this query expects the correct result types.
> 
> So... any thoughts? For now I think I'll just crash through some more
> persistent and esqueleto to end up with the equivalent data. If anyone
> wants to look into this, though, I'm pretty sure you'd get good
> support from upstream.
> 
> -*-*-*- holy crap -*-*-*-
> 
> This is what I want:
> 
> https://github.com/tomjaguarpaw/haskell-opaleye
> 
> Using Opaleye would probably require some deep restructuring of the
> Snowdrift Model code, so I'll hold off for now. I do hope we can move
> towards using it, though.
> 
> 

Not qualified to really weigh in, but Opaleye sounds great on the
surface. Seems reasonable to me to keep it in mind and start using it /
replacing things with it if it's better. As long as we keep all the
Haskell functional / typed goodness, I'm all for whatever is smoothest.


> 
> ___
> Dev mailing list
> Dev@lists.snowdrift.coop
> https://lists.snowdrift.coop/mailman/listinfo/dev
> 

-- 
Aaron Wolf Snowdrift.coop 
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Bryan Richter]

On Thu, Jul 30, 2015 at 05:18:00PM +0300, Nikita Karetnikov wrote:
> I have to admit the current mechanism implementation makes me uneasy.
> Everything is mutable and some functions, like underfundedPatrons, do
> too much, making it hard to test and even follow.

Incidentally, where's what I would like underfundedPatrons to "look"
like:

select * from (
select pl.user
, balance - (outlay.tot - (funded_shares * share_value)) as avail
, pr.id as project
, pr.share_value
, pl.funded_shares
, ac.id account
from pledge pl
join "user" u on pl.user = u.id
join account ac on ac.id = u.account
join project pr on pl.project = pr.id
join (
select u.id as user, sum(funded_shares * share_value) as tot
from pledge pl
join project pr on pl.project = pr.id
join "user" u on pl.user = u.id
join account ac on u.account = ac.id
group by u.id
) outlay on outlay.user = u.id
) as q
where avail < share_value * funded_shares
order by q.user

This is a "pure" function of the underlying tables (except perhaps for
the use of share_value, which is strictly speaking non-normalized and
thus spooky). It could be easily tested in isolation. The subquery "q"
could be turned into a database view and thus encapsulated.

The problem is I don't know how to interface this query with the
Haskell code, yet. I would very much like to. Thanks to years of
experience, I can write queries like this in a matter of minutes, but
it still takes me hours to hammer them into the persistent/esqueleto
mold.

I also want compile time guarantees (types). There's a un(i)typed
multiplication in there between funded_shares and share_value that
should be a compile-time error. The sum itself is another potential
type error, since in general it could return null (Nothing) — although
I think it never will in this case. I should also be able to ensure
that every use of this query expects the correct result types.

So... any thoughts? For now I think I'll just crash through some more
persistent and esqueleto to end up with the equivalent data. If anyone
wants to look into this, though, I'm pretty sure you'd get good
support from upstream.

-*-*-*- holy crap -*-*-*-

This is what I want:

https://github.com/tomjaguarpaw/haskell-opaleye

Using Opaleye would probably require some deep restructuring of the
Snowdrift Model code, so I'll hold off for now. I do hope we can move
towards using it, though.


signature.asc
Description: Digital signature
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Bake on Gitlab

[Bryan Richter]

On Thu, Jul 30, 2015 at 07:38:45PM +0300, Nikita Karetnikov wrote:
> So I'm looking into Bake integration with Gitlab.  Here's the rough
> plan:
> 
> 1. Setup web hooks as described here:
>
> https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/web_hooks/web_hooks.md
> 
> 2. Parse that on the Bake side.
> 
> 3. Give Bake push access to master.

Let's take a step back and start with the goals and purpose. Then we
can be sure the plan fulfills them. Here's my proposal:

Gitlab merge requests are a tool we use for code review. That will
always be a manual process, by definition. The review team manually
reads the code, interacts with the contributor, and ultimately accepts
or rejects the patch.

There are some mechanical steps to a good code review, however,
and CI can perform those automatically. The purpose of doing these
things automatically is to provide quick, automatic feedback to a
contributor, as well as to save time and effort from the review
team.  The two primary mechanical tasks are (1) merging the patch
with master, (2) compiling the merged code, (3) testing it, and (4)
reporting on any errors in the first three steps. The goal for bake,
then, is to automate these tasks.

---

I think to satisfy these goals, we *don't* want bake pushing to
master. That should be a manual step. What bake should do instead is
report, by adding comments or tags to the merge request.

> It'd be ideal if Bake could test merge requests, but there are
> some challenges.  There's only one object type for merge requests,
> which can be in multiple states: open, updated, merged, closed.  So
> there doesn't seem to be a way to clearly indicate: "test this!"

Whether bake does too many builds (e.g. for closed or merged requests)
is not a dealbreaker, but obviously it would be nice to inspect the
web hook payload and only run for open or updated requests.

I take it there is no way to control bake's behavior? It sounds like
that migt be necessary for both how Gitlab interact with bake, and how
bake interacts with Gitlab. If it is not possible, I can think of a
couple possible solutions:

1.  Add hooks on bake's side!
2.  Add a proxy between Gitlab and bake.
3.  Others?

> ...
> Then, the workflow becomes: a user pushes to a feature branch on the
> main repo, Bake takes that as "ready for merge" and tests; merges if
> appropriate.  Is it possible with Gitlab?

Yes, that would be possible, but I think it would not meet the goal of
automating the mechanical aspects of a code review. I think we want to
test, then review, then merge, in that order.

I'm happy that you're working on this, by the way. I think it would be
a great addition to our workflow.


signature.asc
Description: Digital signature
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev

Re: [Dev] Concerns regarding the mechanism

[Nikita Karetnikov]

Bryan, I agree with everything you said.  To summarize: I'm fine with
some technical sloppiness and potential mistakes as long as we don't
think of the code as being production ready.
___
Dev mailing list
Dev@lists.snowdrift.coop
https://lists.snowdrift.coop/mailman/listinfo/dev