Re: Automatic environment filtering for DB queries
Just something to think about, if the new Find and FindId funcs were
mockable, it would make supporting old schemas during upgrade steps much
easier easier. For example, if I could mock out Find at the beginning of
MigrateUnitPortsToOpenedPorts (a 1.21 upgrade step) to not filter by env
uuid and then restore Find to it's original behaviour at the end, it
would save a lot of the code I'm currently having to write.
One step further, the Find and FindId funcs could possibly be version
aware, only filtering by env uuid for those collections that have be
converted by the current version (e.g.
map[version.Number][]string{/*collection names*/}). That could
potentially save a lot of boilerplate code we are currently having to
write to band aid the issue of upgrade steps for old versions using
queries designed for the current schema .
On 19/11/14 10:44, Menno Smits wrote:
Thanks John and Tim. I really like these ideas, especially because it
means the team doesn't need to learn a new way of working (and
remember to keep using the new way). In most cases, the thing returned
by getCollection() will be able to be used in the same way as before,
even though it'll actually be a different type of thing. I'll try this
approach out in the next day or so.
I have thought about what can be done about ensuring env UUIDs are
correctly added or updated during inserts and updates but I don't
think there's anything practical we can do there. I think we need to
rely on diligence and testing to ensure that writes to the DB
correctly handle environment UUIDs.
On 18 November 2014 17:03, Tim Penhey > wrote:
Wrapping the collection objects that are returned from the
`getCollection` method shouldn't be too hard and it could wrap the
Find
and FindId calls. It would have fixed this missed call below:
// aliveUnitsCount returns the number a alive units for the service.
func aliveUnitsCount(service *Service) (int, error) {
units, closer := service.st.getCollection(unitsC)
defer closer()
query := bson.D{{"service", service.doc.Name
}, {"life", Alive}}
return units.Find(query).Count()
}
However it is not just finding that we need to care about, but setting
and updating the collections. Hopefully testing would cover the cases
where we aren't setting what we think we are setting, but that is much
harder to catch as the main execution flow is to just "run these
transaction operations".
Tim
On 18/11/14 16:45, John Meinel wrote:
> I've had this around to think about for a bit. I think it is ok,
though
> sniffing an input parameter to change behavior seems brittle. Mostly
> because the object isn't really design to work that way. Could
we wrap
> the objects so that we just have Find/FindId do the right thing
to start?
>
> I suppose that is a fair amount of more work. I certainly do
like the
> idea of having a common pattern rather than requiring everyone
to know
> exactly whether this object is different than the rest.
>
> John
> =:->
>
>
> On Thu, Nov 13, 2014 at 8:10 AM, Menno Smits
mailto:menno.sm...@canonical.com>
> >> wrote:
>
> Team Onyx has been busy preparing the MongoDB collections
used by
> Juju to support data for multiple environments. For each
collection
> that needs to store data for multiple environments we have been
> adding the a "env-uuid" field to each document and prefixing the
> environment UUID to the document id, with the previous
document id
> being moved to a new field. Apart from the document changes
> themselves, we've been adjusting the state package
implementation to
> match the document changes.
>
> Part of this task is ensuring that all DB queries correctly
filter
> by environment so that we don't end up unintentionally
leaking data
> across environments. To avoid opportunities for us to forget
to add
> environment filtering to the DB queries using by Juju,
sabdfl would
> like us to consider ways to make this filtering happen
> automatically. To this end, I propose we add the following
methods:
>
> func (s *State) Find(coll *mgo.Collection, sel bson.D)
*mgo.Query
> func (s *State) FindId(coll *mgo.Collection, id string)
*mgo.Query
>
>
> The idea is that almost all MongoDB queries performed by
Juju would
> use these 2 methods. They would become the default way that
we do
> queries, used even on collections that don't contain data for
> multiple environments.
>
> Both methods would check the collection passed in again
1.21 critical issue - fixing juju-run
Hi Curtis, https://bugs.launchpad.net/juju-core/+bug/1392745 This needs to be fixed on 1.20, 1.21 and master branches. The fix has landed on 1.20. Hopefully landed on 1.21 by the time you read this. Work needed to tweak for master due to different ways the upgrade steps are being run for now. I think this is a fairly low risk change for 1.21, and easily checked with the standard upgrade tests and a check of 'juju run' after the upgrade has completed. Cheers, Tim -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
> I think you guys have more things to worry about than the spacing style for > comments. I'm happy to let this discussion die at this point. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
I'm watching this thread with interest as a habitual 2-spacer :) This is a really interesting post on the subject: http://www.ditchwalk.com/2011/01/19/two-spaces-after-a-period/ But the best quote is at the end of Nate's reference: "Complaining about the way people space their sentences in their own documents is being an ass." I think you guys have more things to worry about than the spacing style for comments. On Wednesday 19 Nov 2014 05:52:42 Nate Finch wrote: > btw, the typewriter starting the whole two spaces thing is a myth: > http://www.heracliteanriver.com/?p=324 > > I still vote for: this is a waste of time, we have better things to worry > about. > > On Wed, Nov 19, 2014 at 4:05 AM, Dimiter Naydenov < > > dimiter.nayde...@canonical.com> wrote: > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA1 > > > > On 19.11.2014 09:12, Dimiter Naydenov wrote: > > > On 19.11.2014 02:10, Jesse Meek wrote: > > >> I'm coming across double spaces after a full stop in comments. > > >> Some consider this an error, others an intentional style that > > >> makes the comment cleaner to read. > > >> > > >> Yes, it's a minor point but in the name of consistency and > > >> avoiding future review ping pong on the issue, I'm calling for a > > >> vote: > > >> > > >> Should we have a single space after full stops in comments? > > >> http://xkcd.com/1285/ > > >> > > >> Please reply +1 for yes to one space, -1 for no to one space. > > >> I'll tally up votes on Friday and update our style guide > > >> accordingly. > > >> > > >> My vote: +1 > > > > > > -1, come on isn't this just for typewriters? > > > > Sorry, I should've said +1 :) > > > > > > - -- > > Dimiter Naydenov > > juju-core team > > -BEGIN PGP SIGNATURE- > > Version: GnuPG v1 > > > > iQEcBAEBAgAGBQJUbF1MAAoJENzxV2TbLzHwR9UH/0nRdGvDBhKj7rdZprdC/HIX > > TWbXpASMaAzghQMk8gmauvlapgeeBEyEWzHjVA0kdrmuMIDALGVrHxV5LVRmf9kd > > 5kX8iFNv2Sqb3NEuCarvttAzrjzOfkkyKvL0D/Mu8h7zOSCnZOY7pjZC9upBNx/8 > > HLtVwS3HWFm7CUBBRVDA4TR9b2ntbUy334tw+iAvRYbYDNAa2G43VjVnNRnP1yPl > > HbcwK8Dv+iFVwC1gO6+jO/cG3FwMaQ7P9qu171AWz/rDWyxFS3CROVLrI2vNdsKp > > w0a91Us3GkHIYN5GOlUXvUE2rZ3wGYU5r/flTnAaGiWhWW4YO+VWPjU6GqJCOOQ= > > =mPAH > > -END PGP SIGNATURE- > > > > -- > > Juju-dev mailing list > > Juju-dev@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
On Wed, Nov 19, 2014 at 10:31 AM, Eric Snow wrote: > Apparently I'm not the only one. FWIW, Python's PEP 8 dictates the use of two spaces in its stdlib. [1] Presumably this is because code is typically(?) edited/viewed with mono-spaced fonts. Regardless, both are at least part of why I do it. -eric [1] https://www.python.org/dev/peps/pep-0008#comments -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
On Tue, Nov 18, 2014 at 5:13 PM, David Cheney wrote: > To quote Butterick's > http://practicaltypography.com/one-space-between-sentences.html And to quote that same page: "If you have to use a typewriter-style font, you can use two spaces after sentences. Otherwise, don’t." I use a mono-space font. Furthermore, I disagree with the statement "In the second paragraph, the extra spaces disrupt the balance of white space." I find that having each sentence be visually distinct to help with readability. Apparently I'm not the only one. From what I can tell, studies on the matter are inconclusive. [1] Regardless, I don't think it matters all that much, particularly since (as Roger pointed out) godoc automatically converts to single-space for you. Like Nate, I think there are more significant things to worry about, though perhaps not as controversial. :) It seems like the overall consensus is to use single space. If that means it becomes a rule for our code then naturally I'll go along. Otherwise I'd rather not bother. To be honest, this all seems like a waste of time. -eric [1] http://en.wikipedia.org/wiki/Sentence_spacing#Effects_on_readability_and_legibility -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
On Wed, Nov 19, 2014 at 10:52 AM, Nate Finch wrote: > btw, the typewriter starting the whole two spaces thing is a myth: > http://www.heracliteanriver.com/?p=324 well, actually, typewriters DID start the whole "two spaces" thing, before then the amount of space used was a variable block of type, which varied according the the typesetter/publisher/country/fashion. Typewriters quantified that as two "spaces". But that's also a bit irrelevant. It doesn't make a bit of difference to me either way, I just didn't want you to accept some style that was completely at odds with the accepted conventions. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Feature Request: show running relations in 'juju status'
On Tue, Nov 18, 2014 at 9:37 AM, Stuart Bishop wrote: > Ok. If there is a goal state, and I am able to wait until the goal > state is the actual state, then my needs (and amulet and juju-deployer > needs) will be met. It does seem a rather lengthy and long winded way > of getting there though. The question I have always needed juju to > answer is 'are there any hooks running or are there any hooks queued > to run?'. I've always assumed that juju must already know this (or it > would be unable to function), but refuses to communicate this single > bit of information in any way. Juju as a system actually doesn't know this. Unit idleness is known only by the unit agents themselves, and only implicitly at that -- if we're blocking in a particular select clause then we're (probably!) idle, and that's it. I agree that exposing idleness would be good, and I'm doing some of the preliminary work necessary right now, but it's not my current focus: it's just a happy side-effect of what needs to be done for leader election. The impact of exposing goal/active state (and hooks that trigger on changes to same) is rather different: it's internal to a unit, and is essentially an alternative to joined/departed hooks. (There's nothing stopping you having both, but not much point to it either.) > That would work too. If all units are in idle state, then the system > has reached a steady state and my question answered. Sort of. It's steady for now, but will not necessarily still be steady by the time you're reacted to it -- even if you're the only administrator, imagine a cron job that uses juju-run and triggers a wave of relation traffic across the system. > I'm not entirely sure how useful this feature is, given the inherent > race conditions with serialized hooks. Right now, you need to write > charms that gracefully cope with dependent services that have gone > down without notice. With this feature, you will need to write charms > that gracefully cope with dependent services that have gone down and > the notification hasn't reached you yet. Or if the outage is for > non-juju reasons, like a network partition. The window of time waiting > for hooks to bubble through could easily be minutes when you have a > simple chain of services (eg. postgresql -> pgbouncer -> django -> > haproxy -> apache seems common enough). Yeah, you never get away from having to cope gracefully with unexpected failures. But there is still value there -- when one of your remotes takes itself voluntarily out of rotation, you can know not to send it traffic until it tells you it's ready again. > Your example with storage is particularly interesting, as I was just > dealing with this yesterday in my rewrite of the Cassandra charm. The > existing mechanism in the charm is broken. If you add a new unit to > the service, it runs its install and configure hooks and is READY. It > then joins the peer relation, and is still READY. The peer units start > spewing data at it, as the replication ring is rebalanced. We now > have a race. Will the storage hooks fire in time? The new unit unaware > that storage is due to be attached, and does not know that, unless the > storage is attached and the data migrated from local disk soon, the > local disk will fill and the unit will fall over. To solve this with > the current storage-broker subordinate, I could require the operator > to set an 'wait_for_block_storage' boolean in the service > configuration before deploy. But requiring people to read and follow > the documentation is an error prone solution :-( I'm wondering if I > should simply not bother fixing this race, and trust that the block > storage broker hooks will be invoked and completed before local disk > is filled. I understand that work is underway to replace the block > storage broker so it won't be an issue long term, or your goal state > would be useful here if a unit can ask questions like 'is storage > going to be attached' or 'will peers be joining me'. So, in my mind, the goal/active stuff is relevant to all relations, not just peers. So, on the one hand: yes, assuming the relation with storage-broker exists at the time the unit starts up, it should be aware that there will be storage. But on the other... dynamically adding a storage-broker relation would still be hard; and even when storage is in the juju model, handling dynamic storage changes is going to take a bit of effort to migrate your data. Cheers William -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Feature Request: show running relations in 'juju status'
On Tue, Nov 18, 2014 at 7:19 AM, Kapil Thangavelu wrote: > > for clusters... its not a question of futures but being informed of known > unit count to establish quorum. ie 1 to 3 or n+1. leader election helps, but > actually knowing the unit count is critical to being able to establish a > clear state without throwing away data (aka race on peer knowing quorum and > leader) as adhoc leader election has to throw away data from non leaders who > may already be serving clients due to lack of quorum knowledge. > ... > status per future impl helps, as does explicitly marking units.. but pending > cluster count is a missing and important property to properly establish > quorum in a peer rel from one to n that is only resolved by knowing recorded > units count for a svc. Two things: 1) I'm not sure numbers are good enough in general compared to sets of units. 2) I'm not sure a single number|set gives us all the information we need. Leaving (1) aside for now, I *think* that each of the following numbers|sets is potentially relevant: i) "goal": the units that juju expects to be part of the relation once everything's converged (including those not yet running, and not including those that are dying) ii) "active": the units that are in scope for the relation (but might be dying) iii) "current": the units that are *locally known to be* in scope for the relation Today, we only expose "current" -- ie relation-list returns the "current" units, and it might be a complete lie, but it's a consistent lie that is useful for many purposes, and we're not going to remove it. If we expose "active" but not "goal", we don't help anything very much -- the first unit of a cluster to come up will still think it's alone in the world, and we still have all the original problems. If we expose "goal" but not "active", we create new problems when we try to scale: going from 1 unit to 3 puts that first unit in an apparent minority, and is thus likely to effectively take the whole service down. So: I think we definitely need to expose both "goal" and "active" information. The interesting question is whether we can just expose numbers, or whether we need to expose actual sets of units (as we do for "current")... and I *think* we need sets, not just numbers, because: u/0:current=[u/1,u/2] u/0:active=[u/3,u/4] u/0:goal=[u/3,u/4] ...is legitimate, when 3,4 were created a while ago (and have just come up, but 0 has not yet run their joined hooks) and 1,2 were *just* destroyed (and have themselves left scope, but 0 has not yet run their departed hooks)... ...and if all we expose is numbers, there's no way for the charm to tell the difference between that state and a stable 0,1,2 cluster (or any of the other combinations with sets of the given sizes...) at least until more hooks fire. *Maybe* this doesn't matter, but I'm loath to assume that it *never* matters. Thoughts? Cheers William -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
btw, the typewriter starting the whole two spaces thing is a myth: http://www.heracliteanriver.com/?p=324 I still vote for: this is a waste of time, we have better things to worry about. On Wed, Nov 19, 2014 at 4:05 AM, Dimiter Naydenov < dimiter.nayde...@canonical.com> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 19.11.2014 09:12, Dimiter Naydenov wrote: > > On 19.11.2014 02:10, Jesse Meek wrote: > >> I'm coming across double spaces after a full stop in comments. > >> Some consider this an error, others an intentional style that > >> makes the comment cleaner to read. > > > >> Yes, it's a minor point but in the name of consistency and > >> avoiding future review ping pong on the issue, I'm calling for a > >> vote: > > > >> Should we have a single space after full stops in comments? > >> http://xkcd.com/1285/ > > > >> Please reply +1 for yes to one space, -1 for no to one space. > >> I'll tally up votes on Friday and update our style guide > >> accordingly. > > > >> My vote: +1 > > > > -1, come on isn't this just for typewriters? > > > > > Sorry, I should've said +1 :) > > > - -- > Dimiter Naydenov > juju-core team > -BEGIN PGP SIGNATURE- > Version: GnuPG v1 > > iQEcBAEBAgAGBQJUbF1MAAoJENzxV2TbLzHwR9UH/0nRdGvDBhKj7rdZprdC/HIX > TWbXpASMaAzghQMk8gmauvlapgeeBEyEWzHjVA0kdrmuMIDALGVrHxV5LVRmf9kd > 5kX8iFNv2Sqb3NEuCarvttAzrjzOfkkyKvL0D/Mu8h7zOSCnZOY7pjZC9upBNx/8 > HLtVwS3HWFm7CUBBRVDA4TR9b2ntbUy334tw+iAvRYbYDNAa2G43VjVnNRnP1yPl > HbcwK8Dv+iFVwC1gO6+jO/cG3FwMaQ7P9qu171AWz/rDWyxFS3CROVLrI2vNdsKp > w0a91Us3GkHIYN5GOlUXvUE2rZ3wGYU5r/flTnAaGiWhWW4YO+VWPjU6GqJCOOQ= > =mPAH > -END PGP SIGNATURE- > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 19.11.2014 09:12, Dimiter Naydenov wrote: > On 19.11.2014 02:10, Jesse Meek wrote: >> I'm coming across double spaces after a full stop in comments. >> Some consider this an error, others an intentional style that >> makes the comment cleaner to read. > >> Yes, it's a minor point but in the name of consistency and >> avoiding future review ping pong on the issue, I'm calling for a >> vote: > >> Should we have a single space after full stops in comments? >> http://xkcd.com/1285/ > >> Please reply +1 for yes to one space, -1 for no to one space. >> I'll tally up votes on Friday and update our style guide >> accordingly. > >> My vote: +1 > > -1, come on isn't this just for typewriters? > > Sorry, I should've said +1 :) - -- Dimiter Naydenov juju-core team -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJUbF1MAAoJENzxV2TbLzHwR9UH/0nRdGvDBhKj7rdZprdC/HIX TWbXpASMaAzghQMk8gmauvlapgeeBEyEWzHjVA0kdrmuMIDALGVrHxV5LVRmf9kd 5kX8iFNv2Sqb3NEuCarvttAzrjzOfkkyKvL0D/Mu8h7zOSCnZOY7pjZC9upBNx/8 HLtVwS3HWFm7CUBBRVDA4TR9b2ntbUy334tw+iAvRYbYDNAa2G43VjVnNRnP1yPl HbcwK8Dv+iFVwC1gO6+jO/cG3FwMaQ7P9qu171AWz/rDWyxFS3CROVLrI2vNdsKp w0a91Us3GkHIYN5GOlUXvUE2rZ3wGYU5r/flTnAaGiWhWW4YO+VWPjU6GqJCOOQ= =mPAH -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
Learned on a mechanical one too. But here in Germany it always is a single space. So take it as a +1 from my side. ;) mue On Wed, Nov 19, 2014 at 9:08 AM, Matthew Williams < matthew.willi...@canonical.com> wrote: > Believe it or not I also learned to type on a mechanical typewriter and I > did double spaces on that. But you also needed to un jam it if you typed > too fast. > > I'm voting for single space. > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > > -- ** Frank Mueller ** Software Engineer - Juju Development ** Canonical -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Too space, or not two space
Believe it or not I also learned to type on a mechanical typewriter and I did double spaces on that. But you also needed to un jam it if you typed too fast. I'm voting for single space. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
