Summary of IRC Meeting in #aurora at Mon Oct 5 18:05:43 2015:
Attendees: dchung, wickman, jcohen, kts, mkhutornenko, rdelvalle, zmanji,
dlester
- Preface
- AURORA-1503
- AURORA-1504
- Action: wickman to propose json input format on dev list
- AURORA-1376
IRC log follows:
## Preface ##
[Mon Oct 5 18:06:14 2015] <zmanji>: I have a topic if no one else has one
[Mon Oct 5 18:06:19 2015] <jcohen>: Meeting time everyone. As always, everyone
is welcome to participate.
[Mon Oct 5 18:06:25 2015] <jcohen>: Let's start with roll call.
[Mon Oct 5 18:06:32 2015] <dlester>: present :D
[Mon Oct 5 18:06:33 2015] <jcohen>: here.
[Mon Oct 5 18:06:37 2015] <rdelvalle>: present
[Mon Oct 5 18:06:40 2015] <zmanji>: here
[Mon Oct 5 18:06:58 2015] <dchung>: David Chung here.
[Mon Oct 5 18:07:36 2015] <kts>: here
[Mon Oct 5 18:07:40 2015] <mkhutornenko>: here
[Mon Oct 5 18:07:50 2015] <wickman>: here
[Mon Oct 5 18:08:08 2015] <jcohen>: zmanji: you said you had a topic to kick
us off?
## AURORA-1503 ##
[Mon Oct 5 18:08:22 2015] <jcohen>: AURORA-1503
[Mon Oct 5 18:08:35 2015] <jcohen>: excellent topic!
[Mon Oct 5 18:08:44 2015] <zmanji>: as some of us might be aware of the Mesos
release frequency has increased to about one release a month
[Mon Oct 5 18:09:02 2015] <zmanji>: but the deprecation policy has not changed
[Mon Oct 5 18:09:21 2015] <zmanji>: This means Aurora 0.9.0 (linked against
Mesos 0.22) does not work against a Mesos master 0.24
[Mon Oct 5 18:09:30 2015] <jcohen>: can you clarify what the deprecation
policy is for those not aware
[Mon Oct 5 18:09:53 2015] <jcohen>: I believe it's +/- 1 version?
[Mon Oct 5 18:10:00 2015] <zmanji>: Yes, it is +/- 1 version
[Mon Oct 5 18:10:13 2015] <jcohen>: but they're considering moving to a
time-based deprecation policy instead?
[Mon Oct 5 18:10:34 2015] <zmanji>: There is some discussion around a time
based deprecation policy, but there is no clarity on when that will occur
[Mon Oct 5 18:10:55 2015] <zmanji>: Going back to the problem in AURORA-1503,
we ideally solve two problems.
[Mon Oct 5 18:11:08 2015] <zmanji>: 1. Allowing our users to upgrade mesos
installations while still running 0.9.x
[Mon Oct 5 18:11:16 2015] <zmanji>: 2. Figure out an upgrade story for 0.10.0
[Mon Oct 5 18:11:33 2015] <zmanji>: I currently have a plan that I would like
to share. I will put up an email on the mailing list later on this week.
[Mon Oct 5 18:12:00 2015] <zmanji>: I think we can release 0.9.1 linked
against mesos 0.23. And then release 0.10.0 against mesos 0.24.
[Mon Oct 5 18:12:38 2015] <zmanji>: AFAIK, 0.9.1 (with 0.23) will not disrupt
existing installations running 0.22 and provides a clean upgrade path for our
users to 0.24
[Mon Oct 5 18:12:43 2015] <zmanji>: That's all I got
[Mon Oct 5 18:13:29 2015] <jcohen>: I think this probably warrants discussion
on the list as I'm not sure what we consider worthy of a full version bump
[Mon Oct 5 18:13:41 2015] <jcohen>: (i.e. should we release 0.10.0 that does
nothing other than bump mesos version)
[Mon Oct 5 18:13:55 2015] <kts>: i feel like we should err on the side of a
full version bump here
[Mon Oct 5 18:14:01 2015] <mkhutornenko>: +1 to dev list, I also have concerns
about feature fragmentation due to that
[Mon Oct 5 18:14:33 2015] <zmanji>: Anyways, more discussion to come on the
dev list this week, be ready for that
[Mon Oct 5 18:14:42 2015] <jcohen>: thanks zmanji
[Mon Oct 5 18:14:51 2015] <jcohen>: anyone else have a topic for discussion?
[Mon Oct 5 18:14:53 2015] <wickman>: AURORA-1504
[Mon Oct 5 18:15:07 2015] <wickman>: does anybody use any of the
--read-json/--write-json client feautres?
## AURORA-1504 ##
[Mon Oct 5 18:15:29 2015] <wickman>: or specifically, does anyone write
configs using json?
[Mon Oct 5 18:16:29 2015] <wickman>: if the answer is "no", then i'd like to
make the client better at going between .aurora<->json
[Mon Oct 5 18:16:30 2015] <jcohen>: to be clear, this is distinct from the
option of the client outputting json for the sake of parseability in some other
orchestrating toolchain?
[Mon Oct 5 18:16:35 2015] <wickman>: maybe through a separate subcommand
[Mon Oct 5 18:16:41 2015] <wickman>: or through the config subcommand
[Mon Oct 5 18:16:46 2015] <wickman>: right
[Mon Oct 5 18:16:49 2015] <wickman>: this is mostly for configs
[Mon Oct 5 18:17:04 2015] <kts>: wickman: iiuc the schema to do this already
exists and is protected by backwards-compatibility guarantees
[Mon Oct 5 18:17:10 2015] <kts>: i.e. the pystachio schema
[Mon Oct 5 18:17:29 2015] <kts>: so allowing json as an input format is
uncontroversial to me
[Mon Oct 5 18:17:31 2015] <wickman>: 'schema' you mean the client contract?
[Mon Oct 5 18:17:54 2015] <wickman>: but i'm proposing that we change the
input format of --read-json
[Mon Oct 5 18:17:57 2015] <kts>: the .aurora file format has a schema
[Mon Oct 5 18:18:16 2015] <kts>: oh how so?
[Mon Oct 5 18:18:17 2015] <wickman>: since it needs to read more than just the
information defined by the Job schema
[Mon Oct 5 18:18:23 2015] <wickman>: specifically metadata fields for
JobConfiguration
[Mon Oct 5 18:18:38 2015] <wickman>: like those generated by binding helpers
[Mon Oct 5 18:18:52 2015] <wickman>: either that or we just *add* metadata
Map(String,String) to the Job schema
[Mon Oct 5 18:18:57 2015] <wickman>: and have binding helpers mutate it
[Mon Oct 5 18:19:05 2015] <wickman>: that may be the best option
[Mon Oct 5 18:19:11 2015] <kts>: adding that map seems like a good option
[Mon Oct 5 18:19:19 2015] <kts>: then the json input format can also use
binding helpers
[Mon Oct 5 18:19:42 2015] <kts>: for example the (currently future
hypothetical) docker-resolve-to-sha one
[Mon Oct 5 18:20:13 2015] <wickman>: right.
[Mon Oct 5 18:20:40 2015] <kts>: does --read-json currently bypass binding
helpers?
[Mon Oct 5 18:20:50 2015] <wickman>: i don't even know if --read-json works at
all tbh
[Mon Oct 5 18:21:09 2015] <kts>: and if nobody uses it anyway then it's a moot
discussion
[Mon Oct 5 18:21:11 2015] <wickman>: it's unclear to me that tests even exist
either
[Mon Oct 5 18:21:41 2015] <zmanji>: We can just remove --read-json
[Mon Oct 5 18:22:08 2015] <kts>: well it sounds like wickman is a user wanting
to use it
[Mon Oct 5 18:22:12 2015] <wickman>: i want to use it
[Mon Oct 5 18:22:27 2015] <wickman>: or something that allows me to input a
job as job.json instead of <jobkey> <aurorafile>
[Mon Oct 5 18:23:09 2015] <rdelvalle>: this sounds in line with the REST API
changes that have been proposed
[Mon Oct 5 18:23:18 2015] <wickman>: this is ubiquitous with all other cloud
offerings, e.g. kubernetes, cloudformation, hashicorp's new thing etc
[Mon Oct 5 18:23:48 2015] <jcohen>: Yeah, it would definitely be nice for the
REST api to be able to deal with a JSON representation of a job config (and
have it be the client's responsibility to translate from .aurora to JSON)
[Mon Oct 5 18:23:52 2015] <wickman>: yes, being able to use the client to suck
in .aurora + jobkey and spit out json will be essential as we move towards a
REST API
[Mon Oct 5 18:24:04 2015] <wickman>: the 'config' subcommand seems appropriate
for this
[Mon Oct 5 18:24:30 2015] <jcohen>: Nothing here sounds particularly
controversial to me.
[Mon Oct 5 18:24:38 2015] <wickman>: ok.
[Mon Oct 5 18:25:10 2015] <kts>: so we have discussed a need for "strict mode"
[Mon Oct 5 18:25:55 2015] <kts>: it seems like we might want a way to take a
json file with unbound bindings and return a static one
[Mon Oct 5 18:26:32 2015] <wickman>: yes, --bind is still relevant for reading
json blobs
[Mon Oct 5 18:26:50 2015] <wickman>: the client would still do
Job.reads_json(open(filename))
[Mon Oct 5 18:27:02 2015] <wickman>: meaning that bindings can still be made,
and you can still reference {{mustache}} variables
[Mon Oct 5 18:27:31 2015] <wickman>: in fact, .json is, for all intents and
purposes, a manifestation of strictm ode
[Mon Oct 5 18:27:42 2015] <kts>: so im proposing we have a mode that will take
a json with {{package.version[example]}} and return one with 42
[Mon Oct 5 18:28:02 2015] <wickman>: yes, binding helpers should also still
work in this model
[Mon Oct 5 18:28:06 2015] <wickman>: assuming they're linked into the client
[Mon Oct 5 18:28:15 2015] <kts>: so input is a json file or a .aurora file
[Mon Oct 5 18:28:23 2015] <wickman>: (as an aside, we should make it easier to
add binding helpers as plugins from the environment)
[Mon Oct 5 18:28:27 2015] <wickman>: input is .json
[Mon Oct 5 18:28:59 2015] <kts>: i think compiling an .aurora file to a .json
file to be deployed later is a use case we should consider here as well
[Mon Oct 5 18:29:12 2015] <zmanji>: +1 to making binding helpers as plugins
for the environment
[Mon Oct 5 18:29:19 2015] <zmanji>: s/environment/client
[Mon Oct 5 18:29:29 2015] <wickman>: that's exactly what i'm interested in --
because i'd like to save compiled .json files and use them for rollbacks /
rollforwards
[Mon Oct 5 18:29:30 2015] <kts>: it sounds like this proposed .json input
format is equivalent to the .aurora input file (less the python eval())
[Mon Oct 5 18:29:57 2015] <kts>: both of those should be compilable to static
things that facilitate roll-{forward,back}
[Mon Oct 5 18:30:03 2015] <jcohen>: it's a JSON representation of a reified
Aurora config, right?
[Mon Oct 5 18:30:13 2015] <kts>: how do you mean reified?
[Mon Oct 5 18:30:21 2015] <jcohen>: everything is bound?
[Mon Oct 5 18:30:25 2015] <kts>: it's not reified, it still supports binding
helpers
[Mon Oct 5 18:30:31 2015] <wickman>: yes. we already have this, except that
the Job schema does not completely encapsulate everything in the
JobConfiguration thrift struct.
[Mon Oct 5 18:30:36 2015] <kts>: a json file with {{package.version}} is a
valid input format
[Mon Oct 5 18:30:41 2015] <kts>: and will still run the binding helper for that
[Mon Oct 5 18:30:48 2015] <jcohen>: wickman if you want to use it for
rollbacks wouldn't you want it to be fully reified?
[Mon Oct 5 18:30:54 2015] <kts>: but im proposing we have an output that's 0
binding helpers
[Mon Oct 5 18:30:59 2015] <kts>: 2 different json formats
[Mon Oct 5 18:31:04 2015] <wickman>: yes, but you can still do 'config read <
foo.json > foo-compiled.json'
[Mon Oct 5 18:31:10 2015] <kts>: yeah
[Mon Oct 5 18:31:14 2015] <jcohen>: ahh
[Mon Oct 5 18:31:18 2015] <kts>: or config read < foo.aurora >
foo-compiled.json
[Mon Oct 5 18:31:25 2015] <wickman>: yes, either way
[Mon Oct 5 18:31:28 2015] <kts>: +1
[Mon Oct 5 18:31:35 2015] <jcohen>: +1, sounds great to me.
[Mon Oct 5 18:31:37 2015] <wickman>: well it'd be 'config read jobkey <
foo.aurora > foo-compiled.json' technically
[Mon Oct 5 18:32:06 2015] <wickman>: then 'update start' would need to accept
these compiled json blobs
[Mon Oct 5 18:32:12 2015] <wickman>: via --read-json or whatever
[Mon Oct 5 18:32:57 2015] <mkhutornenko>: wickman: itâs not just update
start though, itâs all commands that currently take aurora config, right?
[Mon Oct 5 18:33:16 2015] <wickman>: yes, but for me the important ones are
related to updates
[Mon Oct 5 18:33:38 2015] <kts>: so i'd propose that we auto-detect json and
have --strict flag to disallow --bind and disable binding helpers
[Mon Oct 5 18:34:04 2015] <wickman>: i don't think we should go so far as to
have --strict disable binding helpers
[Mon Oct 5 18:34:12 2015] <kts>: so update start jobkey file.{aurora,json}
allows --bind
[Mon Oct 5 18:34:26 2015] <wickman>: 'jobkey file.json' is redundant though
[Mon Oct 5 18:34:28 2015] <kts>: update start jobkey file.json --strict needs
json input
[Mon Oct 5 18:34:48 2015] <wickman>: perhaps we're getting too much into the
weeds here? i think we can table this for after the meeting.
[Mon Oct 5 18:34:57 2015] <kts>: yeah we don't need to design it here
[Mon Oct 5 18:35:00 2015] <jcohen>: yeah
[Mon Oct 5 18:35:06 2015] <mkhutornenko>: wickman: agree, mind sending this to
dev list?
[Mon Oct 5 18:35:11 2015] <jcohen>: can discuss in jira or in a mailing list
thread
[Mon Oct 5 18:35:14 2015] <wickman>: ok
[Mon Oct 5 18:35:14 2015] <kts>: #action wickman to propose json input format
on dev list
[Mon Oct 5 18:35:33 2015] <jcohen>: any other topics?
[Mon Oct 5 18:35:39 2015] <rdelvalle>: AURORA-1376
## AURORA-1376 ##
[Mon Oct 5 18:36:05 2015] <jcohen>: The floor is yours rdelvalle
[Mon Oct 5 18:36:34 2015] <rdelvalle>: Just real quick, finished switching to
the newest version of Jackson and could use a code review so we can get this
feature in before the next release like Bill wanted to
[Mon Oct 5 18:37:53 2015] <jcohen>: Sounds good. I believe wfarner is back
imminently, so he can hopefully pick the review back up.
[Mon Oct 5 18:38:12 2015] <rdelvalle>: Awesome
[Mon Oct 5 18:38:15 2015] <rdelvalle>: That's all from me
[Mon Oct 5 18:38:25 2015] <jcohen>: thanks rdelvalle
[Mon Oct 5 18:38:33 2015] <jcohen>: Last call for topics...
[Mon Oct 5 18:39:11 2015] <jcohen>: Ok, folks, that'll do it then!
[Mon Oct 5 18:39:34 2015] <kts>: ASFBot: meeting stop
Meeting ended at Mon Oct 5 18:39:34 2015
