I can't do this because all 40 jobs need to be in the same file so they'll share EMR resources. So I need to use composition, not inheritance in order to create a single output file, not 40 output files.
On Monday, January 5, 2015 4:48:29 PM UTC, Brian Coca wrote: > > jinja2 has template inheritance which should work for you for the common > parts > > On Mon, Jan 5, 2015 at 11:33 AM, Mark <[email protected] > <javascript:>> wrote: > > I'm trying to create JSON files that define Amazon data pipeline jobs. > We > > have 40 or so different jobs that all need processing on EMR. There's a > lot > > of shared config between our tasks, with only the input paths and EMR > step > > definition changing per job. However, since a lot of tasks share the > same > > EMR config, I need the ability to reuse the same EMR config for multiple > > tasks, while also having the flexibility to override it per task. > > > > So in the example I first gave, there are 2 jobs using the same default > EMR > > config, and one that will use a custom config. As I say, it's basically > a > > factory pattern, with each job knowing which EMR config it needs, and > the > > whole thing being templated generically. > > > > Here's a sample of the JSON template I'm trying to populate: > > > > {% for definition in definitions %} > > { > > "id": "EmrActivityId_{{ loop.index }}", > > "name": "EmrActivity_{{ definition.suite }}", > > "precondition": { > > "ref": "PreconditionId_{{ loop.index }}" > > }, > > "runsOn": { > > "ref": "EmrClusterId" > > }, > > "type": "EmrActivity", > > "myDate": "{{ date | > default('#{format(minusDays(@scheduledStartTime, > > 1), \'YYYY-MM-dd\')}') }}", > > "step": "{{ {{ definition.step_name }}.step }}" > > }, > > { > > "id": "PreconditionId_{{ loop.index }}", > > "name": "InputExistsPrecondition", > > "s3Prefix": "s3://example-{{ env }}-data{{ > definition.s3_precondition > > }}", > > "type": "S3PrefixNotEmpty" > > }, > > {% endfor %} > > > > > > The rest of the template is identical for all jobs. So I'm really > looking > > for a way to be able to populate the step definition per job so I can > > maximise reuse. As I say, there are around 40 of these particular tasks > I > > need to migrate. > > > > On Monday, January 5, 2015 4:13:59 PM UTC, Brian Coca wrote: > >> > >> sounds like a very complicated process, what are you trying to do in > >> the end? it is normally simpler with ansible, it is rare to need > >> nested variable includes. > >> > >> On Mon, Jan 5, 2015 at 10:09 AM, Mark <[email protected]> wrote: > >> > Thanks for your reply Michael. Can you think of any other way I can > >> > achieve > >> > what I'm trying to? I have tried: > >> > > >> > - templating the vars file before loading it with include_vars, but > >> > jinja > >> > complained about undefined variables > >> > - doing a string replace into the vars file, but then the variables > in > >> > the > >> > default_step.yml file weren't interpolated when it was later loaded > by > >> > ansible > >> > - trying to load the step by name in the json template I'm trying to > >> > create. > >> > I.e. for each product/suite combination I added 'step: default_step', > >> > then > >> > had another vars file containing all of the steps (e..g > >> > steps.default_step, > >> > steps.custom_step) and then in my json template trying "{{ {{ > >> > definition.step }}.step }}", but jinja wouldn't parse that. > >> > > >> > I'm racking my brains but can't think how I can do this. > >> > > >> > Thanks > >> > > >> > On Monday, January 5, 2015 2:14:45 PM UTC, Michael DeHaan wrote: > >> >> > >> >> There is no facility for a variable file including another. > >> >> > >> >> Jinja2 is also not invoked when reading variable files at that time, > so > >> >> include won't help. > >> >> > >> >> > >> >> > >> >> On Mon, Jan 5, 2015 at 6:17 AM, Mark <[email protected]> > wrote: > >> >>> > >> >>> Hi, > >> >>> > >> >>> I'm trying to create some sort of factory pattern I guess in my > vars > >> >>> files for creating Amazon data pipeline jobs. What I want is to > have a > >> >>> list > >> >>> of dicts, and each one will contain a parameter ("step") that I > want > >> >>> to be > >> >>> different per item. However, since this value is quite large & > >> >>> complicated, > >> >>> and will be used for 90% of the items, i don't want to have to > >> >>> duplicate > >> >>> this value 40 or so times. > >> >>> > >> >>> So, is it possible to include a vars file in another one? I tried > >> >>> using > >> >>> jinja2's "include" function, but it complained that certain > variables > >> >>> weren't defined because it was trying to resolve the variables in > the > >> >>> included file. In fact, those variables should be parsed later in > the > >> >>> main > >> >>> play, now when including one vars file into the main one. I can't > use > >> >>> roles > >> >>> because this is part of a larger pattern in which the main vars > file > >> >>> is > >> >>> loaded dynamically depending on another variable. > >> >>> > >> >>> Some examples might make clear what I mean. > >> >>> > >> >>> Here's my playbook, "create-job.yml": > >> >>> > >> >>> - name: "Create a data pipeline and definition for {{ product }} {{ > >> >>> job > >> >>> }}" > >> >>> hosts: localhost > >> >>> gather_facts: True > >> >>> vars_files: > >> >>> - "vars/pipelines/{{ group }}/env/{{ env }}.yml" > >> >>> - "vars/pipelines/{{ group }}/{{ job }}.yml" > >> >>> > >> >>> > >> >>> Here's my vars file, "job1.yml", for the "job1" job: > >> >>> > >> >>> template: multiple-emr > >> >>> startTime: 03:00:00 > >> >>> definitions: > >> >>> - product: web_v2 > >> >>> suite: websuite > >> >>> {% include default_step.yml %} # how to include > >> >>> "default_step.yml"? > >> >>> - product: db2 > >> >>> suite: dbsuite > >> >>> {% include default_step.yml %} > >> >>> - product: custom > >> >>> suite: customsuite > >> >>> {% include custom_step.yml %} > >> >>> ... x 40 > >> >>> > >> >>> > >> >>> And here's the contents of "default_step.yml": > >> >>> > >> >>> s3_precondition: "/raw/data/#{node.myDate}/{{ product }}/" > >> >>> step: > >> >>> - "s3://my-bucket/artifacts/emr-jar-2.1.1.jar" > >> >>> - "com.example.SampleEMR" > >> >>> - "-Dinput=s3n://example-{{ env }}-data{{ s3_precondition | > >> >>> replace('node.', '') }}*{{ suite }}_#{myDate}.*.gz" > >> >>> - "-Doutput=s3n://example-{{ env > >> >>> }}-data/intermediate/data/#{myDate}/{{ > >> >>> product }}/{{ suite }}/", > >> >>> - "-DoutputFormat=json", > >> >>> ... > >> >>> > >> >>> > >> >>> How can I achieve this with ansible? > >> >>> > >> >>> Mark > >> >>> > >> >>> -- > >> >>> You received this message because you are subscribed to the Google > >> >>> Groups > >> >>> "Ansible Project" group. > >> >>> To unsubscribe from this group and stop receiving emails from it, > send > >> >>> an > >> >>> email to [email protected]. > >> >>> To post to this group, send email to [email protected]. > >> >>> To view this discussion on the web visit > >> >>> > >> >>> > https://groups.google.com/d/msgid/ansible-project/cbe7f116-aadf-4d57-94ea-712834cb498a%40googlegroups.com. > > > >> >>> For more options, visit https://groups.google.com/d/optout. > >> >> > >> >> > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Ansible Project" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to [email protected]. > >> > To post to this group, send email to [email protected]. > >> > To view this discussion on the web visit > >> > > >> > > https://groups.google.com/d/msgid/ansible-project/a6197878-8366-4137-8af2-4cbc3b368e07%40googlegroups.com. > > > >> > > >> > For more options, visit https://groups.google.com/d/optout. > >> > >> > >> > >> -- > >> Brian Coca > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Ansible Project" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To post to this group, send email to [email protected] > <javascript:>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/ansible-project/ba8f6a53-67cb-4949-aa2b-029265504770%40googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > Brian Coca > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/c4ac402e-a2e9-47eb-9f19-43974873f0dc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
