OK, after some experimentation, I think I see what your problem might be?
If you do something like:
BOX_IMAGE = "fedora/37-cloud-base"
NODE_COUNT = 2
Vagrant.configure("2") do |config|
(1..NODE_COUNT).each do |i|
config.vm.define "node#{i}" do |subconfig|
subconfig.vm.box = BOX_IMAGE
subconfig.vm.hostname = "node#{i}"
if i == NODE_COUNT
config.vm.provision :ansible do |ansible|
# Disable default limit to connect to all the machines
ansible.limit = "all"
ansible.playbook = "playbook.yml"
end
end
end
end
end
The Vagrant Ansible provisioner fires for every VM causing multiple
discrete runs, you can control that to a degree with ansible.limit, the
hosts statement in the playbook and/or delegate_to but it would be hard to
do stateful cross-cluster config.
If you do something like the following instead, this will provision all 3
Vagrant boxes and then fire the provisioner* once *triggering an Ansible
run just for the final box:
wmcdonald@fedora:~/working/vagrant/fedora-multi$ cat Vagrantfile
Vagrant.configure(2) do |config|
#Define the number of nodes to spin up
N = 3
#Iterate over nodes
(1..N).each do |node_id|
nid = (node_id - 1)
config.vm.define "node#{nid}" do |node|
node.vm.box = "fedora/37-cloud-base"
node.vm.provider "virtualbox" do |vb|
vb.memory = "1024"
end
node.vm.hostname = "node#{nid}"
if node_id == N
node.vm.provision "ansible" do |ansible|
ansible.limit = "all"
ansible.groups = {
"cluster-nodes" => [
"node0",
"node1",
"node2",
]
}
ansible.playbook = "playbook.yml"
end
end
end
end
end
wmcdonald@fedora:~/working/vagrant/fedora-multi130$ cat playbook.yml
- name: Vagrant post-provision
hosts: cluster_nodes
tasks:
- name: Debug vars for hosts
debug:
var: ansible_play_hosts
Note that the provisioner will run once but still parallelise like a normal
Ansible run would and hit each node because we're setting the hosts to the
group members. You could further limit with delegate_to or have one cluster
node in its own 'primary_node' group in addition to the cluster_nodes.
See:
https://everythingshouldbevirtual.com/automation/virtualization/vagrant-ansible-provisioning-multi-nodes/
And another variant with per-box behaviour here:
https://stackoverflow.com/questions/54468546/how-to-run-an-ansible-playbook-on-a-specific-vagrant-host
On Sun, 26 Nov 2023 at 00:22, Will McDonald <[email protected]> wrote:
> There are a couple of ways you could exercise "control over the process to
> pull data from host 1 to be used on host 2 and 3".
>
> If you look at
> https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-clever-way
> 3 nodes are provisioned, one as primary, then two as secondary nodes and
> it'd be relatively trivial to use this to key off the 'primary' node to do
> what you needed, I imagine.
>
> Where I've had scenarios provisioning 3 nodes of something in a 2n+1
> cluster (basically anything like Mongo, Etcd, Zookeeper etc. etc.) and you
> need to at least temporarily choose a semi-deterministic primary I've used
> logic like:
>
> pre_tasks:
> - name: pre_tasks | cluster member role setup for multiple hosts
> block:
> - name: pre_tasks | set cluster role to primary when inventory_hostame
> matches random seed
> set_fact:
> cluster_role: primary
> when: inventory_hostname ==
> ansible_play_hosts|random(seed=ansible_play_hosts | join())
>
> - name: pre_tasks | set mongo replication role to secondary when
> inventory_hostame does not match random seed
> set_fact:
> cluster_role: secondary
> when: inventory_hostname !=
> ansible_play_hosts|random(seed=ansible_play_hosts | join())
>
> - name: pre_tasks | create a custom facts.d directory on the target
> host
> file:
> state: directory
> recurse: true
> path: /etc/ansible/facts.d
>
> - name: pre_tasks | persist the cluster membership role as a custom
> fact
> copy:
> content: |
> {'cluster_role':'{{ cluster_role }}'}
> dest: /etc/ansible/facts.d/cluster.fact
> mode: 0644
> owner: root
> group: root
>
> *Warning! *This sets a *transient value* in facts.d. Which in my cases is
> fine for our purposes. If your cluster membership state changes post-setup,
> the fact would be misleading. (i.e. a node flaps and another cluster member
> assumes leader/primary.)
>
> You would want to replace cluster.fact with something that dynamically
> pulls out the cluster role membership state of a node once the
> cluster/replicaset/whatever topology is provisioned and configured.
>
>
> On Sat, 25 Nov 2023 at 23:25, Evan Hisey <[email protected]> wrote:
>
>> Definitely an edge case. Not an issue in my file atleast as written based
>> on my understanding of the process, but possibly an issue in my
>> understanding of how vagrant is executing ansible as it looks like vagrant
>> runs on each vm as a separate job in either case, just in parallel on each
>> the second time. I still need control over the process to pull data from
>> host 1 to be used on host 2 and 3, which if it is running in parallel as
>> multiple jobs would still be an issue. If it in fact runs a single ansible
>> playbook across the inventory, then that could work, and be the opposite of
>> how I am understanding vagrant ansible provider works. I would need to
>> refactor a large chunk of the application code to support that, but that
>> can be easily done.
>>
>> On Sat, Nov 25, 2023 at 4:44 PM Will McDonald <[email protected]>
>> wrote:
>>
>>> I think you may be misunderstanding me, or I'm misunderstanding you.
>>>
>>> Just for clarity's sake, the flow you would like is:
>>>
>>> 1. An Ansible control node runs a playbook (or role) on
>>> <controlnode> targeting a machine, <targetmachine>
>>> 2. The <targetmachine> is configured to run as a Vagrant host with a
>>> virtualisation provider (Virtualbox, Libvirt or whatever) in order to
>>> support Vagrant box creation
>>> 3. You then have a Vagrantfile which runs on <targetmachine> and
>>> configures multiple Vagrant boxes <vb1>, <vb2>, <vb3>
>>> 4. Once <vb1>, <vb2>, <vb3> are UP* and only then,* you want to run
>>> some Ansible which needs the primary and 2 secondaries to be up
>>>
>>> That being the case, then that is the behaviour that
>>> https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#ansible-parallel-execution
>>> describes. It's slightly poorly worded but to me:
>>>
>>> # Only execute once the Ansible provisioner, # when all the machines
>>> are up and ready.
>>>
>>> Is equivalent to:
>>>
>>> # Provision all Vagrant boxes in the multi-machine setup.
>>> # Only once all the machines are up and ready, run the Ansible
>>> provisioner
>>>
>>> If that's not what's happening, that's likely a Vagrant configuration or
>>> provisioner misbehaviour?
>>>
>>> That's why I'm saying this isn't necessarily an Ansible thing. That
>>> wording, the boxes should all spin up before any Vagrant Ansible
>>> provisioner runs, you're saying that's not the case. That sounds like
>>> either your Vagrantfile is wrong, or your Vagrant VM provisioner or
>>> something else isn't working as expected.
>>>
>>> I'm spinning this up on a test but if you already have a test
>>> case/reproducer, or can provide more info on your Vagrant setup then this
>>> would collectively help people help you. If there's an obvious error in
>>> your Vagrantfile it could be a simple fix rather than an edge case.
>>>
>>> cf:
>>> -
>>> https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-clever-way
>>> -
>>> https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#ansible-parallel-execution
>>>
>>>
>>> On Sat, 25 Nov 2023 at 21:55, Evan Hisey <[email protected]> wrote:
>>>
>>>> Vagrant is behaving fine, so not a vagrant specific problem. It is a
>>>> task problem. I need the vagrant hosts fully installed first because I have
>>>> to collect data from all 3 at once before deploying the software, and
>>>> during software deployment I have to install the master first, collect keys
>>>> and then install the slaves. Vagrant provider provisions does provide this
>>>> kind of control as it assumes the each provisioned VM is self contained. A
>>>> more typical solution would be to directly remote in to the VM's for
>>>> ansible to run after deployment from the remote controller, but that is not
>>>> an available option. Only the vagrant host will have access to the vagrant
>>>> vms, and really only as the vagrant user. The last limitation is not hard
>>>> to deal with, as vagrant provides everything an ansible job would need if
>>>> run from the vagrant host.
>>>>
>>>> That is why I need to trigger to a vagrant host ansible playbook, since
>>>> it can't not run from the initial ansible controller. Yes it is a bit of an
>>>> odd edge case, as the vagrant provider normally would be plenty.
>>>>
>>>> On Sat, Nov 25, 2023 at 2:08 PM Will McDonald <[email protected]>
>>>> wrote:
>>>>
>>>>> It sounds like a Vagrant issue rather than an Ansible issue. Or
>>>>> possibly a niche Vagrant provider problem.
>>>>>
>>>>> Can you share a sample Vagrantfile that's not behaving as it should
>>>>> and details of the target OS of the Vagrant host, and the virtualisation
>>>>> provider you're using?
>>>>>
>>>>>
>>>>> On Sat, 25 Nov 2023 at 19:30, Evan Hisey <[email protected]> wrote:
>>>>>
>>>>>> Already tried it and it does not work, which was why I explicitly
>>>>>> referenced that behavior as not working as not working in this
>>>>>> scenario.While vagrant can run playbooks at provisioning time. it does
>>>>>> not
>>>>>> really proivde a way to to control when the provisioin runs. All 3 hosts
>>>>>> need to be up be for the first host can be provisioned since it requires
>>>>>> the ips of the later hosts. Second option does not work, as the remote
>>>>>> control node does not have access to the VMs, as mentioned. Which is what
>>>>>> lead to the need to trigger a second playbook. otherwise could lust load
>>>>>> the vagrant generated inventory with add_host module.
>>>>>>
>>>>>> IC ould do some ugly sequencing of the "vagrant up --provision" from
>>>>>> a playbook to control the ansible provisioning sequence of the vms, but I
>>>>>> am trying to avoid using ugly shell commands as much as I can. If I
>>>>>> uses a
>>>>>> shell command I could also just trigger an ansible playbook that way,
>>>>>> but
>>>>>> feels wrong.
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 25, 2023 at 12:40 PM Will McDonald <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Quickly skimming the Vagrant Ansible provisioner docs, isn't this
>>>>>>> precisely the behaviour you're looking for:
>>>>>>>
>>>>>>>
>>>>>>> https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#ansible-parallel-execution
>>>>>>>
>>>>>>> # Only execute once the Ansible provisioner, # when all the
>>>>>>> machines are up and ready.
>>>>>>>
>>>>>>> So you would spin up all your Vagrant boxes from your control node,
>>>>>>> wait for that to complete, template out a static inventory of your
>>>>>>> Vagrant
>>>>>>> boxes then run your subsequent Vagrant Ansible provisioner automation?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, 25 Nov 2023 at 18:20, Evan Hisey <[email protected]> wrote:
>>>>>>>
>>>>>>>> I am working on a scenario where the first playbook executes
>>>>>>>> commands on a remote host to create a vagrant host and spins up
>>>>>>>> multiple
>>>>>>>> vms. Vagrant can triggers it's own ansible provisioning runs but they
>>>>>>>> are
>>>>>>>> only single host aware and run when the host is provisioned. That does
>>>>>>>> not
>>>>>>>> work in this case, as I need all VM's running BEFORE the deployment
>>>>>>>> playbook can be triggered. Added wrinkle is the VMs are accessible at
>>>>>>>> this
>>>>>>>> time from outside the vagrant host. If they were, I could simply
>>>>>>>> import the
>>>>>>>> vagrant host list into the controller inventory and refresh.
>>>>>>>>
>>>>>>>> Right now I am looking at the possibility of using
>>>>>>>> ansible.builtin.shell to trigger the new ansible-playbook command on
>>>>>>>> the
>>>>>>>> vagrant host to run the vagrant VM application configuration. But while
>>>>>>>> this works it is not exactly ansible clean. Suggestions on approaches?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Evan Hisey
>>>>>>>> [email protected]
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Ansible Project" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYwsmYmvh%3DWJwSNmJWertkxFRDiKkumnwhzAFupggP58Vg%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYwsmYmvh%3DWJwSNmJWertkxFRDiKkumnwhzAFupggP58Vg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Ansible Project" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/ansible-project/CAKtKohS2LdL4qtTyQF8VdV6pu2tjL3mx868TgvkwvAEUkRbF%3Dg%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAKtKohS2LdL4qtTyQF8VdV6pu2tjL3mx868TgvkwvAEUkRbF%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Ansible Project" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYxdk%3DfA13__d1wEJTiTkpDnaR2eqEpGm%2BXy3v3L1M%3DJ9A%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYxdk%3DfA13__d1wEJTiTkpDnaR2eqEpGm%2BXy3v3L1M%3DJ9A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Ansible Project" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/ansible-project/CAKtKohQ8PP4n7h-ERfE4iONYxZK-nZRLaoMwvRATzSj1hnD%3Dbw%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/ansible-project/CAKtKohQ8PP4n7h-ERfE4iONYxZK-nZRLaoMwvRATzSj1hnD%3Dbw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Ansible Project" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYwxGLhF275kMbBaMMUx6FNQoduaxO7wa_j9AVkUw5ZQVg%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYwxGLhF275kMbBaMMUx6FNQoduaxO7wa_j9AVkUw5ZQVg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Ansible Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/ansible-project/CAKtKohS9rcVZ0rOe_MH9bdb2EgFL7jeBmqwLgyRbZMG2NjmrmQ%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/ansible-project/CAKtKohS9rcVZ0rOe_MH9bdb2EgFL7jeBmqwLgyRbZMG2NjmrmQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Ansible Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYzr7mYhJJimRKsyoV_ikYQoxjHYgOs-pABj4WEPUfoMbg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYzr7mYhJJimRKsyoV_ikYQoxjHYgOs-pABj4WEPUfoMbg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAKtKohSGMsrKNf9rAqafEk3UoLf9AdWK9G491JB87gaAUhUEmg%40mail.gmail.com.
