On Fri, Sep 15, 2017 at 8:56 PM, 'Konstantin Orekhov' via Foreman users <
[email protected]> wrote:
>
>
>> what kind of load do you have? Puppet? Facter? Is that ENC? Something
>> else?
>>
>> Can you tell which requests are slow from logs or monitoring?
>>
>>
> Yes, I should have mentioned that - there's very little puppet and ENC
> work done by this cluster at this point (more is coming soon though). Host
> discovery is by far the largest workload - 7600 discovered systems at this
> point. The last spike that I saw the impact to overall flows was when
> 300-400 systems were trying to register at the same time. Because of the
> deadlocks, about 200-300 systems could not register repeatedly and had to
> keep retrying for a rather long time.
> Rather often these registration attempts would end up creating either
> duplicate entries with the same "mac<mac>" but different IDs in a DB or an
> "empty" discovery host entry. Both of these would prevent a system
> successfully register unless they are removed (I had to write a little
> script that runs from the cron to do so). Here are the examples of an
> "empty" record and duplicate ones (as they get deleted):
>
>
Lukas - how about we change discovery to be async? e.g. import all new
discovered systems into active job and than process than one / multiple at
a time? I assume this would require a image change too (so it knows when
the discovery"job" is done)
> {
> "id": 437923,
> "name": "mac90e2bae6cc70",
> "last_compile": null,
> "last_report": null,
> "updated_at": "2017-08-22T07:08:54.000Z",
> "created_at": "2017-08-22T07:08:54.000Z",
> "root_pass": "<removed>",
> "architecture_id": null,
> "operatingsystem_id": null,
> "environment_id": null,
> "ptable_id": null,
> "medium_id": null,
> "build": false,
> "comment": null,
> "disk": null,
> "installed_at": null,
> "model_id": null,
> "hostgroup_id": null,
> "owner_id": null,
> "owner_type": null,
> "enabled": true,
> "puppet_ca_proxy_id": null,
> "managed": false,
> "use_image": null,
> "image_file": null,
> "uuid": null,
> "compute_resource_id": null,
> "puppet_proxy_id": null,
> "certname": null,
> "image_id": null,
> "organization_id": null,
> "location_id": null,
> "otp": null,
> "realm_id": null,
> "compute_profile_id": null,
> "provision_method": null,
> "grub_pass": "",
> "global_status": 0,
> "lookup_value_matcher": null,
> "discovery_rule_id": null,
> "salt_proxy_id": null,
> "salt_environment_id": null,
> "pxe_loader": null
> }
>
> Duplicates (usually the later duplicate would be an empty one as well, but
> not all the time):
>
> {
> "id": 430090,
> "name": "mac3417ebe3f8f1",
> "last_compile": null,
> "last_report": "2017-09-14T19:47:55.000Z",
> "updated_at": "2017-09-14T19:47:57.000Z",
> "created_at": "2017-03-08T20:24:05.000Z",
> "root_pass": "<removed>",
> "architecture_id": null,
> "operatingsystem_id": null,
> "environment_id": null,
> "ptable_id": null,
> "medium_id": null,
> "build": false,
> "comment": null,
> "disk": null,
> "installed_at": null,
> "model_id": 3,
> "hostgroup_id": null,
> "owner_id": 10,
> "owner_type": "User",
> "enabled": true,
> "puppet_ca_proxy_id": null,
> "managed": false,
> "use_image": null,
> "image_file": null,
> "uuid": null,
> "compute_resource_id": null,
> "puppet_proxy_id": null,
> "certname": null,
> "image_id": null,
> "organization_id": null,
> "location_id": null,
> "otp": null,
> "realm_id": null,
> "compute_profile_id": null,
> "provision_method": null,
> "grub_pass": "",
> "global_status": 0,
> "lookup_value_matcher": null,
> "discovery_rule_id": null,
> "salt_proxy_id": null,
> "salt_environment_id": null,
> "pxe_loader": null
> }
> {
> "id": 438146,
> "name": "mac3417ebe3f8f1",
> "last_compile": null,
> "last_report": "2017-09-11T08:58:05.000Z",
> "updated_at": "2017-09-11T08:58:07.000Z",
> "created_at": "2017-08-24T19:44:23.000Z",
> "root_pass": "<removed>",
> "architecture_id": null,
> "operatingsystem_id": null,
> "environment_id": null,
> "ptable_id": null,
> "medium_id": null,
> "build": false,
> "comment": null,
> "disk": null,
> "installed_at": null,
> "model_id": null,
> "hostgroup_id": null,
> "owner_id": null,
> "owner_type": null,
> "enabled": true,
> "puppet_ca_proxy_id": null,
> "managed": false,
> "use_image": null,
> "image_file": null,
> "uuid": null,
> "compute_resource_id": null,
> "puppet_proxy_id": null,
> "certname": null,
> "image_id": null,
> "organization_id": null,
> "location_id": null,
> "otp": null,
> "realm_id": null,
> "compute_profile_id": null,
> "provision_method": null,
> "grub_pass": "",
> "global_status": 0,
> "lookup_value_matcher": null,
> "discovery_rule_id": null,
> "salt_proxy_id": null,
> "salt_environment_id": null,
> "pxe_loader": null
> }
>
> I can't tell if any queries are slow - can you remind me how to do that?
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
