Jira (PDB-3254) Test performance after undeduplicating fact_values

Rob Browning (JIRA) Thu, 30 Mar 2017 09:12:15 -0700

Title: Message Title

Rob Browning commented on

PDB-3254

Two primary tests were run on pdb-n2-prod.ops.puppetlabs.net using
postgres with a postgresql.conf adjusted by pgtune.

In all cases, "dedup" refers to pdb before the reduplication changes
and "redup" refers to pdb after. The hashes for each source tree were

dedup: 2f7e197961815b7e7de643e4e6d82c39a1ddcd3a
redup: 01c7c0a5c70160f8bd2e7948899f9b50b277d5fc

The first test specifically addressed two of the corner cases that
prompted the original epic.

#!/usr/bin/env python

# -*- coding: utf-8 -*-

from pdb_utils import submit_command

from datetime import datetime

from random import random

import timeit

def generate_dynamic_fact(n=2000):

    fact = {}

    for i in range(n):

        device = "sda" + str(i)

        fact[device] = {"size": str(random()),

                        "size_bytes": random()}

    return {"simulated_mountpoints": fact}

def facts_command(facts):

    return {"certname": "foo.com",

            "environment": "test",

            "values": facts,

            "producer_timestamp": datetime.now().isoformat()}

def test_dynamic_blob():

    for i in range(50):

        submit_command("foo.com", "replace_facts", 4,

                       facts_command(generate_dynamic_fact()))

def generate_dynamic_array(n=2000):

    array = []

    for i in range(n):

        array.append({"size": str(random()),

                      "size_bytes": random()})

    return {"simulated_array": array}

def test_dynamic_array():

    array = generate_dynamic_array()

    for i in range(100):

        submit_command("foo.com", "replace_facts", 4,

                       facts_command(array))

        array["simulated_array"].insert(0, {"size": str(random()),

                                            "size_bytes": random()})

if __name__ == '__main__':

    print("dynamic blobs:", timeit.timeit(test_dynamic_blob, number=1))

    print("dynamic arrays:", timeit.timeit(test_dynamic_array, number=1))

The results for the dedup case:

('dynamic blobs:', 158.71874594688416)
('dynamic arrays:', 557.062803030014)
2.57user 0.35system 11:55.95elapsed 0%CPU (0avgtext+0avgdata 16100maxresident)k
0inputs+0outputs (0major+118401minor)pagefaults 0swaps

and for the redup case:

('dynamic blobs:', 37.07982683181763)
('dynamic arrays:', 66.71684288978577)
2.53user 0.22system 1:44.00elapsed 2%CPU (0avgtext+0avgdata 16100maxresident)k
0inputs+0outputs (0major+35919minor)pagefaults 0swaps

For the second test we generated a stockpile queue of 700k replace
facts commands using benchmark. I believe the generator was:

  lein trampoline run benchmark -c ... \

     -r 10 -i 1 -n 20000 \

     -F resources/puppetlabs/puppetdb/benchmark/samples/facts

Then pdb was started and the global.processing-time .OneMinuteRate was
sampled every 2 seconds until the queue emptied.

The resulting data should be attached to this ticket as
one-min-dedup.log.gz and one-min-redup.log.gz. Each line in those
files is in this format:

UTC_TIMESTAMP ONE_MINUTE_RATE

except that some initial lines in each set may not have a rate (as pdb started up).

Add Comment

This message was sent by Atlassian JIRA

--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.

Jira (PDB-3254) Test performance after undeduplicating fact_values

Reply via email to