Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Zachary Kent updated an issue PuppetDB / PDB-3741 Prioritize catalogs and facts during PDB sync Change By: Zachary Kent Release Notes: Known Issue Add Comment This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Zachary Kent updated an issue PuppetDB / PDB-3741 Prioritize catalogs and facts during PDB sync Change By: Zachary Kent Release Notes Summary: If PuppetDB is out of sync, the two most important datasets to reconcile arecatalogs and facts as catalogs are used for exported resource queries andfacts are used to select groups of nodes for orchestration operations. Priorto this patch we started a sync by transferring reports which can be asignificantly large dataset if two instances are out of sync by several days.This patch re-orders the sync configuration so that entities are synced in thefollowing order:- catalogs- facts- reports- nodesEssentially, catalogs and reports have switched places in the order. Add Comment This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Wyatt Alt updated an issue PuppetDB / PDB-3741 Prioritize catalogs and facts during PDB sync Change By: Wyatt Alt Fix Version/s: 5.1.4 Fix Version/s: PDB 5.2.0 Fix Version/s: PDB 4.2.3.6 Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Wyatt Alt commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync It's in master already. I'll add fix versions and close. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Charlie Sharpsteen commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync Looks like this has been merged into the 4.2.3.x branch. It should get merged up to 5.current and this ticket should get tagged with fix versions. Then we'll be good to close it out. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Wyatt Alt commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync Charlie Sharpsteen it looks to me like you handled this in https://github.com/puppetlabs/pe-puppetdb-extensions/commit/98cdb7876f7e5079c0c511712eb2d484198628a1/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#diff-0 Good to close? Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Charlie Sharpsteen updated an issue PuppetDB / PDB-3741 Prioritize catalogs and facts during PDB sync Change By: Charlie Sharpsteen Affects Version/s: PDB 4.2.3.5 Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Charlie Sharpsteen commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync Good idea. Initial sync should probably just do catalogs and facts since those impact resource collection and querying. Maybe bring over the most recent reports for queries like "re-run task on failed nodes". Beyond that, report history should probably be punted to a background sync because it can take a long time to copy over. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Nick Walker commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync Charlie Sharpsteen agreed, I didn't quite understand the ticket when I first read it. We definitely shouldn't send reports first IMO. I think we should also consider an option that would allow for sync to not send all historical reports and sync a smaller amount so that an initial sync doesn't take as long. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Charlie Sharpsteen commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync PDB-3041 would definitely help. However, the ask here is to copy catalogs and facts over first before we spend a possibly large amount of time copying days and days of report data. Assuming the queue is currently ordered according to insertion, that data would end up ahead of the reports. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Owen Rodabaugh updated an issue PuppetDB / PDB-3741 Prioritize catalogs and facts during PDB sync Change By: Owen Rodabaugh CS Priority: Needs Priority Normal CS Impact: Customers with a lot of data to sync would like the data which impact their catalog compilation to be updated first so that they are using the most current data possible.This could become more important in the future if our HA supported active/active and/or becomes simpler to setup so more customers are using it. CS Severity: 4 - Major CS Business Value: 3 - CS Frequency: 2 - 5-25% of Customers Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Nick Walker commented on PDB-3741 Re: Prioritize catalogs and facts during PDB sync I'd argue that we should implement this generically and not specifically for PDB sync. I think that's in PDB-3041. But essentially you should be able to prioritize certain commands over others and it should probably be catalogs, facts, and then reports. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-3741) Prioritize catalogs and facts during PDB sync
Title: Message Title Charlie Sharpsteen created an issue PuppetDB / PDB-3741 Prioritize catalogs and facts during PDB sync Issue Type: Improvement Affects Versions: PDB 5.1.3 Assignee: Unassigned Created: 2017/11/01 10:01 AM Priority: Normal Reporter: Charlie Sharpsteen Currently, PuppetDB syncs entities in the following order: reports facts catalogs nodes We should consider prioritizing catalogs and facts as catalogs are used in exported resource queries and facts are used for general queries such as targeting nodes for tasks. Additionally, if two PuppetDB nodes become partitioned, there is likely to be a significant amount of report data that needs to be transferred, which means a significant delay in repairing catalogs and facts so that queries are corrected. An updated ordering would be:
