Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Austin Blatt updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Austin Blatt Fix Version/s: PDB 6.16.2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.59886.1623964020026%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Zachary Kent updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Zachary Kent Fix Version/s: PDB 7.4.0 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.38852.1621626960365%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Sprint: ghost-24.03.2021, ghost-7.04.2021, ghost-21.04.2021, Ghost-5.05.2021, ghost-19.05.2021 , HAHA/Grooming 2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.25699.1620224940148%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Sprint: ghost-24.03.2021, ghost-7.04.2021, ghost-21.04.2021, Ghost-5.05.2021 , HAHA/Grooming 2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.25643.162040109%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Sprint: ghost-24.03.2021, ghost-7.04.2021, ghost-21.04.2021 , HAHA/Grooming 2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.15092.1618989960100%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Sebastian Miclea updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Sebastian Miclea Story Points: 3 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.4269.1617885900060%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Sebastian Miclea updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Sebastian Miclea Story Points: 3 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.4267.1617885780328%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Sprint: ghost-24.03.2021, ghost-7.04.2021 , HAHA/Grooming 2 Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.3123.1617800640120%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Sebastian Miclea commented on PDB-5053 Re: Move sync expired resource_event filter to report ingest On a more detailed look, when implementing the filter for resource_events we've noticed that there are a lot of tests that use hard coded dates (example [here|puppetdb/reports.clj at PDB-5053 · sebastian-miclea/puppetdb (github.com)]). This is quite a big problem because at the moment all resource-events are expired and we have about 600 tests failing for similar reasons. We need to find a way to have the timestamps dynamically created so they would be relevant to the tests. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.35.1617353940031%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Sprint: ghost-24.03.2021 , ready for triage 3 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.172786.1616594940233%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Sebastian Miclea assigned an issue to Sebastian Miclea PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Sebastian Miclea Assignee: Sebastian Miclea Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.170341.1616157720037%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie commented on PDB-5053 Re: Move sync expired resource_event filter to report ingest Zachary Kent I split this ticket and moved the sync work to PDB-5058 while this ticket retain the work that has to been done when saving reports. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.163320.1615457520092%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Acceptance Criteria: * sync no longer alters the reports it pulls from the other side * report ingestion accounts for the resource-event-ttl and doesn't separately store resource-events which would be expired locally. The expired events may still live in the main report body * Sync and unit test which check two points checks the point above Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.163315.1615457340096%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie pdbext sync has a filter in the [:clean-up-record-fn|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#L173-L179] for reports which removes any *resource_events* which would be expired locally. This filter was added to account for differing ttls between *reports* and *resource_events*. We wanted to avoid a situation where GC would clean up a *resource_event* partition and then sync would pull a report with *resource_events* that recreated the deleted partition. As a result if an *event* is pulled out of a *resource* in a *report* we could have reports which don't exactly match between two pdbs syncing with one another. This problem could become more pronounced when we add the ability to disable *resource_event* storage in PDB-3653. If the *resource-event-ttl* is set to 0 the sync filter will strip out all *resource_events* in the reports it transfers. It would be better if the check for expired *resource_events* was moved into the *report* ingestion code. That way sync would keep the report identical on both sides.- To do this we would need to remove the filter in the [:clean-up-record-fn|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#L173-L179] for reports and adjust at least the [dont-pull-events-that-would-be-expired-locally|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/test/puppetlabs/pe_puppetdb_extensions/sync/end_to_end_test.clj#L939] test to check that the *resource-events* partitions aren't created not that the events don't exist in the report body. - (this has been moved to PDB-5058) On the FOSS side we'll want to add a check based on the *resource-event-ttl* in the *scf/storage.clj/add-report! \ ** function in the section where [resource_events|https://github.com/puppetlabs/puppetdb/blob/6.x/src/puppetlabs/puppetdb/scf/storage.clj#L1415] are stored. If we add a filter there which discards any *resource_events* which would be expired based on their *timestamp* that should allow us to avoid the issue of sync creating expired partitions and allow us to keep reports the same on both sides of sync.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Sprint: ghost-24.03.2021 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.162081.1615388040124%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Bogdan Irimie updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Bogdan Irimie Story Points: 3 Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.162078.1615387980092%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Zachary Kent commented on PDB-5053 Re: Move sync expired resource_event filter to report ingest As part of this work we'll want to make sure that the resource->skipped-resource-events function can't create resource_events which are older than the resource-event-ttl. This might happen naturally with work described above but we'll want to make sure to double check. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.158028.1614898260027%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Zachary Kent updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Zachary Kent Labels: tsr-pdb-backlog Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.389887.1614814159000.156929.1614814860125%40Atlassian.JIRA.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Zachary Kent updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Zachary Kent pdbext sync has a filter in the [:clean-up-record-fn|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#L173-L179] for reports which removes any *resource_events* which would be expired locally. This filter was added to account for differing ttls between *reports* and *resource_events*. We wanted to avoid a situation where GC would clean up a *resource_event* partition and then sync would pull a report with *resource_events* that recreated the deleted partition. As a result if an *event* is pulled out of a *resource* in a *report* we could have reports which don't exactly match between two pdbs syncing with one another. This problem could become more pronounced when we add the ability to disable *resource_event* storage in PDB-3653. If the *resource-event-ttl* is set to 0 the sync filter will strip out all * resource resource_events * *events* in the reports it transfers. It would be better if the check for expired *resource_events* was moved into the *report* ingestion code. That way sync would keep the report identical on both sides. To do this we would need to remove the filter in the [:clean-up-record-fn|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#L173-L179] for reports and adjust at least the [dont-pull-events-that-would-be-expired-locally|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/test/puppetlabs/pe_puppetdb_extensions/sync/end_to_end_test.clj#L939] test to check that the *resource-events* partitions aren't created not that the events don't exist in the report body. On the FOSS side we'll want to add a check based on the *resource-event-ttl* in the *scf/storage.clj/add-report!\** function in the section where [resource_events|https://github.com/puppetlabs/puppetdb/blob/6.x/src/puppetlabs/puppetdb/scf/storage.clj#L1415] are stored. If we add a filter there which discards any *resource_events* which would be expired based on their *timestamp* that should allow us to avoid the issue of sync creating expired partitions and allow us to keep reports the same on both sides of sync.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Zachary Kent updated an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Change By: Zachary Kent pdbext sync has a filter in the [:clean-up-record-fn|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#L173-L179] for reports which removes any *resource_events* which would be expired locally. This filter was added to account for differing ttls between *reports* and *resource_events*. We wanted to avoid a situation where GC would clean up a *resource_event* partition and then sync would pull a report with *resource_events* that recreated the deleted partition. As a result if an *event* is pulled out of a *resource* in a *report* we could have reports which don't exactly match between two pdbs syncing with one another. This problem could become more pronounced when we add the ability to disable *resource_event* storage in PDB-3653. If the *resource-event-ttl* is set to 0 the sync filter will strip out all * resource_events resource * *events* in the reports it transfers. It would be better if the check for expired *resource_events* was moved into the *report* ingestion code. That way sync would keep the report identical on both sides. To do this we would need to remove the filter in the [:clean-up-record-fn|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/src/puppetlabs/pe_puppetdb_extensions/sync/core.clj#L173-L179] for reports and adjust at least the [dont-pull-events-that-would-be-expired-locally|https://github.com/puppetlabs/pe-puppetdb-extensions/blob/6.x/test/puppetlabs/pe_puppetdb_extensions/sync/end_to_end_test.clj#L939] test to check that the *resource-events* partitions aren't created not that the events don't exist in the report body. On the FOSS side we'll want to add a check based on the *resource-event-ttl* in the *scf/storage.clj/add-report!\** function in the section where [resource_events|https://github.com/puppetlabs/puppetdb/blob/6.x/src/puppetlabs/puppetdb/scf/storage.clj#L1415] are stored. If we add a filter there which discards any *resource_events* which would be expired based on their *timestamp* that should allow us to avoid the issue of sync creating expired partitions and allow us to keep reports the same on both sides of sync.
Jira (PDB-5053) Move sync expired resource_event filter to report ingest
Title: Message Title Zachary Kent created an issue PuppetDB / PDB-5053 Move sync expired resource_event filter to report ingest Issue Type: Bug Assignee: Unassigned Created: 2021/03/03 3:29 PM Priority: Normal Reporter: Zachary Kent pdbext sync has a filter in the :clean-up-record-fn for reports which removes any resource_events which would be expired locally. This filter was added to account for differing ttls between reports and resource_events. We wanted to avoid a situation where GC would clean up a resource_event partition and then sync would pull a report with resource_events that recreated the deleted partition. As a result if an event is pulled out of a resource in a report we could have reports which don't exactly match between two pdbs syncing with one another. This problem could become more pronounced when we add the ability to disable resource_event storage in PDB-3653. If the resource-event-ttl is set to 0 the sync filter will strip out all resource_events in the reports it transfers. It would be better if the check for expired resource_events was moved into the report ingestion code. That way sync would keep the report identical on both sides. To do this we would need to remove the filter in the :clean-up-record-fn for reports and adjust at least the dont-pull-events-that-would-be-expired-locally test to check that the resource-events partitions aren't created not that the events don't exist in the report body. On the FOSS side we'll want to add a check based on the resource-event-ttl in the scf/storage.clj/add-report!* function in the section where resource_events are stored. If we add a filter there which discards any resource_events which would be expired based on their timestamp that should allow us to avoid the issue of sync creating expired partitions and allow us to keep reports the same on both sides of sync.
