[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19430: Affects Version/s: 3.0.0 > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch, > HIVE-19430.03.patch, HIVE-19430.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19430: Component/s: repl HCatalog > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 3.0.0 >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch, > HIVE-19430.03.patch, HIVE-19430.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19430: Resolution: Fixed Status: Resolved (was: Patch Available) +1 04.patch committed to master. Thanks [~ashutosh.bapat] and [~vihangk1]! > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch, > HIVE-19430.03.patch, HIVE-19430.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Attachment: HIVE-19430.04.patch Status: Patch Available (was: In Progress) Patch addresses [~vihangk1]'s comment. > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch, > HIVE-19430.03.patch, HIVE-19430.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Status: In Progress (was: Patch Available) > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch, > HIVE-19430.03.patch, HIVE-19430.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Status: In Progress (was: Patch Available) > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Attachment: HIVE-19430.03.patch Status: Patch Available (was: In Progress) Attaching the .02 patch again to trigger ptest. The testcases that failed in the last ptest run are passing for me locally. > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch, > HIVE-19430.03.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Status: Open (was: Patch Available) > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Attachment: HIVE-19430.02.patch Status: Patch Available (was: Open) > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch, HIVE-19430.02.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Fix Version/s: 4.0.0 Attachment: HIVE-19430.01.patch Target Version/s: 4.0.0 Status: Patch Available (was: In Progress) deletePersistentAll() invokes org.datanucleus.store.rdbms.query.ForwardQueryResult.toArray() on the input collection. ForwardQueryResult is a collection which collects query results in a lazy manner. So, for a very large result set toArray() method would require the entire result set to be materialized consumed a lot of memory. There are three possible solutions # Get small number of events to be cleaned at a time from the PM and delete only those at a time. # If we do not call toArray() on the entire collection, the result set wouldn't get materialized in one shot. We can avoid huge memory consumption by controlling the number of objects (or rather the total memory consumed by the objects) we materialize and pass to ec.deleteObjects(). # we could instead use ec.deleteObject() by iterating over objects in the collection but that would not be as performant as deleting all objects in one go. The third wouldn't be useful since it would put more load on PM. The cleaner thread, which cleans the events, is a low priority task and thus is not expected to put a lot of load on the system and shouldn't cause performance drop in the foreground work. Deleting small number of events at a time, does not put a lot of load on the PM at a time and yet with the default configuration we can delete 600K events per hour which should be more than the rate at which events are produced. That's the first solution. The second solution cleans the events more aggressively but would put a lot of load PM (to the extent that it may become unresponsive momentarily) when there are a lot of events to be cleaned in a go. The first solution handles spikes gracefully by distributing the load over time and yet would catch up with the events in reasonable amount of time. The patch, originally written by [~thejas] uses the first solution. > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19430.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Attachment: (was: HIVE-19430.01.patch) > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-19430: -- Attachment: HIVE-19430.01.patch > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-19430: -- Labels: pull-request-available (was: ) > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
