[
https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-19430:
----------------------------------
Fix Version/s: 4.0.0
Attachment: HIVE-19430.01.patch
Target Version/s: 4.0.0
Status: Patch Available (was: In Progress)
deletePersistentAll() invokes
org.datanucleus.store.rdbms.query.ForwardQueryResult.toArray() on the input
collection. ForwardQueryResult is a collection which collects query results in
a lazy manner. So, for a very large result set toArray() method would require
the entire result set to be materialized consumed a lot of memory.
There are three possible solutions
# Get small number of events to be cleaned at a time from the PM and delete
only those at a time.
# If we do not call toArray() on the entire collection, the result set
wouldn't get materialized in one shot. We can avoid huge memory consumption by
controlling the number of objects (or rather the total memory consumed by the
objects) we materialize and pass to ec.deleteObjects().
# we could instead use ec.deleteObject() by iterating over objects in the
collection but that would not be as performant as deleting all objects in one
go.
The third wouldn't be useful since it would put more load on PM.
The cleaner thread, which cleans the events, is a low priority task and thus is
not expected to put a lot of load on the system and shouldn't cause performance
drop in the foreground work. Deleting small number of events at a time, does
not put a lot of load on the PM at a time and yet with the default
configuration we can delete 600K events per hour which should be more than the
rate at which events are produced. That's the first solution. The second
solution cleans the events more aggressively but would put a lot of load PM (to
the extent that it may become unresponsive momentarily) when there are a lot of
events to be cleaned in a go. The first solution handles spikes gracefully by
distributing the load over time and yet would catch up with the events in
reasonable amount of time.
The patch, originally written by [~thejas] uses the first solution.
> ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending
> events
> ---------------------------------------------------------------------------------
>
> Key: HIVE-19430
> URL: https://issues.apache.org/jira/browse/HIVE-19430
> Project: Hive
> Issue Type: Bug
> Reporter: Thejas M Nair
> Assignee: Ashutosh Bapat
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19430.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If there are large number of events that haven't been cleaned up for some
> reason, then ObjectStore.cleanNotificationEvents() can run out of memory
> while it loads all the events to be deleted.
> It should fetch events in batches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
