Would something like this work (see below)? I normally don't like
hard-coded numbers, but the batch size is really not that crucial. I'm
not well enough versed in Hibernate to know if the query can be reused,
but again, probably not a big deal.
----------
public int purgeEventSessionsBefore(DateTime lastAggregatedEventDate) {
int resultSize;
int totalSize = 0;
final int batchSize = 1000;
TypedQuery<EventSessionImpl> query;
List<EventSessionImpl> resultList;
do {
query = this.createQuery(this.findExpiredEventSessionsQuery);
query.setMaxResults(batchSize);
query.setParameter(this.dateTimeParameter, lastAggregatedEventDate);
resultList = query.getResultList();
resultSize = resultList.size();
for (final EventSessionImpl eventSession : resultList) {
this.getEntityManager().remove(eventSession);
}
totalSize += resultSize;
} while (resultSize == batchSize);
return totalSize;
}
John K. Peterson <[email protected]>
Office of Information Technology Engineering
Brigham Young University - Provo, Utah
You can't stop the waves, but you can learn to surf.
Sent from my Alpine. http://sourceforge.net/projects/re-alpine/
On Mon, 16 Jun 2014, Andrew Petro wrote:
John,
Thanks for reporting this bug.
Captured to https://issues.jasig.org/browse/UP-4143 .
Yes, it should batch. The algorithm probably should be loading a
reasonable number at a go, purging those, and then loading more,
repeating until done with the requested purge.
Want to have a go at coding that?
Kind regards,
Andrew
On 6/16/14, 12:09 PM, John K Peterson wrote:
Hi all,
We had a problem last week where a very large number of sessions
(millions)
were created in uPortal. As a server in a cluster tried to purge expired
events, it eventually ran out of memory.
I think the source of the problem is in the method
purgeEventSessionsBefore
of org.jasig.portal.events.aggr.session.JpaEventSessionDao. It appears to
create a list of all the EventSessionImpl's to purge and then delete them
one by one. In our case, it made it to about 7,000,000 of them
(100,000,000 when you count all the supporting hibernate objects) and
almost 4GB of heap space.
I'm not a hibernate specialist, but is it possible to do this more
efficiently and/or in batches? (My workaround for the time being is
calling
setMaxResults(100000) on the query.)
----------------
@AggrEventsTransactional
@Override
public int purgeEventSessionsBefore(DateTime lastAggregatedEventDate) {
final TypedQuery<EventSessionImpl> query =
this.createQuery(this.findExpiredEventSessionsQuery);
query.setParameter(this.dateTimeParameter, lastAggregatedEventDate);
final List<EventSessionImpl> resultList = query.getResultList();
for (final EventSessionImpl eventSession : resultList) {
this.getEntityManager().remove(eventSession);
}
return resultList.size();
}
--
You are currently subscribed to [email protected] as: [email protected]
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/uportal-dev
--
You are currently subscribed to [email protected] as:
[email protected]
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/uportal-dev
--
You are currently subscribed to [email protected] as:
[email protected]
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/uportal-dev
