[Tracked now as HSEARCH-1474] thanks that's an interesting path to explore; still one problem is consistency during on going changes: the way it works today most of the work is processed by parallel and short lived transactions, but the main stream of primary keys is a forward-only scrollable which needs to select the primary keys only once, or otherwise pagination out of transaction would require a fully locked database.
I simply can't load all those primary keys in memory: I need to page on it while the backend threads make progress and consume the stream to keep the memory usage under control. On 23 December 2013 16:38, Emmanuel Bernard <emman...@hibernate.org> wrote: > I guess you could do it mostly transparently for the user. > Set a reasonably high timeout like say 2 mins, then every n operation look > how much > you've consumed. If you are close to the limit, you commit and restart a > transaction. > So your option would be semi public. > hibernate.search.massindexer.transaction_timeout auto|time in (m)s > > If a user is in trouble with the auto system, we give him the option. > > Emmanuel > > On Mon 2013-12-23 14:54, Sanne Grinovero wrote: >> Thanks that explains why I was missing the correlation. >> To avoid the transaction manager to kill our indexing process, I guess >> our only answer today is to reconfigure the transactions manager to >> use larger timeouts? >> >> That's quite limiting, especially as it requires to change it >> globally; ideally I'd want to ping the TM regularly to let it know >> we're still busy (and doing fine) ? >> >> I couldn't find a method to do that by looking at the standard >> javax.transaction API, so I'm wondering if we should take advantage of >> some proprietary extensions. >> Other than that, on the standard API I see we could set the TX timeout >> for this specific transaction; we should probably add a new option to >> the MassIndexer API to have the user suggest a >> very-high-but-reasonable timeout? >> >> Sanne >> >> On 23 December 2013 10:09, Emmanuel Bernard <emman...@hibernate.org> wrote: >> > Sorry late here. >> > OptionallyWrapInJTATransaction is not about timeout. It's about >> > explicitly starting a JTA transaction like a CMT entity bean would to so >> > that Hibernate ORM does not complain. >> > >> > On Mon 2013-11-11 17:29, Sanne Grinovero wrote: >> >> Hi Emmanuel, >> >> in case you get very bored at Devoxx :) >> >> >> >> I remember you implementing a quite complex fix for my initial >> >> MassIndexer which involved avoiding the transactions we use from >> >> timing out. >> >> This is probably more than a year old, but there is a user on the >> >> forums now using 4.4.0.Final and having a suspiciously similar >> >> problem: >> >> >> >> https://forum.hibernate.org/viewtopic.php?f=9&t=1029562 >> >> >> >> >> >> I've looked into our code, but I'm not understanding how the class >> >> org.hibernate.search.batchindexing.impl.OptionallyWrapInJTATransaction >> >> is supposed to prevent the transaction from timing out? >> >> >> >> Do you have any idea on the problem? >> >> I recently had to apply some refactoring so I might have introduced a >> >> regression but I need another pair of eyes. >> >> >> >> Tia, >> >> Sanne _______________________________________________ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
