Yes, I would. I need time to get back to this. On Sat, Nov 22, 2008 at 5:56 AM, John Wheeler <[EMAIL PROTECTED]> wrote:
> > Hi Ayende, > > Thanks for the response. I tried using clear without committing before > posting this, and i think that detached entities. I'll try it with > transactions and let you know.. > > separate but related--Any idea if you are going to implement > IFullTextSession.FlushToIndexes() ? > > John > > On Nov 21, 7:37 pm, "Ayende Rahien" <[EMAIL PROTECTED]> wrote: > > You need to clear the session after you commit it, to ensure that you are > > only dealing with small number of entities. > > > > On Sat, Nov 22, 2008 at 2:47 AM, John Wheeler <[EMAIL PROTECTED]> > wrote: > > > > > I got trunk, as of yesterday, of NHibernate Search from nhcontrib's > > > svn repo. I tried following the batch indexing advice located here: > > > > >http://www.hibernate.org/hib_docs/search/reference/en/html_single/#se. > .. > > > , > > > which basically uses a JDBC Scrollable ResultSet, which does not exist > > > in ADO. Also, I looked through the code and couldn't find the > > > hibernate.search.worker.batch_size being used anywhere. I saw the > > > constant that declares the parameter but nothing referencing it. Right > > > now, I am indexing 200K Address POCOs at around 3.5 hours with 2.8 > > > Xeon and 8 gigs. I manually commit them in batches of 500. I looked at > > > the log4net logs and noticed that the actual index writing is slow > > > taking about 25 seconds for each batch of 500. > > > > > Anyway, I was wondering if you had any suggestions for making things > > > faster. I realize I'm playing with bleeding edge stuff out of trunk > > > and nothing is guaranteed, but I thought I'd ask anyway. Here is code > > > and config if you are interested: > > > > > public int Import(string filePath) { > > > Stopwatch watch = new Stopwatch(); > > > > > Debug.WriteLine(string.Format("Importing AT&T Address and > > > Phone Number Flat File {0}", filePath)); > > > TextReader reader = new StreamReader(filePath); > > > ISession session = Search.CreateFullTextSession > > > (SessionFactory.OpenSession()); > > > session.CacheMode = CacheMode.Ignore; > > > session.BeginTransaction(); > > > int count = 0; > > > > > try { > > > reader.ReadLine(); // advance cursor past header > > > reader.ReadLine(); // advance cursor past column > > > titles > > > > > string record; > > > while ((record = reader.ReadLine()) != null) { > > > string[] fields = record.Split(','); > > > if (fields.Length != 13) continue; > > > count++; > > > session.Save(ParseRecord(fields)); > > > Debug.WriteLine(string.Format("{0}: Wrote record > > > {1}", DateTime.Now.ToShortTimeString(), count)); > > > > > if (count % BATCH_SIZE == 0) { > > > watch.Start(); > > > Debug.Write(string.Format("Writing batch > > > {0}... ", count)); > > > session.Transaction.Commit > > > (); > > > Debug.WriteLine(string.Format("Completed in > > > {0} milliseconds.", > > > watch.ElapsedMilliseconds)); > > > watch.Reset(); > > > session.BeginTransaction(); > > > } > > > } > > > > > Debug.Write(string.Format("{0}: Comitting final > > > transaction... ", DateTime.Now.ToShortTimeString())); > > > watch.Start(); > > > session.Transaction.Commit(); > > > Debug.WriteLine(string.Format("Completed in {0} > > > milliseconds.", watch.ElapsedMilliseconds)); > > > watch.Reset(); > > > > > return count; > > > } catch (Exception e) { > > > session.Transaction.Rollback(); > > > throw new Exception(string.Format("Error occurred with > > > record {0}:, {1}", count, e.Message), e); > > > } finally { > > > Debug.WriteLine("Committing addresses to database"); > > > session.Close(); > > > reader.Close(); > > > Debug.WriteLine("Done committing addresses to > > > database"); > > > } > > > } > > > > > Here is my relevant config > > > > > <hibernate-configuration xmlns="urn:nhibernate-configuration-2.2"> > > > <session-factory> > > > <property > > > name="dialect">NHibernate.Dialect.MsSql2005Dialect</property> > > > <property > > > > name="connection.provider">NHibernate.Connection.DriverConnectionProvider</ > > > property> > > > <property name="connection.connection_string">[xxx]</ > > > property> > > > <property > > > name="current_session_context_class">thread_static</property> > > > <property name="show_sql">true</property> > > > <property name="hbm2ddl.auto">create</property> > > > <property name="adonet.batch_size">500</property> > > > <listener > > > class="NHibernate.Search.Event.FullTextIndexEventListener, > > > NHibernate.Search" type="post-insert"/> > > > <listener > > > class="NHibernate.Search.Event.FullTextIndexEventListener, > > > NHibernate.Search" type="post-update"/> > > > <listener > > > class="NHibernate.Search.Event.FullTextIndexEventListener, > > > NHibernate.Search" type="post-delete"/> > > > <listener > > > class="NHibernate.Validator.Event.ValidatePreInsertEventListener, > > > NHibernate.Validator" type="pre-insert"/> > > > <listener > > > class="NHibernate.Validator.Event.ValidatePreUpdateEventListener, > > > NHibernate.Validator" type="pre-update"/> > > > </session-factory> > > > </hibernate-configuration> > > > > > <nhs-configuration xmlns="urn:nhs-configuration-1.0"> > > > <search-factory> > > > <property > > > > > > name="hibernate.search.default.directory_provider">NHibernate.Search.Store.FSDirectoryProvider, > > > NHibernate.Search</property> > > > <property name="hibernate.search.default.indexBase">~/App_Data/ > > > Index</property> > > > <property > > > > > > name="hibernate.search.default.analyzer">Lucene.Net.Analysis.Standard.StandardAnalyzer, > > > Lucene.Net</property> > > > <property > > > name="hibernate.search.default.transaction.ram_buffer_size">2048</ > > > property> > > > <property name="hibernate.search.worker.batch_size">500</ > > > property> > > > </search-factory> > > > </nhs-configuration> > > > > > Thanks Much > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NHibernate Contrib - Development Group" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com.ar/group/nhcdevs?hl=en -~----------~----~----~----~------~----~------~--~---
