Mark, my scrolling performance is pretty constant now too. My problem was in incorrect code, actually. You could check out correct version here: https://github.com/bobrik/esreindexer/blob/master/reindexer.go
On 19 January 2014 21:49, LiquidMark <[email protected]> wrote: > Hi Jörg, > > It looks like you know something about scan/scroll I haven't found > documented elsewhere -- how to scan 60 million docs with 1000 documents per > fetch, at constant time per fetch. Other comments I've seen indicate that, > the deeper you get into fetching the results, the slower each fetch gets. > I'm looking at alternatives for implementing a feature which will require > scan/scroll on a similar scale, and knowing that what you've done is > possible is critical for my planning. Could you please share the key parts > of your setup/retrieve code, in addition to the configuration / version > information you've already shared? > > Thanks in advance, > > -Mark > > > On Wednesday, November 27, 2013 8:03:46 AM UTC-6, Jörg Prante wrote: >> >> I executed a scan/scroll over 60 million docs, size of the indices (folder >> 'data' size) is 87G. >> >> java version "1.7.0_25" >> Java(TM) SE Runtime Environment (build 1.7.0_25-b15) >> Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode) >> >> Heap is 2G >> >> Red Hat Enterprise Linux Server release 6.4 (Santiago) >> >> Jörg >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/RYVP-VH8BaA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/705ede01-a19e-4c40-94d4-2e592de72303%40googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. -- Regards, Ian Babrou http://bobrik.name http://twitter.com/ibobrik skype:i.babrou -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANWdNRAdF4uiXSAyTxXq6zgHAHWSF%3Du0nLF1JpYLkJ1Z%2BDw06A%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
