I know it resets the ttl on each scroll call, but since I don't have an automatic resuming process, I need to manually check the last scroll_id (I will log it to a file) and restart the reindexing program using it. That is why I need a longer ttl
I just tested the re-use of the scroll_id. Looks like after the first request, the same scroll_id is returned over and over, returning new docs. So I can't use this approach, since I will always lose the last batch after resuming the reindexing 2014-10-23 18:20 GMT-02:00 John Smith <[email protected]>: > Small ttl is ok (well adjusted properly for you process) because everytime > you call scroll it resets the ttl. So you don't need to put a 60m scroll > time. It just has to be long enough to be able to process the next scroll > id. > > I'm curious if you can re-use the scroll id. It's not specifically > mentioned in the docs but i think scroll is forward only. So not sure once > you got once scroll id you can go back to it. I guess one way to find out :) > > On Thursday, 23 October 2014 15:44:04 UTC-4, Roger de Cordova Farias wrote: >> >> Hmm, I was using a small ttl, just enough to process each scroll call, >> but I could try using a longer time to live and resuming from the last >> scroll_id in case of error >> >> That is a good idea, thanks >> >> 2014-10-23 17:12 GMT-02:00 John Smith <[email protected]>: >> >>> The scroll is available based on a timeout value you give it. >>> Everytimetime you scroll you restart the countdown. >>> >>> You could track the last scroll id you used and try it again from there? >>> >>> On Thursday, 23 October 2014 12:47:02 UTC-4, Roger de Cordova Farias >>> wrote: >>>> >>>> I'm reindexing a ElasticSearch base with 50m docs using the scroll-scan >>>> request to retrieve all docs, but my "reindexer" program stopped at 30m >>>> >>>> Is there a way to redo the query to retrieve the left docs? Like using >>>> offset? >>>> >>>> Would the the internal order of the scan query be the same with a >>>> second request? >>>> >>>> I can assure that no new docs were indexed in the old index since the >>>> beginning of the reindexing >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>> topic/elasticsearch/NbshHCrBHoM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJp2532ZuYCP6%3DxqJeUmZGowAo9dzY%2BQGZkHKbfkTyCWxODF5w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
