Were all of shard replica in active state (green color in admin ui) before starting? Sounds like it otherwise you won't hit the replica that is out of sync.
Replicas can get out of sync, and report being in sync after a sequence of stop start w/o a chance to complete sync. See if it might have happened to you: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201412.mbox/%3CCAOOKt53XTU_e0m2ioJ-S4SfsAp8JC6m-=nybbd4g_mjh60b...@mail.gmail.com%3E On Sep 27, 2015 06:56, "Ravi Solr" <ravis...@gmail.com> wrote: > Erick...There is only one type of String > "sun.org.mozilla.javascript.internal.NativeString:" and no other variations > of that in my index, so no question of missing it. Point taken regarding > the CURSORMARK stuff, yes you are correct, my head so numb at this point > after working 3 days on this, I wasnt thinking straight. > > BTW I found the real issue, I have a total of 8 servers in the solr cloud. > The leader for this specific collection was the one that was returning 0 > for the searches. All other 7 servers had roughly 800K docs still needing > the string replacement. So maybe the real issue is sync among servers. Just > to prove to myself I shutdown the solr that was giving zero results (i.e. > all uuid strings have already been somehow devoid of spurious > sun.org.mozilla.javascript.internal.NativeString on that server). Now it > ran perfectly fine and is about to finish as last 103K are still left when > I was writing this email. > > So the real question is how can we ensure that the Sync is always > maintained and what to do if it ever goes out of Sync, I did see some Jira > tickets from previous 4.10.x versions where Sync was an issue. Can you > please point me to any doc which says how SolrCloud synchs/replicates ? > > Thanks, > > Ravi Kiran Bhaskar > > Thanks > > Rvai Kiran Bhaskar > > On Sat, Sep 26, 2015 at 11:00 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > > > bq: 3. Erick, I wasnt getting all 1.4 mill in one shot. I was initially > > using > > 100 docs batch, which, I later increased to 500 docs per batch. Also it > > would not be a infinite loop if I commit for each batch, right !!?? > > > > That's not the point at all. Look at the basic logic here: > > > > You run for a while processing 100 (or 500 or 1,000) docs per batch > > and change all uuid fields with this statement: > > > > uuid.replace("sun.org.mozilla.javascript.internal.NativeString:", ""); > > > > and then update the doc. You run this as long as you have any docs > > that satisfy the query "q=uuid:sun.org.mozilla*", _changing_ > > every one that has this string! > > > > At that point, theoretically, no document in your index has this string. > So > > running your update program immediately after should find _zero_ > documents. > > > > I've been assuming your complaint is that you don't process 1.4 M docs > (in > > batches), you process some lower number then exit and you think this is > > wrong. > > I'm claiming that you should only expect to find as many docs as have > been > > indexed since the last time the program ran. > > > > As far as the infinite loop is concerned, again trace the logic in the > old > > code. > > Forget about commits and all the mechanics, just look at the logic. > > You're querying on "sun.org.mozilla*". But you only change if you get a > > match on > > "sun.org.mozilla.javascript.internal.NativeString:" > > > > Now imagine you have a doc that has sun.org.mozilla.erick in it. That doc > > gets > > returned from the query but does _not_ get modified because it doesn't > > match your pattern. In the older code, it would be found again and > > returned next > > time you queried. Then not modified again. Eventually you'd be in a > > position > > where you never changed any docs, just kept getting the same docList back > > over and over again. Marching through based on the unique key should not > > have the same potential issue. > > > > You should not be mixing the new query stuff with CURSORMARK. Deep paging > > supposes the exact same query is being run over and over and you're > > _paging_ > > through the results. You're changing the query every time so the results > > aren't > > very predictable. > > > > Best, > > Erick > > > > > > On Sat, Sep 26, 2015 at 5:01 PM, Ravi Solr <ravis...@gmail.com> wrote: > > > Erick & Shawn I incrporated your suggestions. > > > > > > > > > 0. Shut off all other indexing processes. > > > 1. As Shawn mentioned set batch size to 10000. > > > 2. Loved Erick's suggestion about not using filter at all and sort by > > > uniqueId and put last known uinqueId as next queries start while still > > > using cursor marks as follows > > > > > > SolrQuery q = new SolrQuery("+uuid:sun.org.mozilla* +uniqueId:{" + > > > markerSysId + " TO > > > *]").setRows(10000).addSort("uniqueId",ORDER.asc).setFields(new > > > String[]{"uniqueId","uuid"}); > > > q.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark); > > > > > > 3. As per Shawn's advise commented autocommit and soft commit in > > > solrconfig.xml and set openSearcher to false and issued MANUAL COMMIT > for > > > every batch from code as follows > > > > > > client.commit(true, true, true); > > > > > > Here is what the log statement & results - log.info("Indexed " + > count + > > > "/" + docList.getNumFound()); > > > > > > > > > 2015-09-26 17:29:57 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 90000/1344085 > > > 2015-09-26 17:30:30 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 100000/1334085 > > > 2015-09-26 17:33:26 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 110000/1324085 > > > 2015-09-26 17:36:09 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 120000/1314085 > > > 2015-09-26 17:39:42 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 130000/1304085 > > > 2015-09-26 17:43:05 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 140000/1294085 > > > 2015-09-26 17:46:14 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 150000/1284085 > > > 2015-09-26 17:48:22 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 160000/1274085 > > > 2015-09-26 17:48:25 INFO [a.b.c.AdhocCorrectUUID] - Indexed 160000/0 > > > 2015-09-26 17:48:25 INFO [a.b.c.AdhocCorrectUUID] - FINISHED !!! > > > > > > Ran manually a second time to see if first was fluke. Still same. > > > > > > 2015-09-26 17:55:26 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 10000/1264716 > > > 2015-09-26 17:58:07 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 20000/1254716 > > > 2015-09-26 18:03:09 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 30000/1244716 > > > 2015-09-26 18:06:32 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 40000/1234716 > > > 2015-09-26 18:10:35 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 50000/1224716 > > > 2015-09-26 18:15:23 INFO [a.b.c.AdhocCorrectUUID] - Indexed > > 60000/1214716 > > > 2015-09-26 18:15:24 INFO [a.b.c.AdhocCorrectUUID] - Indexed 60000/0 > > > 2015-09-26 18:15:26 INFO [a.b.c.AdhocCorrectUUID] - FINISHED !!! > > > > > > Now changed the autommit in solrconfig.xml as follows...Note the soft > > > commit has been shut off as per Shawn's advise > > > > > > <autoCommit> > > > <!-- <maxDocs>100</maxDocs> --> > > > <maxTime>300000</maxTime> > > > <openSearcher>false</openSearcher> > > > </autoCommit> > > > > > > <!-- > > > <autoSoftCommit> > > > <maxTime>30000</maxTime> > > > </autoSoftCommit> > > > --> > > > > > > 2015-09-26 18:47:44 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > Indexed 10000/1205451 > > > 2015-09-26 18:50:49 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > Indexed 20000/1195451 > > > 2015-09-26 18:54:18 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > Indexed 30000/1185451 > > > 2015-09-26 18:57:04 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > Indexed 40000/1175451 > > > 2015-09-26 19:00:10 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > Indexed 50000/1165451 > > > 2015-09-26 19:00:13 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > Indexed 50000/0 > > > 2015-09-26 19:00:13 INFO > [com.wpost.search.reindexing.AdhocCorrectUUID] > > - > > > FINISHED !!! > > > > > > > > > The query still returned 0 results when they are over million docs > > > available which match uuid:sun.org.mozilla* ...Then why do I get 0 ??? > > > > > > Thanks > > > > > > Ravi Kiran Bhaskar > > > > > > On Sat, Sep 26, 2015 at 3:49 PM, Ravi Solr <ravis...@gmail.com> wrote: > > > > > >> Thank you Erick & Shawn for taking significant time off your weekends > to > > >> debug and explain in great detail. I will try to address the main > points > > >> from your emails to provide more situation context for better > > understanding > > >> of my situation > > >> > > >> 1. Erick, As part of our upgrade from 4.7.2 to 5.3.0 I re-indexed all > > docs > > >> from my old Master-Slave to My SolrCloud using DIH SolrEntityProcessor > > >> which used a Script Transformer. I unwittingly messed up the script > and > > >> hence this 'uuid' (String Type field) got messed up. All records prior > > to > > >> Sep 20 2015 have this issue that I am currently try to rectify. > > >> > > >> 2. Regarding openSearcher=true/false, I had it as false all along in > my > > >> 4.7.2 config. I read somewhere that SolrCloud or 5.x doesn't honor it > > or it > > >> should be left default (Don't exactly remember where I read it), > hence, > > I > > >> removed it from my solrconfig.xml going against my intuition :-) > > >> > > >> 3. Erick, I wasnt getting all 1.4 mill in one shot. I was initially > > using > > >> 100 docs batch, which, I later increased to 500 docs per batch. Also > it > > >> would not be a infinite loop if I commit for each batch, right !!?? > > >> > > >> 4. Shawn, you are correct the uuid is of String Type and its not > unique > > >> key for my schema. My uniqueKey is uniqueId and systemid is of no > > >> consequence here, it's another field for differentiating apps within > my > > >> solr. > > >> > > >> Than you very much again guys. I will incorporate your suggestions and > > >> report back. > > >> > > >> Thanks > > >> > > >> Ravi Kiran Bhaskar > > >> > > >> On Sat, Sep 26, 2015 at 12:58 PM, Erick Erickson < > > erickerick...@gmail.com> > > >> wrote: > > >> > > >>> Oh, one more thing. _assuming_ you can't change the indexing process > > >>> that gets the docs from the system of record, why not just add an > > >>> update processor that does this at index time? See: > > >>> > > > https://cwiki.apache.org/confluence/display/solr/Update+Request+Processors > > >>> , > > >>> in particular the StatelessScriptUpdateProcessorFactory might be a > > >>> good candidate. It just takes a bit of javascript (or other scripting > > >>> language) and changes the record before it gets indexed. > > >>> > > >>> FWIW, > > >>> Erick > > >>> > > >>> On Sat, Sep 26, 2015 at 9:52 AM, Shawn Heisey <apa...@elyograg.org> > > >>> wrote: > > >>> > On 9/26/2015 10:41 AM, Shawn Heisey wrote: > > >>> >> <autoCommit> <maxTime>300000</maxTime> </autoCommit> > > >>> > > > >>> > This needs to include openSearcher=false, as Erick mentioned. I'm > > sorry > > >>> > I screwed that up: > > >>> > > > >>> > <autoCommit> > > >>> > <maxTime>300000</maxTime> > > >>> > <openSearcher>false</openSearcher> > > >>> > </autoCommit> > > >>> > > > >>> > Thanks, > > >>> > Shawn > > >>> > > >> > > >> > > >
