Re: DIH Full Index Issue

2017-03-09 Thread Alexandre Rafalovitch 
t; Opening [Searcher@a4ed679[collectionXXX_shard1_replica2] main]
> 2017-03-09 13:40:28.350 INFO  
> (searcherExecutor-50-thread-1-processing-n:XXX.XXX.XXX.188:8080_solr 
> x:collectionXXX_shard1_replica2 s:shard1 c:collectionXXX r:core_node1) 
> [c:collectionXXX s:shard1 r:core_node1 x:collectionXXX_shard1_replica2] 
> o.a.s.c.QuerySenderListener QuerySenderListener sending requests to 
> Searcher@a4ed679[collectionXXX_shard1_replica2] 
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_oa1(6.3.0):C31847)
>  Uninverting(_oa2(6.3.0):C2520)))}
> 2017-03-09 13:40:28.350 INFO  
> (searcherExecutor-50-thread-1-processing-n:XXX.XXX.XXX.188:8080_solr 
> x:collectionXXX_shard1_replica2 s:shard1 c:collectionXXX r:core_node1) 
> [c:collectionXXX s:shard1 r:core_node1 x:collectionXXX_shard1_replica2] 
> o.a.s.c.QuerySenderListener QuerySenderListener done.
> 2017-03-09 13:40:28.350 INFO  
> (searcherExecutor-50-thread-1-processing-n:XXX.XXX.XXX.188:8080_solr 
> x:collectionXXX_shard1_replica2 s:shard1 c:collectionXXX r:core_node1) 
> [c:collectionXXX s:shard1 r:core_node1 x:collectionXXX_shard1_replica2] 
> o.a.s.c.SolrCore [collectionXXX_shard1_replica2] Registered new searcher 
> Searcher@a4ed679[collectionXXX_shard1_replica2] 
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_oa1(6.3.0):C31847)
>  Uninverting(_oa2(6.3.0):C2520)))}
> 2017-03-09 13:40:28.365 INFO  (qtp2080166188-34463) [c:collectionXXX s:shard1 
> r:core_node1 x:collectionXXX_shard1_replica2] o.a.s.u.DirectUpdateHandler2 
> end_commit_flush
> 2017-03-09 13:40:28.444 INFO  (qtp2080166188-34463) [c:collectionXXX s:shard1 
> r:core_node1 x:collectionXXX_shard1_replica2] 
> o.a.s.u.p.LogUpdateProcessorFactory [collectionXXX_shard1_replica2]  
> webapp=/solr path=/update 
> params={update.distrib=FROMLEADER=true=true=true=false=http://XXX.XXX.XXX.189:8080/solr/collectionXXX_shard1_replica1/_end_point=true=javabin=2=false}{commit=}
>  0 2309
> 2017-03-09 13:40:56.459 INFO  (qtp2080166188-40636) [c:collectionXXX s:shard1 
> r:core_node1 x:collectionXXX_shard1_replica2] 
> o.a.s.u.p.LogUpdateProcessorFactory [collectionXXX_shard1_replica2]  
> webapp=/solr path=/update/json params={commit=true=json}{} 0 0
> 2017-03-09 13:40:56.459 ERROR (qtp2080166188-40636) [c:collectionXXX s:shard1 
> r:core_node1 x:collectionXXX_shard1_replica2] o.a.s.h.RequestHandlerBase 
> org.apache.solr.common.SolrException: [doc=553094612] missing required field: 
> adtype
> 2017-03-09 13:41:00.053 INFO  (qtp2080166188-41928) [c:collectionXXX s:shard1 
> r:core_node1 x:collectionXXX_shard1_replica2] o.a.s.c.S.Request 
> [collectionXXX_shard1_replica2]  webapp=/solr path=/schema params={wt=json} 
> status=0 QTime=0
>
>
>
> AJ
>
> -Original Message-
> From: Alexandre Rafalovitch [mailto:arafa...@gmail.com]
> Sent: Wednesday, March 8, 2017 9:33 AM
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: DIH Full Index Issue
>
> Are you perhaps indexing at the same time from the source other than DIH? 
> Because the commit is global and all the changes from all the sources will 
> become visible.
>
> Check the access logs perhaps to see the requests to /update handler or 
> similar.
>
> Regards,
>Alex.
>
>
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 8 March 2017 at 09:27, AJ Lemke <aj.le...@securitylabs.com> wrote:
>> Good Morning List!
>>
>> I have an issue where my DIH full index is committed after a minute of 
>> indexing.
>> My counts will fall from around 400K to 85K until the import is finished, 
>> usually about four (4) minutes later.
>>
>> This is problematic for us as there are 315K missing items in our searches.
>>
>> Versioning Info:
>> solr-spec - 6.3.0
>> solr-impl - 6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin -
>> 2016-11-02 19:52:42 lucene-spec - 6.3.0 lucene-impl - 6.3.0
>> a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02
>> 19:47:11
>>
>> solrconfig.xml snippet
>>  > class="solr.DirectUpdateHandler2">
>> 
>> ${solr.ulog.dir:}
>> 
>> 
>> -1
>> false
>> 
>> 
>> -1
>> 
>> 
>>
>>
>> Any insights would be greatly appreciated.
>> Let me know if more information is required.
>>
>> AJ

RE: DIH Full Index Issue

2017-03-09 Thread AJ Lemke 
=/solr path=/update/json params={commit=true=json}{} 0 0
2017-03-09 13:40:56.459 ERROR (qtp2080166188-40636) [c:collectionXXX s:shard1 
r:core_node1 x:collectionXXX_shard1_replica2] o.a.s.h.RequestHandlerBase 
org.apache.solr.common.SolrException: [doc=553094612] missing required field: 
adtype
2017-03-09 13:41:00.053 INFO  (qtp2080166188-41928) [c:collectionXXX s:shard1 
r:core_node1 x:collectionXXX_shard1_replica2] o.a.s.c.S.Request 
[collectionXXX_shard1_replica2]  webapp=/solr path=/schema params={wt=json} 
status=0 QTime=0



AJ

-Original Message-
From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] 
Sent: Wednesday, March 8, 2017 9:33 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: DIH Full Index Issue

Are you perhaps indexing at the same time from the source other than DIH? 
Because the commit is global and all the changes from all the sources will 
become visible.

Check the access logs perhaps to see the requests to /update handler or similar.

Regards,
   Alex.



http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 8 March 2017 at 09:27, AJ Lemke <aj.le...@securitylabs.com> wrote:
> Good Morning List!
>
> I have an issue where my DIH full index is committed after a minute of 
> indexing.
> My counts will fall from around 400K to 85K until the import is finished, 
> usually about four (4) minutes later.
>
> This is problematic for us as there are 315K missing items in our searches.
>
> Versioning Info:
> solr-spec - 6.3.0
> solr-impl - 6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 
> 2016-11-02 19:52:42 lucene-spec - 6.3.0 lucene-impl - 6.3.0 
> a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 
> 19:47:11
>
> solrconfig.xml snippet
>   class="solr.DirectUpdateHandler2">
> 
> ${solr.ulog.dir:}
> 
> 
> -1
> false
> 
> 
> -1
> 
> 
>
>
> Any insights would be greatly appreciated.
> Let me know if more information is required.
>
> AJ

Re: DIH Full Index Issue

2017-03-08 Thread Alexandre Rafalovitch 
Are you perhaps indexing at the same time from the source other than
DIH? Because the commit is global and all the changes from all the
sources will become visible.

Check the access logs perhaps to see the requests to /update handler or similar.

Regards,
   Alex.



http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 8 March 2017 at 09:27, AJ Lemke  wrote:
> Good Morning List!
>
> I have an issue where my DIH full index is committed after a minute of 
> indexing.
> My counts will fall from around 400K to 85K until the import is finished, 
> usually about four (4) minutes later.
>
> This is problematic for us as there are 315K missing items in our searches.
>
> Versioning Info:
> solr-spec - 6.3.0
> solr-impl - 6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 
> 2016-11-02 19:52:42
> lucene-spec - 6.3.0
> lucene-impl - 6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 
> 2016-11-02 19:47:11
>
> solrconfig.xml snippet
> 
> 
> 
> ${solr.ulog.dir:}
> 
> 
> -1
> false
> 
> 
> -1
> 
> 
>
>
> Any insights would be greatly appreciated.
> Let me know if more information is required.
>
> AJ