RE: Facetting on a field doesn't work, until i optimized the index

[Pushkar Raste]

Markus,
Can you pick up one of the values in the facets and try running query using
that. Ideally numFound should match the facet count. If those aren't
matching I guess your index is still sorta damaged but you aren't really
noticing it.

On Sep 15, 2016 4:44 AM, "Markus Jelsma" <markus.jel...@openindex.io> wrote:

> Mikhail - yes, there are results for the query. The set is empty as if
> there are no values for the field, {}.
>
> Everything checked out correctly but no facet results, until immediately
> after the optimize.
>
> Thanks,
> Markus
>
> -Original message-
> > From:Mikhail Khludnev <m...@apache.org>
> > Sent: Thursday 15th September 2016 10:12
> > To: solr-user <solr-user@lucene.apache.org>
> > Subject: Re: Facetting on a field doesn't work, until i optimized the
> index
> >
> > Markus,
> >
> > Could you spot more details about this particular field type? How does
> > 'empty set of facets' look like? Is there results in that query?
> >
> > On Wed, Sep 14, 2016 at 4:15 PM, Markus Jelsma <
> markus.jel...@openindex.io>
> > wrote:
> >
> > > Hello - we've just spotted the weirdest issue on Solr 6.1.
> > >
> > > We have a Solr index full of logs, new items are added every few
> minutes.
> > > We also have an application that shows charts based on what's in the
> index,
> > > Banana style.
> > >
> > > Yesterday we saw facets for a specific field were missing. Today we
> > > checked it out until we reduced the facet query just to
> > > facet=true=FIELD, but it returned nothing of use, just an
> empty
> > > set of facets.
> > >
> > > My colleague suggested the crazy idea to optimize the index, i
> protested
> > > because it is no use, numDoc always equals maxDoc and the optimize
> button
> > > was missing anyway. So i forced an optimize via the URL, and it
> worked, the
> > > facets for that field are now back!
> > >
> > > Any ideas? Is there a related ticket?
> > >
> > > Thanks,
> > > Markus
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>

RE: Facetting on a field doesn't work, until i optimized the index

[Markus Jelsma]

Mikhail - yes, there are results for the query. The set is empty as if there 
are no values for the field, {}.

Everything checked out correctly but no facet results, until immediately after 
the optimize.

Thanks,
Markus

-Original message-
> From:Mikhail Khludnev <m...@apache.org>
> Sent: Thursday 15th September 2016 10:12
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: Facetting on a field doesn't work, until i optimized the index
> 
> Markus,
> 
> Could you spot more details about this particular field type? How does
> 'empty set of facets' look like? Is there results in that query?
> 
> On Wed, Sep 14, 2016 at 4:15 PM, Markus Jelsma <markus.jel...@openindex.io>
> wrote:
> 
> > Hello - we've just spotted the weirdest issue on Solr 6.1.
> >
> > We have a Solr index full of logs, new items are added every few minutes.
> > We also have an application that shows charts based on what's in the index,
> > Banana style.
> >
> > Yesterday we saw facets for a specific field were missing. Today we
> > checked it out until we reduced the facet query just to
> > facet=true=FIELD, but it returned nothing of use, just an empty
> > set of facets.
> >
> > My colleague suggested the crazy idea to optimize the index, i protested
> > because it is no use, numDoc always equals maxDoc and the optimize button
> > was missing anyway. So i forced an optimize via the URL, and it worked, the
> > facets for that field are now back!
> >
> > Any ideas? Is there a related ticket?
> >
> > Thanks,
> > Markus
> >
> 
> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev
> 

Re: Facetting on a field doesn't work, until i optimized the index

[Mikhail Khludnev]

Markus,

Could you spot more details about this particular field type? How does
'empty set of facets' look like? Is there results in that query?

On Wed, Sep 14, 2016 at 4:15 PM, Markus Jelsma 
wrote:

> Hello - we've just spotted the weirdest issue on Solr 6.1.
>
> We have a Solr index full of logs, new items are added every few minutes.
> We also have an application that shows charts based on what's in the index,
> Banana style.
>
> Yesterday we saw facets for a specific field were missing. Today we
> checked it out until we reduced the facet query just to
> facet=true=FIELD, but it returned nothing of use, just an empty
> set of facets.
>
> My colleague suggested the crazy idea to optimize the index, i protested
> because it is no use, numDoc always equals maxDoc and the optimize button
> was missing anyway. So i forced an optimize via the URL, and it worked, the
> facets for that field are now back!
>
> Any ideas? Is there a related ticket?
>
> Thanks,
> Markus
>



-- 
Sincerely yours
Mikhail Khludnev

Re: Facetting on a field doesn't work, until i optimized the index

[Pushkar Raste]

Damn I didn't put comments in the ticket but replied to question " Is it
safe to upgrade an existing field to docvalues?" on the mailing list.

Check that out

On Sep 14, 2016 5:59 PM, "Pushkar Raste" <pushkar.ra...@gmail.com> wrote:

> We experienced exact opposite issue on Solr 4.10
>
> Check my comments in https://issues.apache.org/jira/browse/SOLR-9437
>
> I am not sure if issue was fixed in Solr 6
>
> I do be interested in tracking down patch for this.
>
> On Sep 14, 2016 3:04 PM, "Erick Erickson" <erickerick...@gmail.com> wrote:
>
>> Weird indeed. Optimize _shouldn't_ be necessary if the index was
>> rebuilt from scratch after changing something like DV, but in a mixed
>> set of segments I'm not sure what would happen. Perhaps one of the
>> Lucene folks can chime in?
>>
>> Best,
>> Erick
>>
>> On Wed, Sep 14, 2016 at 9:22 AM, Markus Jelsma
>> <markus.jel...@openindex.io> wrote:
>> > Well, it could be that indeed. I know i enabled docValues on that field
>> three and a half months ago. But usually when i do that, i force an
>> optimize.
>> >
>> > On the other hand, i'd reckon that in the past few months, all segments
>> should have been merged with another one at least once because data keeps
>> streaming in. But i'm not sure it would anyway.
>> >
>> > Thanks,
>> > Markus
>> >
>> > -----Original message-
>> >> From:Erick Erickson <erickerick...@gmail.com>
>> >> Sent: Wednesday 14th September 2016 17:22
>> >> To: solr-user <solr-user@lucene.apache.org>
>> >> Subject: Re: Facetting on a field doesn't work, until i optimized the
>> index
>> >>
>> >> That's strange
>> >>
>> >> Is there any chance that the schema changed? This is _really_ a shot
>> >> in the dark, but perhaps the optimize "normalized" the field
>> >> definitions stored with each segment.
>> >>
>> >> Imagine segments 1-5 have one definition, and segments 6-10 have a
>> >> different definition for your field. Optimize would have to resolve
>> >> this somehow, perhaps that process made the magic happen?
>> >>
>> >> NOTE: I'm not conversant with the internals of merge, so this may be
>> >> totally bogus..
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Wed, Sep 14, 2016 at 6:15 AM, Markus Jelsma
>> >> <markus.jel...@openindex.io> wrote:
>> >> > Hello - we've just spotted the weirdest issue on Solr 6.1.
>> >> >
>> >> > We have a Solr index full of logs, new items are added every few
>> minutes. We also have an application that shows charts based on what's in
>> the index, Banana style.
>> >> >
>> >> > Yesterday we saw facets for a specific field were missing. Today we
>> checked it out until we reduced the facet query just to
>> facet=true=FIELD, but it returned nothing of use, just an empty
>> set of facets.
>> >> >
>> >> > My colleague suggested the crazy idea to optimize the index, i
>> protested because it is no use, numDoc always equals maxDoc and the
>> optimize button was missing anyway. So i forced an optimize via the URL,
>> and it worked, the facets for that field are now back!
>> >> >
>> >> > Any ideas? Is there a related ticket?
>> >> >
>> >> > Thanks,
>> >> > Markus
>> >>
>>
>

Re: Facetting on a field doesn't work, until i optimized the index

[Pushkar Raste]

We experienced exact opposite issue on Solr 4.10

Check my comments in https://issues.apache.org/jira/browse/SOLR-9437

I am not sure if issue was fixed in Solr 6

I do be interested in tracking down patch for this.

On Sep 14, 2016 3:04 PM, "Erick Erickson" <erickerick...@gmail.com> wrote:

> Weird indeed. Optimize _shouldn't_ be necessary if the index was
> rebuilt from scratch after changing something like DV, but in a mixed
> set of segments I'm not sure what would happen. Perhaps one of the
> Lucene folks can chime in?
>
> Best,
> Erick
>
> On Wed, Sep 14, 2016 at 9:22 AM, Markus Jelsma
> <markus.jel...@openindex.io> wrote:
> > Well, it could be that indeed. I know i enabled docValues on that field
> three and a half months ago. But usually when i do that, i force an
> optimize.
> >
> > On the other hand, i'd reckon that in the past few months, all segments
> should have been merged with another one at least once because data keeps
> streaming in. But i'm not sure it would anyway.
> >
> > Thanks,
> > Markus
> >
> > -Original message-
> >> From:Erick Erickson <erickerick...@gmail.com>
> >> Sent: Wednesday 14th September 2016 17:22
> >> To: solr-user <solr-user@lucene.apache.org>
> >> Subject: Re: Facetting on a field doesn't work, until i optimized the
> index
> >>
> >> That's strange
> >>
> >> Is there any chance that the schema changed? This is _really_ a shot
> >> in the dark, but perhaps the optimize "normalized" the field
> >> definitions stored with each segment.
> >>
> >> Imagine segments 1-5 have one definition, and segments 6-10 have a
> >> different definition for your field. Optimize would have to resolve
> >> this somehow, perhaps that process made the magic happen?
> >>
> >> NOTE: I'm not conversant with the internals of merge, so this may be
> >> totally bogus..
> >>
> >> Best,
> >> Erick
> >>
> >> On Wed, Sep 14, 2016 at 6:15 AM, Markus Jelsma
> >> <markus.jel...@openindex.io> wrote:
> >> > Hello - we've just spotted the weirdest issue on Solr 6.1.
> >> >
> >> > We have a Solr index full of logs, new items are added every few
> minutes. We also have an application that shows charts based on what's in
> the index, Banana style.
> >> >
> >> > Yesterday we saw facets for a specific field were missing. Today we
> checked it out until we reduced the facet query just to
> facet=true=FIELD, but it returned nothing of use, just an empty
> set of facets.
> >> >
> >> > My colleague suggested the crazy idea to optimize the index, i
> protested because it is no use, numDoc always equals maxDoc and the
> optimize button was missing anyway. So i forced an optimize via the URL,
> and it worked, the facets for that field are now back!
> >> >
> >> > Any ideas? Is there a related ticket?
> >> >
> >> > Thanks,
> >> > Markus
> >>
>

Re: Facetting on a field doesn't work, until i optimized the index

[Erick Erickson]

Weird indeed. Optimize _shouldn't_ be necessary if the index was
rebuilt from scratch after changing something like DV, but in a mixed
set of segments I'm not sure what would happen. Perhaps one of the
Lucene folks can chime in?

Best,
Erick

On Wed, Sep 14, 2016 at 9:22 AM, Markus Jelsma
<markus.jel...@openindex.io> wrote:
> Well, it could be that indeed. I know i enabled docValues on that field three 
> and a half months ago. But usually when i do that, i force an optimize.
>
> On the other hand, i'd reckon that in the past few months, all segments 
> should have been merged with another one at least once because data keeps 
> streaming in. But i'm not sure it would anyway.
>
> Thanks,
> Markus
>
> -Original message-
>> From:Erick Erickson <erickerick...@gmail.com>
>> Sent: Wednesday 14th September 2016 17:22
>> To: solr-user <solr-user@lucene.apache.org>
>> Subject: Re: Facetting on a field doesn't work, until i optimized the index
>>
>> That's strange
>>
>> Is there any chance that the schema changed? This is _really_ a shot
>> in the dark, but perhaps the optimize "normalized" the field
>> definitions stored with each segment.
>>
>> Imagine segments 1-5 have one definition, and segments 6-10 have a
>> different definition for your field. Optimize would have to resolve
>> this somehow, perhaps that process made the magic happen?
>>
>> NOTE: I'm not conversant with the internals of merge, so this may be
>> totally bogus..
>>
>> Best,
>> Erick
>>
>> On Wed, Sep 14, 2016 at 6:15 AM, Markus Jelsma
>> <markus.jel...@openindex.io> wrote:
>> > Hello - we've just spotted the weirdest issue on Solr 6.1.
>> >
>> > We have a Solr index full of logs, new items are added every few minutes. 
>> > We also have an application that shows charts based on what's in the 
>> > index, Banana style.
>> >
>> > Yesterday we saw facets for a specific field were missing. Today we 
>> > checked it out until we reduced the facet query just to 
>> > facet=true=FIELD, but it returned nothing of use, just an 
>> > empty set of facets.
>> >
>> > My colleague suggested the crazy idea to optimize the index, i protested 
>> > because it is no use, numDoc always equals maxDoc and the optimize button 
>> > was missing anyway. So i forced an optimize via the URL, and it worked, 
>> > the facets for that field are now back!
>> >
>> > Any ideas? Is there a related ticket?
>> >
>> > Thanks,
>> > Markus
>>

RE: Facetting on a field doesn't work, until i optimized the index

[Markus Jelsma]

Well, it could be that indeed. I know i enabled docValues on that field three 
and a half months ago. But usually when i do that, i force an optimize.

On the other hand, i'd reckon that in the past few months, all segments should 
have been merged with another one at least once because data keeps streaming 
in. But i'm not sure it would anyway.

Thanks,
Markus

-Original message-
> From:Erick Erickson <erickerick...@gmail.com>
> Sent: Wednesday 14th September 2016 17:22
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: Facetting on a field doesn't work, until i optimized the index
> 
> That's strange
> 
> Is there any chance that the schema changed? This is _really_ a shot
> in the dark, but perhaps the optimize "normalized" the field
> definitions stored with each segment.
> 
> Imagine segments 1-5 have one definition, and segments 6-10 have a
> different definition for your field. Optimize would have to resolve
> this somehow, perhaps that process made the magic happen?
> 
> NOTE: I'm not conversant with the internals of merge, so this may be
> totally bogus..
> 
> Best,
> Erick
> 
> On Wed, Sep 14, 2016 at 6:15 AM, Markus Jelsma
> <markus.jel...@openindex.io> wrote:
> > Hello - we've just spotted the weirdest issue on Solr 6.1.
> >
> > We have a Solr index full of logs, new items are added every few minutes. 
> > We also have an application that shows charts based on what's in the index, 
> > Banana style.
> >
> > Yesterday we saw facets for a specific field were missing. Today we checked 
> > it out until we reduced the facet query just to 
> > facet=true=FIELD, but it returned nothing of use, just an empty 
> > set of facets.
> >
> > My colleague suggested the crazy idea to optimize the index, i protested 
> > because it is no use, numDoc always equals maxDoc and the optimize button 
> > was missing anyway. So i forced an optimize via the URL, and it worked, the 
> > facets for that field are now back!
> >
> > Any ideas? Is there a related ticket?
> >
> > Thanks,
> > Markus
> 

Re: Facetting on a field doesn't work, until i optimized the index

[Erick Erickson]

That's strange

Is there any chance that the schema changed? This is _really_ a shot
in the dark, but perhaps the optimize "normalized" the field
definitions stored with each segment.

Imagine segments 1-5 have one definition, and segments 6-10 have a
different definition for your field. Optimize would have to resolve
this somehow, perhaps that process made the magic happen?

NOTE: I'm not conversant with the internals of merge, so this may be
totally bogus..

Best,
Erick

On Wed, Sep 14, 2016 at 6:15 AM, Markus Jelsma
 wrote:
> Hello - we've just spotted the weirdest issue on Solr 6.1.
>
> We have a Solr index full of logs, new items are added every few minutes. We 
> also have an application that shows charts based on what's in the index, 
> Banana style.
>
> Yesterday we saw facets for a specific field were missing. Today we checked 
> it out until we reduced the facet query just to facet=true=FIELD, 
> but it returned nothing of use, just an empty set of facets.
>
> My colleague suggested the crazy idea to optimize the index, i protested 
> because it is no use, numDoc always equals maxDoc and the optimize button was 
> missing anyway. So i forced an optimize via the URL, and it worked, the 
> facets for that field are now back!
>
> Any ideas? Is there a related ticket?
>
> Thanks,
> Markus

Facetting on a field doesn't work, until i optimized the index

[Markus Jelsma]

Hello - we've just spotted the weirdest issue on Solr 6.1.

We have a Solr index full of logs, new items are added every few minutes. We 
also have an application that shows charts based on what's in the index, Banana 
style.

Yesterday we saw facets for a specific field were missing. Today we checked it 
out until we reduced the facet query just to facet=true=FIELD, but 
it returned nothing of use, just an empty set of facets.

My colleague suggested the crazy idea to optimize the index, i protested 
because it is no use, numDoc always equals maxDoc and the optimize button was 
missing anyway. So i forced an optimize via the URL, and it worked, the facets 
for that field are now back!

Any ideas? Is there a related ticket?

Thanks,
Markus