Re: Searching on dates and time

[Erick Erickson]

There should be a number of these in the example schemas, although perhaps 
without indexed=“true” in the fieldType...

DateRanges are pretty cool, but this in the “keep it simple” category, you 
might just be able to use plain pdates with the standard [time TO time] syntax.

Although when I try your example (this is on 8.1), all your examples return the 
document when using DateRange. I indexed the doc with SolrJ just as you did. I 
was trying queries the admin UI and with quotes didn’t need to escape, with 
quotes I did, don’t know if daterange behavior has changed since 7.3. I don’t 
see any JIRAs on a quick scan that look pertinent.

The range works too if
1> you include the square brackets like all other range fields
2> correctly format the end date, 2019-02-06T12:00 not 2019-02-06:12:00.

BTW, the sweet spot for DateRange is you can, well, index a range, as: 
doc.addField(“dfield", "[2019-02-05T12:04:00Z TO 2019-02-06T12:04:00Z]”);
If all you’re trying to do is index a single point in time, I’d recommend pdate.

Best,
Erick

> On Jul 5, 2019, at 6:19 PM, Steven White  wrote:
> 
> Achieving the use-case is a must.  So if here is an alternative to
> using solr.DateRangeField,
> I'm willing to use it.  What do you mean by "pdate" and what is it?
> 
> I'm reading this link on how to use DateRangeField but yet it is not
> working for me:
> https://lucene.apache.org/solr/guide/6_6/working-with-dates.html  Is the
> issue with my schema the way i set it up?  The way I'm indexing the data?
> Or something else?
> 
> Steven
> 
> On Fri, Jul 5, 2019 at 3:03 PM Erick Erickson 
> wrote:
> 
>> I think what Mikhail is asking is whether your use-case would be satisfied
>> by just indexing a standard pdate rather than a daterange, then querying
>> by
>> 
>> fq=CC_FILE_DATETIME:[some_date/MONTH TO some_maybe_other_full_date].
>> 
>> With regular pdates, you can use “date math” to round to whatever you want
>> on one or both parts of the query.
>> 
>> A note you might be interested in about “fq” clauses and dates in the
>> filter cache:
>> 
>> https://dzone.com/articles/solr-date-math-now-and-filter
>> 
>> Best,
>> Erick
>> 
>>> On Jul 5, 2019, at 11:55 AM, Steven White  wrote:
>>> 
>>> I need both: point in time and range.  In both cases, I need to be able
>> to
>>> search between just 2 years, between year-month to year-month-day-time,
>>> etc.  So getting my schema right, what and how I index right and the
>> search
>>> syntax right are all important.  This is why, in my original post, I
>> shared
>>> my schema, what I'm indexing and search syntax I'm trying to use.  If I
>> got
>>> anything wrong here to get the feature working right, please let me know.
>>> 
>>> Steven.
>>> 
>>> On Fri, Jul 5, 2019 at 2:16 PM Mikhail Khludnev  wrote:
>>> 
 Hold on. Do you need a range or just point in time?
 
 On Fri, Jul 5, 2019 at 6:51 PM Steven White 
>> wrote:
 
> Thanks Mikhail.  I will read those links and switch over to latest
>> Solr.
> 
> Just to be sure, my schema setup and the way I'm indexing the date data
 are
> not the issue, right?
> 
> Steven.
> 
> On Fri, Jul 5, 2019 at 11:05 AM Mikhail Khludnev 
 wrote:
> 
>> Hello,
>> 
>> The indexed daterange value is really narrow, it might not be easy to
> pick
>> per se. I'm in doubts regarding " in queries. At least TO syntax
 expects
> [
>> ]
>> You can start from these baseline cases
>> 
>> 
> 
 
>> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java
>> 
>> and check
>> 
>> 
> 
 
>> https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
>> (also see below) for sure.
>> Also, I remember lack of strictness in 7,2.1 see
>> https://issues.apache.org/jira/browse/LUCENE-8640
>> 
>> On Fri, Jul 5, 2019 at 5:11 PM Steven White 
> wrote:
>> 
>>> Hi everyone,
>>> 
>>> I'm using Solr 7.2.1 but can upgrade if I must.
>>> 
>>> I setup my schema like so:
>>> 
>>>   
>>>    indexed="true"
>>> required="true"stored="false"  multiValued="false" />
>>> 
>>> And indexed my data like so:
>>> 
>>>   doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
>>> 
>>> When I try to search against this field, some search are working,
> others
>>> are not.  Here are examples
>>> 
>>>   I get a hit: CC_FILE_DATETIME:"2019-02-05"
>>>   I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
>>>   I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
>>>   I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
>>> 
>>> I'm seeing issues with range search took, like so:
>>> 
>>>   I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
>>> 2019-02-06:12:00"
>>> 
>>> It looks to me that anytime I include the time part, 

Re: Searching on dates and time

[Steven White]

Achieving the use-case is a must.  So if here is an alternative to
using solr.DateRangeField,
I'm willing to use it.  What do you mean by "pdate" and what is it?

I'm reading this link on how to use DateRangeField but yet it is not
working for me:
https://lucene.apache.org/solr/guide/6_6/working-with-dates.html  Is the
issue with my schema the way i set it up?  The way I'm indexing the data?
Or something else?

Steven

On Fri, Jul 5, 2019 at 3:03 PM Erick Erickson 
wrote:

> I think what Mikhail is asking is whether your use-case would be satisfied
> by just indexing a standard pdate rather than a daterange, then querying
> by
>
> fq=CC_FILE_DATETIME:[some_date/MONTH TO some_maybe_other_full_date].
>
> With regular pdates, you can use “date math” to round to whatever you want
> on one or both parts of the query.
>
> A note you might be interested in about “fq” clauses and dates in the
> filter cache:
>
> https://dzone.com/articles/solr-date-math-now-and-filter
>
> Best,
> Erick
>
> > On Jul 5, 2019, at 11:55 AM, Steven White  wrote:
> >
> > I need both: point in time and range.  In both cases, I need to be able
> to
> > search between just 2 years, between year-month to year-month-day-time,
> > etc.  So getting my schema right, what and how I index right and the
> search
> > syntax right are all important.  This is why, in my original post, I
> shared
> > my schema, what I'm indexing and search syntax I'm trying to use.  If I
> got
> > anything wrong here to get the feature working right, please let me know.
> >
> > Steven.
> >
> > On Fri, Jul 5, 2019 at 2:16 PM Mikhail Khludnev  wrote:
> >
> >> Hold on. Do you need a range or just point in time?
> >>
> >> On Fri, Jul 5, 2019 at 6:51 PM Steven White 
> wrote:
> >>
> >>> Thanks Mikhail.  I will read those links and switch over to latest
> Solr.
> >>>
> >>> Just to be sure, my schema setup and the way I'm indexing the date data
> >> are
> >>> not the issue, right?
> >>>
> >>> Steven.
> >>>
> >>> On Fri, Jul 5, 2019 at 11:05 AM Mikhail Khludnev 
> >> wrote:
> >>>
>  Hello,
> 
>  The indexed daterange value is really narrow, it might not be easy to
> >>> pick
>  per se. I'm in doubts regarding " in queries. At least TO syntax
> >> expects
> >>> [
>  ]
>  You can start from these baseline cases
> 
> 
> >>>
> >>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java
> 
>  and check
> 
> 
> >>>
> >>
> https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
>  (also see below) for sure.
>  Also, I remember lack of strictness in 7,2.1 see
>  https://issues.apache.org/jira/browse/LUCENE-8640
> 
>  On Fri, Jul 5, 2019 at 5:11 PM Steven White 
> >>> wrote:
> 
> > Hi everyone,
> >
> > I'm using Solr 7.2.1 but can upgrade if I must.
> >
> > I setup my schema like so:
> >
> >
> > >>> indexed="true"
> > required="true"stored="false"  multiValued="false" />
> >
> > And indexed my data like so:
> >
> >doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
> >
> > When I try to search against this field, some search are working,
> >>> others
> > are not.  Here are examples
> >
> >I get a hit: CC_FILE_DATETIME:"2019-02-05"
> >I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
> >I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
> >I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
> >
> > I'm seeing issues with range search took, like so:
> >
> >I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
> > 2019-02-06:12:00"
> >
> > It looks to me that anytime I include the time part, it won't work
> >> and
>  yes
> > I tried escaping ":" like so "\:" but that didn't help.
> >
> > Can someone guide me through this?
> >
> > Thank you
> >
> > Steven
> >
> 
> 
>  --
>  Sincerely yours
>  Mikhail Khludnev
> 
> >>>
> >>
> >>
> >> --
> >> Sincerely yours
> >> Mikhail Khludnev
> >>
>
>

Re: Sort on PointFieldType

[Prince Manohar]

Thanks a lot.

On Thu, Jul 4, 2019 at 9:19 PM Shawn Heisey  wrote:

> On 7/4/2019 9:14 AM, Prince Manohar wrote:
> > I am using Solr version *6.4.2*. I got your answers.
> >
> > I have another question. Can we use a *Range query* on point field?
> >
> > I am trying to do something like
> >
> > *fq=abc.pqr_d:[ 1500 TO 2000 ]*
> >
> > Is it a valid filter?
>
> Yes.  In fact, range queries are one of the areas where Point fields are
> faster than Trie fields.
>
> Something that Point fields do slower than Trie is a simple value
> lookup.  A query like this:
>
> abc.pqr_d:5000
>
> Thanks,
> Shawn
>


-- 
*Regards,
*
*Prince Manohar
*
*B.Tech (InformationTechnology)
*
*Bengaluru
*
*+91 7797045315
*

Stored field values don't update after 7 -> 8 upgrade

[lstusr 5u93n4]

Hi All,

We have a collection that was created on Solr 7.5, and then Solr was
upgraded to 8.1 . After the upgrade, we're seeing that the stored values of
the fields of documents that existed before the upgrade aren't being stored
when the record is updated, even though the indexed value is.

For example, one of the stored fields in our schema is filesize.

If we have a record like this that was created on solr 7 and exists before
the upgrade:
{
 id: 'record1'
  filesize: 5
}

and a record like this that was created after the upgrade to 8.1:
{
id: 'record2',
filesize: 20
}

And we do two separate updates using the /update handler on the solr web
UI:
{
id: 'record1',
filesize: 100,
}
and
{
id: 'record2',
filesize: 100,
}

We expect two records with a filesize of 100. If we query for that:
q=filesize:100 (again on the solr UI) we get the following results:
{
id: 'record1',
filesize: 5,
},
{
id: 'record2',
filesize: 100,
}

Note that the filesize of record1 reports as 5, even though it was
successfully queried with 100. What this tells me is that the INDEXED value
of filesize was updated for 'record1' but the STORED value was not.

In our schema, filesize has: [Indexed, Stored, DocValues, UnInvertible,
Omit Norms, Omit Term Frequencies & Positions] all true for Properties and
Schema. The field type is a plong and the index analyzer is the
DefaultAnalyzer. Most of our fileds are the same, but we have a few other
types, including one that's not stored.

Note that we ran the IndexUpgrader tool listed here:
https://lucene.apache.org/solr/guide/8_1/indexupgrader-tool.html against
every replica, but it should be noted that we ran this AFTER solr 8.1 had
already been launched against the existing collection.

So getting to my questions:

 - is the fact that we started solr without upgrading the index the root
cause here?
 - If not, is this behaviour expected or a known issue?
 - If not, is there any more info you need to help debug/reproduce?

Thanks for any help you can provide,

Kyle

Re: Searching on dates and time

[Erick Erickson]

I think what Mikhail is asking is whether your use-case would be satisfied
by just indexing a standard pdate rather than a daterange, then querying
by 

fq=CC_FILE_DATETIME:[some_date/MONTH TO some_maybe_other_full_date].

With regular pdates, you can use “date math” to round to whatever you want
on one or both parts of the query.

A note you might be interested in about “fq” clauses and dates in the
filter cache:

https://dzone.com/articles/solr-date-math-now-and-filter

Best,
Erick

> On Jul 5, 2019, at 11:55 AM, Steven White  wrote:
> 
> I need both: point in time and range.  In both cases, I need to be able to
> search between just 2 years, between year-month to year-month-day-time,
> etc.  So getting my schema right, what and how I index right and the search
> syntax right are all important.  This is why, in my original post, I shared
> my schema, what I'm indexing and search syntax I'm trying to use.  If I got
> anything wrong here to get the feature working right, please let me know.
> 
> Steven.
> 
> On Fri, Jul 5, 2019 at 2:16 PM Mikhail Khludnev  wrote:
> 
>> Hold on. Do you need a range or just point in time?
>> 
>> On Fri, Jul 5, 2019 at 6:51 PM Steven White  wrote:
>> 
>>> Thanks Mikhail.  I will read those links and switch over to latest Solr.
>>> 
>>> Just to be sure, my schema setup and the way I'm indexing the date data
>> are
>>> not the issue, right?
>>> 
>>> Steven.
>>> 
>>> On Fri, Jul 5, 2019 at 11:05 AM Mikhail Khludnev 
>> wrote:
>>> 
 Hello,
 
 The indexed daterange value is really narrow, it might not be easy to
>>> pick
 per se. I'm in doubts regarding " in queries. At least TO syntax
>> expects
>>> [
 ]
 You can start from these baseline cases
 
 
>>> 
>> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java
 
 and check
 
 
>>> 
>> https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
 (also see below) for sure.
 Also, I remember lack of strictness in 7,2.1 see
 https://issues.apache.org/jira/browse/LUCENE-8640
 
 On Fri, Jul 5, 2019 at 5:11 PM Steven White 
>>> wrote:
 
> Hi everyone,
> 
> I'm using Solr 7.2.1 but can upgrade if I must.
> 
> I setup my schema like so:
> 
>
>>> indexed="true"
> required="true"stored="false"  multiValued="false" />
> 
> And indexed my data like so:
> 
>doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
> 
> When I try to search against this field, some search are working,
>>> others
> are not.  Here are examples
> 
>I get a hit: CC_FILE_DATETIME:"2019-02-05"
>I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
>I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
>I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
> 
> I'm seeing issues with range search took, like so:
> 
>I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
> 2019-02-06:12:00"
> 
> It looks to me that anytime I include the time part, it won't work
>> and
 yes
> I tried escaping ":" like so "\:" but that didn't help.
> 
> Can someone guide me through this?
> 
> Thank you
> 
> Steven
> 
 
 
 --
 Sincerely yours
 Mikhail Khludnev
 
>>> 
>> 
>> 
>> --
>> Sincerely yours
>> Mikhail Khludnev
>> 

Re: Searching on dates and time

[Steven White]

I need both: point in time and range.  In both cases, I need to be able to
search between just 2 years, between year-month to year-month-day-time,
etc.  So getting my schema right, what and how I index right and the search
syntax right are all important.  This is why, in my original post, I shared
my schema, what I'm indexing and search syntax I'm trying to use.  If I got
anything wrong here to get the feature working right, please let me know.

Steven.

On Fri, Jul 5, 2019 at 2:16 PM Mikhail Khludnev  wrote:

> Hold on. Do you need a range or just point in time?
>
> On Fri, Jul 5, 2019 at 6:51 PM Steven White  wrote:
>
> > Thanks Mikhail.  I will read those links and switch over to latest Solr.
> >
> > Just to be sure, my schema setup and the way I'm indexing the date data
> are
> > not the issue, right?
> >
> > Steven.
> >
> > On Fri, Jul 5, 2019 at 11:05 AM Mikhail Khludnev 
> wrote:
> >
> > > Hello,
> > >
> > > The indexed daterange value is really narrow, it might not be easy to
> > pick
> > > per se. I'm in doubts regarding " in queries. At least TO syntax
> expects
> > [
> > > ]
> > > You can start from these baseline cases
> > >
> > >
> >
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java
> > >
> > > and check
> > >
> > >
> >
> https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
> > > (also see below) for sure.
> > > Also, I remember lack of strictness in 7,2.1 see
> > > https://issues.apache.org/jira/browse/LUCENE-8640
> > >
> > > On Fri, Jul 5, 2019 at 5:11 PM Steven White 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'm using Solr 7.2.1 but can upgrade if I must.
> > > >
> > > > I setup my schema like so:
> > > >
> > > > 
> > > >  > indexed="true"
> > > > required="true"stored="false"  multiValued="false" />
> > > >
> > > > And indexed my data like so:
> > > >
> > > > doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
> > > >
> > > > When I try to search against this field, some search are working,
> > others
> > > > are not.  Here are examples
> > > >
> > > > I get a hit: CC_FILE_DATETIME:"2019-02-05"
> > > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
> > > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
> > > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
> > > >
> > > > I'm seeing issues with range search took, like so:
> > > >
> > > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
> > > > 2019-02-06:12:00"
> > > >
> > > > It looks to me that anytime I include the time part, it won't work
> and
> > > yes
> > > > I tried escaping ":" like so "\:" but that didn't help.
> > > >
> > > > Can someone guide me through this?
> > > >
> > > > Thank you
> > > >
> > > > Steven
> > > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: Searching on dates and time

[Mikhail Khludnev]

Hold on. Do you need a range or just point in time?

On Fri, Jul 5, 2019 at 6:51 PM Steven White  wrote:

> Thanks Mikhail.  I will read those links and switch over to latest Solr.
>
> Just to be sure, my schema setup and the way I'm indexing the date data are
> not the issue, right?
>
> Steven.
>
> On Fri, Jul 5, 2019 at 11:05 AM Mikhail Khludnev  wrote:
>
> > Hello,
> >
> > The indexed daterange value is really narrow, it might not be easy to
> pick
> > per se. I'm in doubts regarding " in queries. At least TO syntax expects
> [
> > ]
> > You can start from these baseline cases
> >
> >
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java
> >
> > and check
> >
> >
> https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
> > (also see below) for sure.
> > Also, I remember lack of strictness in 7,2.1 see
> > https://issues.apache.org/jira/browse/LUCENE-8640
> >
> > On Fri, Jul 5, 2019 at 5:11 PM Steven White 
> wrote:
> >
> > > Hi everyone,
> > >
> > > I'm using Solr 7.2.1 but can upgrade if I must.
> > >
> > > I setup my schema like so:
> > >
> > > 
> > >  indexed="true"
> > > required="true"stored="false"  multiValued="false" />
> > >
> > > And indexed my data like so:
> > >
> > > doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
> > >
> > > When I try to search against this field, some search are working,
> others
> > > are not.  Here are examples
> > >
> > > I get a hit: CC_FILE_DATETIME:"2019-02-05"
> > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
> > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
> > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
> > >
> > > I'm seeing issues with range search took, like so:
> > >
> > > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
> > > 2019-02-06:12:00"
> > >
> > > It looks to me that anytime I include the time part, it won't work and
> > yes
> > > I tried escaping ":" like so "\:" but that didn't help.
> > >
> > > Can someone guide me through this?
> > >
> > > Thank you
> > >
> > > Steven
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Solr 6.6.0 - DIH - Multiple entities - Multiple DBs

[Charlie Hull]

On 05/07/2019 14:33, Joseph_Tucker wrote:

Thanks for your help / suggestion.

I'm not sure I completely follow in this case.
SolrJ looks like a method to allow Java applications to talk to Solr, or any
other third party application would simply be a communication method between
Solr and the language of your choosing.

I guess what I'm after is, how would using SolrJ improve performance when
indexing?


It's not just about improving performance (although DIH is single 
threaded, so you could obtain a marked indexing performance gain using a 
client such as SolrJ).  With DIH you will embed a lot of SQL code into 
Solr's configuration files, and the more sources you add the more 
complicated, hard to debug and unmaintainable it's going to be. You 
should thus consider writing a proper indexing script in Java, Python or 
whatever language you are most familiar with - this has always been our 
approach.


Best


Charlie



*** I could be wrong in my assumptions as I'm still learning a great deal
about Solr. ***

I appreciate your help

Regards,

Joe



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html



--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: Solr upgrade question

[Charlie Hull]

On 05/07/2019 14:49, Margo Breäs | INDI wrote:


Hi all,

At the moment we are working with Solr version 4.8.1 in combination 
with an older version of Intershop.


We have recently migrated our entire shop to a new party, and so there 
is room for improvements.


Are there any known issues with upgrading over that many versions in 
general, or with an Intershop version specifically?


If so we would appreciate your experiences/stories, so we can mitigate 
things beforehand.


If you're going to migrate from that old a version of Solr, I think you 
will need to re-index completely and also check that all your queries 
work as you expect...there have been a lot of changes since then and 
don't underestimate the task!


Cheers


Charlie


Thanks in advance,

best regards,

Margo Breas | INDI


Met vriendelijke groet / Kind regards,​


Margo Breäs
​Categoriespecialist
​T. +31 88 0666 000
​E. *margo.br...@indi.nl* 
*​W. www.indi.nl* 
 



INDI.nl website 
 





--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: Searching on dates and time

[Steven White]

Thanks Mikhail.  I will read those links and switch over to latest Solr.

Just to be sure, my schema setup and the way I'm indexing the date data are
not the issue, right?

Steven.

On Fri, Jul 5, 2019 at 11:05 AM Mikhail Khludnev  wrote:

> Hello,
>
> The indexed daterange value is really narrow, it might not be easy to pick
> per se. I'm in doubts regarding " in queries. At least TO syntax expects [
> ]
> You can start from these baseline cases
>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java
>
> and check
>
> https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
> (also see below) for sure.
> Also, I remember lack of strictness in 7,2.1 see
> https://issues.apache.org/jira/browse/LUCENE-8640
>
> On Fri, Jul 5, 2019 at 5:11 PM Steven White  wrote:
>
> > Hi everyone,
> >
> > I'm using Solr 7.2.1 but can upgrade if I must.
> >
> > I setup my schema like so:
> >
> > 
> >  > required="true"stored="false"  multiValued="false" />
> >
> > And indexed my data like so:
> >
> > doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
> >
> > When I try to search against this field, some search are working, others
> > are not.  Here are examples
> >
> > I get a hit: CC_FILE_DATETIME:"2019-02-05"
> > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
> > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
> > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
> >
> > I'm seeing issues with range search took, like so:
> >
> > I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
> > 2019-02-06:12:00"
> >
> > It looks to me that anytime I include the time part, it won't work and
> yes
> > I tried escaping ":" like so "\:" but that didn't help.
> >
> > Can someone guide me through this?
> >
> > Thank you
> >
> > Steven
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: Faceting with Stats

[Erick Erickson]

Thanks for bring closure to this. Yeah, “escaping hell” is something that
happens to us all, something that works in a browser doesn’t work
from SolrJ and neither one may work with curl and……

Pretty often, BTW, I look at the Solr log. It takes a little practice to 
reconstruct the query, but it’s not very hard. Then I work 
backwards…

Best,
Erick

> On Jul 4, 2019, at 11:14 PM, Ahmed Adel  wrote:
> 
> Thanks for your reply! Yes, it turned out to be an issue with the way the
> request was being sent, which was cURL that required special handling and
> escaping of spaces and special characters. Using another client cleared
> this issue and the request below worked perfectly now.
> 
> Best,
> A.
> 
> On Thu, Jul 4, 2019 at 4:53 PM Erick Erickson 
> wrote:
> 
>> Might be a formatting error with my mail client, but the very first line
>> is not well formed.
>> 
>> q: * is incorrect
>> 
>> q=*:*
>> 
>> 
>> 
>> I do not see that example on the page either. Looks like you took the bit
>> that starts with stats=true and mis-typed the q clause.
>> 
>> Best,
>> Erick
>>> On Jul 3, 2019, at 5:08 AM, Ahmed Adel  wrote:
>>> 
>>> Hi,
>>> 
>>> As per the documentation recommendation of using pivot with stats
>> component
>>> instead (
>>> 
>> https://lucene.apache.org/solr/guide/8_1/faceting.html#combining-stats-component-with-pivots
>> ),
>>> replacing the stats options that were previously used with the newer
>> pivot
>>> options as follows:
>>> 
>>> q: *
>>> stats=true
>>> stats.field={!tag=piv1 mean=true}average_rating_f
>>> facet=true
>>> facet.pivot={!stats=piv1}author_s
>>> 
>>> returns the following error:
>>> 
>>> Bad Message 400
>>> reason: Illegal character SPACE=' '
>>> 
>>> This is a syntax issue rather than a logical one, however. Any thoughts
>> of
>>> what could be missing would be appreciated.
>>> 
>>> Thanks,
>>> A. Adel
>>> 
>>> On Tue, Jul 2, 2019 at 4:38 PM Ahmed Adel  wrote:
>>> 
 Hi,
 
 How can stats field value be calculated for top facet values? In other
 words, the following request parameters should return the stats.field
 measures for facets sorted by count:
 
 q: *
 wt: json
 stats: true
 stats.facet: authors_s
 stats.field: average_rating_f
 facet.missing: true
 f.authors_s.facet.sort: count
 
 However, the response is not sorted by facet field count. Is there
 something missing?
 
 Best,
 A.
 
>> 
>> --
> Sent from my iPhone

Re: Solr upgrade question

[Erick Erickson]

There are a _lot_ of changes since 4.8.1. 

1> plan on re-indexing the entire corpus. This is required when jumping more 
than one major version.

2> Treat it as a green-field application. In particular do not just copy your 
schema and config files into 8x and start running. Instead, identify any 
changes you made to 4.8.1 and make similar changes to the stock 8x configs.

3> Really read the CHANGES.txt files for both Lucene and Solr. It’ll be mind 
numbing, but you can get a good sense of the changes that affect upgrading by 
reading the “upgrade notes” for each release between 4.8.1 and the current 
version you settle on.

If the upgrade notes generate specific questions, don’t hesitate to ask.

Best,
Erick

> On Jul 5, 2019, at 6:49 AM, Margo Breäs | INDI  wrote:
> 
> Hi all,
>  
> At the moment we are working with Solr version 4.8.1 in combination with an 
> older version of Intershop.
> We have recently migrated our entire shop to a new party, and so there is 
> room for improvements.
>  
> Are there any known issues with upgrading over that many versions in general, 
> or with an Intershop version specifically?
> If so we would appreciate your experiences/stories, so we can mitigate things 
> beforehand.
>  
> Thanks in advance,
>  
> best regards,
>  
> Margo Breas | INDI
>  
>  
> 
> Met vriendelijke groet / Kind regards,​
> 
> Margo Breäs
> ​Categoriespecialist 
> ​T. +31 88 0666 000 
> ​E. margo.br...@indi.nl 
> ​W. www.indi.nl

Re: Searching on dates and time

[Mikhail Khludnev]

Hello,

The indexed daterange value is really narrow, it might not be easy to pick
per se. I'm in doubts regarding " in queries. At least TO syntax expects [
]
You can start from these baseline cases
https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/schema/DateRangeFieldTest.java

and check
https://lucene.apache.org/solr/guide/8_0/working-with-dates.html#date-range-formatting
(also see below) for sure.
Also, I remember lack of strictness in 7,2.1 see
https://issues.apache.org/jira/browse/LUCENE-8640

On Fri, Jul 5, 2019 at 5:11 PM Steven White  wrote:

> Hi everyone,
>
> I'm using Solr 7.2.1 but can upgrade if I must.
>
> I setup my schema like so:
>
> 
>  required="true"stored="false"  multiValued="false" />
>
> And indexed my data like so:
>
> doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;
>
> When I try to search against this field, some search are working, others
> are not.  Here are examples
>
> I get a hit: CC_FILE_DATETIME:"2019-02-05"
> I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
> I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
> I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"
>
> I'm seeing issues with range search took, like so:
>
> I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
> 2019-02-06:12:00"
>
> It looks to me that anytime I include the time part, it won't work and yes
> I tried escaping ":" like so "\:" but that didn't help.
>
> Can someone guide me through this?
>
> Thank you
>
> Steven
>


-- 
Sincerely yours
Mikhail Khludnev

Searching on dates and time

[Steven White]

Hi everyone,

I'm using Solr 7.2.1 but can upgrade if I must.

I setup my schema like so:




And indexed my data like so:

doc.addField("CC_FILE_DATETIME", "2019-02-05T12:04:00Z");;

When I try to search against this field, some search are working, others
are not.  Here are examples

I get a hit: CC_FILE_DATETIME:"2019-02-05"
I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12"
I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04"
I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04:00Z"

I'm seeing issues with range search took, like so:

I don't get a hit: CC_FILE_DATETIME:"2019-02-05T12:04 TO
2019-02-06:12:00"

It looks to me that anytime I include the time part, it won't work and yes
I tried escaping ":" like so "\:" but that didn't help.

Can someone guide me through this?

Thank you

Steven

Solr upgrade question

[Margo Breäs | INDI]

Hi all,

At the moment we are working with Solr version 4.8.1 in combination with an 
older version of Intershop.
We have recently migrated our entire shop to a new party, and so there is room 
for improvements.

Are there any known issues with upgrading over that many versions in general, 
or with an Intershop version specifically?
If so we would appreciate your experiences/stories, so we can mitigate things 
beforehand.

Thanks in advance,

best regards,

Margo Breas | INDI




Met vriendelijke groet / Kind regards,
Margo Breäs
​Categoriespecialist
​T. +31 88 0666 000
​E. margo.br...@indi.nl
​W. www.indi.nl

Re: Solr 6.6.0 - DIH - Multiple entities - Multiple DBs

[Joseph_Tucker]

Thanks for your help / suggestion.

I'm not sure I completely follow in this case.
SolrJ looks like a method to allow Java applications to talk to Solr, or any
other third party application would simply be a communication method between
Solr and the language of your choosing. 

I guess what I'm after is, how would using SolrJ improve performance when
indexing?

*** I could be wrong in my assumptions as I'm still learning a great deal
about Solr. ***

I appreciate your help 

Regards,

Joe



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr 6.6.0 - DIH - Multiple entities - Multiple DBs

[Alexandre Rafalovitch]

I don't think you should be designing this around DIH. It was never planned
for complex scenarios. Or particularly fault tollerant, which you may need.

Either use SolrJ or a third party tools that integrate with Solr.

Regards,
 Alex

On Fri, Jul 5, 2019, 7:43 AM Joseph_Tucker, 
wrote:

> What is the best way - performance wise - to index data from multiple
> databases?
> I'm potentially going to have around 50 different data sources grabbing
> unique data
> Here's what I've roughly designed:
>
> 
>  
>  
>  
>  
>  
>  
>  
>  ...
>  
> 
>
> I've excluded fields but each entity would have a number of fields within.
> The issue I'm seeing here is the full-index is exceedingly slow. Is there a
> better way to go about this?
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Solr 6.6.0 - DIH - Multiple entities - Multiple DBs

[Joseph_Tucker]

What is the best way - performance wise - to index data from multiple
databases?
I'm potentially going to have around 50 different data sources grabbing
unique data
Here's what I've roughly designed:


 
 
 
 
 
 
 
 ... 
 


I've excluded fields but each entity would have a number of fields within.
The issue I'm seeing here is the full-index is exceedingly slow. Is there a
better way to go about this?




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

replication error solr 7.7 simple master/slave no cloud

[Jeff Courtade]

Hi we have a new setup of solr 7.7 without cloud in a master/slave setup

Periodically our core stops responding to queries and must be
restarted on the slave.

Two hosts

is06 solr 7.7 master
ss06 solr 7.7 slave

simple replication is setup no solr cloud


so on the primary is06 we see this error
The file really is not there on the primary it also in not on the slave

/u01/app/solr/var/logs/solr/solr.log.2019-07-03-13-2019-07-03
13:03:24.348 WARN  (qtp142071-2980) [   x:fdsys-core]
o.a.s.h.ReplicationHandler Exception while writing response for
params: 
generation=118615=true=/replication=_320g_Lucene42_0.dvd=filestream=filecontent
/u01/app/solr/var/logs/solr/solr.log.2019-07-03-13:java.nio.file.NoSuchFileException:
/u01/app/solr/indexes/data/index/_320g_Lucene42_0.dvd

we also see these on the slave ss06

/u01/app/solr/var/logs/solr/solr.log.2019-07-03-15-2019-07-03
15:16:30.931 INFO  (qtp142071-2078) [   x:fdsys-core]
o.a.s.c.S.Request [fdsys-core]  webapp=/solr path=/admin/ping
params={wt=javabin=2} status=0 QTime=0
/u01/app/solr/var/logs/solr/solr.log.2019-07-03-15-2019-07-03
15:16:30.931 WARN  (qtp142071-1692) [   x:fdsys-core]
o.a.s.h.ReplicationHandler Exception while writing response for
params: 
generation=118615=true=/replication=_320g_Lucene42_0.dvd=filestream=filecontent
/u01/app/solr/var/logs/solr/solr.log.2019-07-03-15:java.nio.file.NoSuchFileException:
/u01/app/solr/indexes/data/index/_320g_Lucene42_0.dvd

shortly after these occur the fdys core will stop restopnding to queries on ss06

any idea what these messages are about?

Query facet syntax with escaped char

[Bertrand Mahé]

Hello,

I have a collection containing a field called text defined as below in the
schema.xml file:
  
  

Here is an example of text: "pam_unix(cron:session): session closed for user
root"

I want to know the number of documents containing the expression "session
closed".

Below the request I use and which returns 0 documents:
 
http://10.1.1.1:8983/solr/collection/select?q=*:*=json=0=true
=-1={"column0":{"type":"query","q":"text:*session\%
20closed*"}}

But I am sure I have matching documents because the following request
returns 184 documents:
http://10.1.1.1:8983/solr/collection/select?q=text:*session%5C%20closed*=
json=50=true

Any idea?

Thanks

Bertrand

Re: Faceting with Stats

[Ahmed Adel]

Thanks for your reply! Yes, it turned out to be an issue with the way the
request was being sent, which was cURL that required special handling and
escaping of spaces and special characters. Using another client cleared
this issue and the request below worked perfectly now.

Best,
A.

On Thu, Jul 4, 2019 at 4:53 PM Erick Erickson 
wrote:

> Might be a formatting error with my mail client, but the very first line
> is not well formed.
>
> q: * is incorrect
>
> q=*:*
>
>
>
> I do not see that example on the page either. Looks like you took the bit
> that starts with stats=true and mis-typed the q clause.
>
> Best,
> Erick
> > On Jul 3, 2019, at 5:08 AM, Ahmed Adel  wrote:
> >
> > Hi,
> >
> > As per the documentation recommendation of using pivot with stats
> component
> > instead (
> >
> https://lucene.apache.org/solr/guide/8_1/faceting.html#combining-stats-component-with-pivots
> ),
> > replacing the stats options that were previously used with the newer
> pivot
> > options as follows:
> >
> > q: *
> > stats=true
> > stats.field={!tag=piv1 mean=true}average_rating_f
> > facet=true
> > facet.pivot={!stats=piv1}author_s
> >
> > returns the following error:
> >
> > Bad Message 400
> > reason: Illegal character SPACE=' '
> >
> > This is a syntax issue rather than a logical one, however. Any thoughts
> of
> > what could be missing would be appreciated.
> >
> > Thanks,
> > A. Adel
> >
> > On Tue, Jul 2, 2019 at 4:38 PM Ahmed Adel  wrote:
> >
> >> Hi,
> >>
> >> How can stats field value be calculated for top facet values? In other
> >> words, the following request parameters should return the stats.field
> >> measures for facets sorted by count:
> >>
> >> q: *
> >> wt: json
> >> stats: true
> >> stats.facet: authors_s
> >> stats.field: average_rating_f
> >> facet.missing: true
> >> f.authors_s.facet.sort: count
> >>
> >> However, the response is not sorted by facet field count. Is there
> >> something missing?
> >>
> >> Best,
> >> A.
> >>
>
> --
Sent from my iPhone

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Shawn,Erick,
Thank you for the explanation. The merge scheduler params make sense now.

Thanks,
Rahul

On Wed, Jul 3, 2019 at 11:30 AM Erick Erickson 
wrote:

> Two more tidbits to add to Shawn’s explanation:
>
> There are heuristics built in to ConcurrentMergeScheduler.
> From the Javadocs:
> * If it's an SSD,
> *  {@code maxThreadCount} is set to {@code max(1, min(4, cpuCoreCount/2))},
> *  otherwise 1.  Note that detection only currently works on
> *  Linux; other platforms will assume the index is not on an SSD.
>
> Second, TieredMergePolicy (the default) merges in “tiers” that
> are of similar size. So you can have multiple merges going on
> at the same time on disjoint sets of segments.
>
> Best,
> Erick
>
> > On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
> >
> > On 7/2/2019 10:53 PM, Rahul Goswami wrote:
> >> Hi Shawn,
> >> Thank you for the detailed suggestions. Although, I would like to
> >> understand the maxMergeCount and maxThreadCount params better. The
> >> documentation
> >> <
> https://lucene.apache.org/solr/guide/7_3/indexconfig-in-solrconfig.html#mergescheduler
> >
> >> mentions
> >> that
> >> maxMergeCount : The maximum number of simultaneous merges that are
> allowed.
> >> maxThreadCount : The maximum number of simultaneous merge threads that
> >> should be running at once
> >> Since one thread can only do 1 merge at any given point of time, how
> does
> >> maxMergeCount being greater than maxThreadCount help anyway? I am having
> >> difficulty wrapping my head around this, and would appreciate if you
> could
> >> help clear it for me.
> >
> > The maxMergeCount setting controls the number of merges that can be
> *scheduled* at the same time.  As soon as that number of merges is reached,
> the indexing thread(s) will be paused until the number of merges in the
> schedule drops below this number.  This ensures that no more merges will be
> scheduled.
> >
> > By setting maxMergeCount higher than the number of merges that are
> expected in the schedule, you can ensure that indexing will never be
> paused.  It would require very atypical merge policy settings for the
> number of scheduled merges to ever reach six.  On my own indexing, I
> reached three scheduled merges quite frequently.  The default setting for
> maxMergeCount is three.
> >
> > The maxThreadCount setting controls how many of the scheduled merges
> will be simultaneously executed. With index data on standard spinning
> disks, you do not want to increase this number beyond 1, or you will have a
> performance problem due to thrashing disk heads.  If your data is on SSD,
> you can make it larger than 1.
> >
> > Thanks,
> > Shawn
>
>