From: Jonathan Rochkind <rochk...@jhu.edu>
Subject: Re: How to import data with a different date format
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Date: Wednesday, September 8, 2010, 11:35 AM
So the standard 'int' field in Solr
1.4 is a "trie based" field, although the example "int" type
in the default solrconfig.xml has a "precision" set to 0,
which means it's not really doing "trie" things. If you set
the precision to something greater than 0, as in the default
example "tint" type, then it's really using 'trie'
functionality. 'trie' functionality speeds up range
queries by putting each value into 'buckets' (my own term),
per the precision specified, so solr has to do less to grab
all values within a certain range.
That's all tint/non-zero-precision-trie does, speed up
range queries. Your use case involves range queries though,
so it's worth investigating. If you use a string or
other textual type for sorting or range queries, you need to
make sure your values sort the way you want them to as
strings. But yyyy-mm-dd will.
More on trie:
http://www.lucidimagination.com/blog/2009/05/13/exploring-lucene-and-solrs-trierange-capabilities/
I think there probably won't be much of a difference at
query time between non-trie int and string, although I'm not
sure, and it may depend on the nature of your data and
queries. Using a trie int will be faster
for (and only for) range queries, if you have a lot of data.
(There are some cases, depending on the data and the nature
of your queries, where the overhead of a non-zero-precision
trie may outweigh the hypothetical gain, but generally it's
faster).
I don't think there should be any appreciable difference
between how long a non-trie int or a string will take to
index -- at least as far as solr is concerned, if your app
preparing the documents for solr takes longer to prepare one
than another, that's another story. An actual trie
(non-zero-precision) theoretically has indexing-time
overhead, but I doubt it would be noticeable, unless you
have a really really lean mean indexing setup where ever
microsecond counts.
Jonathan
Dennis Gearon wrote:
I'm doing something similar for
dates/times/timestamps.
I'm actually trying to do, "'now' is within the range
of what appointments(date/time from and to combos, i.e.
timestamps).
Fairly simple search of:
What items have a start time BEFORE now,
and an end time AFTER now?
My thoughts were to store:
unix time stamp BIGINTS (64 bit)
"ISO_DATE ISO_TIME" strings
Which is going to be faster:
1/ Indexing?
2/ Searching?
How does the 'tint' field mentioned below apply?
Dennis Gearon
Signature Warning
----------------
EARTH has a Right To Life,
otherwise we all die.
Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php
--- On Wed, 9/8/10, Jonathan Rochkind <rochk...@jhu.edu>
wrote:
From: Jonathan Rochkind <rochk...@jhu.edu>
Subject: Re: How to import data with a different
date format
To: "solr-user@lucene.apache.org"
<solr-user@lucene.apache.org>
Date: Wednesday, September 8, 2010, 10:27 AM
Just throwing it out there, I'd
consider a different approach for an actual real
app,
although it might not be easier to get up quickly.
(For
quickly, yeah, I'd just store it as a string, more
on that
at bottom).
If none of your dates have times, they're all just
full
days, I'm not sure you really need the date type
at all.
Convert the date to number-of-days since epoch
integer. (Most languages will have a way to
do this,
but I don't know about pure XSLT). Store
_that_ in a
1.4 'int' field. On top of that, make it a
"tint"
(precision non-zero) for faster range queries.
But now your actual interface will have to convert
from
"number of days since epoch" to a displayable
date. (And if
you allow user input, convert the input to
number-of-days-since-epoch before making a range
query or
fq, but you'd have to do that anyway even with
solr dates,
users aren't going to be entering W3CDate raw, I
don't
think).
That is probably the most efficient way to have
solr handle
it -- using an actual date field type gives you a
lot more
precision than you need, which is going to hurt
performance
on range queries. Which you can compensate for
with trie
date sure, but if you don't really need that
precision to
begin with, why use it? Also the extra
precision can
end up doing unexpected things and making it
easier to have
bugs (range queries on that high precision stuff,
you need
to make sure your start date has 00:00:00 set and
your end
date has 23:59:59 set, to do what you probably
expect). If
you aren't going to use the extra precision,
makes
everything a lot simpler to not use a date field.
Alternately, for your "get this done quick"
method, yeah,
I'd just store it as a string. With a string
exactly as
you've specified, sorting and range queries won't
work how
you'd want. But if you can make it a string
of the
format "yyyy/mm/dd" instead (always two-digit
month and
year), then you can even sort and do range queries
on your
string dates. For the quick and dirty prototype,
I'd just do
that. In fact, while this might make range
queries and
sorting _slightly_ slower than if you use an int
or a tint,
this might really be good enough even for a real
app (hey,
it's what lots of people did before the trie-based
fields
existed).
Jonathan
Erick Erickson wrote:
I think Markus is spot-on given the fact that
you have
2 days. Using a
string field is quickest.
However, if you absolutely MUST have
functioning
dates, there are three
options I can think of:
1> can you make your XSLT transform the
dates?
Confession; I'm XSLT-ignorant
2> use DIH and DateTransformer, see:
http://wiki.apache.org/solr/DataImportHandler#DateFormatTransformer
you can walk a
directory importing all the XML files with
FileDataSource.
<http://wiki.apache.org/solr/DataImportHandler#DateFormatTransformer>3>
you
could write a program to do this manually.
But given the time constraints, I suspect your
time
would be better spent
doing the other stuff and just using string as
per
Markus. I have no clue
how SOLR-savvy you are, so pardon if this is
something
you already know. But
lots of people trip up over the "string" field
type,
which is NOT tokenized.
You usually want "text" unless it's some sort
of
ID.... So it might be worth
it to do some searching earlier rather than
later
<G>....
Best
Erick
On Wed, Sep 8, 2010 at 12:34 PM, Markus Jelsma
<markus.jel...@buyways.nl>wrote:
No. The Datefield [1] will not accept it
any other
way. You could, however,
fool your boss and dump your dates in an
ordinary
string field. But then you
cannot use some of the nice date
features.
[1]:
http://lucene.apache.org/solr/api/org/apache/solr/schema/DateField.html
-----Original message-----
From: Rico Lelina <rlel...@yahoo.com>
Sent: Wed 08-09-2010 17:36
To: solr-user@lucene.apache.org;
Subject: How to import data with a
different date
format
Hi,
I am attempting to import some of our data
into
SOLR. I did it the quickest
way
I know because I literally only have 2
days to
import the data and do some
queries for a proof-of-concept.
So I have this data in XML format and I
wrote a
short XSLT script to
convert it
to the format in solr/example/exampledocs
(except
I retained the element
names
so I had to modify schema.xml in the conf
directory. So far so good -- the
import works and I can search the data.
One of my
immediate problems is
that
there is a date field with the format
MM/DD/YYYY.
Looking at schema.xml, it
seems SOLR accepts only full date fields
--
everything seems to be
mandatory
including the Z for Zulu/UTC time
according to the
doc. Is there a way to
specify the date format?
Thanks very much.
Rico
