Very cool. I will dig deeper. I really like this, just need to
figure out where it properly belongs. I don't know that it is "core",
but other can help decide.
-Grant
On Apr 22, 2008, at 11:00 AM, Noble Paul (JIRA) wrote:
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591320
#action_12591320 ]
Noble Paul commented on SOLR-469:
---------------------------------
hi Grant,
we started of with something like that and very soon realized that
it cannot scale beyond the very basic usecases.
We need the ability to apply transformations like, splitting,
merging fields etc etc.
sometimes we need to put in a totally different piece of data .
eg: if a value is 1-5 put in the string 'low' , 5-10 put in 'medium'
etc etc.
All these are really driven by the business requirements
And there is the need for joining one table with another from the
values in one table or merging one table with many tables.
Then we had use cases where data comes from a Db and using a key we
have to fetch data from an xml/http datasource etc etc.
So , the fundamental design or the 'kernel' of the system is
supposed to be totally agnostic of the use cases and we let the
users plug in the implemenations in java/JS etc so that they can do
what they actually want. And we want to share some of the components
which can be common for others.
Data Import RequestHandler
--------------------------
Key: SOLR-469
URL: https://issues.apache.org/jira/browse/SOLR-469
Project: Solr
Issue Type: New Feature
Components: update
Affects Versions: 1.3
Reporter: Noble Paul
Assignee: Grant Ingersoll
Fix For: 1.3
Attachments: SOLR-469.patch, SOLR-469.patch, SOLR-469.patch,
SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch,
SOLR-469.patch
We need a RequestHandler Which can import data from a DB or other
dataSources into the Solr index .Think of it as an advanced form of
SqlUpload Plugin (SOLR-103).
The way it works is as follows.
* Provide a configuration file (xml) to the Handler which takes
in the necessary SQL queries and mappings to a solr schema
- It also takes in a properties file for the data source
configuraution
* Given the configuration it can also generate the solr schema.xml
* It is registered as a RequestHandler which can take two
commands do-full-import, do-delta-import
- do-full-import - dumps all the data from the Database
into the index (based on the SQL query in configuration)
- do-delta-import - dumps all the data that has changed
since last import. (We assume a modified-timestamp column in tables)
* It provides a admin page
- where we can schedule it to be run automatically at
regular intervals
- It shows the status of the Handler (idle, full-import,
delta-import)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
--------------------------
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
