Sanity check on dataimport handler -- what are the implications if status request returns error?

[Shawn Heisey]

What might the implications be if a DIH status request returns an error 
response other than a 404?


A 404 says either the handler or the core probably don't exist.

My guess, and I admit that I haven't read the code closely, is that if 
the handler exists but is so broken that it cannot return a status 
response without encountering an error, that there's no possible way an 
import could have been started.


For those who have spent significant time in the DIH code:  Would that 
be an accurate statement?


In case that's an XY problem, here's the "X":

When making a request with curl, the -f parameter will cause errors 
(including 404) to return an exit code of 22 from curl, and there will 
also be no output from the request.


But when using -f, you can't tell the difference between a 404 and any 
other 4XX error or 5XX response code.  I would like to treat an exit 
code of 22 as a "go" condition for the script rather than a "fail" 
condition.  If the statement I made above is accurate, then this should 
be a safe thing to do -- because if an error response on the status 
request means either a 404 or a problem so severe than an import can't 
possibly be running, I'd be OK doing that.


Thanks,
Shawn

404 error on Solr 7.2.1 dataimport handler (on Windows via Cygwin)

[PeterKerk]

I'm running Solr 7.2.1 on Windows via Cygwin.

I've installed Solr7.2.1 but I'm getting a 404 when trying to run the
dataimport handler:
http://localhost:8983/solr/tt-giftsamplecatalog/dataimport?command=full-import
After calling this URL, I don't see any logging in the console.

The error in my browser:
HTTP ERROR 404
Problem accessing /solr/tt-giftsamplecatalog/dataimport. Reason:
Not Found


Bat file to start Solr7.2.1:

@echo off
C:
chdir C:\cygwin\bin
bash -c "cd /cygdrive/C/Databases/solr-7.2.1/server/;java
-Dsolr.solr.home="solr/" -jar -Xms200m -Xmx1600m start.jar --module=http
-OPTIONS=jsp" 

Solr.xml









The folder "tt-giftsamplecatalog" is present in
"C:\Databases\solr-7.2.1\example\example-DIH\solr" 


Console logging:

WARNING: System properties and/or JVM args set.  Consider using --dry-run or
--exec
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: \solr.log (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(Unknown Source)
at java.io.FileOutputStream.(Unknown Source)
at java.io.FileOutputStream.(Unknown Source)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at
org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
at
org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127)
at
org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
at org.eclipse.jetty.util.log.Slf4jLog.(Slf4jLog.java:38)
at org.eclipse.jetty.util.log.Slf4jLog.(Slf4jLog.java:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at org.eclipse.jetty.util.log.Log.initialized(Log.java:177)
at org.eclipse.jetty.util.log.Log.getLogger(Log.java:310)
at org.eclipse.jetty.util.log.Log.getLogger(Log.java:300)
at
org.eclipse.jetty.xml.XmlConfiguration.(XmlConfiguration.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:215)
at org.eclipse.jetty.start.Main.start(Main.java:458)
at org.eclipse.jetty.start.Main.main(Main.java:76)
2018-04-30 11:18:44.765 INFO  (main) [   ] o.e.j.s.Server
jetty-9.3.20.v20170531
2018-04-30 11:18:45.810 ERROR (main) [   ] o.a.s.u.StartupLoggingUtils
Missing Java Option solr.log.dir. Logging may be missing or incomplete.
2018-04-30 11:18:45.816 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___ 
_   Welcome to Apache SolrÖ version 7.2.1
2018-04-30 11:18:45.817 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __|
___| |_ _   Starting in standalone mode on port null
2018-04-30 11:18:45.817 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/
_ \ | '_|  Install dir: null
2018-04-30 11:18:45.846 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter
|___/\___/_|_|Start time: 2018-04-30T09:18:45.821Z
2018-04-30 11:18:45.873 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using
system property solr.solr.home: solr/
2018-04-30 11:18:45.883 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading
container configuration from C:\Databases\solr-7.2.1\server\solr\solr.xml
2018-04-30 11:18:46.978 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator
Found 0 core definitions underneath C:\Databases\solr-7.2.1\server\solr
2018-04-30 11:18:48.171 I

Re: Dataimport handler showing idle status with multiple shards

[Sarah Weissman]

From: Shawn Heisey 
Reply-To: "solr-user@lucene.apache.org" 
Date: Tuesday, December 5, 2017 at 1:31 PM
To: "solr-user@lucene.apache.org" 
Subject: Re: Dataimport handler showing idle status with multiple shards

On 12/5/2017 10:47 AM, Sarah Weissman wrote:
I’ve recently been using the dataimport handler to import records from a 
database into a Solr cloud collection with multiple shards. I have 6 dataimport 
handlers configured on 6 different paths all running simultaneously against the 
same DB. I’ve noticed that when I do this I often get “idle” status from the 
DIH even when the import is still running. The percentage of the time I get an 
“idle” response seems proportional to the number of shards. I.e., with 1 shard 
it always shows me non-idle status, with 2 shards I see idle about half the 
time I check the status, with 96 shards it seems to be showing idle almost all 
the time. I can see the size of each shard increasing, so I’m sure the import 
is still going.

I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1. 
Does anyone know why the DIH would report idle when it’s running?

e.g.:
curl http://myserver:8983/solr/collection/dataimport6



To use DIH with SolrCloud, you should be sending your request directly
to a shard replica core, not the collection, so that you can be
absolutely certain that the import command and the status command are
going to the same place.  You MIGHT need to also have a distrib=false
parameter on the request, but I do not know whether that is required to
prevent the load balancing on the dataimport handler.



Thanks for the information, Shawn. I am relatively new to Solr cloud and I am 
used to running the dataimport from the admin dashboard, where it happens at 
the collection level, so I find it surprising that the right way to do this is 
at the core level. So, if I want to be able to check the status of my data 
import for N cores I would need to create N different data import configs that 
manually partition the collection and start each different config on a 
different core? That seems like it could get confusing. And then if I wanted to 
grow or shrink my shards I’d have to rejigger my data import configs every 
time. I kind of expect a distributed index to hide these details from me.

I only have one node at the moment, and I don’t understand how Solr cloud works 
internally well enough to understand what it means for the data import to be 
running on a shard vs. a node. It would be nice if doing a status query would 
at least tell you something, like the number of documents last indexed on that 
core, even if nothing is currently running. That way at least I could 
extrapolate how much longer the operation will take.

Re: Dataimport handler showing idle status with multiple shards

[Shawn Heisey]

On 12/5/2017 10:47 AM, Sarah Weissman wrote:

I’ve recently been using the dataimport handler to import records from a 
database into a Solr cloud collection with multiple shards. I have 6 dataimport 
handlers configured on 6 different paths all running simultaneously against the 
same DB. I’ve noticed that when I do this I often get “idle” status from the 
DIH even when the import is still running. The percentage of the time I get an 
“idle” response seems proportional to the number of shards. I.e., with 1 shard 
it always shows me non-idle status, with 2 shards I see idle about half the 
time I check the status, with 96 shards it seems to be showing idle almost all 
the time. I can see the size of each shard increasing, so I’m sure the import 
is still going.

I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1. 
Does anyone know why the DIH would report idle when it’s running?

e.g.:
curl http://myserver:8983/solr/collection/dataimport6


When you send a DIH request to the collection name, SolrCloud is going 
to load balance that request across the cloud, just like it would with 
any other request.  Solr will look at the list of all responding nodes 
that host part of the collection and send multiple such requests to 
different cores (shards/replicas) across the cloud.  If there are four 
cores in the collection and the nodes hosting them are all working, then 
each of those cores would only see requests to /dataimport about one 
fourth of the time.


DIH imports happen at the core level, NOT the collection level, so when 
you start an import on a collection with four cores in the cloud, only 
one of those four cores is actually going to be doing the import, the 
rest of them are idle.


This behavior should happen with any version, so I would expect it in 
6.1 as well as 7.1.


To use DIH with SolrCloud, you should be sending your request directly 
to a shard replica core, not the collection, so that you can be 
absolutely certain that the import command and the status command are 
going to the same place.  You MIGHT need to also have a distrib=false 
parameter on the request, but I do not know whether that is required to 
prevent the load balancing on the dataimport handler.


A similar question came to this list two days ago, and I replied to that 
one yesterday.


http://lucene.472066.n3.nabble.com/Dataimporter-status-tp4365602p4365879.html

Somebody did open an issue a LONG time ago about this problem:

https://issues.apache.org/jira/browse/SOLR-3666

I just commented on the issue.

Thanks,
Shawn

Dataimport handler showing idle status with multiple shards

[Sarah Weissman]

Hi,

I’ve recently been using the dataimport handler to import records from a 
database into a Solr cloud collection with multiple shards. I have 6 dataimport 
handlers configured on 6 different paths all running simultaneously against the 
same DB. I’ve noticed that when I do this I often get “idle” status from the 
DIH even when the import is still running. The percentage of the time I get an 
“idle” response seems proportional to the number of shards. I.e., with 1 shard 
it always shows me non-idle status, with 2 shards I see idle about half the 
time I check the status, with 96 shards it seems to be showing idle almost all 
the time. I can see the size of each shard increasing, so I’m sure the import 
is still going.

I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1. 
Does anyone know why the DIH would report idle when it’s running?

e.g.:
curl http://myserver:8983/solr/collection/dataimport6
{
  "responseHeader":{
"status":0,
"QTime":0},
  "initArgs":[
"defaults",[
  "config","data-config6.xml"]],
  "status":"idle",
  "importResponse":"",
  "statusMessages":{}}

Thanks,
Sarah

Re: Solr 7* Sorry, no dataimport-handler defined

[Shawn Heisey]

On 11/7/2017 6:49 AM, richardg wrote:
> vs on the master that shows the error.
>
> 2017-11-07 13:29:14.131 INFO  (qtp1839206329-36) [  
> x:solr_aggregate_production] o.a.s.c.S.Request [solr_aggregate_production] 
> webapp=/solr path=/admin/mbeans
> params={cat=QUERYHANDLER&wt=json&_=1510061366718} status=0 QTime=2

The string "QUERYHANDLER" (all uppercase) only shows up in a 7.1.0
source code checkout in the reference guide, it is not in any code that
builds the program.  Its presence in the reference guide is likely a
documentation error.

If you are seeing QUERYHANDLER in a log for version 7.1.0, then I have
to wonder exactly how you did the upgrade -- because I think there are
only two ways that could happen:  1) Your 7.1.0 install includes at
least some files from a version before 6.4.1.  2) You've got something
(perhaps a load balancer) mixing up requests between two different
versions of Solr.

Thanks,
Shawn

Re: Solr 7* Sorry, no dataimport-handler defined

[richardg]

Yes I am referring to the dataimport tab in the admin UI and issue
SOLR-10035.  My previous setup w/ 6.3 did not show this error.  I then
upgraded to 7.1.0 and the error shows.  I upgraded(downgraded) to versions
6.5.0 and 6.6.2 and I do not see the error.  Version 7.0.1 also shows the
error for me.  I am currently using version 6.6.2 and have been successfully
able to run a data import from the admin UI. 

In my config directory we have 

solcore.properties
solrconfig.xml which defines the dataimport handler (data-config.xml)
schema.xml
dataimport.properties
data-config.xml
some admin-extra*.html files

We copy all the config files over to the slave instances and they do no show
this behavior on 7.1.0, dataimport tab loads fine.  The only thing I notice
is on the slaves I see entries like this in the log:

2017-11-07 13:36:11.200 INFO  (qtp2053591126-35) [  
x:solr_aggregate_production] o.a.s.c.S.Request [solr_aggregate_production] 
webapp=/solr path=/admin/mbeans params={cat=QUERY&wt=json&_=1510061783971}
status=0 QTime=0

vs on the master that shows the error.

2017-11-07 13:29:14.131 INFO  (qtp1839206329-36) [  
x:solr_aggregate_production] o.a.s.c.S.Request [solr_aggregate_production] 
webapp=/solr path=/admin/mbeans
params={cat=QUERYHANDLER&wt=json&_=1510061366718} status=0 QTime=2

I see just "QUERY" in the slave that is working and "QUERYHANDLER" in the
master that isn't.  This is why I referenced the issue w/ 6.4 (SOLR-10035). 
Other than that I do not see anything in the log showing and error for the
dataimport handler.

Thanks



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr 7* Sorry, no dataimport-handler defined

[Shawn Heisey]

On 11/6/2017 4:26 PM, Shawn Heisey wrote:
> If I start Solr "normally" or with the cloud example, then the
> dataimport tab shows that error -- which is exactly as expected.

I have opened an improvement issue so that particular error message
isn't as vague.  It's been labeled with "newdev" because it's an easy
change.

https://issues.apache.org/jira/browse/SOLR-11613

Thanks,
Shawn

Re: Solr 7* Sorry, no dataimport-handler defined

[Shawn Heisey]

On 11/6/2017 1:47 PM, richardg wrote:
> I see where this was an issue w/ 6.4 and fixed.  I keep getting this error w/
> 7.0.1 and 7.1.0.  Works fine up until 6.6.2.  Could this issue have been
> reintroduced?  Is there somewhere to check what might be going on?  I don't
> see anything in the error logs.

You've included almost no information here.  I assume that the subject
is the error?  If so, where do you see it?  Please be aware that without
information, we can't help.

https://wiki.apache.org/solr/UsingMailingLists

My best guess is that you see that error in the admin UI when you click
on dataimport.  The error is pretty self-explanitory -- Solr couldn't
find any defined dataimport handlers.  At least one such handler must be
set up in the config before you can use the dataimport tab.

You mention that this was an issue with 6.4 and fixed ... but didn't
include a link to the issue.  Are you talking about SOLR-10035?

https://issues.apache.org/jira/browse/SOLR-10035

If I start a fresh Solr 7.1.0 download with the DIH example (bin/solr
start -e dih), clicking on the dataimport tab works on all five cores. 
If I start Solr "normally" or with the cloud example, then the
dataimport tab shows that error -- which is exactly as expected.  The
example configsets in the base Solr download do not have a dataimport
handler.  Only the configs specifically in the DIH example have such
handlers defined.

Have you created a core with a dataimport handler that you KNOW is
working, and seeing that error?  If so, we'll need more detailed
information.  Solr version, everything in the core's conf directory,
details about the error and steps taken to see it, etc.  Any error
messages in your logfile would be helpful as well.

Thanks,
Shawn

Solr 7* Sorry, no dataimport-handler defined

[richardg]

I see where this was an issue w/ 6.4 and fixed.  I keep getting this error w/
7.0.1 and 7.1.0.  Works fine up until 6.6.2.  Could this issue have been
reintroduced?  Is there somewhere to check what might be going on?  I don't
see anything in the error logs.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: DataImport Handler Out of Memory

[Allison, Timothy B.]

https://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F


-Original Message-
From: Deeksha Sharma [mailto:dsha...@flexera.com] 
Sent: Wednesday, September 27, 2017 1:40 PM
To: solr-user@lucene.apache.org
Subject: DataImport Handler Out of Memory

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in 
mysql db and the number of records are more than 3.5 million. My solr server 
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of 
RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to 
configure Solr to stream the data from DB or any other solution somewhere may 
have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by 
giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

DataImport Handler Out of Memory

[Deeksha Sharma]

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in 
mysql db and the number of records are more than 3.5 million. My solr server 
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of 
RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to 
configure Solr to stream the data from DB or any other solution somewhere may 
have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by 
giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

Re: no dataimport-handler defined!

[Shawn Heisey]

On 1/26/2017 7:44 AM, Chris Rogers wrote:
> Just tested the DIH example in 6.4 (bin/solr -e dih)
>
> Getting the same “No dataimport-handler defined!” for every one of the cores 
> installed as part of the example.

Repeating a reply already posted elsewhere on this thread:

It's a bug.

https://issues.apache.org/jira/browse/SOLR-10035

Easy enough to fix manually, hopefully 6.4.1 will work out of the box.

Thanks,
Shawn

Re: no dataimport-handler defined!

[Alexandre Rafalovitch]

Chris,

Shawn has already provided a workaround and a JIRA reference earlier
in this thread. Could you review his message and see if his solution
solves it for you. There might be a 6.4.1 soon and it will be fixed
there as well.

Regards,
   Alex

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 26 January 2017 at 09:44, Chris Rogers
 wrote:
> Hi Alex,
>
> Just tested the DIH example in 6.4 (bin/solr -e dih)
>
> Getting the same “No dataimport-handler defined!” for every one of the cores 
> installed as part of the example.
>
> Cheers,
> Chris
>
>
> On 24/01/2017, 15:07, "Alexandre Rafalovitch"  wrote:
>
> Strange.
>
> If you run a pre-built DIH example, do any of the cores work? (not the
> RSS one, that is broken anyway).
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 24 January 2017 at 08:32, Chris Rogers
>  wrote:
> > Hi Alex,
> >
> > I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie 
> the one generated from the configset when the node was created).
> >
> > I’m running standalone, not cloud.
> >
> > I’m restarting sole after every change. Do I need to reload the core 
> instead of restarting?
> >
> > I’ve also tried replacing the relative path to the .jar with an 
> absolute path to the dist directory. Still didn’t work.
> >
> > Thanks,
> > Chris
> >
> > On 24/01/2017, 13:20, "Alexandre Rafalovitch"  
> wrote:
> >
> > Which solrconfig.xml are you editing and what kind of Solr install 
> are
> > you running (cloud?). And did you reload the core.
> >
> > I suspect you are not editing the file that is actually in use. For
> > example, if you are running a cloud setup, the solrconfig.xml on the
> > filesystem is disconnected from the config actually in use that is
> > stored in ZooKeeper. You would need to reupload it for change to 
> take
> > effect.
> >
> > You also may need to reload the core for changes to take effect.
> >
> > Regards,
> >Alex.
> > 
> > http://www.solr-start.com/ - Resources for Solr users, new and 
> experienced
> >
> >
> > On 24 January 2017 at 07:43, Chris Rogers
> >  wrote:
> > > Hi all,
> > >
> > > Having frustrating issues with getting SOLR 6.4.0 to recognize 
> the existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
> > >
> > > The DIH .jar file appears to be loading correctly. There are no 
> errors in the SOLR logs. It just says “Sorry, no dataimport-handler defined” 
> in the SOLR admin UI.
> > >
> > > My config files are listed below. Can anyone spot any mistakes 
> here?
> > >
> > > Many thanks,
> > > Chris
> > >
> > > # solrconfig.xml ##
> > >
> > >regex=".*dataimporthandler-.*\.jar" />
> > >
> > > …
> > >
> > >class="org.apache.solr.handler.dataimport.DataImportHandler">
> > > 
> > >   DIH-data-config.xml
> > > 
> > >   
> > >
> > > # DIH-data-config.xml (in the same dir as solrconfig.xml) 
> ##
> > >
> > > 
> > >   
> > >   
> > > 
> > >  > > fileName=".*xml"
> > > newerThan="'NOW-5YEARS'"
> > > recursive="true"
> > > rootEntity="false"
> > > dataSource="null"
> > > 
> baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
> > >
> > >   
> > >
> > >> >   forEach="/TEI" url="${f.fileAbsolutePath}" 
> transformer="RegexTransformer" >
> > >  xpath="/TEI/teiHeader/fileDesc/titleStmt/title"/>
> > >  xpath="/TEI/teiHeader/fileDesc/publicationStmt/publisher"/>
> > >  xpath="/TEI/teiHeader/fileDesc/sourceDesc/msDesc/msIdentifier/altIdentifier/idno"/>
> > >   
> > >
> > > 
> > >
> > >   
> > > 
> > >
> > >
> > > --
> > > Chris Rogers
> > > Digital Projects Manager
> > > Bodleian Digital Library Systems and Services
> > > chris.rog...@bodleian.ox.ac.uk
> >
> >
>
>

Re: no dataimport-handler defined!

[Chris Rogers]

Hi Alex,

Just tested the DIH example in 6.4 (bin/solr -e dih)

Getting the same “No dataimport-handler defined!” for every one of the cores 
installed as part of the example.

Cheers,
Chris


On 24/01/2017, 15:07, "Alexandre Rafalovitch"  wrote:

Strange.

If you run a pre-built DIH example, do any of the cores work? (not the
RSS one, that is broken anyway).

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 08:32, Chris Rogers
 wrote:
> Hi Alex,
>
> I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie 
the one generated from the configset when the node was created).
>
> I’m running standalone, not cloud.
>
> I’m restarting sole after every change. Do I need to reload the core 
instead of restarting?
>
> I’ve also tried replacing the relative path to the .jar with an absolute 
path to the dist directory. Still didn’t work.
>
> Thanks,
> Chris
>
> On 24/01/2017, 13:20, "Alexandre Rafalovitch"  wrote:
>
> Which solrconfig.xml are you editing and what kind of Solr install are
> you running (cloud?). And did you reload the core.
>
> I suspect you are not editing the file that is actually in use. For
> example, if you are running a cloud setup, the solrconfig.xml on the
> filesystem is disconnected from the config actually in use that is
> stored in ZooKeeper. You would need to reupload it for change to take
> effect.
>
> You also may need to reload the core for changes to take effect.
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and 
experienced
>
>
> On 24 January 2017 at 07:43, Chris Rogers
>  wrote:
> > Hi all,
> >
> > Having frustrating issues with getting SOLR 6.4.0 to recognize the 
existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
> >
> > The DIH .jar file appears to be loading correctly. There are no 
errors in the SOLR logs. It just says “Sorry, no dataimport-handler defined” in 
the SOLR admin UI.
> >
> > My config files are listed below. Can anyone spot any mistakes here?
> >
> > Many thanks,
> > Chris
> >
> > # solrconfig.xml ##
> >
> >   
> >
> > …
> >
> >   
> > 
> >   DIH-data-config.xml
> > 
> >   
> >
> > # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
> >
> > 
> >   
> >   
> > 
> >  > fileName=".*xml"
> > newerThan="'NOW-5YEARS'"
> > recursive="true"
> > rootEntity="false"
> > dataSource="null"
> > 
baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
> >
> >   
> >
> >>   forEach="/TEI" url="${f.fileAbsolutePath}" 
transformer="RegexTransformer" >
> > 
> > 
> > 
> >   
> >
> > 
> >
> >   
> > 
> >
> >
> > --
> > Chris Rogers
> > Digital Projects Manager
> > Bodleian Digital Library Systems and Services
> > chris.rog...@bodleian.ox.ac.uk
>
>

Re: no dataimport-handler defined!

[sudhi]

Did anyone figured out a solution for this? Ran into same issue when I
upgraded to 6.4 from 6.2.1. DIH works perfectly fine in 6.2.1

Tried out of the box example and donot see DIH in the example cloud module
as well.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/no-dataimport-handler-defined-tp4315520p4315787.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: no dataimport-handler defined!

[Shawn Heisey]

On 1/24/2017 5:43 AM, Chris Rogers wrote:
> Having frustrating issues with getting SOLR 6.4.0 to recognize the existence 
> of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no errors in the 
> SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
> admin UI.

A user on the IRC channel has run into what seems to be the same problem.

This bug is caused by a change in the SolrInfoMBean class, part of some
cleanup for new metrics.  The output of the handler that normally lives
at /admin/mbeans changed -- what used to be QUERYHANDLER is now just
QUERY, so the admin UI is asking for the wrong category.  This problem
can be fixed in the binary download by editing the following file to
change QUERYHANDLER to QUERY, being careful to only change the uppercase
version of that string:

solr/server/solr-webapp/webapp/js/angular/controllers/dataimport.js

I have filed an issue to deal with the problem:

https://issues.apache.org/jira/browse/SOLR-10035

Fixing it is easy enough ... but perhaps a test that checks the UI
operation would be a good idea.  I've got no idea how to write that test.

Thanks,
Shawn

Re: no dataimport-handler defined!

[Alexandre Rafalovitch]

Strange.

If you run a pre-built DIH example, do any of the cores work? (not the
RSS one, that is broken anyway).

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 08:32, Chris Rogers
 wrote:
> Hi Alex,
>
> I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie the 
> one generated from the configset when the node was created).
>
> I’m running standalone, not cloud.
>
> I’m restarting sole after every change. Do I need to reload the core instead 
> of restarting?
>
> I’ve also tried replacing the relative path to the .jar with an absolute path 
> to the dist directory. Still didn’t work.
>
> Thanks,
> Chris
>
> On 24/01/2017, 13:20, "Alexandre Rafalovitch"  wrote:
>
> Which solrconfig.xml are you editing and what kind of Solr install are
> you running (cloud?). And did you reload the core.
>
> I suspect you are not editing the file that is actually in use. For
> example, if you are running a cloud setup, the solrconfig.xml on the
> filesystem is disconnected from the config actually in use that is
> stored in ZooKeeper. You would need to reupload it for change to take
> effect.
>
> You also may need to reload the core for changes to take effect.
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 24 January 2017 at 07:43, Chris Rogers
>  wrote:
> > Hi all,
> >
> > Having frustrating issues with getting SOLR 6.4.0 to recognize the 
> existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
> >
> > The DIH .jar file appears to be loading correctly. There are no errors 
> in the SOLR logs. It just says “Sorry, no dataimport-handler defined” in the 
> SOLR admin UI.
> >
> > My config files are listed below. Can anyone spot any mistakes here?
> >
> > Many thanks,
> > Chris
> >
> > # solrconfig.xml ##
> >
> >regex=".*dataimporthandler-.*\.jar" />
> >
> > …
> >
> >class="org.apache.solr.handler.dataimport.DataImportHandler">
> > 
> >   DIH-data-config.xml
> > 
> >   
> >
> > # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
> >
> > 
> >   
> >   
> > 
> >  > fileName=".*xml"
> > newerThan="'NOW-5YEARS'"
> > recursive="true"
> > rootEntity="false"
> > dataSource="null"
> > 
> baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
> >
> >   
> >
> >>   forEach="/TEI" url="${f.fileAbsolutePath}" 
> transformer="RegexTransformer" >
> >  xpath="/TEI/teiHeader/fileDesc/titleStmt/title"/>
> >  xpath="/TEI/teiHeader/fileDesc/publicationStmt/publisher"/>
> >  xpath="/TEI/teiHeader/fileDesc/sourceDesc/msDesc/msIdentifier/altIdentifier/idno"/>
> >   
> >
> > 
> >
> >   
> > 
> >
> >
> > --
> > Chris Rogers
> > Digital Projects Manager
> > Bodleian Digital Library Systems and Services
> > chris.rog...@bodleian.ox.ac.uk
>
>

Re: no dataimport-handler defined!

[Chris Rogers]

A quick update. I rolled back to solr 6.2, and the data import handler is 
recognized there.

So there has either been a change in the config required between 6.2 and 6.4, 
or there’s a bug in 6.4

Any thoughts?   

On 24/01/2017, 13:32, "Chris Rogers"  wrote:

Hi Alex,

I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie the 
one generated from the configset when the node was created).

I’m running standalone, not cloud.

I’m restarting sole after every change. Do I need to reload the core 
instead of restarting?

I’ve also tried replacing the relative path to the .jar with an absolute 
path to the dist directory. Still didn’t work.

Thanks,
Chris

On 24/01/2017, 13:20, "Alexandre Rafalovitch"  wrote:

Which solrconfig.xml are you editing and what kind of Solr install are
you running (cloud?). And did you reload the core.

I suspect you are not editing the file that is actually in use. For
example, if you are running a cloud setup, the solrconfig.xml on the
filesystem is disconnected from the config actually in use that is
stored in ZooKeeper. You would need to reupload it for change to take
effect.

You also may need to reload the core for changes to take effect.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and 
experienced


On 24 January 2017 at 07:43, Chris Rogers
 wrote:
> Hi all,
>
> Having frustrating issues with getting SOLR 6.4.0 to recognize the 
existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no 
errors in the SOLR logs. It just says “Sorry, no dataimport-handler defined” in 
the SOLR admin UI.
>
> My config files are listed below. Can anyone spot any mistakes here?
>
> Many thanks,
> Chris
>
> # solrconfig.xml ##
>
>   
>
> …
>
>   
> 
>   DIH-data-config.xml
> 
>   
>
> # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
>
> 
>   
>   
> 
>  fileName=".*xml"
> newerThan="'NOW-5YEARS'"
> recursive="true"
> rootEntity="false"
> dataSource="null"
> 
baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
>
>   
>
>  forEach="/TEI" url="${f.fileAbsolutePath}" 
transformer="RegexTransformer" >
> 
> 
> 
>   
>
> 
>
>   
> 
>
>
> --
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk

Re: no dataimport-handler defined!

[Chris Rogers]

Hi Alex,

I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie the one 
generated from the configset when the node was created).

I’m running standalone, not cloud.

I’m restarting sole after every change. Do I need to reload the core instead of 
restarting?

I’ve also tried replacing the relative path to the .jar with an absolute path 
to the dist directory. Still didn’t work.

Thanks,
Chris

On 24/01/2017, 13:20, "Alexandre Rafalovitch"  wrote:

Which solrconfig.xml are you editing and what kind of Solr install are
you running (cloud?). And did you reload the core.

I suspect you are not editing the file that is actually in use. For
example, if you are running a cloud setup, the solrconfig.xml on the
filesystem is disconnected from the config actually in use that is
stored in ZooKeeper. You would need to reupload it for change to take
effect.

You also may need to reload the core for changes to take effect.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 07:43, Chris Rogers
 wrote:
> Hi all,
>
> Having frustrating issues with getting SOLR 6.4.0 to recognize the 
existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no errors in 
the SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
admin UI.
>
> My config files are listed below. Can anyone spot any mistakes here?
>
> Many thanks,
> Chris
>
> # solrconfig.xml ##
>
>   
>
> …
>
>   
> 
>   DIH-data-config.xml
> 
>   
>
> # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
>
> 
>   
>   
> 
>  fileName=".*xml"
> newerThan="'NOW-5YEARS'"
> recursive="true"
> rootEntity="false"
> dataSource="null"
> 
baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
>
>   
>
>  forEach="/TEI" url="${f.fileAbsolutePath}" 
transformer="RegexTransformer" >
> 
> 
> 
>   
>
> 
>
>   
> 
>
>
> --
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk

Re: no dataimport-handler defined!

[Alexandre Rafalovitch]

Which solrconfig.xml are you editing and what kind of Solr install are
you running (cloud?). And did you reload the core.

I suspect you are not editing the file that is actually in use. For
example, if you are running a cloud setup, the solrconfig.xml on the
filesystem is disconnected from the config actually in use that is
stored in ZooKeeper. You would need to reupload it for change to take
effect.

You also may need to reload the core for changes to take effect.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 07:43, Chris Rogers
 wrote:
> Hi all,
>
> Having frustrating issues with getting SOLR 6.4.0 to recognize the existence 
> of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no errors in the 
> SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
> admin UI.
>
> My config files are listed below. Can anyone spot any mistakes here?
>
> Many thanks,
> Chris
>
> # solrconfig.xml ##
>
>regex=".*dataimporthandler-.*\.jar" />
>
> …
>
>class="org.apache.solr.handler.dataimport.DataImportHandler">
> 
>   DIH-data-config.xml
> 
>   
>
> # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
>
> 
>   
>   
> 
>  fileName=".*xml"
> newerThan="'NOW-5YEARS'"
> recursive="true"
> rootEntity="false"
> dataSource="null"
> 
> baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
>
>   
>
>  forEach="/TEI" url="${f.fileAbsolutePath}" 
> transformer="RegexTransformer" >
>  xpath="/TEI/teiHeader/fileDesc/titleStmt/title"/>
>  xpath="/TEI/teiHeader/fileDesc/publicationStmt/publisher"/>
>  xpath="/TEI/teiHeader/fileDesc/sourceDesc/msDesc/msIdentifier/altIdentifier/idno"/>
>   
>
> 
>
>   
> 
>
>
> --
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk

no dataimport-handler defined!

[Chris Rogers]

Hi all,

Having frustrating issues with getting SOLR 6.4.0 to recognize the existence of 
my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.

The DIH .jar file appears to be loading correctly. There are no errors in the 
SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
admin UI.

My config files are listed below. Can anyone spot any mistakes here?

Many thanks,
Chris

# solrconfig.xml ##

  

…

  

  DIH-data-config.xml

  

# DIH-data-config.xml (in the same dir as solrconfig.xml) ##


  
  



  

  



  



  



--
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rog...@bodleian.ox.ac.uk

Re: trying DIH but get 'Sorry, no dataimport-handler defined!'

[scott.chu]

I happen to find the problem. The problem seems to come from the html file that 
shows DIH function page. I use Maxthon browser, it has a function that can 
switch between IE mode and non-IE mode (actually the Webkit engine). I happen 
to switch back to non-IE mode and the error message is gone and everything is 
ok now! TOO WEIRD!


scott.chu，scott@udngroup.com
2016/5/24 (週二)
- Original Message - 
From: kostali hassan 
To: solr-user ; scott(自己) 
CC: 
Date: 2016/5/24 (週二) 16:55
Subject: Re: trying DIH but get 'Sorry, no dataimport-handler defined!'


if you have in this path server/solr/configsets/testdih/conf you shoud 
right this in your line commande: 
'bin\solr>solr create -c your_core -d testdih -p 8983 to create a core with 
an exemple config testdih. 

2016-05-24 9:35 GMT+01:00 scott.chu : 

> 
> I do following things: 
> 
> * I create folder : D:\solr-6.0.0\myconfigsets\testdih. 
> * Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to 
> D:\solr-6.0.0\myconfigsets\testdih. 
> * Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit 
> db-data-config.xml as follows (I am pretty sure mysql environment is ok): 
> 
>  
>  url="jdbc:mysql://localhost:3306/test" user="hello" password="hellothere" /> 
>  
>  

>  
>  
>  
>  
>  
>  
>  
> 
> * Then I copy mysql-connector-java-5.0.8-bin.jar to 
> D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib. 
> * I check solrconfig.xml and see these relevant lines: 
> 
>  regex="solr-dataimporthandler-.*\.jar" /> 
> ... 
> ... 
>  
>  
> db-data-config.xml 
>  
>  
> 
> * cd to D:solr-6.0.0, issue 'bin\solr start', it starts ok. 
> * Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to 
> create a core. It's ok, too. 
> 
> * The solr.log has these log messages: 
> 
> 2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
> o.a.s.c.SolrResourceLoader Adding 
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to 
> classloader 
> 2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
> o.a.s.c.SolrResourceLoader Adding 
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar'
>  
> to classloader 
> 
> * So I think dih jars are loaded ok. 
> 
> I go to localhost:893 in browser and select core 'testdih', then click 
> 'DataImport' item but rightpane shows "Sorry, no dataimport-handler 
> defined!". 
> 
> What do I miss? 
> 
> 
> scott.chu，scott@udngroup.com 
> 2016/5/24 (週二) 
> 



- 
未在此訊息中找到病毒。 
已透過 AVG 檢查 - www.avg.com 
版本: 2015.0.6201 / 病毒庫: 4568/12285 - 發佈日期: 05/23/16

Re: trying DIH but get 'Sorry, no dataimport-handler defined!'

[kostali hassan]

if you have in  this path server/solr/configsets/testdih/conf you shoud
right this in your line commande:
'bin\solr>solr create -c your_core -d testdih -p 8983 to create a core with
an exemple config testdih.

2016-05-24 9:35 GMT+01:00 scott.chu :

>
> I do following things:
>
> * I create folder : D:\solr-6.0.0\myconfigsets\testdih.
> * Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to
> D:\solr-6.0.0\myconfigsets\testdih.
> * Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit
> db-data-config.xml as follows (I am pretty sure mysql environment is ok):
>
>   
>url="jdbc:mysql://localhost:3306/test" user="hello" password="hellothere" />
>   
>   
>   
>   
>   
>   
>   
>   
>   
>
> * Then I copy mysql-connector-java-5.0.8-bin.jar to
> D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib.
> * I check solrconfig.xml  and see these relevant lines:
>
>  regex="solr-dataimporthandler-.*\.jar" />
>   ...
>   ...
>   
>   
> db-data-config.xml
>   
> 
>
> * cd to  D:solr-6.0.0, issue 'bin\solr start', it starts ok.
> * Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to
> create a core. It's ok, too.
>
> * The solr.log has these log messages:
>
> 2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ]
> o.a.s.c.SolrResourceLoader Adding
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to
> classloader
> 2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ]
> o.a.s.c.SolrResourceLoader Adding
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar'
> to classloader
>
> * So I think dih jars are loaded ok.
>
> I go to localhost:893 in browser and select core 'testdih', then click
> 'DataImport' item but rightpane shows "Sorry, no dataimport-handler
> defined!".
>
>  What do I miss?
>
>
> scott.chu，scott@udngroup.com
> 2016/5/24 (週二)
>

Re: trying DIH but get 'Sorry, no dataimport-handler defined!'

[scott.chu]

I try run the example by issuing "bin\solr create_core -c exampledih -d 
example\example-DIH\solr\db\conf". It also shows same error. Do I issue wrong 
command?

scott.chu，scott@udngroup.com
2016/5/24 (週二)
- Original Message - 
From: scott(自己) 
To: solr-user 
CC: 
Date: 2016/5/24 (週二) 16:35
Subject: trying DIH but get 'Sorry, no dataimport-handler defined!'



I do following things: 

* I create folder : D:\solr-6.0.0\myconfigsets\testdih. 
* Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to 
D:\solr-6.0.0\myconfigsets\testdih. 
* Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit db-data-config.xml 
as follows (I am pretty sure mysql environment is ok): 

   
   
   
   
   
   
   
   
   
   
   

* Then I copy mysql-connector-java-5.0.8-bin.jar to 
D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib. 
* I check solrconfig.xml and see these relevant lines: 

 
  ... 
  ... 
   
   
db-data-config.xml 
   
 

* cd to D:solr-6.0.0, issue 'bin\solr start', it starts ok. 
* Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to 
create a core. It's ok, too. 

* The solr.log has these log messages: 

2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to 
classloader 
2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar' 
to classloader 

* So I think dih jars are loaded ok. 

I go to localhost:893 in browser and select core 'testdih', then click 
'DataImport' item but rightpane shows "Sorry, no dataimport-handler defined!". 

 What do I miss? 


scott.chu，scott@udngroup.com 
2016/5/24 (週二) 


- 
未在此訊息中找到病毒。 
已透過 AVG 檢查 - www.avg.com 
版本: 2015.0.6201 / 病毒庫: 4568/12285 - 發佈日期: 05/23/16

trying DIH but get 'Sorry, no dataimport-handler defined!'

[scott.chu]

I do following things:

* I create folder : D:\solr-6.0.0\myconfigsets\testdih.
* Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to 
D:\solr-6.0.0\myconfigsets\testdih.
* Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit db-data-config.xml 
as follows (I am pretty sure mysql environment is ok):

  
  
  
  
  
  
  
  
  
  
  

* Then I copy mysql-connector-java-5.0.8-bin.jar to 
D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib.
* I check solrconfig.xml  and see these relevant lines:


  ...
  ...
  
  
db-data-config.xml
  


* cd to  D:solr-6.0.0, issue 'bin\solr start', it starts ok.
* Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to 
create a core. It's ok, too.

* The solr.log has these log messages:

2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to 
classloader
2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar' 
to classloader

* So I think dih jars are loaded ok.

I go to localhost:893 in browser and select core 'testdih', then click 
'DataImport' item but rightpane shows "Sorry, no dataimport-handler defined!".

 What do I miss?


scott.chu，scott@udngroup.com
2016/5/24 (週二)

Re: sorry, no dataimport-handler defined!

[kostali hassan]

in request data import handler for solrconfig.xml do :



  tika-data-config.xml

  

and define your file tika-data-config.xml and put this file in the
directory config from your core.

2016-02-02 17:35 GMT+00:00 Jean-Jacques Monot :

> Exact. Newbie user !
>
> OK i have seen what is missing ...
>
> Le 2 févr. 2016 15:40, "Davis, Daniel (NIH/NLM) [C]" 
> a écrit :
> >
> > It sounds a bit like you are just exploring Solr for the first time.
> To use the Data Import Handler, you need to create an XML file that
> configures it, data-config.xml by default.
> >
> > But before we go into details, what are you trying to accomplish with
> Solr?
> >
> > -Original Message-
> > From: Jean-Jacques MONOT [mailto:jj_mo...@yahoo.fr]
> > Sent: Monday, February 01, 2016 2:31 PM
> > To: solr-user@lucene.apache.org
> > Subject: Potential SPAM:sorry, no dataimport-handler defined!
> >
> > Hello
> >
> > I am using SOLR 5.4.1 and the graphical admin UI.
> >
> > I successfully created multiples cores and indexed various documents,
> using in line commands : (create -c) and (post.jar) on W10.
> >
> > But in the GUI, when I click on "Dataimport", I get the following
> message : "sorry, no dataimport-handler defined!"
> >
> > I get the same message even on 5.3.1 or for different cores.
> >
> > What is wrong ?
> >
> > JJM
> >
> > ---
> > L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> > https://www.avast.com/antivirus
> >
> >
>

Re: sorry, no dataimport-handler defined!

[Jean-Jacques Monot]

Exact. Newbie user !

OK i have seen what is missing ...

Le 2 févr. 2016 15:40, "Davis, Daniel (NIH/NLM) [C]"  a 
écrit :
>
> It sounds a bit like you are just exploring Solr for the first time.   To use 
> the Data Import Handler, you need to create an XML file that configures it, 
> data-config.xml by default. 
>
> But before we go into details, what are you trying to accomplish with Solr? 
>
> -Original Message- 
> From: Jean-Jacques MONOT [mailto:jj_mo...@yahoo.fr] 
> Sent: Monday, February 01, 2016 2:31 PM 
> To: solr-user@lucene.apache.org 
> Subject: Potential SPAM:sorry, no dataimport-handler defined! 
>
> Hello 
>
> I am using SOLR 5.4.1 and the graphical admin UI. 
>
> I successfully created multiples cores and indexed various documents, using 
> in line commands : (create -c) and (post.jar) on W10. 
>
> But in the GUI, when I click on "Dataimport", I get the following message : 
> "sorry, no dataimport-handler defined!" 
>
> I get the same message even on 5.3.1 or for different cores. 
>
> What is wrong ? 
>
> JJM 
>
> --- 
> L'absence de virus dans ce courrier électronique a été vérifiée par le 
> logiciel antivirus Avast. 
> https://www.avast.com/antivirus 
>
>

sorry, no dataimport-handler defined!

[Davis, Daniel (NIH/NLM) [C]]

It sounds a bit like you are just exploring Solr for the first time.   To use 
the Data Import Handler, you need to create an XML file that configures it, 
data-config.xml by default.

But before we go into details, what are you trying to accomplish with Solr?

-Original Message-
From: Jean-Jacques MONOT [mailto:jj_mo...@yahoo.fr] 
Sent: Monday, February 01, 2016 2:31 PM
To: solr-user@lucene.apache.org
Subject: Potential SPAM:sorry, no dataimport-handler defined!

Hello

I am using SOLR 5.4.1 and the graphical admin UI.

I successfully created multiples cores and indexed various documents, using in 
line commands : (create -c) and (post.jar) on W10.

But in the GUI, when I click on "Dataimport", I get the following message : 
"sorry, no dataimport-handler defined!"

I get the same message even on 5.3.1 or for different cores.

What is wrong ?

JJM

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

Re: sorry, no dataimport-handler defined!

[Susheel Kumar]

Please register Data Import Handler to work with it
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler


On Mon, Feb 1, 2016 at 2:31 PM, Jean-Jacques MONOT 
wrote:

> Hello
>
> I am using SOLR 5.4.1 and the graphical admin UI.
>
> I successfully created multiples cores and indexed various documents,
> using in line commands : (create -c) and (post.jar) on W10.
>
> But in the GUI, when I click on "Dataimport", I get the following message
> : "sorry, no dataimport-handler defined!"
>
> I get the same message even on 5.3.1 or for different cores.
>
> What is wrong ?
>
> JJM
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
>

sorry, no dataimport-handler defined!

[Jean-Jacques MONOT]

Hello

I am using SOLR 5.4.1 and the graphical admin UI.

I successfully created multiples cores and indexed various documents,
using in line commands : (create -c) and (post.jar) on W10.

But in the GUI, when I click on "Dataimport", I get the following
message : "sorry, no dataimport-handler defined!"

I get the same message even on 5.3.1 or for different cores.

What is wrong ?

JJM

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

Re: exclude folder in dataimport handler.

[coolmals]

I used this to exclude files from folders of templatedata. But it still
couldnt remove these files from indexing 

  

When I save the value of this expression in temp variable, all files in
templatedata have value true but still they are not skipped to index.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/exclude-folder-in-dataimport-handler-tp4224267p4224920.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: exclude folder in dataimport handler.

[Alexandre Rafalovitch]

A transformer on the outer entity will run before the inner entity is
invoked. So, you might be able to remove the list of files to ignore
before the inner entity starts extracting from them.

You could also pre-generate a list of files by doing ls/find with your
requirements and then just read that file directly with DIH.

Regards,
  Alex.

Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/


On 21 August 2015 at 12:23, coolmals  wrote:
> Can you give me a brief description on how to use a transformer to excludes
> docs from a folder?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/exclude-folder-in-dataimport-handler-tp4224267p4224489.html
> Sent from the Solr - User mailing list archive at Nabble.com.

RE: exclude folder in dataimport handler.

[coolmals]

Can you give me a brief description on how to use a transformer to excludes
docs from a folder?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/exclude-folder-in-dataimport-handler-tp4224267p4224489.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: exclude folder in dataimport handler.

[Dyer, James]

I took a quick look at FileListEntityProcessor#init, and it looks like it 
applies the "excludes" regex to the filename element of the path only, and not 
to the directories.

If your filenames do not have a naming convention that would let you use it 
this way, you might be able to write a transformer to get what you want.

James Dyer
Ingram Content Group


-Original Message-
From: coolmals [mailto:coolm...@gmail.com] 
Sent: Thursday, August 20, 2015 12:57 PM
To: solr-user@lucene.apache.org
Subject: exclude folder in dataimport handler.

I am importing files from my file system and want to exclude import of files
from folder called templatedata. How do i configure that in entity. 
excludes="templatedata" doesnt seem to work.

 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/exclude-folder-in-dataimport-handler-tp4224267.html
Sent from the Solr - User mailing list archive at Nabble.com.

exclude folder in dataimport handler.

[coolmals]

I am importing files from my file system and want to exclude import of files
from folder called templatedata. How do i configure that in entity. 
excludes="templatedata" doesnt seem to work.

 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/exclude-folder-in-dataimport-handler-tp4224267.html
Sent from the Solr - User mailing list archive at Nabble.com.

SOLR 5.1.0 DB dataimport handler from orientdb

[Nauman Ramzan]

Hi everyone !
I want to import data from orientdb in solr 5.1.0.
here is my configurations

*data-config.xml*

> 
>
> > driver="com.orientechnologies.orient.jdbc.OrientJdbcDriver"
>> url="jdbc:orient:remote:localhost/emallates_combine" user="root"
>> password="root" batchSize="-1"/>
>
> 
>
> 
> deltaQuery="select * from sellings where updatedAt >
>> '${dataimporter.last_index_time}'">
>
> 
>
> 
>
> 
>
> 
>
> 
>
>
*JDBC* driver link http://orientdb.com/download/
and I paste this driver in* {solr_install_dir}/dist/orientdb-jdbc-2.0.5.jar*

my configuration is not showing any error nor any output.
here is log of solr after full/delta import call.

INFO  - 2015-06-29 12:37:24.894; [   DB]
> org.apache.solr.handler.dataimport.DataImporter; Loading DIH Configuration:
> db-data-config.xml

INFO  - 2015-06-29 12:37:24.899; [   DB]
> org.apache.solr.handler.dataimport.DataImporter; Data Configuration loaded
> successfully

INFO  - 2015-06-29 12:37:24.900; [   DB] org.apache.solr.core.SolrCore;
> [DB] webapp=/solr path=/dataimport
> params={debug=false&optimize=false&indent=true&commit=true&clean=true&wt=json&command=full-import&verbose=false}
> status=0 QTime=7

INFO  - 2015-06-29 12:37:24.902; [   DB]
> org.apache.solr.handler.dataimport.DataImporter; Starting Full Import

*WARN  - 2015-06-29 12:37:24.912; [   DB]
> org.apache.solr.handler.dataimport.SimplePropertiesWriter; Unable to read:
> dataimport.properties*

INFO  - 2015-06-29 12:37:24.914; [   DB] org.apache.solr.core.SolrCore;
> [DB] webapp=/solr path=/dataimport
> params={indent=true&wt=json&command=status&_=1435567044911} status=0
> QTime=1

INFO  - 2015-06-29 12:37:24.942; [   DB]
> org.apache.solr.handler.dataimport.JdbcDataSource$1; Creating a connection
> for entity item with URL: jdbc:orient:remote:localhost/emallates_combine

INFO  - 2015-06-29 12:37:24.942; [   DB]
> org.apache.solr.update.processor.LogUpdateProcessor; [DB] webapp=/solr
> path=/dataimport
> params={debug=false&optimize=false&indent=true&commit=true&clean=true&wt=json&command=full-import&verbose=false}
> status=0 QTime=7 {deleteByQuery=*:* (-1505301149686693888)} 0 49

INFO  - 2015-06-29 12:37:32.992; [   DB] org.apache.solr.core.SolrCore;
> [DB] webapp=/solr path=/dataimport
> params={wt=json&command=abort&_=1435567052987} status=0 QTime=1

INFO  - 2015-06-29 12:37:33.000; [   DB] org.apache.solr.core.SolrCore;
> [DB] webapp=/solr path=/dataimport
> params={indent=true&wt=json&command=status&_=1435567052997} status=0
> QTime=0


SOLR is not importing any data. What am I doing wrong ?
and second why am I getting above warming ?

Thank you
Regards
Nauman Ramzan

Re: Problem with numeric "math" types and the dataimport handler

[Shawn Heisey]

On 5/26/2015 2:37 PM, Shawn Heisey wrote:
> On 5/20/2015 12:06 AM, Shalin Shekhar Mangar wrote:
>> Sounds similar to https://issues.apache.org/jira/browse/SOLR-6165 which I
>> fixed in 4.10. Can you try a newer release?
> Looks like that didn't fix it.
>
> I applied the patch on SOLR-6165 to the lucene_solr_4_9_1 tag, built a
> new war, and when it was done, restarted Solr with that war.  The
> solr-impl version in the dashboard is now
>
> 4.9-SNAPSHOT 1680667 - solr - 2015-05-20 14:23:11
>
> After some importing with DIH and a Solr restart, this is the most
> recent error in the log:
>
> WARN  - 2015-05-26 14:28:09.289;
> org.apache.solr.update.UpdateLog$LogReplayer; REYPLAY_ERR: IOException
> reading log org.apache.solr.common.SolrException: ERROR:
> [doc=usatphotos084190] Error adding field
> 'did'='java.math.BigInteger:1214221' msg=For input string:
> "java.math.BigInteger:1214221"
>
> Looks like we'll need a new issue.  I'm not in a position right now to
> try a newer Solr version than 4.9.1.

Given the way that I use Solr, this is honestly not really a major
problem for me.  Within five minutes or so after DIH is done, my
transaction logs will only contain data indexed via SolrJ, so this
problem will be gone.

The reason I think it's worth fixing, assuming it's still a problem in
5.2: There are people that use DIH *exclusively* for indexing, and for
those people, this could become a real problem, because tlog replay
won't work.

Thanks,
Shawn

Re: Problem with numeric "math" types and the dataimport handler

[Shawn Heisey]

On 5/20/2015 12:06 AM, Shalin Shekhar Mangar wrote:
> Sounds similar to https://issues.apache.org/jira/browse/SOLR-6165 which I
> fixed in 4.10. Can you try a newer release?

Looks like that didn't fix it.

I applied the patch on SOLR-6165 to the lucene_solr_4_9_1 tag, built a
new war, and when it was done, restarted Solr with that war.  The
solr-impl version in the dashboard is now


4.9-SNAPSHOT 1680667 - solr - 2015-05-20 14:23:11

After some importing with DIH and a Solr restart, this is the most
recent error in the log:

WARN  - 2015-05-26 14:28:09.289;
org.apache.solr.update.UpdateLog$LogReplayer; REYPLAY_ERR: IOException
reading log org.apache.solr.common.SolrException: ERROR:
[doc=usatphotos084190] Error adding field
'did'='java.math.BigInteger:1214221' msg=For input string:
"java.math.BigInteger:1214221"

Looks like we'll need a new issue.  I'm not in a position right now to
try a newer Solr version than 4.9.1.

Thanks,
Shawn

Re: Problem with numeric "math" types and the dataimport handler

[Shawn Heisey]

On 5/20/2015 12:06 AM, Shalin Shekhar Mangar wrote:
> Sounds similar to https://issues.apache.org/jira/browse/SOLR-6165 which I
> fixed in 4.10. Can you try a newer release?

I can't upgrade yet.  I am using a plugin that hasn't been verified
against anything newer than 4.9.  When a new version becomes available,
I will begin testing 5.x.

The patch does look like it will fix the issue perfectly ... so I am
very likely to patch 4.9.1 and build a custom war.

Thanks,
Shawn

Re: Problem with numeric "math" types and the dataimport handler

[Shalin Shekhar Mangar]

Sounds similar to https://issues.apache.org/jira/browse/SOLR-6165 which I
fixed in 4.10. Can you try a newer release?

On Wed, May 20, 2015 at 6:51 AM, Shawn Heisey  wrote:

> An unusual problem is happening with the DIH on a field that is an
> unsigned BIGINT in the MySQL database.  This is Solr 4.9.1 without
> SolrCloud, running on OpenJDK 7u79.
>
> During actual import, everything is fine.  The problem comes when I
> restart Solr and the transaction logs are replayed.  I get the following
> exception for every document replayed:
>
> WARN  - 2015-05-19 18:52:44.461;
> org.apache.solr.update.UpdateLog$LogReplayer; REYPLAY_ERR: IOException
> reading log
> org.apache.solr.common.SolrException: ERROR: [doc=getty26025060] Error
> adding field 'file_size'='java.math.BigInteger:5934053' msg=For input
> string: "java.math.BigInteger:5934053"
>
> I believe I need one of two things to solve this problem:
>
> 1) A connection parameter for the MySQL JDBC driver that will force the
> use of java.lang.* objects and exclude the java.math.* classes.
>
> 2) Write the actual imported value into the transaction log rather than
> include the class name in the string representation.  Testing shows that
> the toString() method on BigInteger does *NOT* include the class name,
> so I am confused about why the class name is being recorded in the
> transaction log.
>
> For the first solution, I've been looking for a MySQL connection
> parameter to change the Java object types that get used, but so far I
> haven't found one.  For the second, I should probably open an issue in
> Jira, but I wanted to run it by everyone before taking that step.
>
> I have another index (building from a different database) where this
> isn't happening, because the MySQL column is *NOT* unsigned, which
> causes the JDBC driver to use java.lang.Long instead of
> java.math.BigInteger.
>
> Thanks,
> Shawn
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Problem with numeric "math" types and the dataimport handler

[Shawn Heisey]

An unusual problem is happening with the DIH on a field that is an
unsigned BIGINT in the MySQL database.  This is Solr 4.9.1 without
SolrCloud, running on OpenJDK 7u79.

During actual import, everything is fine.  The problem comes when I
restart Solr and the transaction logs are replayed.  I get the following
exception for every document replayed:

WARN  - 2015-05-19 18:52:44.461;
org.apache.solr.update.UpdateLog$LogReplayer; REYPLAY_ERR: IOException
reading log
org.apache.solr.common.SolrException: ERROR: [doc=getty26025060] Error
adding field 'file_size'='java.math.BigInteger:5934053' msg=For input
string: "java.math.BigInteger:5934053"

I believe I need one of two things to solve this problem:

1) A connection parameter for the MySQL JDBC driver that will force the
use of java.lang.* objects and exclude the java.math.* classes.

2) Write the actual imported value into the transaction log rather than
include the class name in the string representation.  Testing shows that
the toString() method on BigInteger does *NOT* include the class name,
so I am confused about why the class name is being recorded in the
transaction log.

For the first solution, I've been looking for a MySQL connection
parameter to change the Java object types that get used, but so far I
haven't found one.  For the second, I should probably open an issue in
Jira, but I wanted to run it by everyone before taking that step.

I have another index (building from a different database) where this
isn't happening, because the MySQL column is *NOT* unsigned, which
causes the JDBC driver to use java.lang.Long instead of
java.math.BigInteger.

Thanks,
Shawn

Re: Solr error : sorry, no dataimport-handler defined!

[Tim Dunphy]

s, so they are being ignored. But
> still very very wrong. Take those out all together.
>
> You should just have 3 things tying together:
> 1) jars loaded in the lib statement in solrconfig.xml
> 2) handler definition that points at your data-config file
> 3) data-config file itself.
>
> If you are still having troubles, I strongly recommend getting the
> shipped example to work and then adding your own stuff until you get
> that working. Then, try to create a standalone configuration.
> Sometimes, this is an easier approach for the first time user.
>
> Regards,
>Alex.
> P.s. I also cover that in my Solr book. A relevant example is here:
> https://github.com/arafalov/solr-indexing-book/tree/master/published/dihdb
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 3 November 2014 13:40, Tim Dunphy  wrote:
> > Hi Alexandre,
> >
> >  Thanks so much for your input and examples! Ok so here's what I've done
> so
> > far with no luck as of yet unfortunately.
> >
> >   Inside of solrconfig.xml I put the following:
> >
> >   
> >   
> >   
> >
> >   
> >   
> >   
> >   
> >
> >   
> >   
> >
> > As you can see, I've replaced the relative paths with absolute ones. So
> as
> > of now, my solr 4 server is no longer complaining about not being able to
> > find directories and modules. So we're off to a good start! And now I can
> > list the 'dist' directory and in my case find the jar files I'm looking
> for.
> >
> >
> > [root@solr1:/opt/solr/collection1/conf] #ls /opt/solr/dist/ | grep
> > dataimporthandler
> > solr-dataimporthandler-4.10.1.jar
> > solr-dataimporthandler-extras-4.10.1.jar
> >
> > So far so good.
> >
> > I next tried this db-data-config file in the same directory as
> > solrconfig.xml
> >
> > [root@solr1:/opt/solr/collection1/conf] #cat db-data-config.xml.bak
> > 
> >
> > 
> >
> >  > url="jdbc:mysql://web1.mydomain.com:3306/jokefire" user="admin"
> > password="secret" batchSize="1" />
> > 
> >
> >     
> > 
> > 
> > 
> > 
> > 
> > 
> >  name="user_activation_key"
> > />
> > 
> > 
> > 
> >
> > 
> > 
> > 
> >
> > Restarted tomcat, and with this setup I wasn't getting any errors in the
> > browser or logs and the web interface was still working. Always a good
> sign!
> >
> > So then I went down to Core Selector -> collection1 -> data import. And
> it
> > was quite frustrating, cuz I was getting the same error as before!
> >
> >  sorry, no dataimport-handler defined!
> >
> > So then I tried the exact db-data-config.xml file from your example.
> >
> > 
> >  > url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
> > 
> >  > deltaQuery="select id from item where last_modified >
> > '${dataimporter.last_index_time}'">
> > 
> >
> >  > query="select DESCRIPTION from FEATURE where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID from FEATURE where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${feature.ITEM_ID}">
> > 
> > 
> >
> >  > query="select CATEGORY_ID from item_category where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${item_category.ITEM_ID}">
> >  > query="select DESCRIPTION from category where ID
> =
> > '${item_category.CATEGORY_ID}'"
> > deltaQuery="select ID from category where
> > last_modified > '${dataimporter.last_index_time}'"

Re: Solr error : sorry, no dataimport-handler defined!

[Alexandre Rafalovitch]

Two problems:
1) You have  (span) elements in your solrconfig.xml. They just
do not belong there. The original tutorial screwed up. Your element
should be on the same level as the other elements in that example.
2) You also seem to have another random piece of data configuration in
the solrconfig.xml. Also in the spans, so they are being ignored. But
still very very wrong. Take those out all together.

You should just have 3 things tying together:
1) jars loaded in the lib statement in solrconfig.xml
2) handler definition that points at your data-config file
3) data-config file itself.

If you are still having troubles, I strongly recommend getting the
shipped example to work and then adding your own stuff until you get
that working. Then, try to create a standalone configuration.
Sometimes, this is an easier approach for the first time user.

Regards,
   Alex.
P.s. I also cover that in my Solr book. A relevant example is here:
https://github.com/arafalov/solr-indexing-book/tree/master/published/dihdb
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 3 November 2014 13:40, Tim Dunphy  wrote:
> Hi Alexandre,
>
>  Thanks so much for your input and examples! Ok so here's what I've done so
> far with no luck as of yet unfortunately.
>
>   Inside of solrconfig.xml I put the following:
>
>   
>   
>   
>
>   
>   
>   
>   
>
>   
>   
>
> As you can see, I've replaced the relative paths with absolute ones. So as
> of now, my solr 4 server is no longer complaining about not being able to
> find directories and modules. So we're off to a good start! And now I can
> list the 'dist' directory and in my case find the jar files I'm looking for.
>
>
> [root@solr1:/opt/solr/collection1/conf] #ls /opt/solr/dist/ | grep
> dataimporthandler
> solr-dataimporthandler-4.10.1.jar
> solr-dataimporthandler-extras-4.10.1.jar
>
> So far so good.
>
> I next tried this db-data-config file in the same directory as
> solrconfig.xml
>
> [root@solr1:/opt/solr/collection1/conf] #cat db-data-config.xml.bak
> 
>
> 
>
>  url="jdbc:mysql://web1.mydomain.com:3306/jokefire" user="admin"
> password="secret" batchSize="1" />
> 
>
> 
> 
> 
> 
> 
> 
> 
>  />
> 
> 
> 
>
> 
> 
> 
>
> Restarted tomcat, and with this setup I wasn't getting any errors in the
> browser or logs and the web interface was still working. Always a good sign!
>
> So then I went down to Core Selector -> collection1 -> data import. And it
> was quite frustrating, cuz I was getting the same error as before!
>
>  sorry, no dataimport-handler defined!
>
> So then I tried the exact db-data-config.xml file from your example.
>
> 
>  url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
> 
>  deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'">
> 
>
>  query="select DESCRIPTION from FEATURE where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID from FEATURE where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${feature.ITEM_ID}">
> 
> 
>
>  query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
>  query="select DESCRIPTION from category where ID =
> '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}">
> 
> 
> 
> 
> 
> 
>
> Knowing full well it wouldn't actually work, as I"m using a remote mysql
> database instead of a local 

Re: Solr error : sorry, no dataimport-handler defined!

[Tim Dunphy]

Hi Alexandre,

 Thanks so much for your input and examples! Ok so here's what I've done so
far with no luck as of yet unfortunately.

  Inside of solrconfig.xml I put the following:

  **
  
  

  
  
  
  

  
  

As you can see, I've replaced the relative paths with absolute ones. So as
of now, my solr 4 server is no longer complaining about not being able to
find directories and modules. So we're off to a good start! And now I can
list the 'dist' directory and in my case find the jar files I'm looking for.


[root@solr1:/opt/solr/collection1/conf] #ls /opt/solr/dist/ | grep
dataimporthandler
*solr-dataimporthandler-4.10.1.jar*
*solr-dataimporthandler-extras-4.10.1.jar*

So far so good.

I next tried this db-data-config file in the same directory as
solrconfig.xml

[root@solr1:/opt/solr/collection1/conf] #cat db-data-config.xml.bak























Restarted tomcat, and with this setup I wasn't getting any errors in the
browser or logs and the web interface was still working. Always a good sign!

So then I went down to Core Selector -> collection1 -> data import. And it
was quite frustrating, cuz I was getting the same error as before!

 *sorry, no dataimport-handler defined!*

So then I tried the exact db-data-config.xml file from your example.




















Knowing full well it wouldn't actually work, as I"m using a remote mysql
database instead of a local hsqldb database.But at this point, my only goal
was to get the data import to show up as an option. I'd tweak the
db-data-config.xml file at a later point if this in fact worked!

But alas, I was still getting the same result...

*sorry, no dataimport-handler defined!*

G.. so annoying after all that work. Anyway, I really do appreciate
your kindness and help. :) I'm enclosing my solrconfig.xml and both
versions of my db-data-config.xml in hopes that we can make some progress
here!


Thank


Tim









On Sun, Nov 2, 2014 at 9:50 PM, Alexandre Rafalovitch 
wrote:

> That tutorial seems to be somewhat dodgy. You need at least one more
> step of adding DIH library in solrconfig.xml:
>
> https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/solrconfig.xml#L75
> (I recommend using absolute path though).
>
> Also, you should not need to spell the full class out. See lower down
> in the same class:
>
> https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/solrconfig.xml#L823
>
> Finally, in the config file, I don't remember document element having
> a name. Again, the working example can be found in the same directory:
>
> https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/db-data-config.xml#L3
>
> Solr ships with a bunch of examples. If you are using/download
> standard distribution, you could start from those until you understand
> how it all hangs together.
>
> Regards,
>Alex.
>
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 2 November 2014 21:26, Tim Dunphy  wrote:
> > Hi Alex,
> >
> >
> >> I thought the ""
> >> and the ending span were broken email thing but they seem to be in the
> >> solrconfig.xml file as well. I would start from removing those and
> >> leaving just the actual definition.
> >
> >
> > Thanks for your response!
> >
> > OK so I tried your suggestion of removing those span tags like so:
> >
>



-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B





  

  
  4.10.1

  

  
  
  
  

  
  
  
  

  
  

  
  
  
  
  ${solr.data.dir:}


  
  

 
  
 
${solr.hdfs.home:}

${solr.hdfs.confdir:}

${solr.hdfs.blockcache.enabled:true}

${solr.hdfs.blockcache.global:true}

   

  
  

  
  

  
  
















   







${solr.lock.type:native}












  
  
  
  
  
  



 true


 false
  


  
  
  
  
  
  

  
  

 

  ${solr.ulog.dir:}

 

  
   ${solr.autoCommit.maxTime:15000} 
   false 
 



  
   ${solr.autoSoftCommit.maxTime:-1} 
 






 

Re: Solr error : sorry, no dataimport-handler defined!

[Alexandre Rafalovitch]

That tutorial seems to be somewhat dodgy. You need at least one more
step of adding DIH library in solrconfig.xml:
https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/solrconfig.xml#L75
(I recommend using absolute path though).

Also, you should not need to spell the full class out. See lower down
in the same class:
https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/solrconfig.xml#L823

Finally, in the config file, I don't remember document element having
a name. Again, the working example can be found in the same directory:
https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/db-data-config.xml#L3

Solr ships with a bunch of examples. If you are using/download
standard distribution, you could start from those until you understand
how it all hangs together.

Regards,
   Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 2 November 2014 21:26, Tim Dunphy  wrote:
> Hi Alex,
>
>
>> I thought the ""
>> and the ending span were broken email thing but they seem to be in the
>> solrconfig.xml file as well. I would start from removing those and
>> leaving just the actual definition.
>
>
> Thanks for your response!
>
> OK so I tried your suggestion of removing those span tags like so:
>

Re: Solr error : sorry, no dataimport-handler defined!

[Tim Dunphy]

 that, I still don't have the ability to import data
from mysql. :(

Any other ideas?

Thanks,

Tim




On Sun, Nov 2, 2014 at 8:56 PM, Alexandre Rafalovitch 
wrote:

> Well,
>
> I thought the ""
> and the ending span were broken email thing but they seem to be in the
> solrconfig.xml file as well. I would start from removing those and
> leaving just the actual definition.
>
> Regards,
>Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 2 November 2014 20:50, Tim Dunphy  wrote:
> >
> > Hey guys,
> >
> >  I'm real new at working with Solr. But I need to get up to speed and I
> > appreciate your bearing with me.
> >
> >  I've installed solr 4 and am running it under tomcat 7. The install went
> > perfectly fine and everything seems to work, up to a point. I've even
> > automated the installation with puppet which gets everything up and
> running
> > perfectly as well.
> >
> >  However my problem is that I need to be able to import some data from a
> > mysql database.
> >
> >  I've followed this tutorial to try and do this:
> >
> >
> >
> http://www.beingjavaguys.com/2013/01/how-to-use-solr-data-import-handler-to.html
> >
> >
> > I've added a file called data-config.xml to the following location under
> my
> > solr root:
> >
> > [root@solr1:/opt/solr/collection1/conf] #cat data-config.xml
> >
> >  
> >
> >   > url="jdbc:mysql://web1.mydomain.com:3306/mydomain" user="admin"
> > password=“secret” batchSize="1" />
> >
> >
> >
> > 
> >
> > 
> >
> > 
> >
> > 
> >
> >
> >
> >
> >
> >
> >
> > name="user_activation_key" />
> >
> >
> >
> >    
> >
> >
> >
> >
> >
> >
> >
> > 
> >
> > And added the following section to my
> > /opt/solr/collection1/conf/solrconfig.xml
> >
> >> name="/dataimport"
> > class="org.apache.solr.handler.dataimport.DataImportHandler">
> >
> >  
> >
> >data-config.xml
> >
> >
> >
> > 
> >
> >
> >
> > Then restart tomcat. I then navigate to collection1 -> data import in the
> > solr admin interface and see the following response:
> >
> > sorry, no dataimport-handler defined!
> >
> > I am ok with parsing XML with my eyes. I've worked in some big
> environments,
> > where I've had to read it until my eyes bled! hah.. but I am not sure if
> I
> > am placing the section in solrconfig.xml that it needs to be. I'm
> probably
> > missing something obvious since I'm so new at using solr. I'm hoping
> someone
> > with more experience can point me in the right direction.
> >
> > I'm enclosing my solrconfig.xml files and data-config.xml in case someone
> > wants to get a sense of the context that I'm working with.
> >
> > Thanks!
> >
> > --
> > GPG me!!
> >
> > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> >
>



-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: Solr error : sorry, no dataimport-handler defined!

[Alexandre Rafalovitch]

Well,

I thought the ""
and the ending span were broken email thing but they seem to be in the
solrconfig.xml file as well. I would start from removing those and
leaving just the actual definition.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 2 November 2014 20:50, Tim Dunphy  wrote:
>
> Hey guys,
>
>  I'm real new at working with Solr. But I need to get up to speed and I
> appreciate your bearing with me.
>
>  I've installed solr 4 and am running it under tomcat 7. The install went
> perfectly fine and everything seems to work, up to a point. I've even
> automated the installation with puppet which gets everything up and running
> perfectly as well.
>
>  However my problem is that I need to be able to import some data from a
> mysql database.
>
>  I've followed this tutorial to try and do this:
>
>
> http://www.beingjavaguys.com/2013/01/how-to-use-solr-data-import-handler-to.html
>
>
> I've added a file called data-config.xml to the following location under my
> solr root:
>
> [root@solr1:/opt/solr/collection1/conf] #cat data-config.xml
>
>  
>
>   url="jdbc:mysql://web1.mydomain.com:3306/mydomain" user="admin"
> password=“secret” batchSize="1" />
>
>
>
> 
>
> 
>
> 
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 
>
> And added the following section to my
> /opt/solr/collection1/conf/solrconfig.xml
>
>name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
>
>  
>
>data-config.xml
>
>
>
> 
>
>
>
> Then restart tomcat. I then navigate to collection1 -> data import in the
> solr admin interface and see the following response:
>
> sorry, no dataimport-handler defined!
>
> I am ok with parsing XML with my eyes. I've worked in some big environments,
> where I've had to read it until my eyes bled! hah.. but I am not sure if I
> am placing the section in solrconfig.xml that it needs to be. I'm probably
> missing something obvious since I'm so new at using solr. I'm hoping someone
> with more experience can point me in the right direction.
>
> I'm enclosing my solrconfig.xml files and data-config.xml in case someone
> wants to get a sense of the context that I'm working with.
>
> Thanks!
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>

Solr error : sorry, no dataimport-handler defined!

[Tim Dunphy]

Hey guys,

 I'm real new at working with Solr. But I need to get up to speed and I
appreciate your bearing with me.

 I've installed solr 4 and am running it under tomcat 7. The install went
perfectly fine and everything seems to work, up to a point. I've even
automated the installation with puppet which gets everything up and running
perfectly as well.

 However my problem is that I need to be able to import some data from a
mysql database.

 I've followed this tutorial to try and do this:


http://www.beingjavaguys.com/2013/01/how-to-use-solr-data-import-handler-to.html


I've added a file called data-config.xml to the following location under my
solr root:

[root@solr1:*/opt/solr/collection1/con*f] #cat data-config.xml

 

 

   









   

   

   

   

   

   

   

   

   



And added the following section to
my /opt/solr/collection1/conf/solrconfig.xml

  

 

   data-config.xml

   



   

Then restart tomcat. I then navigate to collection1 -> data import in the
solr admin interface and see the following response:

sorry, no dataimport-handler defined!

I am ok with parsing XML with my eyes. I've worked in some big
environments, where I've had to read it until my eyes bled! hah.. but I am
not sure if I am placing the section in solrconfig.xml that it needs to be.
I'm probably missing something obvious since I'm so new at using solr. I'm
hoping someone with more experience can point me in the right direction.

I'm enclosing my solrconfig.xml files and data-config.xml in case someone
wants to get a sense of the context that I'm working with.

Thanks!
-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
 

 
   

        
            
            

   
   
   
   
   
   
       

   
   






  

  
  4.10.1

  

  
  
  

  
  

  
  

  
  

  
  
  
  
  ${solr.data.dir:}


  
  

 
  
 
${solr.hdfs.home:}

${solr.hdfs.confdir:}

${solr.hdfs.blockcache.enabled:true}

${solr.hdfs.blockcache.global:true}

   

  
  

  
  

  
  
















   







${solr.lock.type:native}












  
  
  
  
  
  



 true


 false
  


  
  
  
  
  
  

  
  

 

  ${solr.ulog.dir:}

 

  
   ${solr.autoCommit.maxTime:15000} 
   false 
 



  
   ${solr.autoSoftCommit.maxTime:-1} 
 






  
  
  
  
  
  

  
  

1024









   



 










true

   
   

   
   20

   
   200

   


  

  


  

  static firstSearcher warming in solrconfig.xml

  



false


2

  


  
  
 








  

  
  
  

 
   explicit
   10
   text
 









  
  
 
   explicit
   json
   true
   text
 
  

     

   
   data-config.xml



   


  
  
 
   true
   json
   true
 
  

  

  

  {!xport}
  xsort
  false



  query

  






  
  
 
   explicit

   
   velocity
   browse
   layout
   Solritas

   
   edismax
   
  text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
  title^10.0 description^5.0 keywords^5.0 author^2.0 resourcename^1.0
   
   text
   100%
   *:*
   10
   *,score

   
 text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
 title^10.0 description^5.0 keywords^5.0 author^2.0 resourcename^1.0
   
   text,features,name,sku,id,manu,cat,title,description,keywords,author,resourcename
   3

   
   on
   true
   cat
   manu_exact
   content_type
   author_s
   ipod
   GB
   1
   cat,inStock
   after
   price
   0
   600
   50
   popularity
   0
   10
   3
   manufacturedate_dt
   NOW/YEAR-10YEARS
   NOW
   +1YEAR
   before
   after

   
   on
   content features title name
   true
   html
   <b>
   </b>
   0
   title
   0
   name
   3
   200
   content
   750

   
   on
   false   
   5
   2
   5   
   true
   true  
   5
   3   
 

 
 
   spellcheck
 
  


  
  


  

  

  
  
 

Re: Dataimport handler Date

[Gora Mohanty]

On 7 March 2014 08:50, Pritesh Patel  wrote:
> I'm using the dataimporthandler to index data from a mysql DB.  Been
> running it just fine. I've been using full-imports. I'm now trying
> implement the delta import functionality.
>
> To implement the delta query, you need to be reading the last_index_time
> from a properties file to know what new to index.  So I'm using the
> parameter:
> {dataimporter.last_index_time} within my query.
>
> The problem is when I use this, the date always is : "Thu Jan 01 00:00:00
> UTC 1970".  It's never actually reading the correct date stored in the
> dataimport.properties file.
[...]

I take it that you have verified that the dataimport.properties file exists.
What are its contents?

Please share the exact DIH configuration file that you use, obfuscating
DB password/username. Your cut-and-paste seems to have a syntax
error in the deltaQuery (notice the 'jgkg' string):
deltaQuery="SELECT node.nid from node where node.type = 'news' and
node.status = 1 and (node.changed >
UNIX_TIMESTAMP('${
dataimporter.last_index_time}'jgkg) or node.created >
UNIX_TIMESTAMP('${dataimporter.last_index_time}'))"

What response do you get fromm the delta-import URL?
Are there any error messages in your Solr log?

Regards,
Gora

Dataimport handler Date

[Pritesh Patel]

I'm using the dataimporthandler to index data from a mysql DB.  Been
running it just fine. I've been using full-imports. I'm now trying
implement the delta import functionality.

To implement the delta query, you need to be reading the last_index_time
from a properties file to know what new to index.  So I'm using the
parameter:
{dataimporter.last_index_time} within my query.

The problem is when I use this, the date always is : "Thu Jan 01 00:00:00
UTC 1970".  It's never actually reading the correct date stored in the
dataimport.properties file.

So my delta query does not work.  Has anybody see this issue?

Seems like its always using the beginning date for epoch or unix timestamp
code 0.

--Pritesh

P.S.  If you want to see the delta query, see below.

deltaQuery="SELECT node.nid from node where node.type = 'news' and
node.status = 1 and (node.changed >
UNIX_TIMESTAMP('${dataimporter.last_index_time}'jgkg) or node.created >
UNIX_TIMESTAMP('${dataimporter.last_index_time}'))"

deltaImportQuery="SELECT node.nid, node.vid, node.type, node.language,
node.title, node.uid, node.status,
FROM_UNIXTIME(node.created,'%Y-%m-%dT%TZ') as created,
FROM_UNIXTIME(node.changed,'%Y-%m-%dT%TZ') as changed, node.comment,
node.promote, node.moderate, node.sticky, node.tnid, node.translate,
content_type_news.field_image_credit_value,
content_type_news.field_image_caption_value,
content_type_news.field_subhead_value,
content_type_news.field_author_value,
content_type_news.field_dateline_value,
content_type_news.field_article_image_fid,
content_type_news.field_article_image_list,
content_type_news.field_article_image_data,
content_type_news.field_news_blurb_value,
content_type_news.field_news_blurb_format,
content_type_news.field_news_syndicate_value,
content_type_news.field_news_video_reference_nid,
content_type_news.field_news_inline_location_value,
content_type_news.field_article_contributor_nid,
content_type_news.field_news_title_value, page_title.page_title FROM node
LEFT JOIN content_type_news ON node.nid = content_type_news.nid LEFT JOIN
page_title ON node.nid = page_title.id where node.type = 'news' and
node.status = 1 and node.nid = '${deltaimport.delta.nid}'"

Re: dataimport handler

[Shalin Shekhar Mangar]

I'm guessing that "id" in your schema.xml is also a unique key field.
If so, each document must have an id field or Solr will refuse to
index them.

DataImportHandler will map the id field in your table to Solr schema's
id field only if you have not specified a mapping.

On Thu, Jan 23, 2014 at 3:01 AM, tom  wrote:
> Hi,
> I am trying to use dataimporthandler(Solr 4.6) from oracle database, but I
> have some issues in mapping the data.
> I have 3 columns in the test_table,
>  column1,
>  column2,
>  id
>
> dataconfig.xml
>
>query="select * from test_table" >
> 
> 
> 
>
> Issue is,
> - if I remove the id column from the table, index fails, solr is looking for
> id column even though it is not mapped in dataconfig.xml.
> - if I add, it directly maps the id column form the db to solr id, it
> ignores the column1, even though it is mapped.
>
> my problem is I don't have ID in every table, I should be able to map the
> column I choose from the table to solr Id,  any solution will be greatly
> appreciated.
>
> `Tom
>
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/dataimport-handler-tp4112830.html
> Sent from the Solr - User mailing list archive at Nabble.com.



-- 
Regards,
Shalin Shekhar Mangar.

dataimport handler

[tom]

Hi,
I am trying to use dataimporthandler(Solr 4.6) from oracle database, but I
have some issues in mapping the data.
I have 3 columns in the test_table,
 column1,
 column2,
 id

dataconfig.xml

  


   


Issue is,
- if I remove the id column from the table, index fails, solr is looking for
id column even though it is not mapped in dataconfig.xml.
- if I add, it directly maps the id column form the db to solr id, it
ignores the column1, even though it is mapped.

my problem is I don't have ID in every table, I should be able to map the
column I choose from the table to solr Id,  any solution will be greatly
appreciated.

`Tom




--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-handler-tp4112830.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Dataimport handler exception when migrating from 4.4 to 4.6. Help needed

[William Bell]

The best practice for upgrading is take the distribution and expand it.
Then take your cores and replace it.

Then you are guaranteed to get the jars and not have other WARs/JARs
hanging around.



On Sun, Dec 22, 2013 at 7:24 PM, Shawn Heisey  wrote:

> On 12/22/2013 9:51 AM, William Pierce wrote:
> > My configurations works nicely with solr 4.4. I am encountering a
> configuration error when I try to upgrade from 4.4 to 4.6.  All I did was
> the following:
> >
> > a) Replace the 4.4 solr.war file with the 4.6 solr.war in the tomcat/lib
> folder. I am using version 6.0.36 of tomcat.
> > b) I replaced the solr-dataimporthandler-4.4.0.jar and
> solr-dataimporthandler-extras-4.4.0.jar with the corresponding 4.6
> counterparts in the collection/lib folder.
> >
> > I restarted tomcat.   I get the following stack trace (full log is also
> given below) – there are no other warnings/errors in my log.  I have gone
> through the 4.5 changes to see if I needed to add/modify my DIH
> configuration – but I am stymied.  Any help will be greatly appreciated.
> >
> > ERROR - 2013-12-22 08:05:09.824;
> org.apache.solr.handler.dataimport.DataImportHandler; Exception while
> loading DataImporter
> > java.lang.NoSuchMethodError:
> org.apache.solr.core.SolrCore.getLatestSchema()Lorg/apache/solr/schema/IndexSchema;
>
> The method it's complaining about not being there is
> org.apache.solr.core.SolrCore.getLatestSchema() ... which is in Solr
> itself, not the dataimport handler.  I did some checking.  This method
> did not exist before 4.4.0, so my best guess is that your classloader is
> loading a SolrCore class from 4.3.1 or earlier, which probably means one
> of two things: 1) The Solr war you're extracting is not actually version
> 4.6.0, or 2) you've got jars in your system from one or more older
> versions.
>
> It's a good idea to delete the extracted war data whenever you upgrade
> Solr -- stop the container, delete the extracted data and all old jars,
> then replace the .war file and start it back up.
>
> Thanks,
> Shawn
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Dataimport handler exception when migrating from 4.4 to 4.6. Help needed

[Shawn Heisey]

On 12/22/2013 9:51 AM, William Pierce wrote:
> My configurations works nicely with solr 4.4. I am encountering a 
> configuration error when I try to upgrade from 4.4 to 4.6.  All I did was the 
> following:
> 
> a) Replace the 4.4 solr.war file with the 4.6 solr.war in the tomcat/lib 
> folder. I am using version 6.0.36 of tomcat.
> b) I replaced the solr-dataimporthandler-4.4.0.jar and 
> solr-dataimporthandler-extras-4.4.0.jar with the corresponding 4.6 
> counterparts in the collection/lib folder.
> 
> I restarted tomcat.   I get the following stack trace (full log is also given 
> below) – there are no other warnings/errors in my log.  I have gone through 
> the 4.5 changes to see if I needed to add/modify my DIH configuration – but I 
> am stymied.  Any help will be greatly appreciated.
> 
> ERROR - 2013-12-22 08:05:09.824; 
> org.apache.solr.handler.dataimport.DataImportHandler; Exception while loading 
> DataImporter
> java.lang.NoSuchMethodError: 
> org.apache.solr.core.SolrCore.getLatestSchema()Lorg/apache/solr/schema/IndexSchema;

The method it's complaining about not being there is
org.apache.solr.core.SolrCore.getLatestSchema() ... which is in Solr
itself, not the dataimport handler.  I did some checking.  This method
did not exist before 4.4.0, so my best guess is that your classloader is
loading a SolrCore class from 4.3.1 or earlier, which probably means one
of two things: 1) The Solr war you're extracting is not actually version
4.6.0, or 2) you've got jars in your system from one or more older versions.

It's a good idea to delete the extracted war data whenever you upgrade
Solr -- stop the container, delete the extracted data and all old jars,
then replace the .war file and start it back up.

Thanks,
Shawn

Dataimport handler exception when migrating from 4.4 to 4.6. Help needed

[William Pierce]

Hello, all:

My configurations works nicely with solr 4.4. I am encountering a configuration 
error when I try to upgrade from 4.4 to 4.6.  All I did was the following:

a) Replace the 4.4 solr.war file with the 4.6 solr.war in the tomcat/lib 
folder. I am using version 6.0.36 of tomcat.
b) I replaced the solr-dataimporthandler-4.4.0.jar and 
solr-dataimporthandler-extras-4.4.0.jar with the corresponding 4.6 counterparts 
in the collection/lib folder.

I restarted tomcat.   I get the following stack trace (full log is also given 
below) – there are no other warnings/errors in my log.  I have gone through the 
4.5 changes to see if I needed to add/modify my DIH configuration – but I am 
stymied.  Any help will be greatly appreciated.

ERROR - 2013-12-22 08:05:09.824; 
org.apache.solr.handler.dataimport.DataImportHandler; Exception while loading 
DataImporter
java.lang.NoSuchMethodError: 
org.apache.solr.core.SolrCore.getLatestSchema()Lorg/apache/solr/schema/IndexSchema;
at 
org.apache.solr.handler.dataimport.DataImporter.(DataImporter.java:103)
at 
org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:103)
at 
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:616)
at org.apache.solr.core.SolrCore.(SolrCore.java:816)
at org.apache.solr.core.SolrCore.(SolrCore.java:618)
at 
org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

The full solr.log (until the exception) is as follows:

INFO  - 2013-12-22 08:05:08.261; org.apache.solr.servlet.SolrDispatchFilter; 
SolrDispatchFilter.init()
INFO  - 2013-12-22 08:05:08.277; org.apache.solr.core.SolrResourceLoader; Using 
JNDI solr.home: c:\tomcatweb\postingsmaster
INFO  - 2013-12-22 08:05:08.292; 
org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: 
c:\tomcatweb\postingsmaster\solr.xml
INFO  - 2013-12-22 08:05:08.292; org.apache.solr.core.CoreContainer; New 
CoreContainer 20804623
INFO  - 2013-12-22 08:05:08.292; 
org.apache.solr.core.CoreContainer$Initializer; no solr.xml found. using 
default old-style solr.xml
INFO  - 2013-12-22 08:05:08.292; org.apache.solr.core.CoreContainer; Loading 
CoreContainer using Solr Home: 'c:\tomcatweb\postingsmaster\'
INFO  - 2013-12-22 08:05:08.292; org.apache.solr.core.SolrResourceLoader; new 
SolrResourceLoader for directory: 'c:\tomcatweb\postingsmaster\'
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting 
socketTimeout to: 0
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme 
to: http://
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout 
to: 0
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting 
maxConnectionsPerHost to: 20
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize 
to: 0
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting 
maximumPoolSize to: 2147483647
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting 
maxThreadIdleTime to: 5
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue 
to: -1
INFO  - 2013-12-22 08:05:08.605; 
org.apache.solr.handler.component.HttpShardHandlerFactory; Setting 
fairnessPolicy to: false
INFO  - 2013-12-22 08:05:08.621; 
org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, 
config:maxConnectionsPerHost=20&maxConnections=1&socketTimeout=0&connTimeout=0&retry=false
INFO  - 2013-12-22 08:05:08.761; org.apache.solr.core.CoreContainer; 
Registering Log Listener
INFO  - 2013-12-22 08:05:08.792; org.apache.solr.core.CoreContainer; Creating 
SolrCore 'collection1' using instanceDir: 
c:\tomcatweb\postingsmaster\collection1
INFO  - 2013-12-22 08:05:08.792; org.apache.solr.core.SolrResourceLoader; new 
SolrResourceLoader for directory: 'c:\tomcatweb\postingsmaster\collection1\'
INFO  - 2013-12-22 08:05:08.792; org.apache.solr.core.SolrResourceLoader; 
Adding 
'file:/c:/tomcatweb/postingsmaster/col

Re: DataImport Handler, writing a new EntityProcessor

[Mathias Lux]

Hi!

Thanks for all the advice! I finally did it, the most annoying error
that took me the best of a day to figure out was that the state
variable here had to be reset:
https://bitbucket.org/dermotte/liresolr/src/d27878a71c63842cb72b84162b599d99c4408965/src/main/java/net/semanticmetadata/lire/solr/LireEntityProcessor.java?at=master#cl-56

The EntityProcessor is part of this image search plugin if anyone is
interested: https://bitbucket.org/dermotte/liresolr/

:) It's always the small things that are hard to find

cheers and thanks, Mathias

On Wed, Dec 18, 2013 at 7:26 PM, P Williams
 wrote:
> Hi Mathias,
>
> I'd recommend testing one thing at a time.  See if you can get it to work
> for one image before you try a directory of images.  Also try testing using
> the solr-testframework using your ide (I use Eclipse) to debug rather than
> your browser/print statements.  Hopefully that will give you some more
> specific knowledge of what's happening around your plugin.
>
> I also wrote an EntityProcessor plugin to read from a properties
> file.
>  Hopefully that'll give you some insight about this kind of Solr plugin and
> testing them.
>
> Cheers,
> Tricia
>
>
>
>
> On Wed, Dec 18, 2013 at 3:03 AM, Mathias Lux wrote:
>
>> Hi all!
>>
>> I've got a question regarding writing a new EntityProcessor, in the
>> same sense as the Tika one. My EntityProcessor should analyze jpg
>> images and create document fields to be used with the LIRE Solr plugin
>> (https://bitbucket.org/dermotte/liresolr). Basically I've taken the
>> same approach as the TikaEntityProcessor, but my setup just indexes
>> the first of 1000 images. I'm using a FileListEntityProcessor to get
>> all JPEGs from a directory and then I'm handing them over (see [2]).
>> My code for the EntityProcessor is at [1]. I've tried to use the
>> DataSource as well as the filePath attribute, but it ends up all the
>> same. However, the FileListEntityProcessor is able to read all the
>> files according to the debug output, but I'm missing the link from the
>> FileListEntityProcessor to the LireEntityProcessor.
>>
>> I'd appreciate any pointer or help :)
>>
>> cheers,
>>   Mathias
>>
>> [1] LireEntityProcessor http://pastebin.com/JFajkNtf
>> [2] dataConfig http://pastebin.com/vSHucatJ
>>
>> --
>> Dr. Mathias Lux
>> Klagenfurt University, Austria
>> http://tinyurl.com/mlux-itec
>>



-- 
PD Dr. Mathias Lux
Klagenfurt University, Austria
http://tinyurl.com/mlux-itec

Re: DataImport Handler, writing a new EntityProcessor

[P Williams]

Hi Mathias,

I'd recommend testing one thing at a time.  See if you can get it to work
for one image before you try a directory of images.  Also try testing using
the solr-testframework using your ide (I use Eclipse) to debug rather than
your browser/print statements.  Hopefully that will give you some more
specific knowledge of what's happening around your plugin.

I also wrote an EntityProcessor plugin to read from a properties
file.
 Hopefully that'll give you some insight about this kind of Solr plugin and
testing them.

Cheers,
Tricia




On Wed, Dec 18, 2013 at 3:03 AM, Mathias Lux wrote:

> Hi all!
>
> I've got a question regarding writing a new EntityProcessor, in the
> same sense as the Tika one. My EntityProcessor should analyze jpg
> images and create document fields to be used with the LIRE Solr plugin
> (https://bitbucket.org/dermotte/liresolr). Basically I've taken the
> same approach as the TikaEntityProcessor, but my setup just indexes
> the first of 1000 images. I'm using a FileListEntityProcessor to get
> all JPEGs from a directory and then I'm handing them over (see [2]).
> My code for the EntityProcessor is at [1]. I've tried to use the
> DataSource as well as the filePath attribute, but it ends up all the
> same. However, the FileListEntityProcessor is able to read all the
> files according to the debug output, but I'm missing the link from the
> FileListEntityProcessor to the LireEntityProcessor.
>
> I'd appreciate any pointer or help :)
>
> cheers,
>   Mathias
>
> [1] LireEntityProcessor http://pastebin.com/JFajkNtf
> [2] dataConfig http://pastebin.com/vSHucatJ
>
> --
> Dr. Mathias Lux
> Klagenfurt University, Austria
> http://tinyurl.com/mlux-itec
>

Re: DataImport Handler, writing a new EntityProcessor

[Mathias Lux]

Unfortunately it is the same in non-debug, just the first document. I
also output the params to sout, but it seems only the first one is
ever arriving at my custom class. I've the feeling that I'm doing
something seriously wrong here, based on a complete misunderstanding
:) I basically assume that the nested entity processor will be called
for each of the rows that come out from its parent. I've read
somewhere, that the data has to be taken from the data source, and
I've implemented that, but it doesn't seem to change anything.

cheers,
Mathias

On Wed, Dec 18, 2013 at 3:05 PM, Dyer, James
 wrote:
> The first thing I would suggest is to try and run it not in debug mode.  
> DIH's debug mode limits the number of documents it will take in, so that 
> might be all that is wrong here.
>
> James Dyer
> Ingram Content Group
> (615) 213-4311
>
>
> -Original Message-
> From: mathias@gmail.com [mailto:mathias@gmail.com] On Behalf Of 
> Mathias Lux
> Sent: Wednesday, December 18, 2013 4:04 AM
> To: solr-user@lucene.apache.org
> Subject: DataImport Handler, writing a new EntityProcessor
>
> Hi all!
>
> I've got a question regarding writing a new EntityProcessor, in the
> same sense as the Tika one. My EntityProcessor should analyze jpg
> images and create document fields to be used with the LIRE Solr plugin
> (https://bitbucket.org/dermotte/liresolr). Basically I've taken the
> same approach as the TikaEntityProcessor, but my setup just indexes
> the first of 1000 images. I'm using a FileListEntityProcessor to get
> all JPEGs from a directory and then I'm handing them over (see [2]).
> My code for the EntityProcessor is at [1]. I've tried to use the
> DataSource as well as the filePath attribute, but it ends up all the
> same. However, the FileListEntityProcessor is able to read all the
> files according to the debug output, but I'm missing the link from the
> FileListEntityProcessor to the LireEntityProcessor.
>
> I'd appreciate any pointer or help :)
>
> cheers,
>   Mathias
>
> [1] LireEntityProcessor http://pastebin.com/JFajkNtf
> [2] dataConfig http://pastebin.com/vSHucatJ
>
> --
> Dr. Mathias Lux
> Klagenfurt University, Austria
> http://tinyurl.com/mlux-itec
>



-- 
PD Dr. Mathias Lux
Klagenfurt University, Austria
http://tinyurl.com/mlux-itec

RE: DataImport Handler, writing a new EntityProcessor

[Dyer, James]

The first thing I would suggest is to try and run it not in debug mode.  DIH's 
debug mode limits the number of documents it will take in, so that might be all 
that is wrong here.

James Dyer
Ingram Content Group
(615) 213-4311


-Original Message-
From: mathias@gmail.com [mailto:mathias@gmail.com] On Behalf Of Mathias 
Lux
Sent: Wednesday, December 18, 2013 4:04 AM
To: solr-user@lucene.apache.org
Subject: DataImport Handler, writing a new EntityProcessor

Hi all!

I've got a question regarding writing a new EntityProcessor, in the
same sense as the Tika one. My EntityProcessor should analyze jpg
images and create document fields to be used with the LIRE Solr plugin
(https://bitbucket.org/dermotte/liresolr). Basically I've taken the
same approach as the TikaEntityProcessor, but my setup just indexes
the first of 1000 images. I'm using a FileListEntityProcessor to get
all JPEGs from a directory and then I'm handing them over (see [2]).
My code for the EntityProcessor is at [1]. I've tried to use the
DataSource as well as the filePath attribute, but it ends up all the
same. However, the FileListEntityProcessor is able to read all the
files according to the debug output, but I'm missing the link from the
FileListEntityProcessor to the LireEntityProcessor.

I'd appreciate any pointer or help :)

cheers,
  Mathias

[1] LireEntityProcessor http://pastebin.com/JFajkNtf
[2] dataConfig http://pastebin.com/vSHucatJ

-- 
Dr. Mathias Lux
Klagenfurt University, Austria
http://tinyurl.com/mlux-itec

DataImport Handler, writing a new EntityProcessor

[Mathias Lux]

Hi all!

I've got a question regarding writing a new EntityProcessor, in the
same sense as the Tika one. My EntityProcessor should analyze jpg
images and create document fields to be used with the LIRE Solr plugin
(https://bitbucket.org/dermotte/liresolr). Basically I've taken the
same approach as the TikaEntityProcessor, but my setup just indexes
the first of 1000 images. I'm using a FileListEntityProcessor to get
all JPEGs from a directory and then I'm handing them over (see [2]).
My code for the EntityProcessor is at [1]. I've tried to use the
DataSource as well as the filePath attribute, but it ends up all the
same. However, the FileListEntityProcessor is able to read all the
files according to the debug output, but I'm missing the link from the
FileListEntityProcessor to the LireEntityProcessor.

I'd appreciate any pointer or help :)

cheers,
  Mathias

[1] LireEntityProcessor http://pastebin.com/JFajkNtf
[2] dataConfig http://pastebin.com/vSHucatJ

-- 
Dr. Mathias Lux
Klagenfurt University, Austria
http://tinyurl.com/mlux-itec

Re: dataimport handler

[Shalin Shekhar Mangar]

Hmm, I will fix.

https://issues.apache.org/jira/browse/SOLR-4788


On Thu, May 9, 2013 at 8:35 PM, William Bell  wrote:

> It does not work anymore in 4.x.
>
> ${dih.last_index_time} does work, but the entity version does not.
>
> Bill
>
>
>
> On Tue, May 7, 2013 at 4:19 PM, Shalin Shekhar Mangar <
> shalinman...@gmail.com> wrote:
>
> > Using ${dih..last_index_time} should work. Make sure you put
> > it in quotes in your query.
> >
> >
> > On Tue, May 7, 2013 at 12:07 PM, Eric Myers 
> wrote:
> >
> > > In the  data import handler  I have multiple entities.  Each one
> > > generates a date in the
> > > dataimport.properties i.e. entityname.last_index_time.
> > >
> > > How do I reference the specific entity time in my delta queries?
> > >
> > > Thanks
> > >
> > > Eric
> > >
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Regards,
Shalin Shekhar Mangar.

Re: dataimport handler

[William Bell]

It does not work anymore in 4.x.

${dih.last_index_time} does work, but the entity version does not.

Bill



On Tue, May 7, 2013 at 4:19 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> Using ${dih..last_index_time} should work. Make sure you put
> it in quotes in your query.
>
>
> On Tue, May 7, 2013 at 12:07 PM, Eric Myers  wrote:
>
> > In the  data import handler  I have multiple entities.  Each one
> > generates a date in the
> > dataimport.properties i.e. entityname.last_index_time.
> >
> > How do I reference the specific entity time in my delta queries?
> >
> > Thanks
> >
> > Eric
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: dataimport handler

[Shalin Shekhar Mangar]

Using ${dih..last_index_time} should work. Make sure you put
it in quotes in your query.


On Tue, May 7, 2013 at 12:07 PM, Eric Myers  wrote:

> In the  data import handler  I have multiple entities.  Each one
> generates a date in the
> dataimport.properties i.e. entityname.last_index_time.
>
> How do I reference the specific entity time in my delta queries?
>
> Thanks
>
> Eric
>



-- 
Regards,
Shalin Shekhar Mangar.

dataimport handler

[Eric Myers]

In the  data import handler  I have multiple entities.  Each one
generates a date in the
dataimport.properties i.e. entityname.last_index_time.

How do I reference the specific entity time in my delta queries?

Thanks

Eric

Re: Dataimport handler

[William Bell]

I also get this. 4.2+


On Fri, Apr 19, 2013 at 10:43 PM, Eric Myers  wrote:

> I have multiple parallel entities in my document and when I run an import
> there are times like
> xxx.last_index_time
> where xxx is the name of the entity.
>
> I tried accessing these using dih.xxx.last_index_time but receive a null
> value.
>
> Is there a way to reference these in my queries.
>
> Thanks
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: html parsing with dataimport handler from sql column

[Rohan Thakur]

hi

yes I dont have html as documents I have data saved in sql data base in
HTML format and I want to index it on solr but not as complete string that
is with tags but just want to index the actual text in it...that is strip
off the tags.

regards
Rohan

On Wed, Feb 20, 2013 at 6:40 PM, Gora Mohanty  wrote:

> On 20 February 2013 18:31, Rohan Thakur  wrote:
> > hi all
> >
> > I have data stored in HTML format in a column in sql database and want to
> > index the data from that field to solr how can I do that any one has idea
> > please help. right now i am treating it as a string which is indexing
> > complete HTML with tags as one string to solr.
>
> How do you want to process the HTML? If you simply want to
> strip HTML tags, please take a look at the HTMLStripTransformer
> http://wiki.apache.org/solr/DataImportHandler#HTMLStripTransformer
>
> Your title implies that you want to parse the HTML in some
> fashion. If so, you will need to do that on your own, e.g., by
> using a transformer.
>
> Regards,
> Gora
>

Re: html parsing with dataimport handler from sql column

[Gora Mohanty]

On 20 February 2013 18:31, Rohan Thakur  wrote:
> hi all
>
> I have data stored in HTML format in a column in sql database and want to
> index the data from that field to solr how can I do that any one has idea
> please help. right now i am treating it as a string which is indexing
> complete HTML with tags as one string to solr.

How do you want to process the HTML? If you simply want to
strip HTML tags, please take a look at the HTMLStripTransformer
http://wiki.apache.org/solr/DataImportHandler#HTMLStripTransformer

Your title implies that you want to parse the HTML in some
fashion. If so, you will need to do that on your own, e.g., by
using a transformer.

Regards,
Gora

RE: DataImport Handler : Transformer Function Eval Failed Error

[Mishra, Shikhar]

Looks like it will be helpful. I'm going to give it a shot. Thanks, Otis.

Shikhar

From: Otis Gospodnetic [otis.gospodne...@gmail.com]
Sent: Friday, November 02, 2012 4:36 PM
To: solr-user@lucene.apache.org
Subject: Re: DataImport Handler : Transformer Function Eval Failed Error

Would http://wiki.apache.org/solr/Join do anything for you?

Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html


On Fri, Nov 2, 2012 at 10:06 AM, Mishra, Shikhar <
shikhar.mis...@telcobuy.com> wrote:

> We have a scenario where the same products are available from multiple
> vendors at different prices. We want to store these prices along with the
> products in the index (product has many prices), so that we can apply
> dynamic filtering on the prices at the time of search.
>
>
> Thanks,
> Shikhar
>
> -Original Message-
> From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com]
> Sent: Thursday, November 01, 2012 8:13 PM
> To: solr-user@lucene.apache.org
> Subject: Re: DataImport Handler : Transformer Function Eval Failed Error
>
> Hi,
>
> That looks a little painful... what are you trying to achieve by storing
> JSON in there? Maybe there's a simpler way to get there...
>
> Otis
> --
> Performance Monitoring - http://sematext.com/spm On Nov 1, 2012 6:16 PM,
> "Mishra, Shikhar" 
> wrote:
>
> > Hi,
> >
> > I'm trying to store a list of JSON objects as stored value for the
> > field prices (see below).
> >
> > I'm getting the following error from the custom transformer function
> > (see the data-config file at the end) of data import handler.
> >
> > Error Message
> >
> > --
> > - Caused by:
> > org.apache.solr.handler.dataimport.DataImportHandlerException:
> > 'eval' failed with language: JavaScript and script:
> > function vendorPrices(row){
> >
> > var wwtCost = row.get('WWT_COST');
> > var listPrice = row.get('LIST_PRICE');
> > var vendorName = row.get('VENDOR_NAME');
> >
> > //Below approach fails
> > var prices = [];
> >
> > prices.push({'vendor':vendorName});
> > prices.push({'wwtCost':wwtCost});
> > prices.push({'listPrice':listPrice});
> >
> > row.put('prices':prices);
> >
> > //Below approach works
> > //row.put('prices', '{' + 'vendor:' + vendorName +
> > ', ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> > return row;
> > } Processing Document # 1
> > at
> > org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndT
> > hrow(DataImportHandlerException.java:71)
> >
> > Data Import Handler Configuration File 
> >
> > 
> > <![CDATA[
> > function vendorPrices(row){
> >
> > var wwtCost = row.get('WWT_COST');
> > var listPrice = row.get('LIST_PRICE');
> > var vendorName = row.get('VENDOR_NAME');
> >
> > //Below approach fails
> > var prices = [];
> >
> > prices.push({'vendor':vendorName});
> > prices.push({'wwtCost':wwtCost});
> > prices.push({'listPrice':listPrice});
> >
> > row.put('prices':prices);
> >
> > //Below approach works
> > //row.put('prices', '{' + 'vendor:' + vendorName +
> > ', ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> > return row;
> > }
> > ]]>
> > 
> >
> >  > url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=
> > rac-scan.somr.com)(PORT=3465))(CONNECT_DATA=(SERVICE_NAME=
> > ERP_GENERAL.SOMR.ORG)))" user="dummy" password="xx"/>
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >
> >  > transformer="script:vendorPrices" query="SELECT PRICE.WWT_COST,
> > PRICE.LIST_PRICE, VEND.VENDOR_NAME, AVAIL.LEAD_TIME,
> > AVAIL.QTY_AVAILABLE FROM wwt_catalog.wwt_product prod,
> > wwt_catalog.wwt_product_pricing price, wwt_catalog.wwt_vendor vend,
> > wwt_catalog.wwt_product_availability avail WHERE  PROD.PRODUCT_ID =
> > price.product_id(+) AND price.vendor_id =
> > vend.vendor_id(+) AND PRICE.PRODUCT_ID = avail.product_id(+) AND
> > PRICE.VENDOR_ID = AVAIL.VENDOR_ID(+) AND prod.PRODUCT_ID =
> > '${item.PRODUCT_ID}'">
> >
> > 
> > 
> >
> > 
> > 
> >
> >
> > Are there any syntactic errors in the JavaScript code above? Thanks.
> >
> > Shikhar
> >
> >
> >
>

Re: DataImport Handler : Transformer Function Eval Failed Error

[Otis Gospodnetic]

Would http://wiki.apache.org/solr/Join do anything for you?

Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html


On Fri, Nov 2, 2012 at 10:06 AM, Mishra, Shikhar <
shikhar.mis...@telcobuy.com> wrote:

> We have a scenario where the same products are available from multiple
> vendors at different prices. We want to store these prices along with the
> products in the index (product has many prices), so that we can apply
> dynamic filtering on the prices at the time of search.
>
>
> Thanks,
> Shikhar
>
> -Original Message-
> From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com]
> Sent: Thursday, November 01, 2012 8:13 PM
> To: solr-user@lucene.apache.org
> Subject: Re: DataImport Handler : Transformer Function Eval Failed Error
>
> Hi,
>
> That looks a little painful... what are you trying to achieve by storing
> JSON in there? Maybe there's a simpler way to get there...
>
> Otis
> --
> Performance Monitoring - http://sematext.com/spm On Nov 1, 2012 6:16 PM,
> "Mishra, Shikhar" 
> wrote:
>
> > Hi,
> >
> > I'm trying to store a list of JSON objects as stored value for the
> > field prices (see below).
> >
> > I'm getting the following error from the custom transformer function
> > (see the data-config file at the end) of data import handler.
> >
> > Error Message
> >
> > --
> > - Caused by:
> > org.apache.solr.handler.dataimport.DataImportHandlerException:
> > 'eval' failed with language: JavaScript and script:
> > function vendorPrices(row){
> >
> > var wwtCost = row.get('WWT_COST');
> > var listPrice = row.get('LIST_PRICE');
> > var vendorName = row.get('VENDOR_NAME');
> >
> > //Below approach fails
> > var prices = [];
> >
> > prices.push({'vendor':vendorName});
> > prices.push({'wwtCost':wwtCost});
> > prices.push({'listPrice':listPrice});
> >
> > row.put('prices':prices);
> >
> > //Below approach works
> > //row.put('prices', '{' + 'vendor:' + vendorName +
> > ', ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> > return row;
> > } Processing Document # 1
> > at
> > org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndT
> > hrow(DataImportHandlerException.java:71)
> >
> > Data Import Handler Configuration File 
> >
> > 
> > <![CDATA[
> > function vendorPrices(row){
> >
> > var wwtCost = row.get('WWT_COST');
> > var listPrice = row.get('LIST_PRICE');
> > var vendorName = row.get('VENDOR_NAME');
> >
> > //Below approach fails
> > var prices = [];
> >
> > prices.push({'vendor':vendorName});
> > prices.push({'wwtCost':wwtCost});
> > prices.push({'listPrice':listPrice});
> >
> > row.put('prices':prices);
> >
> > //Below approach works
> > //row.put('prices', '{' + 'vendor:' + vendorName +
> > ', ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> > return row;
> > }
> > ]]>
> > 
> >
> >  > url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=
> > rac-scan.somr.com)(PORT=3465))(CONNECT_DATA=(SERVICE_NAME=
> > ERP_GENERAL.SOMR.ORG)))" user="dummy" password="xx"/>
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >
> >  > transformer="script:vendorPrices" query="SELECT PRICE.WWT_COST,
> > PRICE.LIST_PRICE, VEND.VENDOR_NAME, AVAIL.LEAD_TIME,
> > AVAIL.QTY_AVAILABLE FROM wwt_catalog.wwt_product prod,
> > wwt_catalog.wwt_product_pricing price, wwt_catalog.wwt_vendor vend,
> > wwt_catalog.wwt_product_availability avail WHERE  PROD.PRODUCT_ID =
> > price.product_id(+) AND price.vendor_id =
> > vend.vendor_id(+) AND PRICE.PRODUCT_ID = avail.product_id(+) AND
> > PRICE.VENDOR_ID = AVAIL.VENDOR_ID(+) AND prod.PRODUCT_ID =
> > '${item.PRODUCT_ID}'">
> >
> > 
> > 
> >
> > 
> > 
> >
> >
> > Are there any syntactic errors in the JavaScript code above? Thanks.
> >
> > Shikhar
> >
> >
> >
>

RE: DataImport Handler : Transformer Function Eval Failed Error

[Mishra, Shikhar]

We have a scenario where the same products are available from multiple vendors 
at different prices. We want to store these prices along with the products in 
the index (product has many prices), so that we can apply dynamic filtering on 
the prices at the time of search.


Thanks,
Shikhar

-Original Message-
From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] 
Sent: Thursday, November 01, 2012 8:13 PM
To: solr-user@lucene.apache.org
Subject: Re: DataImport Handler : Transformer Function Eval Failed Error

Hi,

That looks a little painful... what are you trying to achieve by storing JSON 
in there? Maybe there's a simpler way to get there...

Otis
--
Performance Monitoring - http://sematext.com/spm On Nov 1, 2012 6:16 PM, 
"Mishra, Shikhar" 
wrote:

> Hi,
>
> I'm trying to store a list of JSON objects as stored value for the 
> field prices (see below).
>
> I'm getting the following error from the custom transformer function 
> (see the data-config file at the end) of data import handler.
>
> Error Message
>
> --
> - Caused by: 
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> 'eval' failed with language: JavaScript and script:
> function vendorPrices(row){
>
> var wwtCost = row.get('WWT_COST');
> var listPrice = row.get('LIST_PRICE');
> var vendorName = row.get('VENDOR_NAME');
>
> //Below approach fails
> var prices = [];
>
> prices.push({'vendor':vendorName});
> prices.push({'wwtCost':wwtCost});
> prices.push({'listPrice':listPrice});
>
> row.put('prices':prices);
>
> //Below approach works
> //row.put('prices', '{' + 'vendor:' + vendorName + 
> ', ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> return row;
> } Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndT
> hrow(DataImportHandlerException.java:71)
>
> Data Import Handler Configuration File 
>
> 
> <![CDATA[
> function vendorPrices(row){
>
> var wwtCost = row.get('WWT_COST');
> var listPrice = row.get('LIST_PRICE');
> var vendorName = row.get('VENDOR_NAME');
>
> //Below approach fails
> var prices = [];
>
> prices.push({'vendor':vendorName});
> prices.push({'wwtCost':wwtCost});
> prices.push({'listPrice':listPrice});
>
> row.put('prices':prices);
>
> //Below approach works
> //row.put('prices', '{' + 'vendor:' + vendorName + 
> ', ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> return row;
> }
> ]]>
> 
>
>  url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=
> rac-scan.somr.com)(PORT=3465))(CONNECT_DATA=(SERVICE_NAME=
> ERP_GENERAL.SOMR.ORG)))" user="dummy" password="xx"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
>  transformer="script:vendorPrices" query="SELECT PRICE.WWT_COST, 
> PRICE.LIST_PRICE, VEND.VENDOR_NAME, AVAIL.LEAD_TIME, 
> AVAIL.QTY_AVAILABLE FROM wwt_catalog.wwt_product prod, 
> wwt_catalog.wwt_product_pricing price, wwt_catalog.wwt_vendor vend, 
> wwt_catalog.wwt_product_availability avail WHERE  PROD.PRODUCT_ID = 
> price.product_id(+) AND price.vendor_id =
> vend.vendor_id(+) AND PRICE.PRODUCT_ID = avail.product_id(+) AND 
> PRICE.VENDOR_ID = AVAIL.VENDOR_ID(+) AND prod.PRODUCT_ID = 
> '${item.PRODUCT_ID}'">
>
> 
> 
>
> 
> 
>
>
> Are there any syntactic errors in the JavaScript code above? Thanks.
>
> Shikhar
>
>
>

Re: DataImport Handler : Transformer Function Eval Failed Error

[Otis Gospodnetic]

Hi,

That looks a little painful... what are you trying to achieve by storing
JSON in there? Maybe there's a simpler way to get there...

Otis
--
Performance Monitoring - http://sematext.com/spm
On Nov 1, 2012 6:16 PM, "Mishra, Shikhar" 
wrote:

> Hi,
>
> I'm trying to store a list of JSON objects as stored value for the field
> prices (see below).
>
> I'm getting the following error from the custom transformer function (see
> the data-config file at the end) of data import handler.
>
> Error Message
>
> ---
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> 'eval' failed with language: JavaScript and script:
> function vendorPrices(row){
>
> var wwtCost = row.get('WWT_COST');
> var listPrice = row.get('LIST_PRICE');
> var vendorName = row.get('VENDOR_NAME');
>
> //Below approach fails
> var prices = [];
>
> prices.push({'vendor':vendorName});
> prices.push({'wwtCost':wwtCost});
> prices.push({'listPrice':listPrice});
>
> row.put('prices':prices);
>
> //Below approach works
> //row.put('prices', '{' + 'vendor:' + vendorName + ',
> ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> return row;
> } Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
>
> Data Import Handler Configuration File
> 
>
> 
>  function vendorPrices(row){
>
> var wwtCost = row.get('WWT_COST');
> var listPrice = row.get('LIST_PRICE');
> var vendorName = row.get('VENDOR_NAME');
>
> //Below approach fails
> var prices = [];
>
> prices.push({'vendor':vendorName});
> prices.push({'wwtCost':wwtCost});
> prices.push({'listPrice':listPrice});
>
> row.put('prices':prices);
>
> //Below approach works
> //row.put('prices', '{' + 'vendor:' + vendorName + ',
> ' + 'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
> return row;
> }
> ]]>
> 
>
>  url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=
> rac-scan.somr.com)(PORT=3465))(CONNECT_DATA=(SERVICE_NAME=
> ERP_GENERAL.SOMR.ORG)))" user="dummy" password="xx"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
>  transformer="script:vendorPrices" query="SELECT PRICE.WWT_COST,
> PRICE.LIST_PRICE, VEND.VENDOR_NAME, AVAIL.LEAD_TIME, AVAIL.QTY_AVAILABLE
> FROM wwt_catalog.wwt_product prod, wwt_catalog.wwt_product_pricing price,
> wwt_catalog.wwt_vendor vend, wwt_catalog.wwt_product_availability avail
> WHERE  PROD.PRODUCT_ID = price.product_id(+) AND price.vendor_id =
> vend.vendor_id(+) AND PRICE.PRODUCT_ID = avail.product_id(+) AND
> PRICE.VENDOR_ID = AVAIL.VENDOR_ID(+) AND prod.PRODUCT_ID =
> '${item.PRODUCT_ID}'">
>
> 
> 
>
> 
> 
>
>
> Are there any syntactic errors in the JavaScript code above? Thanks.
>
> Shikhar
>
>
>

DataImport Handler : Transformer Function Eval Failed Error

[Mishra, Shikhar]

Hi,

I'm trying to store a list of JSON objects as stored value for the field prices 
(see below).

I'm getting the following error from the custom transformer function (see the 
data-config file at the end) of data import handler.

Error Message
---
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: 
'eval' failed with language: JavaScript and script:
function vendorPrices(row){

var wwtCost = row.get('WWT_COST');
var listPrice = row.get('LIST_PRICE');
var vendorName = row.get('VENDOR_NAME');

//Below approach fails
var prices = [];

prices.push({'vendor':vendorName});
prices.push({'wwtCost':wwtCost});
prices.push({'listPrice':listPrice});

row.put('prices':prices);

//Below approach works
//row.put('prices', '{' + 'vendor:' + vendorName + ', ' + 
'wwtCost:' + wwtCost + ', ' + 'listPrice:' + listPrice + '}');
return row;
} Processing Document # 1
at 
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)

Data Import Handler Configuration File


























Are there any syntactic errors in the JavaScript code above? Thanks.

Shikhar

RE: Dataimport Handler in solr 3.6.1

[Dyer, James]

There were 2 major changes to DIH Cache functionality in Solr 3.6, only 1 of 
which was carried to Solr 4.0:

- Solr 3.6 had 2 MAJOR changes:

1. We support pluggable caches so that you can write your own cache 
implemetations and cache however you want.  The goal here is to allow you to 
cache to disk when you had to do large, complex joins and an in-memory cache 
could result in an OOM.  Also, you can specify "cacheImpl" with any 
EntityProcessor, not just SqlEntityProcessor.  So you can join child entities 
that come from XML, flat files, etc.  CachedSqlEntityProcessor is technically 
deprecated as using it is the same as SqlEntityProcessor with 
cacheImpl="SortedMapBackedCache" specified.  This does a simple in-memory cache 
very similar to Solr3.5 and prior. (see 
https://issues.apache.org/jira/browse/SOLR-2382)

2. Extensive work was done to try and make the "threads" parameter work in more 
situations.  This involved some rather invasive changes to the DIH Cache 
functionality. (see https://issues.apache.org/jira/browse/SOLR-3011)

- Solr 4.0 has #1 above, BUT NOT #2.  Rather the "threads" functionality was 
entirely removed.

Subsequently, if the problem is due to #2 (SOLR-3011), this isn't as big a 
problem because 3.x users can simply use the 3.5 DIH jar (but some use-cases 
involding "threads" work with the 3.6(.1) jar and not at all with 3.5, so users 
will have to pick & choose the best version to use for their instance).

My concern is there are issues with #1 (SOLR-2382).  That's why I'm asking if 
at all possible you can try this with SOLR 4.0.  I have tested Solr 4.0 
extensively here and it seems caching works exactly as it ought.  However, DIH 
is flexible on how it can be configured and there could be somethat that was 
broken that I have not uncovered myself.  Any issues that may exist with 
SOLR-2382 need to be identified and fixed in the 4.x branch as soon as possible.

I apologize for the late response.  I was away the past week.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311

-Original Message-
From: mechravi25 [mailto:mechrav...@yahoo.co.in] 
Sent: Tuesday, August 21, 2012 7:47 AM
To: solr-user@lucene.apache.org
Subject: RE: Dataimport Handler in solr 3.6.1

Hi James,

Thanks for the suggestions. 

Actually it is cacheLookup="ent1.id" . had misspelt it. Also, I will be
needing the transformers mentioned as there are other columns as well.

Actually tried using the 3.5 DIH jars in 3.6.1 and indexed the same and the
indexing was successful. But I wanted this to work with 3.6.1 DIH. Just came
across the SOLR-2382 patch. I tried giving the following 

processor="CachedSqlEntityProcessor" cacheImpl="SortedMapBackedCache" 

in my DIH.xml file. In case of static fields in child entities ,the indexing
happended fine but in case of dynamic fields, only one of the dynamic fields
was indexed and the rest was skipped even though the total rows fetched from
datasource was correct.

Following are my questions

1.) Is there a big difference in solr 3.5 and 3.6.1 DIH handler files? like
is any new feature added in 3.6 DIH that is not present in 3.5?
2.) Am i missing something while giving the cacheImpl="SortedMapBackedCache"
in my DIH.xml because of which dynamic fields are not indexed properly?
There is no change to my DIH file from my previous post apart from this
cacheImpl addition and also the dynamic fields are indexed properly if I do
not give this cacheImpl. Am I missing something here?

Thanks.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Dataimport-Handler-in-solr-3-6-1-tp4001149p4002421.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Dataimport Handler in solr 3.6.1

[mechravi25]

Hi James,

Thanks for the suggestions. 

Actually it is cacheLookup="ent1.id" . had misspelt it. Also, I will be
needing the transformers mentioned as there are other columns as well.

Actually tried using the 3.5 DIH jars in 3.6.1 and indexed the same and the
indexing was successful. But I wanted this to work with 3.6.1 DIH. Just came
across the SOLR-2382 patch. I tried giving the following 

processor="CachedSqlEntityProcessor" cacheImpl="SortedMapBackedCache" 

in my DIH.xml file. In case of static fields in child entities ,the indexing
happended fine but in case of dynamic fields, only one of the dynamic fields
was indexed and the rest was skipped even though the total rows fetched from
datasource was correct.

Following are my questions

1.) Is there a big difference in solr 3.5 and 3.6.1 DIH handler files? like
is any new feature added in 3.6 DIH that is not present in 3.5?
2.) Am i missing something while giving the cacheImpl="SortedMapBackedCache"
in my DIH.xml because of which dynamic fields are not indexed properly?
There is no change to my DIH file from my previous post apart from this
cacheImpl addition and also the dynamic fields are indexed properly if I do
not give this cacheImpl. Am I missing something here?

Thanks.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Dataimport-Handler-in-solr-3-6-1-tp4001149p4002421.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Dataimport Handler in solr 3.6.1

[Dyer, James]

One thing I notice in your configuration...the child entity has this:

cacheLookup="ent1.uid"

but your parent entity doesn't have a "uid" field.  

Also, you have these 3 transformers:  
RegexTransformer,DateFormatTransformer,TemplateTransformer

but none of your columns seem to make use of these.  Are you sure you need them?

In any case I am suspicious there may still be bugs in 3.6.1 related to 
CachedSqlEntityProcessor, so if you are able to create a failing unit test and 
post it to JIRA that would be helpful.  If you need to, you can use the 3.5 DIH 
jar with Solr 3.6.1.  Also, I do not think the SOLR-3360 should affect you 
unless you're using the "threads" parameter.  Both SOLR-3360 & SOLR-3430 fixed 
bugs related to CachedSqlEntityProcessor that were introduced in 3.6.0 (from 
SOLR-3411 and SOLR-2482 respectively).

Finally, if you are at all able to test this on 4.0-beta, I would greatly 
appreciate it!  SOLR-3411/SOLR-3360 were never applied to version 4.0 because 
"threadS" support was removed entirely.  However, SOLR-2482/SOLR-3430 were 
applied to 4.0 also.  If we have any more SOLR-2482 bugs lingering in 4.0 these 
really need to be fixed so any testing help would be much appreciated.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-Original Message-
From: mechravi25 [mailto:mechrav...@yahoo.co.in] 
Sent: Tuesday, August 14, 2012 8:04 AM
To: solr-user@lucene.apache.org
Subject: Dataimport Handler in solr 3.6.1

I am indexing some data using dataimport handler files in solr 3.6.1. I using
a nested entity in my handler file. 
I noticed a scenario where-in instead of the records which is to be fetched
for a document, 
all the records present in the table are indexed.

Following is the ideal scenario how the data has to be indexed.
For a document A, I am trying to index the 2 values B,C as a multivalued
field

A

B
C


This is how the output should be. I have used the same DIH file for solr
1.4,3.5 versions 
and the data was indexed fine like the one mentioned above in both the
versions.

But in solr 3.6.1 version, data was indexed differently. In my table, there
are 4 values(B,C,D,E) in related_id field.
This is how the data is indexed in 3.6.1

A

B
C
D
E


Ideally, the values D and E should not get indexed under id "A". This is the
same for the other id records.


Following is the content of the DIH file



 


  





   








 I tried changing the CachedSqlEntityProcessor to SqlEntityProcessor and
then indexed the same but still I faced the same issue.
 
 When I googled a bit, I found this url
https://issues.apache.org/jira/browse/SOLR-3360


I am not sure if the issue 3360 is the same as the scenario as I have
mentioned above.

Please guid me.

Thanks.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Dataimport-Handler-in-solr-3-6-1-tp4001149.html
Sent from the Solr - User mailing list archive at Nabble.com.

Dataimport Handler in solr 3.6.1

[mechravi25]

I am indexing some data using dataimport handler files in solr 3.6.1. I using
a nested entity in my handler file. 
I noticed a scenario where-in instead of the records which is to be fetched
for a document, 
all the records present in the table are indexed.

Following is the ideal scenario how the data has to be indexed.
For a document A, I am trying to index the 2 values B,C as a multivalued
field

A

B
C


This is how the output should be. I have used the same DIH file for solr
1.4,3.5 versions 
and the data was indexed fine like the one mentioned above in both the
versions.

But in solr 3.6.1 version, data was indexed differently. In my table, there
are 4 values(B,C,D,E) in related_id field.
This is how the data is indexed in 3.6.1

A

B
C
D
E


Ideally, the values D and E should not get indexed under id "A". This is the
same for the other id records.


Following is the content of the DIH file



 


  





   








 I tried changing the CachedSqlEntityProcessor to SqlEntityProcessor and
then indexed the same but still I faced the same issue.
 
 When I googled a bit, I found this url
https://issues.apache.org/jira/browse/SOLR-3360


I am not sure if the issue 3360 is the same as the scenario as I have
mentioned above.

Please guid me.

Thanks.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Dataimport-Handler-in-solr-3-6-1-tp4001149.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport handler (DIH) - notify when it has finished?

[Gora Mohanty]

On 1 May 2012 23:12, geeky2  wrote:
> Hello all,
>
> is there a notification / trigger / callback mechanism people use that
> allows them to know when a dataimport process has finished?
>
> we will be doing daily delta-imports and i need some way for an operations
> group to know when the DIH has finished.
>

Never tried it myself, but this should meet your needs:
http://wiki.apache.org/solr/DataImportHandler#EventListeners

Regards,
Gora

dataimport handler (DIH) - notify when it has finished?

[geeky2]

Hello all,

is there a notification / trigger / callback mechanism people use that
allows them to know when a dataimport process has finished?

we will be doing daily delta-imports and i need some way for an operations
group to know when the DIH has finished.

thank you,



--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-handler-DIH-notify-when-it-has-finished-tp3953339.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: PlainTextEntityProcessor and RegexTransformer in DataImport Handler

[meghana]

Thanks Matthew ,

Its really helped a lot. i am about to done with this. 

--
View this message in context: 
http://lucene.472066.n3.nabble.com/PlainTextEntityProcessor-and-RegexTransformer-in-DataImport-Handler-tp3608449p3612674.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: PlainTextEntityProcessor and RegexTransformer in DataImport Handler

[Matthew Parker]

I would try something like the following:



<![CDATA[
function format(row){
var text = row.get("plainText")

// do regex processsing with Javascript's RegExp object.

row.put("all_text", results );   // store results in
the "all_text" field.
return row;
}
]]>








On Fri, Dec 23, 2011 at 7:41 AM, meghana  wrote:

> Hi..
>
> Plz anybody have any idea? how can i achieve this?
>
> also if it is possible to convert multivalued field to non-multicalued
> field
> then it would aslo work for me.
>
> I have custom mustivalued field ArrText, which have value as shown below
> 
>12 : Hello World!!
>14 : Welcome to Solr.
>15 : Enjoy
> 
>
> if we can convert this as my desired result then it would be great.
> Thanks in Adcance.
> Meghana
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/PlainTextEntityProcessor-and-RegexTransformer-in-DataImport-Handler-tp3608449p3608726.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

--
This e-mail and any files transmitted with it may be proprietary.  Please note 
that any views or opinions presented in this e-mail are solely those of the 
author and do not necessarily represent those of Apogee Integration.

Re: PlainTextEntityProcessor and RegexTransformer in DataImport Handler

[meghana]

Hi..

Plz anybody have any idea? how can i achieve this? 

also if it is possible to convert multivalued field to non-multicalued field
then it would aslo work for me.

I have custom mustivalued field ArrText, which have value as shown below

12 : Hello World!!
14 : Welcome to Solr.
15 : Enjoy


if we can convert this as my desired result then it would be great.
Thanks in Adcance.
Meghana

--
View this message in context: 
http://lucene.472066.n3.nabble.com/PlainTextEntityProcessor-and-RegexTransformer-in-DataImport-Handler-tp3608449p3608726.html
Sent from the Solr - User mailing list archive at Nabble.com.

PlainTextEntityProcessor and RegexTransformer in DataImport Handler

[meghana]

Hi all, 

I need to import data from my text file (which have HTML text). and need to
apply some formatting on it. i want all text with in  tag , and i want it
to be preceded by one element of p tag in my output,  like below.

Original Text
--
Hello World!!Welcome to Solr.Enjoy


Needed Text After Formattting
--
12 : Hello World!!
14 : Welcome to Solr.
15 : Enjoy

I have applied combination of PlainTextEntityProcessor with RegexTransformer
and TemplateTransformer for that as below. but i am receiving
ConfigurationError when i set that.


   
   
   

I like to add here that i am able do this using TemplateTransformer and
multivalued field by setting foreach on entity, but i need above format in
single valued field, for which i am failed to do it.

Can any body help me, how can i get my desired result? or what i am doing
wrong on above transformer?
Thanks
Meghana

--
View this message in context: 
http://lucene.472066.n3.nabble.com/PlainTextEntityProcessor-and-RegexTransformer-in-DataImport-Handler-tp3608449p3608449.html
Sent from the Solr - User mailing list archive at Nabble.com.

PlainTexttransformer and RegexTransformer in DataImport Handler

[meghana]

Hi all, 

I need to import data from my text file (which have HTML text). and need to
apply some formatting on it. i want all text with in  tag , and i want it
to be preceded by one element of p tag in my output,  like below.

Original Text
--
Hello World!!Welcome to Solr.Enjoy


Needed Text After Formattting
--
12 : Hello World!!
14 : Welcome to Solr.
15 : Enjoy

I have applied combination of PlainTextTransformer , RegexTransformer and
TemplateTransformer for that as below. but i am receiving ConfigurationError
when i set that.


   
   
   

I like to add here that i am able do this using TempleteTransformer and
multivalued field. but i need above format in signle valued field, for which
i am failed to do it.

Can any body help me, how can i get my desired result? or what i am doing
wrong on above transformer?
Thanks
Meghana

--
View this message in context: 
http://lucene.472066.n3.nabble.com/PlainTexttransformer-and-RegexTransformer-in-DataImport-Handler-tp3608415p3608415.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to disable Auto Commit and Auto optimize operation after addition of few documents through dataimport handler

[Shawn Heisey]

On 12/16/2011 5:57 AM, mechravi25 wrote:

I would like to know how can we disable the commit and optimize operation is
called by deafult after addition of few documents through dataimport
handlers.


Add this to the url you use to call the handler:

&commit=false&optimize=false

Thanks,
Shawn

How to disable Auto Commit and Auto optimize operation after addition of few documents through dataimport handler

[mechravi25]

Hi,

I would like to know how can we disable the commit and optimize operation is
called by deafult after addition of few documents through dataimport
handlers.

In our application, the master solr instance is used for indexing purpose
and the slave solr is for user search request. Hence the replication has to
happen on regular interval of time. Master solr has around 1.4 million
document (Size : 2.7 GB). We have frequent addition/deletion of documents in
the master solr. After each addition/deletion commit and optimize operation
are called by default, which tends to be a costly operation. Also this makes
the replication time longer. So what I thought is that the commit operation
should be performed after certain amount of documents are added and optimize
operation should performed only once in a day or manually done.

Please let me know how to customize the setting for commit and optimize
operation in solrConfig.xml. do we have any documentation regarding the
same. Any pointers would be of great help. Thanks in advances.

Thanks & Regards,
Sivaganesh


--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-disable-Auto-Commit-and-Auto-optimize-operation-after-addition-of-few-documents-through-datair-tp3591560p3591560.html
Sent from the Solr - User mailing list archive at Nabble.com.

dataImport handler: how to figure out errors

[Julian Davchev]

Hello folks,

I have two questions regarding dataimport handler.  Using mysql
datasource with mysql java connector.

1.   Full import works fine
http://example.com:8983/solr/dataimport?command=full-import

delta-import as well

Question is how can I figure out if there are errors when running
delta-import
Cause I have the feeling url is for trigger only...status is always
idle... Is status idle good enough that things are smooth?

I plan to make wget on crontab or something to frequently delta-import
and need reliable way to catch errors.

2. Second one regards data-config.xml and RegexpTransformer
I know howto use it when I set it to entity
But in entity I am forced to supply query.   I would like not to make
extra query for this but use the value
I tried setting transformer directly on 

Re: dataimport handler with mysql: wrong field mapping

[Luca Molteni]

Have you tried using the



options in the schema.xml? After the indexing, take a look to the
fields DIH has generated.

Bye,

L.M.



2008/12/15 jokkmokk :
>
> HI,
>
> I'm desperately trying to get the dataimport handler to work, however it
> seems that it just ignores the field name mapping.
> I have the fields "body" and "subject" in the database and those are called
> "title" and "content" in the solr schema, so I use the following import
> config:
>
> 
>
> type="JdbcDataSource"
>driver="com.mysql.jdbc.Driver"
>url="jdbc:mysql://localhost/mydb"
>user="root"
>password=""/>
>
>
> 
>
>
>
>
> 
>
> 
>
> however I always get the following exception:
>
> org.apache.solr.common.SolrException: ERROR:unknown field 'body'
>at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:274)
>at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:59)
>at
> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
>at
> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:279)
>at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:317)
>at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:137)
>at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:326)
>at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
>at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:367)
>
>
> but according to the documentation it should add a document with "title" and
> "content" not "body" and "subject"?!
>
> I'd appreciate any help as I can't see anything wrong with my
> configuration...
>
> TIA,
>
> Stefan
> --
> View this message in context: 
> http://www.nabble.com/dataimport-handler-with-mysql%3A-wrong-field-mapping-tp21013109p21013109.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: dataimport handler with mysql: wrong field mapping

[jokkmokk]

sorry, I'm using the 1.3.0 release. I've now worked around that issue by
using aliases in the sql statement so that no mapping is needed. This way it
works perfectly.

best regards

Stefan


Shalin Shekhar Mangar wrote:
> 
> Which solr version are you using?
> 
-- 
View this message in context: 
http://www.nabble.com/dataimport-handler-with-mysql%3A-wrong-field-mapping-tp21013109p21013639.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport handler with mysql: wrong field mapping

[Shalin Shekhar Mangar]

Which solr version are you using?

On Mon, Dec 15, 2008 at 6:04 PM, jokkmokk  wrote:

>
> HI,
>
> I'm desperately trying to get the dataimport handler to work, however it
> seems that it just ignores the field name mapping.
> I have the fields "body" and "subject" in the database and those are called
> "title" and "content" in the solr schema, so I use the following import
> config:
>
> 
>
> type="JdbcDataSource"
>driver="com.mysql.jdbc.Driver"
>url="jdbc:mysql://localhost/mydb"
>user="root"
>password=""/>
>
>
> 
>
>
>
>
> 
>
> 
>
> however I always get the following exception:
>
> org.apache.solr.common.SolrException: ERROR:unknown field 'body'
>at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:274)
>at
>
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:59)
>at
> org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
>at
>
> org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:279)
>at
>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:317)
>at
>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:137)
>at
>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:326)
>at
>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
>at
>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:367)
>
>
> but according to the documentation it should add a document with "title"
> and
> "content" not "body" and "subject"?!
>
> I'd appreciate any help as I can't see anything wrong with my
> configuration...
>
> TIA,
>
> Stefan
> --
> View this message in context:
> http://www.nabble.com/dataimport-handler-with-mysql%3A-wrong-field-mapping-tp21013109p21013109.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
Regards,
Shalin Shekhar Mangar.

dataimport handler with mysql: wrong field mapping

[jokkmokk]

HI,

I'm desperately trying to get the dataimport handler to work, however it
seems that it just ignores the field name mapping.
I have the fields "body" and "subject" in the database and those are called
"title" and "content" in the solr schema, so I use the following import
config:















however I always get the following exception:

org.apache.solr.common.SolrException: ERROR:unknown field 'body'
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:274)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:59)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:279)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:317)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:137)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:326)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:367)


but according to the documentation it should add a document with "title" and
"content" not "body" and "subject"?!

I'd appreciate any help as I can't see anything wrong with my
configuration...

TIA,

Stefan
-- 
View this message in context: 
http://www.nabble.com/dataimport-handler-with-mysql%3A-wrong-field-mapping-tp21013109p21013109.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport handler multiple databases

[Noble Paul നോബിള്‍ नोब्ळ्]

each entity has an optional attribute called dataSource.
If you have multiple dataSources give them a name and use the name is
dataSource .So you solrconfig must look like

   
 /home/username/data-config.xml
 
datasource-1
com.mysql.jdbc.Driver

 
  
datasource-2
com.mysql.jdbc.Driver

 
   
 

and each entity can have its dataSource attribute refer to something
eg:






But as I see you have a usecase where prod and qa uses different dbs. But

So betweenprod and qa us can change the solrconfig xml
--Noble

On undefined, Ismail Siddiqui <[EMAIL PROTECTED]> wrote:
> Hi I have a situaion where I am using dataimport handler with development db
>  and  going to use it with production database in production environment
>
>  I have entry in solr-config.xml like this
>
>class="org.apache.solr.handler.dataimport.DataImportHandler">
> 
>   /home/username/data-config.xml
>   
>  com.mysql.jdbc.Driver
>  jdbc:mysql://localhost/dbname
>  db_username
>  db_password
>   
> 
>   
>
>  I understand i can add  another datasource called datasource-2 . but how can
>  I can use this datasource to index data
>
>  currently i am colling somethign  /dataimport?command=full-import or
>  /dataimport?command=delta-import.How can i define a particular db to be
>  called
>  so it indexes dev db on development machine and prod db in production
>  environmnt.
>
>
>  thanks
>



-- 
--Noble Paul

dataimport handler multiple databases

[Ismail Siddiqui]

Hi I have a situaion where I am using dataimport handler with development db
and  going to use it with production database in production environment

I have entry in solr-config.xml like this



  /home/username/data-config.xml
  
 com.mysql.jdbc.Driver
 jdbc:mysql://localhost/dbname
 db_username
 db_password
  

  

I understand i can add  another datasource called datasource-2 . but how can
I can use this datasource to index data

currently i am colling somethign  /dataimport?command=full-import or
/dataimport?command=delta-import.How can i define a particular db to be
called
so it indexes dev db on development machine and prod db in production
environmnt.


thanks