SolrCloud with custom package in dataimport

[stefan]

Hey,

Is it possible to reference a custom java class during the dataimport? The 
dataimport looks something like this:

```






db-data-config.xml


```

Sadly I was unable to find any information on this topic.

Thanks for your help!

Re: Dataimport: Could not load driver: com.mysql.jdbc.Driver

[Alexandre Rafalovitch]

Try referencing the jar directly (by absolute path) with a statement
in the solrconfig.xml (and reloading the core).
The DIH example shipped with Solr shows how it works.
This will help to see if the problem with not finding the jar or something else.

Regards,
   Alex.

On Wed, 9 Oct 2019 at 09:14, Erick Erickson  wrote:
>
> Try starting Solr with the “-v” option. That will echo all the jars that are 
> loaded and the paths.
>
> Where _exactly_ is the jar file? You say “in the lib folder of my core”, but 
> that leaves a lot of room for interpretation.
>
> Are you running stand-alone or SolrCloud? Exactly how do you start Solr?
>
> Details matter
>
> Best,
> Erick
>
> > On Oct 9, 2019, at 3:07 AM, guptavaibhav35  wrote:
> >
> > Hi,
> > Kindly help me solve the issue when I am connecting NEO4j with solr. I am
> > facing this issue in my log file while I have the jar file of neo4j driver
> > in the lib folder of my core.
> >
> > Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException:
> > org.apache.solr.handler.dataimport.DataImportHandlerException: Could not
> > load driver: org.neo4j.jdbc.Driver Processing Document # 1
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:271)
> >   at
> > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
> >   at
> > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
> >   at
> > org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
> >   at java.base/java.lang.Thread.run(Thread.java:835)
> > Caused by: java.lang.RuntimeException:
> > org.apache.solr.handler.dataimport.DataImportHandlerException: Could not
> > load driver: org.neo4j.jdbc.Driver Processing Document # 1
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:417)
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
> >   ... 4 more
> > Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> > Could not load driver: org.neo4j.jdbc.Driver Processing Document # 1
> >   at
> > org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
> >   at
> > org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:159)
> >   at
> > org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:80)
> >   at
> > org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:397)
> >   at
> > org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:100)
> >   at
> > org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)
> >   at
> > org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:77)
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:434)
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
> >   ... 6 more
> > Caused by: java.lang.ClassNotFoundException: Unable to load
> > org.neo4j.jdbc.Driver or
> > org.apache.solr.handler.dataimport.org.neo4j.jdbc.Driver
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
> >   at
> > org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:157)
> >   ... 13 more
> > Caused by: org.apache.solr.common.SolrException: Error loading class
> > 'org.neo4j.jdbc.Driver'
> >   at
> > org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:557)
> >   at
> > org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:488)
> >   at
> > org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:926)
> >   ... 14 more
> > Caused by: java.lang.ClassNotFoundException: org.neo4j.jdbc.Driver
> >   at 
> > java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
> >   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
> >   at
> > java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:864)
> >   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> >   at java.base/java.lang.Class.forName0(Native Method)
> >   at java.base/java.lang.Class.forName(Class.java:415)
> >   at
> > org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:541)
> >   ... 16 more
> >
> >
> >
> > --
> > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Dataimport: Could not load driver: com.mysql.jdbc.Driver

[Erick Erickson]

Try starting Solr with the “-v” option. That will echo all the jars that are 
loaded and the paths.

Where _exactly_ is the jar file? You say “in the lib folder of my core”, but 
that leaves a lot of room for interpretation.

Are you running stand-alone or SolrCloud? Exactly how do you start Solr?

Details matter

Best,
Erick

> On Oct 9, 2019, at 3:07 AM, guptavaibhav35  wrote:
> 
> Hi,
> Kindly help me solve the issue when I am connecting NEO4j with solr. I am
> facing this issue in my log file while I have the jar file of neo4j driver
> in the lib folder of my core. 
> 
> Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException:
> org.apache.solr.handler.dataimport.DataImportHandlerException: Could not
> load driver: org.neo4j.jdbc.Driver Processing Document # 1
>   at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:271)
>   at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
>   at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
>   at
> org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
>   at java.base/java.lang.Thread.run(Thread.java:835)
> Caused by: java.lang.RuntimeException:
> org.apache.solr.handler.dataimport.DataImportHandlerException: Could not
> load driver: org.neo4j.jdbc.Driver Processing Document # 1
>   at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:417)
>   at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
>   at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
>   ... 4 more
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> Could not load driver: org.neo4j.jdbc.Driver Processing Document # 1
>   at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
>   at
> org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:159)
>   at
> org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:80)
>   at
> org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:397)
>   at
> org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:100)
>   at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)
>   at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:77)
>   at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:434)
>   at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
>   ... 6 more
> Caused by: java.lang.ClassNotFoundException: Unable to load
> org.neo4j.jdbc.Driver or
> org.apache.solr.handler.dataimport.org.neo4j.jdbc.Driver
>   at
> org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
>   at
> org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:157)
>   ... 13 more
> Caused by: org.apache.solr.common.SolrException: Error loading class
> 'org.neo4j.jdbc.Driver'
>   at
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:557)
>   at
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:488)
>   at
> org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:926)
>   ... 14 more
> Caused by: java.lang.ClassNotFoundException: org.neo4j.jdbc.Driver
>   at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
>   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
>   at
> java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:864)
>   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
>   at java.base/java.lang.Class.forName0(Native Method)
>   at java.base/java.lang.Class.forName(Class.java:415)
>   at
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:541)
>   ... 16 more
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Dataimport: Could not load driver: com.mysql.jdbc.Driver

[guptavaibhav35]

Hi,
Kindly help me solve the issue when I am connecting NEO4j with solr. I am
facing this issue in my log file while I have the jar file of neo4j driver
in the lib folder of my core. 

Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could not
load driver: org.neo4j.jdbc.Driver Processing Document # 1
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:271)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Could not
load driver: org.neo4j.jdbc.Driver Processing Document # 1
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:417)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
... 4 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
Could not load driver: org.neo4j.jdbc.Driver Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
at
org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:159)
at
org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:80)
at
org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:397)
at
org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:100)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:434)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
... 6 more
Caused by: java.lang.ClassNotFoundException: Unable to load
org.neo4j.jdbc.Driver or
org.apache.solr.handler.dataimport.org.neo4j.jdbc.Driver
at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
at
org.apache.solr.handler.dataimport.JdbcDataSource.createConnectionFactory(JdbcDataSource.java:157)
... 13 more
Caused by: org.apache.solr.common.SolrException: Error loading class
'org.neo4j.jdbc.Driver'
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:557)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:488)
at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:926)
... 14 more
Caused by: java.lang.ClassNotFoundException: org.neo4j.jdbc.Driver
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at
java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:864)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:415)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:541)
... 16 more



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Error in Dataimport without reason or log

[Daniel Carrasco]

Hello,

I write because I'm having problems importing some data from a MariaDB
database to my Solr Cloud cluster, and I'm not able to see the data or
where's the import problem.
My Solr has a dataimport that query a MariaDB database and index the data,
but seems to be not working.
When the dataimport ends, I see a message saying that was OK (in green
letters) with this message:
*Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.*

But after activate the debug checkbox I've seen an error at end:
"Full Import failed": "2019-08-27 11:32:27"

So looks like the import is not right.
I've tried to look at the Solr logs but I only get GC messages, so I don't
know what else I can do to debug the problem.

Someone can help me to debug the problem?

I'm using the Solr version 7.7.2 (7.7.2
d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48),
I've 11 nodes in NTR and all looks healthy.

Thanks, and greetings.

-- 
_

  Daniel Carrasco Marín
  Ingeniería para la Innovación i2TIC, S.L.
  Tlf:  +34 911 12 32 84 Ext: 223
  www.i2tic.com
_

Re: Dataimport problem

[Alexandre Rafalovitch]

I wonder if you have some sort of JDBC pool enabled and/or the number
of worker threads is configured differently. Compare tomcat level
configuration and/or try thread dump of the java runtime when you are
stuck.

Or maybe something similar on the Postgres side.

Regards,
   Alex.

On Wed, 31 Jul 2019 at 10:36, Srinivas Kashyap  wrote:
>
> Hi,
> Hi,
>
> 1)Have you tried running _just_ your SQL queries to see how long they take to 
> respond and whether it responds with the full result set of batches
>
> The 9th request returns only 2 rows. This behaviour is happening for all the 
> cores which have more than 8 SQL requests. But the same is working fine with 
> AWS hosting. Really baffled.
>
> Thanks and Regards,
> Srinivas Kashyap
>
> -Original Message-
> From: Erick Erickson 
> Sent: 31 July 2019 08:00 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Dataimport problem
>
> This code is a little old, but should give you a place to start:
>
> https://lucidworks.com/post/indexing-with-solrj/
>
> As for DIH, my guess is that when you moved to Azure, your connectivity to 
> the DB changed, possibly the driver Solr uses etc., and your SQL query in 
> step 9 went from, maybe, batching rows to returning the entire result set or 
> similar weirdness. Have you tried running _just_ your SQL queries to see how 
> long they take to respond and whether it responds with the full result set of 
> batches?
>
> Best,
> Erick
>
> > On Jul 31, 2019, at 10:18 AM, Srinivas Kashyap  
> > wrote:
> >
> > Hi,
> >
> > 1) Solr on Tomcat has not been an option for quite a while. So, you must be 
> > running an old version of Solr. Which one?
> >
> > We are using Solr 5.2.1(WAR based deployment so)
> >
> >
> > 5) DIH is not actually recommended for production, more for exploration; 
> > you may want to consider moving to a stronger architecture given the 
> > complexity of your needs
> >
> > Can you please give pointers to look into, We are using DIH for production 
> > and facing few issues. We need to start phasing out
> >
> >
> > Thanks and Regards,
> > Srinivas Kashyap
> >
> > -Original Message-
> > From: Alexandre Rafalovitch 
> > Sent: 31 July 2019 07:41 PM
> > To: solr-user 
> > Subject: Re: Dataimport problem
> >
> > A couple of things:
> > 1) Solr on Tomcat has not been an option for quite a while. So, you must be 
> > running an old version of Solr. Which one?
> > 2) Compare that you have the same Solr config. In Admin UI, there will be 
> > all O/S variables passed to the Java runtime, I would check them 
> > side-by-side
> > 3) You can enable Dataimport(DIH) debug in Admin UI, so perhaps you can run 
> > a subset (1?) of the queries and see the difference
> > 4) Worst case, you may want to track this in between Solr and DB by using 
> > network analyzer (e.g. Wireshark). That may show you the actual queries, 
> > timing, connection issues, etc
> > 5) DIH is not actually recommended for production, more for exploration; 
> > you may want to consider moving to a stronger architecture given the 
> > complexity of your needs
> >
> > Regards,
> >   Alex.
> >
> > On Wed, 31 Jul 2019 at 10:04, Srinivas Kashyap  
> > wrote:
> >>
> >> Hello,
> >>
> >> We are trying to run Solr(Tomcat) on Azure instance and postgres being the 
> >> DB. When I run full import(my core has 18 SQL queries), for some reason, 
> >> the requests will go till 9 and it gets hung for eternity.
> >>
> >> But the same setup, solr(tomcat) and postgres database works fine with AWS 
> >> hosting.
> >>
> >> Am I missing some configuration? Please let me know.
> >>
> >> Thanks and Regards,
> >> Srinivas Kashyap
> >> 
> 
> DISCLAIMER:
> E-mails and attachments from Bamboo Rose, LLC are confidential.
> If you are not the intended recipient, please notify the sender immediately 
> by replying to the e-mail, and then delete it without making copies or using 
> it in any way.
> No representation is made that this email or any attachments are free of 
> viruses. Virus scanning is recommended and is the responsibility of the 
> recipient.

RE: Dataimport problem

[Srinivas Kashyap]

Hi,
Hi,

1)Have you tried running _just_ your SQL queries to see how long they take to 
respond and whether it responds with the full result set of batches

The 9th request returns only 2 rows. This behaviour is happening for all the 
cores which have more than 8 SQL requests. But the same is working fine with 
AWS hosting. Really baffled.

Thanks and Regards,
Srinivas Kashyap

-Original Message-
From: Erick Erickson 
Sent: 31 July 2019 08:00 PM
To: solr-user@lucene.apache.org
Subject: Re: Dataimport problem

This code is a little old, but should give you a place to start:

https://lucidworks.com/post/indexing-with-solrj/

As for DIH, my guess is that when you moved to Azure, your connectivity to the 
DB changed, possibly the driver Solr uses etc., and your SQL query in step 9 
went from, maybe, batching rows to returning the entire result set or similar 
weirdness. Have you tried running _just_ your SQL queries to see how long they 
take to respond and whether it responds with the full result set of batches?

Best,
Erick

> On Jul 31, 2019, at 10:18 AM, Srinivas Kashyap  
> wrote:
>
> Hi,
>
> 1) Solr on Tomcat has not been an option for quite a while. So, you must be 
> running an old version of Solr. Which one?
>
> We are using Solr 5.2.1(WAR based deployment so)
>
>
> 5) DIH is not actually recommended for production, more for exploration; you 
> may want to consider moving to a stronger architecture given the complexity 
> of your needs
>
> Can you please give pointers to look into, We are using DIH for production 
> and facing few issues. We need to start phasing out
>
>
> Thanks and Regards,
> Srinivas Kashyap
>
> -Original Message-
> From: Alexandre Rafalovitch 
> Sent: 31 July 2019 07:41 PM
> To: solr-user 
> Subject: Re: Dataimport problem
>
> A couple of things:
> 1) Solr on Tomcat has not been an option for quite a while. So, you must be 
> running an old version of Solr. Which one?
> 2) Compare that you have the same Solr config. In Admin UI, there will be all 
> O/S variables passed to the Java runtime, I would check them side-by-side
> 3) You can enable Dataimport(DIH) debug in Admin UI, so perhaps you can run a 
> subset (1?) of the queries and see the difference
> 4) Worst case, you may want to track this in between Solr and DB by using 
> network analyzer (e.g. Wireshark). That may show you the actual queries, 
> timing, connection issues, etc
> 5) DIH is not actually recommended for production, more for exploration; you 
> may want to consider moving to a stronger architecture given the complexity 
> of your needs
>
> Regards,
>   Alex.
>
> On Wed, 31 Jul 2019 at 10:04, Srinivas Kashyap  
> wrote:
>>
>> Hello,
>>
>> We are trying to run Solr(Tomcat) on Azure instance and postgres being the 
>> DB. When I run full import(my core has 18 SQL queries), for some reason, the 
>> requests will go till 9 and it gets hung for eternity.
>>
>> But the same setup, solr(tomcat) and postgres database works fine with AWS 
>> hosting.
>>
>> Am I missing some configuration? Please let me know.
>>
>> Thanks and Regards,
>> Srinivas Kashyap
>> 

DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

RE: Dataimport problem

[Srinivas Kashyap]

Hi,

1) Solr on Tomcat has not been an option for quite a while. So, you must be 
running an old version of Solr. Which one?

We are using Solr 5.2.1(WAR based deployment so)


5) DIH is not actually recommended for production, more for exploration; you 
may want to consider moving to a stronger architecture given the complexity of 
your needs

Can you please give pointers to look into, We are using DIH for production and 
facing few issues. We need to start phasing out


Thanks and Regards,
Srinivas Kashyap
            
-Original Message-
From: Alexandre Rafalovitch  
Sent: 31 July 2019 07:41 PM
To: solr-user 
Subject: Re: Dataimport problem

A couple of things:
1) Solr on Tomcat has not been an option for quite a while. So, you must be 
running an old version of Solr. Which one?
2) Compare that you have the same Solr config. In Admin UI, there will be all 
O/S variables passed to the Java runtime, I would check them side-by-side
3) You can enable Dataimport(DIH) debug in Admin UI, so perhaps you can run a 
subset (1?) of the queries and see the difference
4) Worst case, you may want to track this in between Solr and DB by using 
network analyzer (e.g. Wireshark). That may show you the actual queries, 
timing, connection issues, etc
5) DIH is not actually recommended for production, more for exploration; you 
may want to consider moving to a stronger architecture given the complexity of 
your needs

Regards,
   Alex.

On Wed, 31 Jul 2019 at 10:04, Srinivas Kashyap  wrote:
>
> Hello,
>
> We are trying to run Solr(Tomcat) on Azure instance and postgres being the 
> DB. When I run full import(my core has 18 SQL queries), for some reason, the 
> requests will go till 9 and it gets hung for eternity.
>
> But the same setup, solr(tomcat) and postgres database works fine with AWS 
> hosting.
>
> Am I missing some configuration? Please let me know.
>
> Thanks and Regards,
> Srinivas Kashyap
> 
> DISCLAIMER:
> E-mails and attachments from Bamboo Rose, LLC are confidential.
> If you are not the intended recipient, please notify the sender immediately 
> by replying to the e-mail, and then delete it without making copies or using 
> it in any way.
> No representation is made that this email or any attachments are free of 
> viruses. Virus scanning is recommended and is the responsibility of the 
> recipient.

Re: Dataimport problem

[Alexandre Rafalovitch]

A couple of things:
1) Solr on Tomcat has not been an option for quite a while. So, you
must be running an old version of Solr. Which one?
2) Compare that you have the same Solr config. In Admin UI, there will
be all O/S variables passed to the Java runtime, I would check them
side-by-side
3) You can enable Dataimport(DIH) debug in Admin UI, so perhaps you
can run a subset (1?) of the queries and see the difference
4) Worst case, you may want to track this in between Solr and DB by
using network analyzer (e.g. Wireshark). That may show you the actual
queries, timing, connection issues, etc
5) DIH is not actually recommended for production, more for
exploration; you may want to consider moving to a stronger
architecture given the complexity of your needs

Regards,
   Alex.

On Wed, 31 Jul 2019 at 10:04, Srinivas Kashyap  wrote:
>
> Hello,
>
> We are trying to run Solr(Tomcat) on Azure instance and postgres being the 
> DB. When I run full import(my core has 18 SQL queries), for some reason, the 
> requests will go till 9 and it gets hung for eternity.
>
> But the same setup, solr(tomcat) and postgres database works fine with AWS 
> hosting.
>
> Am I missing some configuration? Please let me know.
>
> Thanks and Regards,
> Srinivas Kashyap
> 
> DISCLAIMER:
> E-mails and attachments from Bamboo Rose, LLC are confidential.
> If you are not the intended recipient, please notify the sender immediately 
> by replying to the e-mail, and then delete it without making copies or using 
> it in any way.
> No representation is made that this email or any attachments are free of 
> viruses. Virus scanning is recommended and is the responsibility of the 
> recipient.

Dataimport problem

[Srinivas Kashyap]

Hello,

We are trying to run Solr(Tomcat) on Azure instance and postgres being the DB. 
When I run full import(my core has 18 SQL queries), for some reason, the 
requests will go till 9 and it gets hung for eternity.

But the same setup, solr(tomcat) and postgres database works fine with AWS 
hosting.

Am I missing some configuration? Please let me know.

Thanks and Regards,
Srinivas Kashyap

DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Re: dataimport for full-import

[Alexandre Rafalovitch]

It is probably autocommit setting in your solrconfig.xml.

But you may also want to consider indexing into a new core and then doing a
core swap at the end. Or re-aliasing if you are running a multiCore
collection.

Regards,
 Alex

On Fri, Mar 29, 2019, 2:25 AM 黄云尧,  wrote:

> when I do the full-import , it may take about 1 hours,but these old
> documents will be deleted after 10 minite, it cause query nothing 。what do
> something method to controller the old documents  will be deleted after
> longer?
>
>
>
>

dataimport for full-import

[黄云尧]

when I do the full-import , it may take about 1 hours,but these old  documents 
will be deleted after 10 minite, it cause query nothing 。what do something 
method to controller the old documents  will be deleted after longer?

Issue with dataimport xml validation with dtd and jetty: conflict of use for user.dir variable

[jerome . dupont]

Hello,

I use solr and dataimport to index xml files with a dtd.
The dtd is referenced like this


Previously we were using solr4 in a tomcat container.
During the import process, solr tries to validate the xml file with the 
dtd.
To find it we were defining -Duser.dir=pathToDtD and solr could find te 
dtd and validation was working

Now, we are migrating to solr7 (and jetty embedded)
When we start solr  with -a "-Duser.dir=pathToDtd", solr doesn't start and 
returns an error: Cannot find jetty main class

So I removed the a "-Duser.dir=pathToDtd" option, and solr starts. 
BUT
Now solr cannot anymore open xml file, because it doesn't find the dtd 
during validation stage.

Is there a way to:
- activate an xml catalog file to indicate where the dtd is? (Seems it 
would be the better way, fat I didn't find how to do)
- disable dtd validation 

Regards,
---
Jérôme Dupont
Bibliothèque Nationale de France
Département des Systèmes d'Information
Tour T3 - Quai François Mauriac
75706 Paris Cedex 13
téléphone: 33 (0)1 53 79 45 40
e-mail: jerome.dup...@bnf.fr
---

Pass BnF lecture/culture : bibliothèques, expositions, conférences, concerts en 
illimité pour 15 € / an  –  Acheter en ligne Avant d'imprimer, pensez à 
l'environnement. 

Re: Dataimport UI - shows green even when import fails

[Jan Høydahl]

I have seen the same if the JDBC jar is not found, you cannot tell from the UI, 
you have to go to Solr logs. We should fix this!

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 30. nov. 2018 kl. 00:46 skrev Shawn Heisey :
> 
> I'm looking into a problem where the admin UI dataimport screen has a green 
> status summary bar even though an import has failed.
> 
> Here's a screenshot:
> 
> https://www.dropbox.com/s/94baej11nn75746/dih-green-import-failed.png?dl=0
> 
> What I did to get this:
> Downloaded 7.5.0.
> Extracted the archive.
> On windows 10, in a command prompt at the root of the extracted files:
> bin\solr -e dih
> 
> Then I edited the DIH config for the "db" core, changing the URL to this 
> (just added a "2" near the end):
> 
> url="jdbc:hsqldb:${solr.install.dir}/example/example-DIH/hsqldb/e2x"
> 
> Once that was done, I just clicked the "Execute" button in the dataimport UI 
> for the db core.  The import failed, because the database name in the 
> modified URL doesn't exist.  But the page still shows the status summary in 
> green, with a green check mark.  The screenshot shows "Full Import failed" in 
> the raw status output.  A quick glance at this page will leave a typical user 
> with the incorrect impression that everything is fine with their import.
> 
> I thought I should just go ahead and file a bug, but before I do that, I'd 
> like to know if I should have expected something different here.
> 
> There's been a lot of issues on problems with the fact that the DIH status 
> response is extremely difficult for computers to parse.  It's probably just 
> as hard for the admin UI to parse as it is for most users.  I once wrote some 
> SolrJ code to handle parsing that response.  There was so much code that it 
> needed its own class.
> 
> https://issues.apache.org/jira/browse/SOLR-2728
> https://issues.apache.org/jira/browse/SOLR-2729
> https://issues.apache.org/jira/browse/SOLR-3319
> https://issues.apache.org/jira/browse/SOLR-3689
> https://issues.apache.org/jira/browse/SOLR-4241
> 
> Thanks,
> Shawn
> 

Dataimport UI - shows green even when import fails

[Shawn Heisey]

I'm looking into a problem where the admin UI dataimport screen has a 
green status summary bar even though an import has failed.


Here's a screenshot:

https://www.dropbox.com/s/94baej11nn75746/dih-green-import-failed.png?dl=0

What I did to get this:
Downloaded 7.5.0.
Extracted the archive.
On windows 10, in a command prompt at the root of the extracted files:
bin\solr -e dih

Then I edited the DIH config for the "db" core, changing the URL to this 
(just added a "2" near the end):


url="jdbc:hsqldb:${solr.install.dir}/example/example-DIH/hsqldb/e2x"

Once that was done, I just clicked the "Execute" button in the 
dataimport UI for the db core.  The import failed, because the database 
name in the modified URL doesn't exist.  But the page still shows the 
status summary in green, with a green check mark.  The screenshot shows 
"Full Import failed" in the raw status output.  A quick glance at this 
page will leave a typical user with the incorrect impression that 
everything is fine with their import.


I thought I should just go ahead and file a bug, but before I do that, 
I'd like to know if I should have expected something different here.


There's been a lot of issues on problems with the fact that the DIH 
status response is extremely difficult for computers to parse.  It's 
probably just as hard for the admin UI to parse as it is for most 
users.  I once wrote some SolrJ code to handle parsing that response.  
There was so much code that it needed its own class.


https://issues.apache.org/jira/browse/SOLR-2728
https://issues.apache.org/jira/browse/SOLR-2729
https://issues.apache.org/jira/browse/SOLR-3319
https://issues.apache.org/jira/browse/SOLR-3689
https://issues.apache.org/jira/browse/SOLR-4241

Thanks,
Shawn

Sanity check on dataimport handler -- what are the implications if status request returns error?

[Shawn Heisey]

What might the implications be if a DIH status request returns an error 
response other than a 404?


A 404 says either the handler or the core probably don't exist.

My guess, and I admit that I haven't read the code closely, is that if 
the handler exists but is so broken that it cannot return a status 
response without encountering an error, that there's no possible way an 
import could have been started.


For those who have spent significant time in the DIH code:  Would that 
be an accurate statement?


In case that's an XY problem, here's the "X":

When making a request with curl, the -f parameter will cause errors 
(including 404) to return an exit code of 22 from curl, and there will 
also be no output from the request.


But when using -f, you can't tell the difference between a 404 and any 
other 4XX error or 5XX response code.  I would like to treat an exit 
code of 22 as a "go" condition for the script rather than a "fail" 
condition.  If the statement I made above is accurate, then this should 
be a safe thing to do -- because if an error response on the status 
request means either a 404 or a problem so severe than an import can't 
possibly be running, I'd be OK doing that.


Thanks,
Shawn

Re: Dataimport not working on solrcloud

[Shawn Heisey]

On 8/20/2018 10:00 PM, Sushant Vengurlekar wrote:

I have a dataimport working on standalone solr instance but the same
doesn't work on solrcloud. I keep on hitting this error

Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
Exception in invoking url


There will be more to this error than what you've shared. Look in 
solr.log, and share all the ERROR/WARN entries from the correct 
timeframe.  Some of them can be quite long.  We will need *all* of that 
information.  Will also need the exact Solr version.



The url is returning well formed xml. I have verified that. The solr nodes
can fully resolve this url. I checked that out. I have the following params
set in xml-import.xml

connectionTimeout="50" readTimeout="5000"


We'll need to see the full dataimport config and the handler config from 
solrconfig.xml.


Thanks,
Shawn

Dataimport not working on solrcloud

[Sushant Vengurlekar]

I have a dataimport working on standalone solr instance but the same
doesn't work on solrcloud. I keep on hitting this error

Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
Exception in invoking url


The url is returning well formed xml. I have verified that. The solr nodes
can fully resolve this url. I checked that out. I have the following params
set in xml-import.xml

connectionTimeout="50" readTimeout="5000"


Any ideas as to what could be going wrong.


Thanks

Re: Dataimport performance

[Shawn Heisey]

On 6/7/2018 12:19 AM, kotekaman wrote:

sorry. may i know how to code it?


Code *what*?

Here's the same wiki page that I gave you for your last message:

https://wiki.apache.org/solr/UsingMailingLists

Even if I go to the Nabble website and discover that you've replied to a 
topic that's SEVEN AND A HALF YEARS OLD, that information doesn't help 
me understand exactly what it is you want to know.  The previous 
information in the topic is a question and answer about what kind of 
performance can be expected from the dataimport handler.  There's 
nothing about coding in it.


Thanks,
Shawn

Re: Dataimport performance

[kotekaman]

sorry. may i know how to code it?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

404 error on Solr 7.2.1 dataimport handler (on Windows via Cygwin)

[PeterKerk]

I'm running Solr 7.2.1 on Windows via Cygwin.

I've installed Solr7.2.1 but I'm getting a 404 when trying to run the
dataimport handler:
http://localhost:8983/solr/tt-giftsamplecatalog/dataimport?command=full-import
After calling this URL, I don't see any logging in the console.

The error in my browser:
HTTP ERROR 404
Problem accessing /solr/tt-giftsamplecatalog/dataimport. Reason:
Not Found


Bat file to start Solr7.2.1:

@echo off
C:
chdir C:\cygwin\bin
bash -c "cd /cygdrive/C/Databases/solr-7.2.1/server/;java
-Dsolr.solr.home="solr/" -jar -Xms200m -Xmx1600m start.jar --module=http
-OPTIONS=jsp" 

Solr.xml









The folder "tt-giftsamplecatalog" is present in
"C:\Databases\solr-7.2.1\example\example-DIH\solr" 


Console logging:

WARNING: System properties and/or JVM args set.  Consider using --dry-run or
--exec
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: \solr.log (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(Unknown Source)
at java.io.FileOutputStream.(Unknown Source)
at java.io.FileOutputStream.(Unknown Source)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at
org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
at
org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127)
at
org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
at org.eclipse.jetty.util.log.Slf4jLog.(Slf4jLog.java:38)
at org.eclipse.jetty.util.log.Slf4jLog.(Slf4jLog.java:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at org.eclipse.jetty.util.log.Log.initialized(Log.java:177)
at org.eclipse.jetty.util.log.Log.getLogger(Log.java:310)
at org.eclipse.jetty.util.log.Log.getLogger(Log.java:300)
at
org.eclipse.jetty.xml.XmlConfiguration.(XmlConfiguration.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:215)
at org.eclipse.jetty.start.Main.start(Main.java:458)
at org.eclipse.jetty.start.Main.main(Main.java:76)
2018-04-30 11:18:44.765 INFO  (main) [   ] o.e.j.s.Server
jetty-9.3.20.v20170531
2018-04-30 11:18:45.810 ERROR (main) [   ] o.a.s.u.StartupLoggingUtils
Missing Java Option solr.log.dir. Logging may be missing or incomplete.
2018-04-30 11:18:45.816 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___ 
_   Welcome to Apache SolrÖ version 7.2.1
2018-04-30 11:18:45.817 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __|
___| |_ _   Starting in standalone mode on port null
2018-04-30 11:18:45.817 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/
_ \ | '_|  Install dir: null
2018-04-30 11:18:45.846 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter
|___/\___/_|_|Start time: 2018-04-30T09:18:45.821Z
2018-04-30 11:18:45.873 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using
system property solr.solr.home: solr/
2018-04-30 11:18:45.883 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading
container configuration from C:\Databases\solr-7.2.1\server\solr\solr.xml
2018-04-30 11:18:46.978 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator
Found 0 core definitions underneath C:\Databases\solr-7.2.1\server\solr
2018-04-30 11:18:48.171 INFO  (main) [   ] o.e.j.s

Re: Uncheck dataimport checkboxes by default

[Daniel Carrasco]

Thanks Erick, I'll take a look into the js.


Greetings!!

2018-01-15 17:46 GMT+01:00 Erick Erickson <erickerick...@gmail.com>:

> Daniel:
>
> There's no preferences section in the admin UI. That said, it's
> all angular js and the source is there wherever you unpacked
> the package you could just change it. There's no need to
> rebuild Solr etc
>
> BTW, the mail server is pretty aggressive about stripping attachments,
> your (presumed) screenshot is blank
>
> Best,
> Erick
>
> On Mon, Jan 15, 2018 at 2:30 AM, Daniel Carrasco <d.carra...@i2tic.com>
> wrote:
>
> > Hello,
> >
> > My question is just what I've summarized on the subject: Is there any way
> > to change the default state of the checkboxes on dataimport admin page?
> >
> > I want to change the default state of the "clean" checkbox to uncheck
> > because sometimes I import incremental data and I forgot to uncheck that
> > box, then all data is cleared and I've to import all again.
> >
> >
> >
> > Thanks!!​
> >
> > --
> > _
> >
> >   Daniel Carrasco Marín
> >   Ingeniería para la Innovación i2TIC, S.L.
> >   Tlf:  +34 911 12 32 84 Ext: 223
> >   www.i2tic.com
> > _
> >
>



-- 
_

  Daniel Carrasco Marín
  Ingeniería para la Innovación i2TIC, S.L.
  Tlf:  +34 911 12 32 84 Ext: 223
  www.i2tic.com
_

Re: Uncheck dataimport checkboxes by default

[Erick Erickson]

Daniel:

There's no preferences section in the admin UI. That said, it's
all angular js and the source is there wherever you unpacked
the package you could just change it. There's no need to
rebuild Solr etc

BTW, the mail server is pretty aggressive about stripping attachments,
your (presumed) screenshot is blank

Best,
Erick

On Mon, Jan 15, 2018 at 2:30 AM, Daniel Carrasco <d.carra...@i2tic.com>
wrote:

> Hello,
>
> My question is just what I've summarized on the subject: Is there any way
> to change the default state of the checkboxes on dataimport admin page?
>
> I want to change the default state of the "clean" checkbox to uncheck
> because sometimes I import incremental data and I forgot to uncheck that
> box, then all data is cleared and I've to import all again.
>
>
>
> Thanks!!​
>
> --
> _
>
>   Daniel Carrasco Marín
>   Ingeniería para la Innovación i2TIC, S.L.
>   Tlf:  +34 911 12 32 84 Ext: 223
>   www.i2tic.com
> _
>

Uncheck dataimport checkboxes by default

[Daniel Carrasco]

Hello,

My question is just what I've summarized on the subject: Is there any way
to change the default state of the checkboxes on dataimport admin page?

I want to change the default state of the "clean" checkbox to uncheck
because sometimes I import incremental data and I forgot to uncheck that
box, then all data is cleared and I've to import all again.



Thanks!!​

-- 
_

  Daniel Carrasco Marín
  Ingeniería para la Innovación i2TIC, S.L.
  Tlf:  +34 911 12 32 84 Ext: 223
  www.i2tic.com
_

Re: SOLR nested dataimport issues

[Emir Arnautović]

Hi,
I did not check it but it seems to me that it might be related to using full 
path in your fields xpath: you are iterating hash-es and you should probable 
set field paths assuming it is the new root. E.g. for id it would be:
> 

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 15 Dec 2017, at 11:07, Triveni  wrote:
> 
> I am trying to import a nested xml using URLdatasource. But indexing is not
> happening.
> XML:
> 
> ABC
> 1512016450886
> XYZ
> 
> 
> access
> public
> 
> 
> access12
> public12
> 
> 
> 
> 
> My data-config.xml:
> 
> 
>  
>  
>url="http://abc:123/api/sample_api.xml;
>   processor="XPathEntityProcessor" 
>   forEach="/hash" >
> 
> 
>  url="http://abc:123/api/sample_api.xml;
>   processor="XPathEntityProcessor" forEach="/hash/xyz/xyz"
> transformer="script:f1">
>xpath="/hash/xyz/xyz/name"/>
>   xpath="/hash/xyz/xyz/value"/>
>  
>  
>  
>  
> 
> I am seeing below message when indexing:
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> Requests: 2 , Fetched: 3 , Skipped: 0 , Processed: 0 
> Started: less than a minute ago
> 
> And in solr.log file below error:
> 2017-12-15 09:38:53.254 WARN  (qtp223684-13) [   x:xml_data]
> o.a.s.h.d.SolrWriter Error creating document : SolrInputDocument(fields:
> [createdBy=XYZ, id=ABC, _version_=1586842286933671936, _root_=ABC],
> children: [SolrInputDocument(fields: [attr.attrValue=public, attr.Id=1,
> attr.attrName=access, _root_=ABC, _version_=1586842286933671936]),
> SolrInputDocument(fields: [attr.attrValue=public12, attr.Id=2,
> attr.attrName=access12, _root_=ABC, _version_=1586842286933671936])])
> org.apache.solr.common.SolrException: [doc=null] missing required field: id
>   at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:265)
>   at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:107)
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SOLR nested dataimport issues

[Triveni]

I am trying to import a nested xml using URLdatasource. But indexing is not
happening.
XML:

ABC
1512016450886
XYZ


access
public


access12
public12




My data-config.xml:


  
  
http://abc:123/api/sample_api.xml;
processor="XPathEntityProcessor" 
forEach="/hash" >
 
  
   http://abc:123/api/sample_api.xml;
processor="XPathEntityProcessor" forEach="/hash/xyz/xyz"
transformer="script:f1">

   
   
   
  
 

I am seeing below message when indexing:
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 2 , Fetched: 3 , Skipped: 0 , Processed: 0 
Started: less than a minute ago

And in solr.log file below error:
2017-12-15 09:38:53.254 WARN  (qtp223684-13) [   x:xml_data]
o.a.s.h.d.SolrWriter Error creating document : SolrInputDocument(fields:
[createdBy=XYZ, id=ABC, _version_=1586842286933671936, _root_=ABC],
children: [SolrInputDocument(fields: [attr.attrValue=public, attr.Id=1,
attr.attrName=access, _root_=ABC, _version_=1586842286933671936]),
SolrInputDocument(fields: [attr.attrValue=public12, attr.Id=2,
attr.attrName=access12, _root_=ABC, _version_=1586842286933671936])])
org.apache.solr.common.SolrException: [doc=null] missing required field: id
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:265)
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:107)



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Dataimport handler showing idle status with multiple shards

[Sarah Weissman]

From: Shawn Heisey <elyog...@elyograg.org>
Reply-To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Date: Tuesday, December 5, 2017 at 1:31 PM
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Subject: Re: Dataimport handler showing idle status with multiple shards

On 12/5/2017 10:47 AM, Sarah Weissman wrote:
I’ve recently been using the dataimport handler to import records from a 
database into a Solr cloud collection with multiple shards. I have 6 dataimport 
handlers configured on 6 different paths all running simultaneously against the 
same DB. I’ve noticed that when I do this I often get “idle” status from the 
DIH even when the import is still running. The percentage of the time I get an 
“idle” response seems proportional to the number of shards. I.e., with 1 shard 
it always shows me non-idle status, with 2 shards I see idle about half the 
time I check the status, with 96 shards it seems to be showing idle almost all 
the time. I can see the size of each shard increasing, so I’m sure the import 
is still going.

I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1. 
Does anyone know why the DIH would report idle when it’s running?

e.g.:
curl http://myserver:8983/solr/collection/dataimport6



To use DIH with SolrCloud, you should be sending your request directly
to a shard replica core, not the collection, so that you can be
absolutely certain that the import command and the status command are
going to the same place.  You MIGHT need to also have a distrib=false
parameter on the request, but I do not know whether that is required to
prevent the load balancing on the dataimport handler.



Thanks for the information, Shawn. I am relatively new to Solr cloud and I am 
used to running the dataimport from the admin dashboard, where it happens at 
the collection level, so I find it surprising that the right way to do this is 
at the core level. So, if I want to be able to check the status of my data 
import for N cores I would need to create N different data import configs that 
manually partition the collection and start each different config on a 
different core? That seems like it could get confusing. And then if I wanted to 
grow or shrink my shards I’d have to rejigger my data import configs every 
time. I kind of expect a distributed index to hide these details from me.

I only have one node at the moment, and I don’t understand how Solr cloud works 
internally well enough to understand what it means for the data import to be 
running on a shard vs. a node. It would be nice if doing a status query would 
at least tell you something, like the number of documents last indexed on that 
core, even if nothing is currently running. That way at least I could 
extrapolate how much longer the operation will take.

Re: Dataimport handler showing idle status with multiple shards

[Shawn Heisey]

On 12/5/2017 10:47 AM, Sarah Weissman wrote:

I’ve recently been using the dataimport handler to import records from a 
database into a Solr cloud collection with multiple shards. I have 6 dataimport 
handlers configured on 6 different paths all running simultaneously against the 
same DB. I’ve noticed that when I do this I often get “idle” status from the 
DIH even when the import is still running. The percentage of the time I get an 
“idle” response seems proportional to the number of shards. I.e., with 1 shard 
it always shows me non-idle status, with 2 shards I see idle about half the 
time I check the status, with 96 shards it seems to be showing idle almost all 
the time. I can see the size of each shard increasing, so I’m sure the import 
is still going.

I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1. 
Does anyone know why the DIH would report idle when it’s running?

e.g.:
curl http://myserver:8983/solr/collection/dataimport6


When you send a DIH request to the collection name, SolrCloud is going 
to load balance that request across the cloud, just like it would with 
any other request.  Solr will look at the list of all responding nodes 
that host part of the collection and send multiple such requests to 
different cores (shards/replicas) across the cloud.  If there are four 
cores in the collection and the nodes hosting them are all working, then 
each of those cores would only see requests to /dataimport about one 
fourth of the time.


DIH imports happen at the core level, NOT the collection level, so when 
you start an import on a collection with four cores in the cloud, only 
one of those four cores is actually going to be doing the import, the 
rest of them are idle.


This behavior should happen with any version, so I would expect it in 
6.1 as well as 7.1.


To use DIH with SolrCloud, you should be sending your request directly 
to a shard replica core, not the collection, so that you can be 
absolutely certain that the import command and the status command are 
going to the same place.  You MIGHT need to also have a distrib=false 
parameter on the request, but I do not know whether that is required to 
prevent the load balancing on the dataimport handler.


A similar question came to this list two days ago, and I replied to that 
one yesterday.


http://lucene.472066.n3.nabble.com/Dataimporter-status-tp4365602p4365879.html

Somebody did open an issue a LONG time ago about this problem:

https://issues.apache.org/jira/browse/SOLR-3666

I just commented on the issue.

Thanks,
Shawn

Dataimport handler showing idle status with multiple shards

[Sarah Weissman]

Hi,

I’ve recently been using the dataimport handler to import records from a 
database into a Solr cloud collection with multiple shards. I have 6 dataimport 
handlers configured on 6 different paths all running simultaneously against the 
same DB. I’ve noticed that when I do this I often get “idle” status from the 
DIH even when the import is still running. The percentage of the time I get an 
“idle” response seems proportional to the number of shards. I.e., with 1 shard 
it always shows me non-idle status, with 2 shards I see idle about half the 
time I check the status, with 96 shards it seems to be showing idle almost all 
the time. I can see the size of each shard increasing, so I’m sure the import 
is still going.

I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1. 
Does anyone know why the DIH would report idle when it’s running?

e.g.:
curl http://myserver:8983/solr/collection/dataimport6
{
  "responseHeader":{
"status":0,
"QTime":0},
  "initArgs":[
"defaults",[
  "config","data-config6.xml"]],
  "status":"idle",
  "importResponse":"",
  "statusMessages":{}}

Thanks,
Sarah

Re: Solr 7* Sorry, no dataimport-handler defined

[Shawn Heisey]

On 11/7/2017 6:49 AM, richardg wrote:
> vs on the master that shows the error.
>
> 2017-11-07 13:29:14.131 INFO  (qtp1839206329-36) [  
> x:solr_aggregate_production] o.a.s.c.S.Request [solr_aggregate_production] 
> webapp=/solr path=/admin/mbeans
> params={cat=QUERYHANDLER=json&_=1510061366718} status=0 QTime=2

The string "QUERYHANDLER" (all uppercase) only shows up in a 7.1.0
source code checkout in the reference guide, it is not in any code that
builds the program.  Its presence in the reference guide is likely a
documentation error.

If you are seeing QUERYHANDLER in a log for version 7.1.0, then I have
to wonder exactly how you did the upgrade -- because I think there are
only two ways that could happen:  1) Your 7.1.0 install includes at
least some files from a version before 6.4.1.  2) You've got something
(perhaps a load balancer) mixing up requests between two different
versions of Solr.

Thanks,
Shawn

Re: Solr 7* Sorry, no dataimport-handler defined

[richardg]

Yes I am referring to the dataimport tab in the admin UI and issue
SOLR-10035.  My previous setup w/ 6.3 did not show this error.  I then
upgraded to 7.1.0 and the error shows.  I upgraded(downgraded) to versions
6.5.0 and 6.6.2 and I do not see the error.  Version 7.0.1 also shows the
error for me.  I am currently using version 6.6.2 and have been successfully
able to run a data import from the admin UI. 

In my config directory we have 

solcore.properties
solrconfig.xml which defines the dataimport handler (data-config.xml)
schema.xml
dataimport.properties
data-config.xml
some admin-extra*.html files

We copy all the config files over to the slave instances and they do no show
this behavior on 7.1.0, dataimport tab loads fine.  The only thing I notice
is on the slaves I see entries like this in the log:

2017-11-07 13:36:11.200 INFO  (qtp2053591126-35) [  
x:solr_aggregate_production] o.a.s.c.S.Request [solr_aggregate_production] 
webapp=/solr path=/admin/mbeans params={cat=QUERY=json&_=1510061783971}
status=0 QTime=0

vs on the master that shows the error.

2017-11-07 13:29:14.131 INFO  (qtp1839206329-36) [  
x:solr_aggregate_production] o.a.s.c.S.Request [solr_aggregate_production] 
webapp=/solr path=/admin/mbeans
params={cat=QUERYHANDLER=json&_=1510061366718} status=0 QTime=2

I see just "QUERY" in the slave that is working and "QUERYHANDLER" in the
master that isn't.  This is why I referenced the issue w/ 6.4 (SOLR-10035). 
Other than that I do not see anything in the log showing and error for the
dataimport handler.

Thanks



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr 7* Sorry, no dataimport-handler defined

[Shawn Heisey]

On 11/6/2017 4:26 PM, Shawn Heisey wrote:
> If I start Solr "normally" or with the cloud example, then the
> dataimport tab shows that error -- which is exactly as expected.

I have opened an improvement issue so that particular error message
isn't as vague.  It's been labeled with "newdev" because it's an easy
change.

https://issues.apache.org/jira/browse/SOLR-11613

Thanks,
Shawn

Re: Solr 7* Sorry, no dataimport-handler defined

[Shawn Heisey]

On 11/6/2017 1:47 PM, richardg wrote:
> I see where this was an issue w/ 6.4 and fixed.  I keep getting this error w/
> 7.0.1 and 7.1.0.  Works fine up until 6.6.2.  Could this issue have been
> reintroduced?  Is there somewhere to check what might be going on?  I don't
> see anything in the error logs.

You've included almost no information here.  I assume that the subject
is the error?  If so, where do you see it?  Please be aware that without
information, we can't help.

https://wiki.apache.org/solr/UsingMailingLists

My best guess is that you see that error in the admin UI when you click
on dataimport.  The error is pretty self-explanitory -- Solr couldn't
find any defined dataimport handlers.  At least one such handler must be
set up in the config before you can use the dataimport tab.

You mention that this was an issue with 6.4 and fixed ... but didn't
include a link to the issue.  Are you talking about SOLR-10035?

https://issues.apache.org/jira/browse/SOLR-10035

If I start a fresh Solr 7.1.0 download with the DIH example (bin/solr
start -e dih), clicking on the dataimport tab works on all five cores. 
If I start Solr "normally" or with the cloud example, then the
dataimport tab shows that error -- which is exactly as expected.  The
example configsets in the base Solr download do not have a dataimport
handler.  Only the configs specifically in the DIH example have such
handlers defined.

Have you created a core with a dataimport handler that you KNOW is
working, and seeing that error?  If so, we'll need more detailed
information.  Solr version, everything in the core's conf directory,
details about the error and steps taken to see it, etc.  Any error
messages in your logfile would be helpful as well.

Thanks,
Shawn

Solr 7* Sorry, no dataimport-handler defined

[richardg]

I see where this was an issue w/ 6.4 and fixed.  I keep getting this error w/
7.0.1 and 7.1.0.  Works fine up until 6.6.2.  Could this issue have been
reintroduced?  Is there somewhere to check what might be going on?  I don't
see anything in the error logs.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: DataImport Handler Out of Memory

[Allison, Timothy B.]

https://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F


-Original Message-
From: Deeksha Sharma [mailto:dsha...@flexera.com] 
Sent: Wednesday, September 27, 2017 1:40 PM
To: solr-user@lucene.apache.org
Subject: DataImport Handler Out of Memory

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in 
mysql db and the number of records are more than 3.5 million. My solr server 
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of 
RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to 
configure Solr to stream the data from DB or any other solution somewhere may 
have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by 
giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

DataImport Handler Out of Memory

[Deeksha Sharma]

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in 
mysql db and the number of records are more than 3.5 million. My solr server 
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of 
RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to 
configure Solr to stream the data from DB or any other solution somewhere may 
have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by 
giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

[bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Mannott, Birgit]

Thanks to all for your support. Many good ideas!
But meanwhile I found a way using the Oracle "AFTER LOGON" Trigger where I am 
able to identify if it is a connection from solr and then switch to cursor 
sharing. 

By the way cursor sharing reduced the import time from 12 to 6 hours !!
So it would be really great if the Data Import Handler would be able to use 
binding variables in selects so that cursor sharing could be done easier.

Regards,
Birgit


-Original Message-
From: Mannott, Birgit [mailto:b.mann...@klopotek.de] 
Sent: Tuesday, August 15, 2017 4:58 PM
To: solr-user@lucene.apache.org
Subject: [bulk]: RE: [bulk]: Re: Optimizing Dataimport from Oracle; cursor 
sharing; changing oracle session parameters

Yes, I'm using Data Import Handler and I would prefer a solution for this way 
of import because it's already tested and the imported data is ok and 
everything is fine. 
I just have to speed it up a little...

But thanks for your info. Next time I'll try indexing with solrj.

Regards,
Birgit


-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, August 15, 2017 4:33 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: [bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing; 
changing oracle session parameters

I presume you're using Data Import Handler? An alternative when you get into 
complex imports is to use a SolrJ client, here's a sample.
That way you can use whatever tools the particular JDBC connector will allow 
and can be much faster.

https://lucidworks.com/2012/02/14/indexing-with-solrj/

Best,
Erick

On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit <b.mann...@klopotek.de> wrote:
> Hi,
>
> I'm using solr 6.6.0 and I have to do a complex data import from an oracle db 
> concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52 
> Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by 
> creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.
>
> I think, the main remaining problem is that oracle cursor sharing is not used 
> and that for every select a hard parse is done.
> Solr does not use binding variables. This would be the easiest way to use 
> oracle cursor sharing. But I didn't find anything about influencing the way 
> solr builds select statements.
> I could force oracle cursor sharing without binding variables but I have to 
> do this configuration for the session. I'm not allowed to change the 
> configuration of the whole database system.
>
> Is there a way to execute a command like "ALTER SESSION SET cursor_sharing = 
> FORCE;" after getting the connection for processing an entity?
>
> Thanks,
> Birgit

RE: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Phil Scadden]

Perhaps there is potential to optimize with some PLSQL functions on Oracle side 
to do as much work within database as possible and have the text indexers only 
access a view referencing that function. Also, the obvious optimization is a 
record-updated timestamp so that every time indexer runs, only changed data is 
managed.

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: Wednesday, 16 August 2017 5:42 a.m.
To: solr-user@lucene.apache.org
Subject: Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle 
session parameters

On 8/15/2017 8:09 AM, Mannott, Birgit wrote:
> I'm using solr 6.6.0 and I have to do a complex data import from an oracle db 
> concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52 
> Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by 
> creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.

I think the reason it takes 12 hours is because there are 52 million SELECT 
statements.  That many statements over 12 hours is an average of
1200 per second.  This sounds like pretty good database performance.

> Is there a way to execute a command like "ALTER SESSION SET cursor_sharing = 
> FORCE;" after getting the connection for processing an entity?

I think that most JDBC drivers (by default) don't allow multiple SQL statements 
to be sent in a single request, so commands like "SELECT FOO; SELECT BAR" won't 
work.  The idea behind denying this kind of command is protection against SQL 
injection attacks.  There is likely a JDBC URL parameter for the Oracle driver 
that would allow that ... and if there is, then you could add that to the 
connection URL in the DIH config to allow putting the ALTER SESSION statement 
before SELECT in your DIH entity.

The Oracle driver might also have a JDBC URL parameter to turn on the cursor 
sharing you're interested in.  That would be the best way to handle it, if that 
is an option.  You're going to need to consult Oracle documentation or an 
Oracle support resource to find out what URL parameter options there are for 
their driver.

I have near zero experience with Oracle databases, but I suspect that even with 
cursor sharing, you're still going to have the sheer number of SELECT 
statements as a bottleneck.  If there is a performance improvement, it probably 
won't be dramatic.

Thanks,
Shawn

Notice: This email and any attachments are confidential and may not be used, 
published or redistributed without the prior written consent of the Institute 
of Geological and Nuclear Sciences Limited (GNS Science). If received in error 
please destroy and immediately notify GNS Science. Do not copy or disclose the 
contents.

Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Walter Underwood]

This might be a hack, but the CSV importer is really fast. Run the query in 
your favorite command line and export to CSV, then load it.

You can even make batches. Maybe use ranges of the ID, then delete by query for 
that range.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Aug 15, 2017, at 8:46 AM, Stefan Matheis  wrote:
> 
> Birgit,
> 
> any chance to utilise one of the caching strategies that DIH offers?
> 
> Like building a complete map for one of the subentities? That would mean
> reading the whole table at the beginning and then only doing lookups by key.
> 
> Or getting data from subentities with joins in your main entity?
> 
> Heavily depends on the amount of data we're talking about - but might be
> worth a thought.
> 
> Best
> -Stefan
> 
> On Aug 15, 2017 4:33 PM, "Erick Erickson"  wrote:
> 
> I presume you're using Data Import Handler? An alternative when you
> get into complex imports is to use a SolrJ client, here's a sample.
> That way you can use whatever tools the particular JDBC connector will
> allow and can be much faster.
> 
> https://lucidworks.com/2012/02/14/indexing-with-solrj/
> 
> Best,
> Erick
> 
> On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit 
> wrote:
>> Hi,
>> 
>> I'm using solr 6.6.0 and I have to do a complex data import from an
> oracle db concerning 3.500.000 data rows.
>> For each row I have 15 additional entities. That means that more than 52
> Million selects are send to the database.
>> For every select that is done I optimized the oracle execution path by
> creating indizes.
>> The execution plans are ok.
>> But the import still lasts 12 hours.
>> 
>> I think, the main remaining problem is that oracle cursor sharing is not
> used and that for every select a hard parse is done.
>> Solr does not use binding variables. This would be the easiest way to use
> oracle cursor sharing. But I didn't find anything about influencing the way
> solr builds select statements.
>> I could force oracle cursor sharing without binding variables but I have
> to do this configuration for the session. I'm not allowed to change the
> configuration of the whole database system.
>> 
>> Is there a way to execute a command like "ALTER SESSION SET
> cursor_sharing = FORCE;" after getting the connection for processing an
> entity?
>> 
>> Thanks,
>> Birgit

Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Shawn Heisey]

On 8/15/2017 8:09 AM, Mannott, Birgit wrote:
> I'm using solr 6.6.0 and I have to do a complex data import from an oracle db 
> concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52 
> Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by 
> creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.

I think the reason it takes 12 hours is because there are 52 million
SELECT statements.  That many statements over 12 hours is an average of
1200 per second.  This sounds like pretty good database performance.

> Is there a way to execute a command like "ALTER SESSION SET cursor_sharing = 
> FORCE;" after getting the connection for processing an entity?

I think that most JDBC drivers (by default) don't allow multiple SQL
statements to be sent in a single request, so commands like "SELECT FOO;
SELECT BAR" won't work.  The idea behind denying this kind of command is
protection against SQL injection attacks.  There is likely a JDBC URL
parameter for the Oracle driver that would allow that ... and if there
is, then you could add that to the connection URL in the DIH config to
allow putting the ALTER SESSION statement before SELECT in your DIH entity.

The Oracle driver might also have a JDBC URL parameter to turn on the
cursor sharing you're interested in.  That would be the best way to
handle it, if that is an option.  You're going to need to consult Oracle
documentation or an Oracle support resource to find out what URL
parameter options there are for their driver.

I have near zero experience with Oracle databases, but I suspect that
even with cursor sharing, you're still going to have the sheer number of
SELECT statements as a bottleneck.  If there is a performance
improvement, it probably won't be dramatic.

Thanks,
Shawn

Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Stefan Matheis]

Birgit,

any chance to utilise one of the caching strategies that DIH offers?

Like building a complete map for one of the subentities? That would mean
reading the whole table at the beginning and then only doing lookups by key.

Or getting data from subentities with joins in your main entity?

Heavily depends on the amount of data we're talking about - but might be
worth a thought.

Best
-Stefan

On Aug 15, 2017 4:33 PM, "Erick Erickson"  wrote:

I presume you're using Data Import Handler? An alternative when you
get into complex imports is to use a SolrJ client, here's a sample.
That way you can use whatever tools the particular JDBC connector will
allow and can be much faster.

https://lucidworks.com/2012/02/14/indexing-with-solrj/

Best,
Erick

On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit 
wrote:
> Hi,
>
> I'm using solr 6.6.0 and I have to do a complex data import from an
oracle db concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52
Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by
creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.
>
> I think, the main remaining problem is that oracle cursor sharing is not
used and that for every select a hard parse is done.
> Solr does not use binding variables. This would be the easiest way to use
oracle cursor sharing. But I didn't find anything about influencing the way
solr builds select statements.
> I could force oracle cursor sharing without binding variables but I have
to do this configuration for the session. I'm not allowed to change the
configuration of the whole database system.
>
> Is there a way to execute a command like "ALTER SESSION SET
cursor_sharing = FORCE;" after getting the connection for processing an
entity?
>
> Thanks,
> Birgit

Re: [bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[David Hastings]

If you dont want to use your own Solj code, why not try many concurrent
indexers that index different data sets.  So run seven indexers each
getting 500,000 rows at the exact same time perhaps.  Its a hack, if it
works, but if you have the machinery to do it, why not. or use the
deltaquery, but i havent used the DIH that much to know if that will help
you.

On Tue, Aug 15, 2017 at 10:57 AM, Mannott, Birgit <b.mann...@klopotek.de>
wrote:

> Yes, I'm using Data Import Handler and I would prefer a solution for this
> way of import because it's already tested and the imported data is ok and
> everything is fine.
> I just have to speed it up a little...
>
> But thanks for your info. Next time I'll try indexing with solrj.
>
> Regards,
> Birgit
>
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Tuesday, August 15, 2017 4:33 PM
> To: solr-user <solr-user@lucene.apache.org>
> Subject: [bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing;
> changing oracle session parameters
>
> I presume you're using Data Import Handler? An alternative when you get
> into complex imports is to use a SolrJ client, here's a sample.
> That way you can use whatever tools the particular JDBC connector will
> allow and can be much faster.
>
> https://lucidworks.com/2012/02/14/indexing-with-solrj/
>
> Best,
> Erick
>
> On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit <b.mann...@klopotek.de>
> wrote:
> > Hi,
> >
> > I'm using solr 6.6.0 and I have to do a complex data import from an
> oracle db concerning 3.500.000 data rows.
> > For each row I have 15 additional entities. That means that more than 52
> Million selects are send to the database.
> > For every select that is done I optimized the oracle execution path by
> creating indizes.
> > The execution plans are ok.
> > But the import still lasts 12 hours.
> >
> > I think, the main remaining problem is that oracle cursor sharing is not
> used and that for every select a hard parse is done.
> > Solr does not use binding variables. This would be the easiest way to
> use oracle cursor sharing. But I didn't find anything about influencing the
> way solr builds select statements.
> > I could force oracle cursor sharing without binding variables but I have
> to do this configuration for the session. I'm not allowed to change the
> configuration of the whole database system.
> >
> > Is there a way to execute a command like "ALTER SESSION SET
> cursor_sharing = FORCE;" after getting the connection for processing an
> entity?
> >
> > Thanks,
> > Birgit
>

RE: [bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Mannott, Birgit]

Yes, I'm using Data Import Handler and I would prefer a solution for this way 
of import because it's already tested and the imported data is ok and 
everything is fine. 
I just have to speed it up a little...

But thanks for your info. Next time I'll try indexing with solrj.

Regards,
Birgit


-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, August 15, 2017 4:33 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: [bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing; 
changing oracle session parameters

I presume you're using Data Import Handler? An alternative when you get into 
complex imports is to use a SolrJ client, here's a sample.
That way you can use whatever tools the particular JDBC connector will allow 
and can be much faster.

https://lucidworks.com/2012/02/14/indexing-with-solrj/

Best,
Erick

On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit <b.mann...@klopotek.de> wrote:
> Hi,
>
> I'm using solr 6.6.0 and I have to do a complex data import from an oracle db 
> concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52 
> Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by 
> creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.
>
> I think, the main remaining problem is that oracle cursor sharing is not used 
> and that for every select a hard parse is done.
> Solr does not use binding variables. This would be the easiest way to use 
> oracle cursor sharing. But I didn't find anything about influencing the way 
> solr builds select statements.
> I could force oracle cursor sharing without binding variables but I have to 
> do this configuration for the session. I'm not allowed to change the 
> configuration of the whole database system.
>
> Is there a way to execute a command like "ALTER SESSION SET cursor_sharing = 
> FORCE;" after getting the connection for processing an entity?
>
> Thanks,
> Birgit

Re: Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Erick Erickson]

I presume you're using Data Import Handler? An alternative when you
get into complex imports is to use a SolrJ client, here's a sample.
That way you can use whatever tools the particular JDBC connector will
allow and can be much faster.

https://lucidworks.com/2012/02/14/indexing-with-solrj/

Best,
Erick

On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit  wrote:
> Hi,
>
> I'm using solr 6.6.0 and I have to do a complex data import from an oracle db 
> concerning 3.500.000 data rows.
> For each row I have 15 additional entities. That means that more than 52 
> Million selects are send to the database.
> For every select that is done I optimized the oracle execution path by 
> creating indizes.
> The execution plans are ok.
> But the import still lasts 12 hours.
>
> I think, the main remaining problem is that oracle cursor sharing is not used 
> and that for every select a hard parse is done.
> Solr does not use binding variables. This would be the easiest way to use 
> oracle cursor sharing. But I didn't find anything about influencing the way 
> solr builds select statements.
> I could force oracle cursor sharing without binding variables but I have to 
> do this configuration for the session. I'm not allowed to change the 
> configuration of the whole database system.
>
> Is there a way to execute a command like "ALTER SESSION SET cursor_sharing = 
> FORCE;" after getting the connection for processing an entity?
>
> Thanks,
> Birgit

Optimizing Dataimport from Oracle; cursor sharing; changing oracle session parameters

[Mannott, Birgit]

Hi,

I'm using solr 6.6.0 and I have to do a complex data import from an oracle db 
concerning 3.500.000 data rows.
For each row I have 15 additional entities. That means that more than 52 
Million selects are send to the database.
For every select that is done I optimized the oracle execution path by creating 
indizes.
The execution plans are ok.
But the import still lasts 12 hours.

I think, the main remaining problem is that oracle cursor sharing is not used 
and that for every select a hard parse is done.
Solr does not use binding variables. This would be the easiest way to use 
oracle cursor sharing. But I didn't find anything about influencing the way 
solr builds select statements.
I could force oracle cursor sharing without binding variables but I have to do 
this configuration for the session. I'm not allowed to change the configuration 
of the whole database system.

Is there a way to execute a command like "ALTER SESSION SET cursor_sharing = 
FORCE;" after getting the connection for processing an entity?

Thanks,
Birgit

Re: Different DateTime format in dataimport and index

[SOLR4189]

I don't use DB. I do dataimport from one collection of SOLR to another
collection with the same configuration. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Different-DateTime-format-in-dataimport-and-index-tp4339230p4339244.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Different DateTime format in dataimport and index

[Erick Erickson]

My guess would be that your DIH query is interacting weirdly with your db
and doing time one transforms before Solr gets it.

On Jun 6, 2017 8:38 AM, "SOLR4189" <klin892...@yandex.ru> wrote:

> Let's say I have SolrDoc:
> *{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*,
> where type of id is int, type of price is float, type of name is string and
> type of pickupTime is tdate/date. And let's say I have my update processor
> that writes to log indexed item.
>
> So, my question is why in indexing of item I see in log:
> *{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*
> and in reindex or dataimport I see in log:
> *{id: test1, price: 100.0, name: pizza, pickupTime: Tue Jun 6 19:00:00 IDT
> 2017}*
>
> Why do float and date have different format in index and dataimport? Is it
> SOLR bug?
> How can I change dataimport format to index format?
> Which are types have different format like float and date?
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Different-DateTime-format-in-dataimport-and-
> index-tp4339230.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Different DateTime format in dataimport and index

[SOLR4189]

Let's say I have SolrDoc: 
*{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*,
where type of id is int, type of price is float, type of name is string and
type of pickupTime is tdate/date. And let's say I have my update processor
that writes to log indexed item. 

So, my question is why in indexing of item I see in log:
*{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*
and in reindex or dataimport I see in log:
*{id: test1, price: 100.0, name: pizza, pickupTime: Tue Jun 6 19:00:00 IDT
2017}*

Why do float and date have different format in index and dataimport? Is it
SOLR bug?
How can I change dataimport format to index format?
Which are types have different format like float and date?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Different-DateTime-format-in-dataimport-and-index-tp4339230.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Closed connection issue while doing dataimport

[santosh sidnal]

Thanks Shwan, that was good point to consider but we had porblem with
holdability param in data-config.xml and Oracle 12c version of DB and
client.

BY remove Holdability parameter we are able to solve the issue.

On 28 March 2017 at 18:19, Shawn Heisey  wrote:

> On 3/27/2017 7:13 PM, santosh sidnal wrote:
> > i am facing closed connection issue while doing dataimporter, any
> solution
> > to this> stack trace is as below
> >
> >
> > [3/27/17 8:54:41:399 CDT] 00b4 OracleDataSto >  findMappingClass for
> :
> > Entry
> >  java.sql.SQLRecoverableException:
> Closed
> > Connection
>
> Does the import appear to work correctly at first, then stop before it's
> done with this exception after a few hours?  If so, then I think you may
> be running into a situation where the merge scheduler has multiple
> merges scheduled and stops the incoming indexing thread until the
> largest merge is done.  If the indexing thread is stopped for long
> enough, JDBC will disconnect the database, and when the indexing thread
> finally starts back up, the dataimporter finds that it cannot read from
> the database any more.
>
> If this is what's happening, then the solution is to allow the merge
> scheduler to schedule more merges simultaneously.  Here's the
> indexConfig that I use in solrconfig.xml:
>
> 
>   
> 35
> 35
> 105
>   
>   
> 1
> 6
>   
> 
>
> You don't need the mergePolicy part of this config to solve this issue.
> That is the settings that I use, but this greatly increases the number
> of files that can be in an index, which may require OS tuning for max
> open files.
>
> The mergeScheduler is the important part, and specifically
> maxMergeCount.  These settings are for standard spinning-magnetic
> disks.  I increase the maxMergeCount to 6, so more merges can be
> scheduled without halting the indexing thread.  The maxThreadCount value
> should be set to 1 if you're using standard disks.  If you're using SSD,
> then you can bump it a little bit, because SSD can easily handle
> multiple threads randomly writing to the disk.  I don't know what values
> are appropriate for SSD, but I would probably start with 2-4.
>
> Thanks,
> Shawn
>
>


-- 
Regards,
Santosh Sidnal

Re: Closed connection issue while doing dataimport

[Shawn Heisey]

On 3/27/2017 7:13 PM, santosh sidnal wrote:
> i am facing closed connection issue while doing dataimporter, any solution
> to this> stack trace is as below
>
>
> [3/27/17 8:54:41:399 CDT] 00b4 OracleDataSto >  findMappingClass for :
> Entry
>  java.sql.SQLRecoverableException: Closed
> Connection

Does the import appear to work correctly at first, then stop before it's
done with this exception after a few hours?  If so, then I think you may
be running into a situation where the merge scheduler has multiple
merges scheduled and stops the incoming indexing thread until the
largest merge is done.  If the indexing thread is stopped for long
enough, JDBC will disconnect the database, and when the indexing thread
finally starts back up, the dataimporter finds that it cannot read from
the database any more.

If this is what's happening, then the solution is to allow the merge
scheduler to schedule more merges simultaneously.  Here's the
indexConfig that I use in solrconfig.xml:


  
35
35
105
  
  
1
6
  


You don't need the mergePolicy part of this config to solve this issue. 
That is the settings that I use, but this greatly increases the number
of files that can be in an index, which may require OS tuning for max
open files.

The mergeScheduler is the important part, and specifically
maxMergeCount.  These settings are for standard spinning-magnetic
disks.  I increase the maxMergeCount to 6, so more merges can be
scheduled without halting the indexing thread.  The maxThreadCount value
should be set to 1 if you're using standard disks.  If you're using SSD,
then you can bump it a little bit, because SSD can easily handle
multiple threads randomly writing to the disk.  I don't know what values
are appropriate for SSD, but I would probably start with 2-4.

Thanks,
Shawn

Closed connection issue while doing dataimport

[santosh sidnal]

Hi All,

i am facing closed connection issue while doing dataimporter, any solution
to this> stack trace is as below


[3/27/17 8:54:41:399 CDT] 00b4 OracleDataSto >  findMappingClass for :
Entry
 java.sql.SQLRecoverableException: Closed
Connection
at
oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3640)
at
oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3680)
at
oracle.jdbc.OracleConnectionWrapper.commit(OracleConnectionWrapper.java:140)
at
com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.commit(WSJdbcConnection.java:1113)
at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:432)
at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:421)
at
com.ibm.commerce.solr.handler.SchemaJdbcDataSource.close(SchemaJdbcDataSource.java:289)
at
org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:294)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:283)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)

[3/27/17 8:54:41:399 CDT] 00b4 OracleDataSto <  findMappingClass return
 Exit
 class
com.ibm.websphere.ce.cm.StaleConnectionException
[3/27/17 8:54:41:401 CDT] 00b4 StaleConnecti 3   The stack trace for
the staleConn is:
 java.sql.SQLRecoverableException: Closed
Connection
at
oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3640)
at
oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3680)
at
oracle.jdbc.OracleConnectionWrapper.commit(OracleConnectionWrapper.java:140)
at
com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.commit(WSJdbcConnection.java:1113)
at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:432)
at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:421)
at
com.ibm.commerce.solr.handler.SchemaJdbcDataSource.close(SchemaJdbcDataSource.java:289)
at
org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:294)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:283)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)

[3/27/17 8:54:41:401 CDT] 00b4 GenericDataSt <  mapExceptionHelper:
Mapping was done returning: Exit

 com.ibm.websphere.ce.cm.StaleConnectionException: Closed Connection
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at
com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:620)
at
com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:682)
at com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2112)
at com.ibm.ws.rsadapter.jdbc.WSJdbcUtil.mapException(WSJdbcUtil.java:1047)
at
com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.commit(WSJdbcConnection.java:1151)
at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:432)
at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:421)
at
com.ibm.commerce.solr.handler.SchemaJdbcDataSource.close(SchemaJdbcDataSource.java:289)
at
org.apache.solr.handler.dataimport.DocBuilder.closeEntityProcessorWrappers(DocBuilder.java:294)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:283)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)
 Begin backtrace for Nested Throwables
java.sql.SQLRecoverableException: Closed Connection
at
oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3640)
at
oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3680)
at
oracle.jdbc.OracleConnectionWrapper.commit(OracleConnectionWrapper.java:140)
at
com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.commit(WSJdbcConnection.java:1113)
at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:432)
at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:421)
at
com.ibm.commerce.solr.handler.SchemaJdbcDataSource.close(SchemaJdbcDataSource.java:289)
at

Re: dataimport to a smaller Solr farm

[Mikhail Khludnev]

Hello, Dean.

DIH is shard agnostic. How do you try to specify "a shard from the new
collection"?

On Tue, Mar 21, 2017 at 8:24 PM, deansg <dea...@gmail.com> wrote:

> Hello,
> My team often uses the /dataimport & /dih handlers to move items from one
> Solr collection to another. However, all the times we did that, the number
> of shards in the new collection was always the same or higher than in the
> old.
> Can /dataimport work if I have less shards in the new collection than in
> the
> old one? I tried specifying a shard from the new collection multiple times
> in the data-config file, and it didn't seem to work - there were no visible
> exceptions, but most items simply didn't enter the new collection.
> Dean.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/dataimport-to-a-smaller-Solr-farm-tp4326067.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Sincerely yours
Mikhail Khludnev

dataimport to a smaller Solr farm

[deansg]

Hello,
My team often uses the /dataimport & /dih handlers to move items from one
Solr collection to another. However, all the times we did that, the number
of shards in the new collection was always the same or higher than in the
old. 
Can /dataimport work if I have less shards in the new collection than in the
old one? I tried specifying a shard from the new collection multiple times
in the data-config file, and it didn't seem to work - there were no visible
exceptions, but most items simply didn't enter the new collection.
Dean.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-to-a-smaller-Solr-farm-tp4326067.html
Sent from the Solr - User mailing list archive at Nabble.com.

??????Aw: ??????Solr 5.5.0 Configure global jndi DS for dataimport

[alias]

I just tested, the success of the need to amend two points
1??the jetty.xml  comment out
2??Jndi's name is this: java:comp/env/jdbc/mysqlds 
The complete steps are as follows:


jetty.xml

   
 java:comp/env/jdbc/mysqlds
 
 
 jdbc:mysql://localhost:3306/ka_ka
 root
 123456
 
 
 


db-data-config.xml

 
 
  




I wish you success





--  --
??: "Per Newgro";<per.new...@gmx.ch>;
: 2017??2??7??(??) 8:44
??: "solr-user"<solr-user@lucene.apache.org>; 

: Aw: ??Solr 5.5.0 Configure global jndi DS for dataimport



Maybe someone is interested in solution:

AND
java:comp/env/jdbc/myds

Both need the complete lookup name.

Per


> Gesendet: Dienstag, 07. Februar 2017 um 10:29 Uhr
> Von: "Per Newgro" <per.new...@gmx.ch>
> An: solr-user@lucene.apache.org
> Betreff: Aw: ??Solr 5.5.0 Configure global jndi DS for dataimport
>
> Changed db-data-config.xml
> 
> 
> This leads to
> Caused by: javax.naming.NameNotFoundException; remaining name 'env/jdbc/myds'
> at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:538)
> at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:569)
> at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:584)
> at 
> org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:108)
> at javax.naming.InitialContext.lookup(InitialContext.java:417)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
> ... 39 more
> 
> Do i need to install other libraries? Do i need to enable jndi?
> Can i configure something useful for logging?
> 
> Thanks for your support
> Per
> 
> > Gesendet: Dienstag, 07. Februar 2017 um 10:02 Uhr
> > Von: alias <524839...@qq.com>
> > An: solr-user <solr-user@lucene.apache.org>
> > Betreff: ??Solr 5.5.0 Configure global jndi DS for dataimport
> >
> > jndiName="java:comp/env/jdbc/myds"
> > 
> > 
> > --  --
> > ??: "Per Newgro";<per.new...@gmx.ch>;
> > ????: 2017??2??7??(??) 4:47
> > ??: "solr-user-group"<solr-user@lucene.apache.org>; 
> > 
> > : Solr 5.5.0 Configure global jndi DS for dataimport
> > 
> > 
> > 
> > Hello,
> > 
> > I would like to configure a JNDI datasource for use in dataimport. From the 
> > documentation it shall be possible and easy.
> > 
> > My environment:
> > Debian
> > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
> > Solr 5.5.0 downloaded and installed as service in /opt/solr
> > Installed core in /var/lib/solr/data/collection1
> > 
> > Solr is running and core can be managed.
> > 
> > Put into /opt/solr/server/lib
> > jetty-jndi-9.2.13.v20150730.jar
> > jetty-plus-9.2.13.v20150730.jar
> > Put into /opt/solr/server/lib/ext
> > sqljdbc4-4.0.jar
> > 
> > /opt/solr/server/etc/jetty.xml
> > ...
> > 
> > 
> > jdbc/myds
> > 
> > 
> >  > name="URL">jdbc:sqlserver://;databaseName=dbname;
> > user
> > password
> > 
> > 
> > 
> > ...
> > 
> > /var/lib/solr/data/collection1/conf/db-data-config.xml
> > 
> > 
> > 
> >  > name="bodyshop"
> > query="SELECT b.id as ID,
> >   customer_number as CUSTOMER_NUMBER,
> >   customer_name as CUSTOMER_NAME
> > FROM  schema.body_shops b
> >WHERE  '${dataimporter.request.clean}' != 'false'
> >   OR  b.last_modified > 
> > '${dataimporter.last_index_time}'">
> > ...
> > 
> > But all i get is an exception
> > Caused by

Aw: 回复：Solr 5.5.0 Configure global jndi DS for dataimport

[Per Newgro]

Maybe someone is interested in solution:

AND
java:comp/env/jdbc/myds

Both need the complete lookup name.

Per


> Gesendet: Dienstag, 07. Februar 2017 um 10:29 Uhr
> Von: "Per Newgro" <per.new...@gmx.ch>
> An: solr-user@lucene.apache.org
> Betreff: Aw: 回复：Solr 5.5.0 Configure global jndi DS for dataimport
>
> Changed db-data-config.xml
> 
> 
> This leads to
> Caused by: javax.naming.NameNotFoundException; remaining name 'env/jdbc/myds'
> at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:538)
> at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:569)
> at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:584)
> at 
> org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:108)
> at javax.naming.InitialContext.lookup(InitialContext.java:417)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
> ... 39 more
> 
> Do i need to install other libraries? Do i need to enable jndi?
> Can i configure something useful for logging?
> 
> Thanks for your support
> Per
> 
> > Gesendet: Dienstag, 07. Februar 2017 um 10:02 Uhr
> > Von: alias <524839...@qq.com>
> > An: solr-user <solr-user@lucene.apache.org>
> > Betreff: 回复：Solr 5.5.0 Configure global jndi DS for dataimport
> >
> > jndiName="java:comp/env/jdbc/myds"
> > 
> > 
> > -- 原始邮件 ------
> > 发件人: "Per Newgro";<per.new...@gmx.ch>;
> > 发送时间: 2017年2月7日(星期二) 下午4:47
> > 收件人: "solr-user-group"<solr-user@lucene.apache.org>; 
> > 
> > 主题: Solr 5.5.0 Configure global jndi DS for dataimport
> > 
> > 
> > 
> > Hello,
> > 
> > I would like to configure a JNDI datasource for use in dataimport. From the 
> > documentation it shall be possible and easy.
> > 
> > My environment:
> > Debian
> > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
> > Solr 5.5.0 downloaded and installed as service in /opt/solr
> > Installed core in /var/lib/solr/data/collection1
> > 
> > Solr is running and core can be managed.
> > 
> > Put into /opt/solr/server/lib
> > jetty-jndi-9.2.13.v20150730.jar
> > jetty-plus-9.2.13.v20150730.jar
> > Put into /opt/solr/server/lib/ext
> > sqljdbc4-4.0.jar
> > 
> > /opt/solr/server/etc/jetty.xml
> > ...
> > 
> > 
> > jdbc/myds
> > 
> > 
> >  > name="URL">jdbc:sqlserver://;databaseName=dbname;
> > user
> > password
> > 
> > 
> > 
> > ...
> > 
> > /var/lib/solr/data/collection1/conf/db-data-config.xml
> > 
> > 
> > 
> >  > name="bodyshop"
> > query="SELECT b.id as ID,
> >   customer_number as CUSTOMER_NUMBER,
> >   customer_name as CUSTOMER_NAME
> > FROM  schema.body_shops b
> >WHERE  '${dataimporter.request.clean}' != 'false'
> >   OR  b.last_modified > 
> > '${dataimporter.last_index_time}'">
> > ...
> > 
> > But all i get is an exception
> > Caused by: javax.naming.NameNotFoundException; remaining name 'jdbc/myds'
> > at 
> > org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:487)
> > at 
> > org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:533)
> > at javax.naming.InitialContext.lookup(InitialContext.java:417)
> > at 
> > org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
> > at 
> > org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
> > at 
> > org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
> > at 
> > org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
> > at 
> > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
> > ... 39 more
> > 
> > I've searched across the web for a solution but all i found did not work.
> > It would be great if someone could help me out.
> > 
> > Thanks
> > Per
>

Aw: 回复：Solr 5.5.0 Configure global jndi DS for dataimport

[Per Newgro]

Changed db-data-config.xml


This leads to
Caused by: javax.naming.NameNotFoundException; remaining name 'env/jdbc/myds'
at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:538)
at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:569)
at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:584)
at 
org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:108)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
... 39 more

Do i need to install other libraries? Do i need to enable jndi?
Can i configure something useful for logging?

Thanks for your support
Per

> Gesendet: Dienstag, 07. Februar 2017 um 10:02 Uhr
> Von: alias <524839...@qq.com>
> An: solr-user <solr-user@lucene.apache.org>
> Betreff: 回复：Solr 5.5.0 Configure global jndi DS for dataimport
>
> jndiName="java:comp/env/jdbc/myds"
> 
> 
> -- 原始邮件 --
> 发件人: "Per Newgro";<per.new...@gmx.ch>;
> 发送时间: 2017年2月7日(星期二) 下午4:47
> 收件人: "solr-user-group"<solr-user@lucene.apache.org>; 
> 
> 主题: Solr 5.5.0 Configure global jndi DS for dataimport
> 
> 
> 
> Hello,
> 
> I would like to configure a JNDI datasource for use in dataimport. From the 
> documentation it shall be possible and easy.
> 
> My environment:
> Debian
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
> Solr 5.5.0 downloaded and installed as service in /opt/solr
> Installed core in /var/lib/solr/data/collection1
> 
> Solr is running and core can be managed.
> 
> Put into /opt/solr/server/lib
> jetty-jndi-9.2.13.v20150730.jar
> jetty-plus-9.2.13.v20150730.jar
> Put into /opt/solr/server/lib/ext
> sqljdbc4-4.0.jar
> 
> /opt/solr/server/etc/jetty.xml
> ...
> 
> 
> jdbc/myds
> 
> 
>  name="URL">jdbc:sqlserver://;databaseName=dbname;
> user
> password
> 
> 
> 
> ...
> 
> /var/lib/solr/data/collection1/conf/db-data-config.xml
> 
> 
> 
>  name="bodyshop"
> query="SELECT b.id as ID,
>   customer_number as CUSTOMER_NUMBER,
>   customer_name as CUSTOMER_NAME
> FROM  schema.body_shops b
>WHERE  '${dataimporter.request.clean}' != 'false'
>   OR  b.last_modified > 
> '${dataimporter.last_index_time}'">
> ...
> 
> But all i get is an exception
> Caused by: javax.naming.NameNotFoundException; remaining name 'jdbc/myds'
> at 
> org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:487)
> at 
> org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:533)
> at javax.naming.InitialContext.lookup(InitialContext.java:417)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
> at 
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
> ... 39 more
> 
> I've searched across the web for a solution but all i found did not work.
> It would be great if someone could help me out.
> 
> Thanks
> Per

??????Solr 5.5.0 Configure global jndi DS for dataimport

[alias]

jndiName="java:comp/env/jdbc/myds"


--  --
??: "Per Newgro";<per.new...@gmx.ch>;
: 2017??2??7??(??) 4:47
??: "solr-user-group"<solr-user@lucene.apache.org>; 

: Solr 5.5.0 Configure global jndi DS for dataimport



Hello,

I would like to configure a JNDI datasource for use in dataimport. From the 
documentation it shall be possible and easy.

My environment:
Debian
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
Solr 5.5.0 downloaded and installed as service in /opt/solr
Installed core in /var/lib/solr/data/collection1

Solr is running and core can be managed.

Put into /opt/solr/server/lib
jetty-jndi-9.2.13.v20150730.jar
jetty-plus-9.2.13.v20150730.jar
Put into /opt/solr/server/lib/ext
sqljdbc4-4.0.jar

/opt/solr/server/etc/jetty.xml
...


jdbc/myds


jdbc:sqlserver://;databaseName=dbname;
user
password



...

/var/lib/solr/data/collection1/conf/db-data-config.xml




...

But all i get is an exception
Caused by: javax.naming.NameNotFoundException; remaining name 'jdbc/myds'
at 
org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:487)
at 
org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:533)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
... 39 more

I've searched across the web for a solution but all i found did not work.
It would be great if someone could help me out.

Thanks
Per

Solr 5.5.0 Configure global jndi DS for dataimport

[Per Newgro]

Hello,

I would like to configure a JNDI datasource for use in dataimport. From the 
documentation it shall be possible and easy.

My environment:
Debian
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
Solr 5.5.0 downloaded and installed as service in /opt/solr
Installed core in /var/lib/solr/data/collection1

Solr is running and core can be managed.

Put into /opt/solr/server/lib
jetty-jndi-9.2.13.v20150730.jar
jetty-plus-9.2.13.v20150730.jar
Put into /opt/solr/server/lib/ext
sqljdbc4-4.0.jar

/opt/solr/server/etc/jetty.xml
...


jdbc/myds


jdbc:sqlserver://;databaseName=dbname;
user
password



...

/var/lib/solr/data/collection1/conf/db-data-config.xml




...

But all i get is an exception
Caused by: javax.naming.NameNotFoundException; remaining name 'jdbc/myds'
at 
org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:487)
at 
org.eclipse.jetty.jndi.local.localContextRoot.lookup(localContextRoot.java:533)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.getFromJndi(JdbcDataSource.java:250)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:182)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:463)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:309)
... 39 more

I've searched across the web for a solution but all i found did not work.
It would be great if someone could help me out.

Thanks
Per

Re: no dataimport-handler defined!

[Shawn Heisey]

On 1/26/2017 7:44 AM, Chris Rogers wrote:
> Just tested the DIH example in 6.4 (bin/solr -e dih)
>
> Getting the same “No dataimport-handler defined!” for every one of the cores 
> installed as part of the example.

Repeating a reply already posted elsewhere on this thread:

It's a bug.

https://issues.apache.org/jira/browse/SOLR-10035

Easy enough to fix manually, hopefully 6.4.1 will work out of the box.

Thanks,
Shawn

Re: no dataimport-handler defined!

[Alexandre Rafalovitch]

Chris,

Shawn has already provided a workaround and a JIRA reference earlier
in this thread. Could you review his message and see if his solution
solves it for you. There might be a 6.4.1 soon and it will be fixed
there as well.

Regards,
   Alex

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 26 January 2017 at 09:44, Chris Rogers
<chris.rog...@bodleian.ox.ac.uk> wrote:
> Hi Alex,
>
> Just tested the DIH example in 6.4 (bin/solr -e dih)
>
> Getting the same “No dataimport-handler defined!” for every one of the cores 
> installed as part of the example.
>
> Cheers,
> Chris
>
>
> On 24/01/2017, 15:07, "Alexandre Rafalovitch" <arafa...@gmail.com> wrote:
>
> Strange.
>
> If you run a pre-built DIH example, do any of the cores work? (not the
> RSS one, that is broken anyway).
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 24 January 2017 at 08:32, Chris Rogers
> <chris.rog...@bodleian.ox.ac.uk> wrote:
> > Hi Alex,
> >
> > I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie 
> the one generated from the configset when the node was created).
> >
> > I’m running standalone, not cloud.
> >
> > I’m restarting sole after every change. Do I need to reload the core 
> instead of restarting?
> >
> > I’ve also tried replacing the relative path to the .jar with an 
> absolute path to the dist directory. Still didn’t work.
> >
> > Thanks,
> > Chris
> >
> > On 24/01/2017, 13:20, "Alexandre Rafalovitch" <arafa...@gmail.com> 
> wrote:
> >
> > Which solrconfig.xml are you editing and what kind of Solr install 
> are
> > you running (cloud?). And did you reload the core.
> >
> > I suspect you are not editing the file that is actually in use. For
> > example, if you are running a cloud setup, the solrconfig.xml on the
> > filesystem is disconnected from the config actually in use that is
> > stored in ZooKeeper. You would need to reupload it for change to 
> take
> > effect.
> >
> > You also may need to reload the core for changes to take effect.
> >
> > Regards,
> >Alex.
> > 
> > http://www.solr-start.com/ - Resources for Solr users, new and 
> experienced
> >
> >
> > On 24 January 2017 at 07:43, Chris Rogers
> > <chris.rog...@bodleian.ox.ac.uk> wrote:
> > > Hi all,
> > >
> > > Having frustrating issues with getting SOLR 6.4.0 to recognize 
> the existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
> > >
> > > The DIH .jar file appears to be loading correctly. There are no 
> errors in the SOLR logs. It just says “Sorry, no dataimport-handler defined” 
> in the SOLR admin UI.
> > >
> > > My config files are listed below. Can anyone spot any mistakes 
> here?
> > >
> > > Many thanks,
> > > Chris
> > >
> > > # solrconfig.xml ##
> > >
> > >regex=".*dataimporthandler-.*\.jar" />
> > >
> > > …
> > >
> > >class="org.apache.solr.handler.dataimport.DataImportHandler">
> > > 
> > >   DIH-data-config.xml
> > > 
> > >   
> > >
> > > # DIH-data-config.xml (in the same dir as solrconfig.xml) 
> ##
> > >
> > > 
> > >   
> > >   
> > > 
> > >  > > fileName=".*xml"
> > > newerThan="'NOW-5YEARS'"
> > > recursive="true"
> > > rootEntity="false"
> > > dataSource="null"
> > > 
> baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
> > >
> > >   
> > >
> > >> >   forEach="/TEI" url="${f.fileAbsolutePath}" 
> transformer="RegexTransformer" >
> > >  xpath="/TEI/teiHeader/fileDesc/titleStmt/title"/>
> > >  xpath="/TEI/teiHeader/fileDesc/publicationStmt/publisher"/>
> > >  xpath="/TEI/teiHeader/fileDesc/sourceDesc/msDesc/msIdentifier/altIdentifier/idno"/>
> > >   
> > >
> > > 
> > >
> > >   
> > > 
> > >
> > >
> > > --
> > > Chris Rogers
> > > Digital Projects Manager
> > > Bodleian Digital Library Systems and Services
> > > chris.rog...@bodleian.ox.ac.uk
> >
> >
>
>

Re: no dataimport-handler defined!

[Chris Rogers]

Hi Alex,

Just tested the DIH example in 6.4 (bin/solr -e dih)

Getting the same “No dataimport-handler defined!” for every one of the cores 
installed as part of the example.

Cheers,
Chris


On 24/01/2017, 15:07, "Alexandre Rafalovitch" <arafa...@gmail.com> wrote:

Strange.

If you run a pre-built DIH example, do any of the cores work? (not the
RSS one, that is broken anyway).

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 08:32, Chris Rogers
<chris.rog...@bodleian.ox.ac.uk> wrote:
> Hi Alex,
>
> I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie 
the one generated from the configset when the node was created).
>
> I’m running standalone, not cloud.
>
> I’m restarting sole after every change. Do I need to reload the core 
instead of restarting?
>
> I’ve also tried replacing the relative path to the .jar with an absolute 
path to the dist directory. Still didn’t work.
>
> Thanks,
> Chris
>
> On 24/01/2017, 13:20, "Alexandre Rafalovitch" <arafa...@gmail.com> wrote:
>
> Which solrconfig.xml are you editing and what kind of Solr install are
> you running (cloud?). And did you reload the core.
>
> I suspect you are not editing the file that is actually in use. For
> example, if you are running a cloud setup, the solrconfig.xml on the
> filesystem is disconnected from the config actually in use that is
> stored in ZooKeeper. You would need to reupload it for change to take
> effect.
>
> You also may need to reload the core for changes to take effect.
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and 
experienced
>
>
> On 24 January 2017 at 07:43, Chris Rogers
> <chris.rog...@bodleian.ox.ac.uk> wrote:
> > Hi all,
> >
> > Having frustrating issues with getting SOLR 6.4.0 to recognize the 
existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
> >
> > The DIH .jar file appears to be loading correctly. There are no 
errors in the SOLR logs. It just says “Sorry, no dataimport-handler defined” in 
the SOLR admin UI.
> >
> > My config files are listed below. Can anyone spot any mistakes here?
> >
> > Many thanks,
> > Chris
> >
> > # solrconfig.xml ##
> >
> >   
> >
> > …
> >
> >   
> > 
> >   DIH-data-config.xml
> > 
> >   
> >
> > # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
> >
> > 
> >   
> >   
> > 
> >  > fileName=".*xml"
> > newerThan="'NOW-5YEARS'"
> > recursive="true"
> > rootEntity="false"
> > dataSource="null"
> > 
baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
> >
> >   
> >
> >>   forEach="/TEI" url="${f.fileAbsolutePath}" 
transformer="RegexTransformer" >
> > 
> > 
> > 
> >   
> >
> > 
> >
> >   
> > 
> >
> >
> > --
> > Chris Rogers
> > Digital Projects Manager
> > Bodleian Digital Library Systems and Services
> > chris.rog...@bodleian.ox.ac.uk
>
>

Re: no dataimport-handler defined!

[sudhi]

Did anyone figured out a solution for this? Ran into same issue when I
upgraded to 6.4 from 6.2.1. DIH works perfectly fine in 6.2.1

Tried out of the box example and donot see DIH in the example cloud module
as well.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/no-dataimport-handler-defined-tp4315520p4315787.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: no dataimport-handler defined!

[Shawn Heisey]

On 1/24/2017 5:43 AM, Chris Rogers wrote:
> Having frustrating issues with getting SOLR 6.4.0 to recognize the existence 
> of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no errors in the 
> SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
> admin UI.

A user on the IRC channel has run into what seems to be the same problem.

This bug is caused by a change in the SolrInfoMBean class, part of some
cleanup for new metrics.  The output of the handler that normally lives
at /admin/mbeans changed -- what used to be QUERYHANDLER is now just
QUERY, so the admin UI is asking for the wrong category.  This problem
can be fixed in the binary download by editing the following file to
change QUERYHANDLER to QUERY, being careful to only change the uppercase
version of that string:

solr/server/solr-webapp/webapp/js/angular/controllers/dataimport.js

I have filed an issue to deal with the problem:

https://issues.apache.org/jira/browse/SOLR-10035

Fixing it is easy enough ... but perhaps a test that checks the UI
operation would be a good idea.  I've got no idea how to write that test.

Thanks,
Shawn

Re: no dataimport-handler defined!

[Alexandre Rafalovitch]

Strange.

If you run a pre-built DIH example, do any of the cores work? (not the
RSS one, that is broken anyway).

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 08:32, Chris Rogers
<chris.rog...@bodleian.ox.ac.uk> wrote:
> Hi Alex,
>
> I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie the 
> one generated from the configset when the node was created).
>
> I’m running standalone, not cloud.
>
> I’m restarting sole after every change. Do I need to reload the core instead 
> of restarting?
>
> I’ve also tried replacing the relative path to the .jar with an absolute path 
> to the dist directory. Still didn’t work.
>
> Thanks,
> Chris
>
> On 24/01/2017, 13:20, "Alexandre Rafalovitch" <arafa...@gmail.com> wrote:
>
> Which solrconfig.xml are you editing and what kind of Solr install are
> you running (cloud?). And did you reload the core.
>
> I suspect you are not editing the file that is actually in use. For
> example, if you are running a cloud setup, the solrconfig.xml on the
> filesystem is disconnected from the config actually in use that is
> stored in ZooKeeper. You would need to reupload it for change to take
> effect.
>
> You also may need to reload the core for changes to take effect.
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 24 January 2017 at 07:43, Chris Rogers
> <chris.rog...@bodleian.ox.ac.uk> wrote:
> > Hi all,
> >
> > Having frustrating issues with getting SOLR 6.4.0 to recognize the 
> existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
> >
> > The DIH .jar file appears to be loading correctly. There are no errors 
> in the SOLR logs. It just says “Sorry, no dataimport-handler defined” in the 
> SOLR admin UI.
> >
> > My config files are listed below. Can anyone spot any mistakes here?
> >
> > Many thanks,
> > Chris
> >
> > # solrconfig.xml ##
> >
> >regex=".*dataimporthandler-.*\.jar" />
> >
> > …
> >
> >class="org.apache.solr.handler.dataimport.DataImportHandler">
> > 
> >   DIH-data-config.xml
> > 
> >   
> >
> > # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
> >
> > 
> >   
> >   
> > 
> >  > fileName=".*xml"
> > newerThan="'NOW-5YEARS'"
> > recursive="true"
> > rootEntity="false"
> > dataSource="null"
> > 
> baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
> >
> >   
> >
> >>   forEach="/TEI" url="${f.fileAbsolutePath}" 
> transformer="RegexTransformer" >
> >  xpath="/TEI/teiHeader/fileDesc/titleStmt/title"/>
> >  xpath="/TEI/teiHeader/fileDesc/publicationStmt/publisher"/>
> >  xpath="/TEI/teiHeader/fileDesc/sourceDesc/msDesc/msIdentifier/altIdentifier/idno"/>
> >   
> >
> > 
> >
> >   
> > 
> >
> >
> > --
> > Chris Rogers
> > Digital Projects Manager
> > Bodleian Digital Library Systems and Services
> > chris.rog...@bodleian.ox.ac.uk
>
>

Re: no dataimport-handler defined!

[Chris Rogers]

A quick update. I rolled back to solr 6.2, and the data import handler is 
recognized there.

So there has either been a change in the config required between 6.2 and 6.4, 
or there’s a bug in 6.4

Any thoughts?   

On 24/01/2017, 13:32, "Chris Rogers" <chris.rog...@bodleian.ox.ac.uk> wrote:

Hi Alex,

I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie the 
one generated from the configset when the node was created).

I’m running standalone, not cloud.

I’m restarting sole after every change. Do I need to reload the core 
instead of restarting?

I’ve also tried replacing the relative path to the .jar with an absolute 
path to the dist directory. Still didn’t work.

Thanks,
Chris

On 24/01/2017, 13:20, "Alexandre Rafalovitch" <arafa...@gmail.com> wrote:

Which solrconfig.xml are you editing and what kind of Solr install are
you running (cloud?). And did you reload the core.

I suspect you are not editing the file that is actually in use. For
example, if you are running a cloud setup, the solrconfig.xml on the
filesystem is disconnected from the config actually in use that is
stored in ZooKeeper. You would need to reupload it for change to take
effect.

You also may need to reload the core for changes to take effect.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and 
experienced


On 24 January 2017 at 07:43, Chris Rogers
<chris.rog...@bodleian.ox.ac.uk> wrote:
> Hi all,
>
> Having frustrating issues with getting SOLR 6.4.0 to recognize the 
existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no 
errors in the SOLR logs. It just says “Sorry, no dataimport-handler defined” in 
the SOLR admin UI.
>
> My config files are listed below. Can anyone spot any mistakes here?
>
> Many thanks,
> Chris
>
> # solrconfig.xml ##
>
>   
>
> …
>
>   
> 
>   DIH-data-config.xml
> 
>   
>
> # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
>
> 
>   
>   
> 
>  fileName=".*xml"
> newerThan="'NOW-5YEARS'"
> recursive="true"
> rootEntity="false"
> dataSource="null"
> 
baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
>
>   
>
>  forEach="/TEI" url="${f.fileAbsolutePath}" 
transformer="RegexTransformer" >
> 
> 
> 
>   
>
> 
>
>   
> 
>
>
> --
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk

Re: no dataimport-handler defined!

[Chris Rogers]

Hi Alex,

I’m editing the solrconfig.xml file at /solr/server/solr/tei_config (ie the one 
generated from the configset when the node was created).

I’m running standalone, not cloud.

I’m restarting sole after every change. Do I need to reload the core instead of 
restarting?

I’ve also tried replacing the relative path to the .jar with an absolute path 
to the dist directory. Still didn’t work.

Thanks,
Chris

On 24/01/2017, 13:20, "Alexandre Rafalovitch" <arafa...@gmail.com> wrote:

Which solrconfig.xml are you editing and what kind of Solr install are
you running (cloud?). And did you reload the core.

I suspect you are not editing the file that is actually in use. For
example, if you are running a cloud setup, the solrconfig.xml on the
filesystem is disconnected from the config actually in use that is
stored in ZooKeeper. You would need to reupload it for change to take
effect.

You also may need to reload the core for changes to take effect.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 07:43, Chris Rogers
<chris.rog...@bodleian.ox.ac.uk> wrote:
> Hi all,
>
> Having frustrating issues with getting SOLR 6.4.0 to recognize the 
existence of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no errors in 
the SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
admin UI.
>
> My config files are listed below. Can anyone spot any mistakes here?
>
> Many thanks,
> Chris
>
> # solrconfig.xml ##
>
>   
>
> …
>
>   
> 
>   DIH-data-config.xml
> 
>   
>
> # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
>
> 
>   
>   
> 
>  fileName=".*xml"
> newerThan="'NOW-5YEARS'"
> recursive="true"
> rootEntity="false"
> dataSource="null"
> 
baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
>
>   
>
>  forEach="/TEI" url="${f.fileAbsolutePath}" 
transformer="RegexTransformer" >
> 
> 
> 
>   
>
> 
>
>   
> 
>
>
> --
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk

Re: no dataimport-handler defined!

[Alexandre Rafalovitch]

Which solrconfig.xml are you editing and what kind of Solr install are
you running (cloud?). And did you reload the core.

I suspect you are not editing the file that is actually in use. For
example, if you are running a cloud setup, the solrconfig.xml on the
filesystem is disconnected from the config actually in use that is
stored in ZooKeeper. You would need to reupload it for change to take
effect.

You also may need to reload the core for changes to take effect.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 24 January 2017 at 07:43, Chris Rogers
<chris.rog...@bodleian.ox.ac.uk> wrote:
> Hi all,
>
> Having frustrating issues with getting SOLR 6.4.0 to recognize the existence 
> of my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.
>
> The DIH .jar file appears to be loading correctly. There are no errors in the 
> SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
> admin UI.
>
> My config files are listed below. Can anyone spot any mistakes here?
>
> Many thanks,
> Chris
>
> # solrconfig.xml ##
>
>regex=".*dataimporthandler-.*\.jar" />
>
> …
>
>class="org.apache.solr.handler.dataimport.DataImportHandler">
> 
>   DIH-data-config.xml
> 
>   
>
> # DIH-data-config.xml (in the same dir as solrconfig.xml) ##
>
> 
>   
>   
> 
>  fileName=".*xml"
> newerThan="'NOW-5YEARS'"
> recursive="true"
> rootEntity="false"
> dataSource="null"
> 
> baseDir="/home/bodl-tei-svc/sites/bodl-tei-svc/var/data/tolkein_tei">
>
>   
>
>  forEach="/TEI" url="${f.fileAbsolutePath}" 
> transformer="RegexTransformer" >
>  xpath="/TEI/teiHeader/fileDesc/titleStmt/title"/>
>  xpath="/TEI/teiHeader/fileDesc/publicationStmt/publisher"/>
>  xpath="/TEI/teiHeader/fileDesc/sourceDesc/msDesc/msIdentifier/altIdentifier/idno"/>
>   
>
> 
>
>   
> 
>
>
> --
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk

no dataimport-handler defined!

[Chris Rogers]

Hi all,

Having frustrating issues with getting SOLR 6.4.0 to recognize the existence of 
my DIH config. I’m using Oracle Java8 jdk on Ubuntu 14.04.

The DIH .jar file appears to be loading correctly. There are no errors in the 
SOLR logs. It just says “Sorry, no dataimport-handler defined” in the SOLR 
admin UI.

My config files are listed below. Can anyone spot any mistakes here?

Many thanks,
Chris

# solrconfig.xml ##

  

…

  

  DIH-data-config.xml

  

# DIH-data-config.xml (in the same dir as solrconfig.xml) ##


  
  



  

  



  



  



--
Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rog...@bodleian.ox.ac.uk

Re: Retaining a field value during DataImport

[Selvam]

Hi,

Thanks, I will look into options specified.


On Mon, Sep 26, 2016 at 4:35 PM, Alexandre Rafalovitch 
wrote:

> Transformers do not see what's in the Solr index, they are too early
> in the processing chain.
>
> You could probably do something by exporting that field's value,
> caching it and injecting it back with transformer from that cache.
> Messy but doable.
>
> UpdateRequestProcessor would be able to do it, but your request from
> DIH is coming as a new document, not an update. So the old one would
> be overidden.
>
> SOLR-9530 could be an answer to that, but it is just a design so far -
> no implementation. You could write one yourself or see if showing
> excitement on the JIRA and being ready to debug the patch would get
> the committer's attention.
>
>
> Regards,
> Alex.
> 
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 26 September 2016 at 17:36, Selvam  wrote:
> > Hi All,
> >
> > We use DataImportHandler to import data from Redshift. We want to
> overwrite
> > some 250M existing records (that has around 350 columns) while retaining
> > the field value of only one column in those 250M records. The reason is,
> > that one column is a multi-valued and requires a costly query to build
> that
> > values again.
> >
> > I learned about Transformers, I am not sure if it is possible to get the
> > old document value during that process. Any help would be appreciated.
> >
> >
> > --
> > Regards,
> > Selvam
>



-- 
Regards,
Selvam
KnackForge 

Re: Retaining a field value during DataImport

[Alexandre Rafalovitch]

Transformers do not see what's in the Solr index, they are too early
in the processing chain.

You could probably do something by exporting that field's value,
caching it and injecting it back with transformer from that cache.
Messy but doable.

UpdateRequestProcessor would be able to do it, but your request from
DIH is coming as a new document, not an update. So the old one would
be overidden.

SOLR-9530 could be an answer to that, but it is just a design so far -
no implementation. You could write one yourself or see if showing
excitement on the JIRA and being ready to debug the patch would get
the committer's attention.


Regards,
Alex.

Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 26 September 2016 at 17:36, Selvam  wrote:
> Hi All,
>
> We use DataImportHandler to import data from Redshift. We want to overwrite
> some 250M existing records (that has around 350 columns) while retaining
> the field value of only one column in those 250M records. The reason is,
> that one column is a multi-valued and requires a costly query to build that
> values again.
>
> I learned about Transformers, I am not sure if it is possible to get the
> old document value during that process. Any help would be appreciated.
>
>
> --
> Regards,
> Selvam

Retaining a field value during DataImport

[Selvam]

Hi All,

We use DataImportHandler to import data from Redshift. We want to overwrite
some 250M existing records (that has around 350 columns) while retaining
the field value of only one column in those 250M records. The reason is,
that one column is a multi-valued and requires a costly query to build that
values again.

I learned about Transformers, I am not sure if it is possible to get the
old document value during that process. Any help would be appreciated.


-- 
Regards,
Selvam

Solr : DataImport failed from Cassandra due to TimedOut Exception

[saravanan]

Hi 

I am new to solr just exploring how the solr is working . I did the import
from cassandra database to solr . It imported successfully for 500
records(i.e one records = 36 doc in solr) .For next 100 records it is
throwing"java.sql.SQLTransientConnectionException: TimedOutException()"
Below attached is the image of exception
  <http://lucene.472066.n3.nabble.com/file/n4297088/solr_timeout.png> 

Version of Solr : 5.0.0

Dependency jars for Cassandra : cassandra-jdbc-1.2.5.jar
,cassandra-thrift-1.2.5.jar,libthrift-0.7.0.jar,cassandra-clientutil-1.2.6.jar

I dont know whether i need to increase the documents size that the solr is
processing or i need to increase the timout . Please suggest some recovery 

Data-config.xml









 


 

  
 
 
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 








Schema.xml



























   
   
  

SolrConfig.xml



data-config.xml




Thanks
Saravanan



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-DataImport-failed-from-Cassandra-due-to-TimedOut-Exception-tp4297088.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: DataImport-Solr6: Nested Entities

[Shawn Heisey]

On 8/18/2016 5:10 PM, Peri Subrahmanya wrote:
> Hi,
>
> I have a simple one-to-many relationship setup in the data-import.xml and 
> when I try to index it using the dataImportHandler, Solr complains of “no 
> unique id found”. 
>
> managed-schema.xml
> id
> solrconfig,xml:
> 
>   
> id
>   

>  query=“select blah blah from course where 
> catalog_id=‘${catalog.catalog_id}'">
> 

Can you get the full error message(s) from the solr.log file, including
the full java stacktrace(s)?  Many error messages are dozens of lines
long, because they include Java stacktraces.  For correct
interpretation, we also need the exact version of Solr that you're
running.  Your subject indicates Solr6, but there are three releases so
far in the 6.x series.

If you want your update processor chain to be used by DIH, I think you
need to make it the default chain with 'default="true"' in the opening
tag.  There might be a way to apply a specific update chain in DIH, but
if there is, you need to give it a name, which yours doesn't have.

I am using a custom update chain with both DIH and explicit update
requests, which I do like this:



Thanks,
Shawn

Re: DataImport-Solr6: Nested Entities

[Alexandre Rafalovitch]

Well, do both parent and child entity have a field called 'id'
containing their corresponding unique ids? That would be the first
step.

Regards,
   Alex.

Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 19 August 2016 at 09:10, Peri Subrahmanya
 wrote:
> Hi,
>
> I have a simple one-to-many relationship setup in the data-import.xml and 
> when I try to index it using the dataImportHandler, Solr complains of “no 
> unique id found”.
>
> managed-schema.xml
> id
> solrconfig,xml:
> 
>   
> id
>   
>   
>   
> 
>
> data-import.xml
> 
>  driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://x.x.x.x:3306/xx"
> user=“blah"
> password=“blah"/>
>
> 
>  query="select blah blah from catalog">
>
>  query=“select blah blah from course where 
> catalog_id=‘${catalog.catalog_id}'">
> 
>
> 
> 
>
> 
>
> Could someone please advise?
>
> Thanks
> -Peri

DataImport-Solr6: Nested Entities

[Peri Subrahmanya]

Hi,

I have a simple one-to-many relationship setup in the data-import.xml and when 
I try to index it using the dataImportHandler, Solr complains of “no unique id 
found”. 

managed-schema.xml
id
solrconfig,xml:

  
id
  
  
  


data-import.xml














Could someone please advise?

Thanks
-Peri

Re: trying DIH but get 'Sorry, no dataimport-handler defined!'

[scott.chu]

I happen to find the problem. The problem seems to come from the html file that 
shows DIH function page. I use Maxthon browser, it has a function that can 
switch between IE mode and non-IE mode (actually the Webkit engine). I happen 
to switch back to non-IE mode and the error message is gone and everything is 
ok now! TOO WEIRD!


scott.chu，scott@udngroup.com
2016/5/24 (週二)
- Original Message - 
From: kostali hassan 
To: solr-user ; scott(自己) 
CC: 
Date: 2016/5/24 (週二) 16:55
Subject: Re: trying DIH but get 'Sorry, no dataimport-handler defined!'


if you have in this path server/solr/configsets/testdih/conf you shoud 
right this in your line commande: 
'bin\solr>solr create -c your_core -d testdih -p 8983 to create a core with 
an exemple config testdih. 

2016-05-24 9:35 GMT+01:00 scott.chu <scott@udngroup.com>: 

> 
> I do following things: 
> 
> * I create folder : D:\solr-6.0.0\myconfigsets\testdih. 
> * Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to 
> D:\solr-6.0.0\myconfigsets\testdih. 
> * Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit 
> db-data-config.xml as follows (I am pretty sure mysql environment is ok): 
> 
>  
>  url="jdbc:mysql://localhost:3306/test" user="hello" password="hellothere" /> 
>  
>  

>  
>  
>  
>  
>  
>  
>  
> 
> * Then I copy mysql-connector-java-5.0.8-bin.jar to 
> D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib. 
> * I check solrconfig.xml and see these relevant lines: 
> 
>  regex="solr-dataimporthandler-.*\.jar" /> 
> ... 
> ... 
>  
>  
> db-data-config.xml 
>  
>  
> 
> * cd to D:solr-6.0.0, issue 'bin\solr start', it starts ok. 
> * Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to 
> create a core. It's ok, too. 
> 
> * The solr.log has these log messages: 
> 
> 2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
> o.a.s.c.SolrResourceLoader Adding 
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to 
> classloader 
> 2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
> o.a.s.c.SolrResourceLoader Adding 
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar'
>  
> to classloader 
> 
> * So I think dih jars are loaded ok. 
> 
> I go to localhost:893 in browser and select core 'testdih', then click 
> 'DataImport' item but rightpane shows "Sorry, no dataimport-handler 
> defined!". 
> 
> What do I miss? 
> 
> 
> scott.chu，scott@udngroup.com 
> 2016/5/24 (週二) 
> 



- 
未在此訊息中找到病毒。 
已透過 AVG 檢查 - www.avg.com 
版本: 2015.0.6201 / 病毒庫: 4568/12285 - 發佈日期: 05/23/16

Re: trying DIH but get 'Sorry, no dataimport-handler defined!'

[kostali hassan]

if you have in  this path server/solr/configsets/testdih/conf you shoud
right this in your line commande:
'bin\solr>solr create -c your_core -d testdih -p 8983 to create a core with
an exemple config testdih.

2016-05-24 9:35 GMT+01:00 scott.chu <scott@udngroup.com>:

>
> I do following things:
>
> * I create folder : D:\solr-6.0.0\myconfigsets\testdih.
> * Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to
> D:\solr-6.0.0\myconfigsets\testdih.
> * Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit
> db-data-config.xml as follows (I am pretty sure mysql environment is ok):
>
>   
>url="jdbc:mysql://localhost:3306/test" user="hello" password="hellothere" />
>   
>   
>   
>   
>   
>   
>   
>   
>   
>
> * Then I copy mysql-connector-java-5.0.8-bin.jar to
> D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib.
> * I check solrconfig.xml  and see these relevant lines:
>
>  regex="solr-dataimporthandler-.*\.jar" />
>   ...
>   ...
>   
>   
> db-data-config.xml
>   
> 
>
> * cd to  D:solr-6.0.0, issue 'bin\solr start', it starts ok.
> * Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to
> create a core. It's ok, too.
>
> * The solr.log has these log messages:
>
> 2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ]
> o.a.s.c.SolrResourceLoader Adding
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to
> classloader
> 2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ]
> o.a.s.c.SolrResourceLoader Adding
> 'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar'
> to classloader
>
> * So I think dih jars are loaded ok.
>
> I go to localhost:893 in browser and select core 'testdih', then click
> 'DataImport' item but rightpane shows "Sorry, no dataimport-handler
> defined!".
>
>  What do I miss?
>
>
> scott.chu，scott@udngroup.com
> 2016/5/24 (週二)
>

Re: trying DIH but get 'Sorry, no dataimport-handler defined!'

[scott.chu]

I try run the example by issuing "bin\solr create_core -c exampledih -d 
example\example-DIH\solr\db\conf". It also shows same error. Do I issue wrong 
command?

scott.chu，scott@udngroup.com
2016/5/24 (週二)
- Original Message - 
From: scott(自己) 
To: solr-user 
CC: 
Date: 2016/5/24 (週二) 16:35
Subject: trying DIH but get 'Sorry, no dataimport-handler defined!'



I do following things: 

* I create folder : D:\solr-6.0.0\myconfigsets\testdih. 
* Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to 
D:\solr-6.0.0\myconfigsets\testdih. 
* Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit db-data-config.xml 
as follows (I am pretty sure mysql environment is ok): 

   
   
   
   
   
   
   
   
   
   
   

* Then I copy mysql-connector-java-5.0.8-bin.jar to 
D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib. 
* I check solrconfig.xml and see these relevant lines: 

 
  ... 
  ... 
   
   
db-data-config.xml 
   
 

* cd to D:solr-6.0.0, issue 'bin\solr start', it starts ok. 
* Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to 
create a core. It's ok, too. 

* The solr.log has these log messages: 

2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to 
classloader 
2016-05-24 15:59:24,781 INFO (coreLoadExecutor-6-thread-1) [ ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar' 
to classloader 

* So I think dih jars are loaded ok. 

I go to localhost:893 in browser and select core 'testdih', then click 
'DataImport' item but rightpane shows "Sorry, no dataimport-handler defined!". 

 What do I miss? 


scott.chu，scott@udngroup.com 
2016/5/24 (週二) 


- 
未在此訊息中找到病毒。 
已透過 AVG 檢查 - www.avg.com 
版本: 2015.0.6201 / 病毒庫: 4568/12285 - 發佈日期: 05/23/16

trying DIH but get 'Sorry, no dataimport-handler defined!'

[scott.chu]

I do following things:

* I create folder : D:\solr-6.0.0\myconfigsets\testdih.
* Copy D:\portable_sw\solr-6.0.0\example\example-DIH\solr\db\conf to 
D:\solr-6.0.0\myconfigsets\testdih.
* Go into D:\solr-6.0.0\myconfigsets\testdih\conf and edit db-data-config.xml 
as follows (I am pretty sure mysql environment is ok):

  
  
  
  
  
  
  
  
  
  
  

* Then I copy mysql-connector-java-5.0.8-bin.jar to 
D:\portable_sw\solr-6.0.0\server\solr-webapp\webapp\WEB-INF\lib.
* I check solrconfig.xml  and see these relevant lines:


  ...
  ...
  
  
db-data-config.xml
  


* cd to  D:solr-6.0.0, issue 'bin\solr start', it starts ok.
* Issue 'bin\solr create_core -c testdih -d myconfigsets\testdih\conf' to 
create a core. It's ok, too.

* The solr.log has these log messages:

2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-6.0.0.jar' to 
classloader
2016-05-24 15:59:24,781 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.SolrResourceLoader Adding 
'file:/D:/portable_sw/solr-6.0.0/dist/solr-dataimporthandler-extras-6.0.0.jar' 
to classloader

* So I think dih jars are loaded ok.

I go to localhost:893 in browser and select core 'testdih', then click 
'DataImport' item but rightpane shows "Sorry, no dataimport-handler defined!".

 What do I miss?


scott.chu，scott@udngroup.com
2016/5/24 (週二)

RE: dataimport db-data-config.xml

[Davis, Daniel (NIH/NLM) [C]]

Kishor,

Data Import Handler doesn't know how to randomly access rows from the CSV to 
"JOIN" them to rows from the MySQL table at indexing time.
However, both MySQL and Solr know how to JOIN rows/documents from multiple 
tables/collections/cores.

Data Import Handler could read the CSV first, and query MySQL within that, but 
I don't think that's a great architecture because it depends on the business 
requirements in a rather brittle way (more on this below).

So, I see three basic architectures:

Use MySQL to do the JOIN:
--
- Your indexing isn't just DIH, but a script that first.
- Imports the CSV into a MySQL table, validating that the id in the CSV table 
is found in the MySQL table.
- Your DIH has either an  for one SQL query that contains an  
for the other SQL query, or it has a JOIN query/query on a MySQL view.

This is ideal if:
- Your resources (including you) are more familiar with RDBMS technology than 
Solr.
- You have no business requirement to return rows from just the MySQL table or 
just the CSV as search results.
- The data is small enough that the processing time to import into MySQL each 
time you index is acceptable.

Use Solr to do the JOIN:
--
- Index all the rows from the CSV as documents within Solr, 
- Index all the rows from the MySQL table as documents within Solr,
- Use JOIN queries to query them together.

This is ideal if:
- You don't control the MySQL database, and have no way at all to add a table 
to it.
- You have a business requirement to return either or both results from the 
MySQL table or the CSV.
- You want Solr JOIN queries on your Solr resume ;)   Not a terribly good 
reason, I guess.


Use Data Import Handler to do the JOIN:
---
If you absolutely want to join the data using Data Import Handler, then:
- Have DIH loop through the CSV *first*, and then make queries based on the id 
into the MySQL table.
- In this case, the  for the MySQL query will appear within the 
 for the CSV row, which will appear within an  for the CSV file 
within the filesystem.
- The  for the CSV row would be the primary document entity.

This is only appropriate if:
- There is no business requirement to search for results directly from the 
MySQL table on its own.
- Your business requirements suggest one result for each row from the CSV, 
rather than from the MySQL table or either way.
- The CSV contains every id in the MySQL table, or the entries within the MySQL 
table that don't have anything from the CSV shouldn't appear in the results 
anyway.


-Original Message-
From: kishor [mailto:krajus...@gmail.com] 
Sent: Friday, April 29, 2016 4:58 AM
To: solr-user@lucene.apache.org
Subject: dataimport db-data-config.xml

I want to import data from mysql-table and csv file ata the same time beacuse 
some data are in mysql tables and some are in csv file . I want to match 
specific id from mysql table in csv file then add the data in solar.

What i think or wnat to do








   




   

Is this possible in solr? 

Please suggest me How to import data from csv and mysql table at the same time.









--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-db-data-config-xml-tp4270673p4273614.html
Sent from the Solr - User mailing list archive at Nabble.com.

dataimport db-data-config.xml

[kishor]

I want to import data from mysql-table and csv file ata the same time beacuse
some data are in mysql tables and some are in csv file . I want to match
specific id from mysql table in csv file then add the data in solar.

What i think or wnat to do








   




   

Is this possible in solr? 

Please suggest me How to import data from csv and mysql table at the same
time.









--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-db-data-config-xml-tp4270673p4273614.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport db-data-config.xml

[Reth RM]

 What are the errors reported?  Errors can be either seen on admin page
logging tab or log file under solr_home.
If you follow the steps mentioned on the blog precisely, it should almost
work
http://solr.pl/en/2010/10/11/data-import-handler-%E2%80%93-how-to-import-data-from-sql-databases-part-1/
<http://solr.pl/en/2010/10/11/data-import-handler-%E2%80%93-how-to-import-data-from-sql-databases-part-1/>
If you encounter errors at any step, lets us know.




On Sat, Apr 16, 2016 at 10:49 AM, kishor <krajus...@gmail.com> wrote:

> I am try to run two pgsql query on same data-source. is this possible in
> db-data-config.xml.
>
>
> 
>
>  url="jdbc:postgresql://0.0.0.0:5432/iboats"
> user="iboats"
> password="root" />
>
> 
>  transformer="TemplateTransformer">
>
>  template="user1-${user1.id}"/>
>
>
> This code is not working please suggest any more example
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/dataimport-db-data-config-xml-tp4270673.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

dataimport db-data-config.xml

[kishor]

I am try to run two pgsql query on same data-source. is this possible in
db-data-config.xml.










   

This code is not working please suggest any more example




--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-db-data-config-xml-tp4270673.html
Sent from the Solr - User mailing list archive at Nabble.com.

dataimport db-data-config.xml

[kishor]

I am try to run two pgsql query on same data-source. is this possible in
db-data-config.xml.










   

This code is not working please suggest any more example




--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-db-data-config-xml-tp4270674.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Not able to do DataImport from MSSQL server to solr server

[Shawn Heisey]

On 3/14/2016 2:31 AM, Adi@GTN wrote:
> Am using the below config details in order to connect the database, and pull
> the query but in the connection details it is mentioned as
> authenticateMethod=ntlm, am bit confused here am not sure what is this
> authentication method, now am not able to connect the database so am not
> able to debug and understand what is the username it is being used in-order
> to connect the database. Can anyone help me understanding and give some
> inputs on how to debug. 
>
>  driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
> url="jdbc:sqlserver://localhost;databaseName=testdb;integratedSecurity=true;responseBuffering=adaptive;selectMethod=direct;;authenticationMethod=ntlm;"
> batchsize="1" readnonly="true"/>   

I'm not an expert on SQL Server, and honestly I hope that I never am.

>From what I have been able to determine with Google, the
integratedSecurity option (which you have) should pull the credentials
from the user on the Windows machine that is running Solr.  Note that if
you figure out how to install Solr as a service and use the System or
Network user to run it, this probably is NOT going to do what you want.

You might need to remove the authenticationMethod parameter entirely,
and if that doesn't work, you're going to need to consult Microsoft
about how to properly use their JDBC driver.  You also might want to
consult the logs from SQL Server to determine what credentials it is
seeing, as well as the Solr server log to see whether any errors are
logged there.

Note that you have mistyped the batchSize and readOnly parameters. 
These parameter names are case sensitive and readOnly is misspelled. 
You should probably remove the batchsize parameter entirely, and fix
readOnly.  The problem that most people are trying to avoid with
batchSize is controlled in a different way:

http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F

Thanks,
Shawn

Not able to do DataImport from MSSQL server to solr server

[Adi@GTN]

Hi All,

Am using the below config details in order to connect the database, and pull
the query but in the connection details it is mentioned as
authenticateMethod=ntlm, am bit confused here am not sure what is this
authentication method, now am not able to connect the database so am not
able to debug and understand what is the username it is being used in-order
to connect the database. Can anyone help me understanding and give some
inputs on how to debug. 

   

Thanks 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Not-able-to-do-DataImport-from-MSSQL-server-to-solr-server-tp4263567.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: sorry, no dataimport-handler defined!

[kostali hassan]

in request data import handler for solrconfig.xml do :



  tika-data-config.xml

  

and define your file tika-data-config.xml and put this file in the
directory config from your core.

2016-02-02 17:35 GMT+00:00 Jean-Jacques Monot <jj_mo...@yahoo.fr>:

> Exact. Newbie user !
>
> OK i have seen what is missing ...
>
> Le 2 févr. 2016 15:40, "Davis, Daniel (NIH/NLM) [C]" <daniel.da...@nih.gov>
> a écrit :
> >
> > It sounds a bit like you are just exploring Solr for the first time.
> To use the Data Import Handler, you need to create an XML file that
> configures it, data-config.xml by default.
> >
> > But before we go into details, what are you trying to accomplish with
> Solr?
> >
> > -Original Message-
> > From: Jean-Jacques MONOT [mailto:jj_mo...@yahoo.fr]
> > Sent: Monday, February 01, 2016 2:31 PM
> > To: solr-user@lucene.apache.org
> > Subject: Potential SPAM:sorry, no dataimport-handler defined!
> >
> > Hello
> >
> > I am using SOLR 5.4.1 and the graphical admin UI.
> >
> > I successfully created multiples cores and indexed various documents,
> using in line commands : (create -c) and (post.jar) on W10.
> >
> > But in the GUI, when I click on "Dataimport", I get the following
> message : "sorry, no dataimport-handler defined!"
> >
> > I get the same message even on 5.3.1 or for different cores.
> >
> > What is wrong ?
> >
> > JJM
> >
> > ---
> > L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> > https://www.avast.com/antivirus
> >
> >
>

sorry, no dataimport-handler defined!

[Davis, Daniel (NIH/NLM) [C]]

It sounds a bit like you are just exploring Solr for the first time.   To use 
the Data Import Handler, you need to create an XML file that configures it, 
data-config.xml by default.

But before we go into details, what are you trying to accomplish with Solr?

-Original Message-
From: Jean-Jacques MONOT [mailto:jj_mo...@yahoo.fr] 
Sent: Monday, February 01, 2016 2:31 PM
To: solr-user@lucene.apache.org
Subject: Potential SPAM:sorry, no dataimport-handler defined!

Hello

I am using SOLR 5.4.1 and the graphical admin UI.

I successfully created multiples cores and indexed various documents, using in 
line commands : (create -c) and (post.jar) on W10.

But in the GUI, when I click on "Dataimport", I get the following message : 
"sorry, no dataimport-handler defined!"

I get the same message even on 5.3.1 or for different cores.

What is wrong ?

JJM

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

Re: sorry, no dataimport-handler defined!

[Jean-Jacques Monot]

Exact. Newbie user !

OK i have seen what is missing ...

Le 2 févr. 2016 15:40, "Davis, Daniel (NIH/NLM) [C]" <daniel.da...@nih.gov> a 
écrit :
>
> It sounds a bit like you are just exploring Solr for the first time.   To use 
> the Data Import Handler, you need to create an XML file that configures it, 
> data-config.xml by default. 
>
> But before we go into details, what are you trying to accomplish with Solr? 
>
> -Original Message- 
> From: Jean-Jacques MONOT [mailto:jj_mo...@yahoo.fr] 
> Sent: Monday, February 01, 2016 2:31 PM 
> To: solr-user@lucene.apache.org 
> Subject: Potential SPAM:sorry, no dataimport-handler defined! 
>
> Hello 
>
> I am using SOLR 5.4.1 and the graphical admin UI. 
>
> I successfully created multiples cores and indexed various documents, using 
> in line commands : (create -c) and (post.jar) on W10. 
>
> But in the GUI, when I click on "Dataimport", I get the following message : 
> "sorry, no dataimport-handler defined!" 
>
> I get the same message even on 5.3.1 or for different cores. 
>
> What is wrong ? 
>
> JJM 
>
> --- 
> L'absence de virus dans ce courrier électronique a été vérifiée par le 
> logiciel antivirus Avast. 
> https://www.avast.com/antivirus 
>
>

sorry, no dataimport-handler defined!

[Jean-Jacques MONOT]

Hello

I am using SOLR 5.4.1 and the graphical admin UI.

I successfully created multiples cores and indexed various documents,
using in line commands : (create -c) and (post.jar) on W10.

But in the GUI, when I click on "Dataimport", I get the following
message : "sorry, no dataimport-handler defined!"

I get the same message even on 5.3.1 or for different cores.

What is wrong ?

JJM

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel 
antivirus Avast.
https://www.avast.com/antivirus

Re: sorry, no dataimport-handler defined!

[Susheel Kumar]

Please register Data Import Handler to work with it
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler


On Mon, Feb 1, 2016 at 2:31 PM, Jean-Jacques MONOT <jj_mo...@yahoo.fr>
wrote:

> Hello
>
> I am using SOLR 5.4.1 and the graphical admin UI.
>
> I successfully created multiples cores and indexed various documents,
> using in line commands : (create -c) and (post.jar) on W10.
>
> But in the GUI, when I click on "Dataimport", I get the following message
> : "sorry, no dataimport-handler defined!"
>
> I get the same message even on 5.3.1 or for different cores.
>
> What is wrong ?
>
> JJM
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
>

Re: Many files /dataImport in same project

[fabigol]

So I can run a script containing a fixed time for each file. It is a
solution. is that good and only solution?

My solrConfig.xml file, I declare a data-import files for 6.
can i group in the same data-import?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Many-files-dataImport-in-same-project-tp4237731p4237868.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Many files /dataImport in same project

[fabigol]

Hi,
My problem is that I got a an old 3-year project without knowing whether it
worked
I passed the stage of understanding and I managed to make it work. At the
beginning I think about why 6 files
I'm in a new stage which is optimization.
I have 6-import data files and am currently forced to launch after each
other.
So exite there a way to launch to launch 6 once.
Thank you still hoping to have been clear enough.
thank a lot



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Many-files-dataImport-in-same-project-tp4237731p4237865.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Many files /dataImport in same project

[Gora Mohanty]

On 2 November 2015 at 22:38, Alexandre Rafalovitch  wrote:
> On 2 November 2015 at 11:30, Gora Mohanty  wrote:
>> As per my last
>> follow-up, there is currently no way to have DIH automatically pick up
>> different data-config files without manually editing the DIH
>> configuration each time.
>
> I missed previous discussions, but the DIH config file is given in a
> query parameter. So, if there is a bunch of them on a file system, one
> could probably do
> find . - name "*.dihconf" | xargs curl .

Sorry, I might be missing something, or things might have changed with
newer Solr versions. I thought that the DIH data-config file had to be
specified in solrconfig.xml.

Regards,
Gora

Re: Many files /dataImport in same project

[Erick Erickson]

A possibility: Define 6 different request handlers, something
like:

 

  /home/username/data-config-1.xml

  

 

  /home/username/data-config-2.xml

  

And fire off 6 separate commands, one to each end point.

WARNING! I have not tried this personally, so it might be an "adventure"!

Another alternative would be moving to SolrJ rather than using DIH, it's
much more flexible and you have total control over,
say, how many clients you run etc.

Here's a code sample to get you started:

https://lucidworks.com/blog/2012/02/14/indexing-with-solrj/

That has some Tika processing as well, but that should be
easy to pull out. I'm envisioning a stand-alone program that
has a parameter which one of your jobs to execute.

NOTE: I'm not suggesting that you can just use the current
DIH config files, rather that you express the DIH functioning
in SolrJ.

Best,
Erick

On Tue, Nov 3, 2015 at 1:07 AM, fabigol <fabien.stou...@vialtis.com> wrote:
> So I can run a script containing a fixed time for each file. It is a
> solution. is that good and only solution?
>
> My solrConfig.xml file, I declare a data-import files for 6.
> can i group in the same data-import?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Many-files-dataImport-in-same-project-tp4237731p4237868.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Many files /dataImport in same project

[Gora Mohanty]

On 3 November 2015 at 21:25, Alexandre Rafalovitch  wrote:
> On 3 November 2015 at 10:38, Gora Mohanty  wrote:
>>> I missed previous discussions, but the DIH config file is given in a
>>> query parameter. So, if there is a bunch of them on a file system, one
>>> could probably do
>>> find . - name "*.dihconf" | xargs curl .
>>
>> Sorry, I might be missing something, or things might have changed with
>> newer Solr versions. I thought that the DIH data-config file had to be
>> specified in solrconfig.xml.
>
> It is just a parameter to the request handler. I am 99.5% sure you
> just pass config= parameter to the call and it picks it up then.
> That's why you can keep updating the definition file and not need to
> reload the core, it gets reloaded on each DIH call.
[...]

OK, looks like you are right. Had never had reason to use this, so
hadn't realised that this was possible.

Regards,
Gora

Re: Many files /dataImport in same project

[Alexandre Rafalovitch]

On 3 November 2015 at 10:38, Gora Mohanty  wrote:
>> I missed previous discussions, but the DIH config file is given in a
>> query parameter. So, if there is a bunch of them on a file system, one
>> could probably do
>> find . - name "*.dihconf" | xargs curl .
>
> Sorry, I might be missing something, or things might have changed with
> newer Solr versions. I thought that the DIH data-config file had to be
> specified in solrconfig.xml.

It is just a parameter to the request handler. I am 99.5% sure you
just pass config= parameter to the call and it picks it up then.
That's why you can keep updating the definition file and not need to
reload the core, it gets reloaded on each DIH call.

Now, I am not sure what happens if you run DIH several time with
different files and then look for status. It is asynchronous, so.


Regards,
   Alex.



Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/

Re: Many files /dataImport in same project

[Alexandre Rafalovitch]

On 2 November 2015 at 11:30, Gora Mohanty  wrote:
> As per my last
> follow-up, there is currently no way to have DIH automatically pick up
> different data-config files without manually editing the DIH
> configuration each time.

I missed previous discussions, but the DIH config file is given in a
query parameter. So, if there is a bunch of them on a file system, one
could probably do
find . - name "*.dihconf" | xargs curl .

Regards,
   Alex.

Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/

Re: Many files /dataImport in same project

[Gora Mohanty]

On 2 November 2015 at 21:50, fabigol <fabien.stou...@vialtis.com> wrote:
> Hi,
>  i have many files of config dataImport
> I want to start at once instead of launching DataImport for each file.
> is it possible??

Not to be antagonistic, but did you not ask this before, and have
various people not tried to help you?

With all due respect, it seems that you need to understand your
specific setup better in order to ask more specific questions. It
would be good if you stuck to one thread for that. As per my last
follow-up, there is currently no way to have DIH automatically pick up
different data-config files without manually editing the DIH
configuration each time. This is probably unlikely to get fixed as one
can put all DIH entities into one file, and import each as needed.
Further, if what you need is complex requirements in populating Solr,
it is advisable to use SolrJ, or similar libraries for other
languages.

Regards,
Gora

Many files /dataImport in same project

[fabigol]

Hi,
 i have many files of config dataImport
I want to start at once instead of launching DataImport for each file.
is it possible??



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Many-files-dataImport-in-same-project-tp4237731.html
Sent from the Solr - User mailing list archive at Nabble.com.