Re: Data Import Handler (DIH) - Installing and running

[Erick Erickson]

Have you done what the message says and looked at your Solr log? If so,
what information is there?

> On Dec 23, 2020, at 5:13 AM, DINSD | SPAutores 
>  wrote:
> 
> Hi,
> 
> I'm trying to install the package "data-import-handler", since it was 
> discontinued from core SolR distro.
> 
> https://github.com/rohitbemax/dataimporthandler
> 
> However, as soon as the first command is carried out
> 
> solr -c -Denable.packages=true
> 
> I get this screen in web interface
> 
> 
> 
> Has anyone been through this, or have any idea why it's happening ?
> 
> Thanks for any help
> Rui Pimentel
> 
> 
> 
> DINSD - Departamento de Informática / SPA Digital
> Av. Duque de Loulé, 31 - 1069-153 Lisboa  PORTUGAL
> T (+ 351) 21 359 44 36 / (+ 351) 21 359 44 00  F (+ 351) 21 353 02 57
>  informat...@spautores.pt
>  www.SPAutores.pt
> 
> Please consider the environment before printing this email 
> 
> Esta mensagem electrónica, incluindo qualquer dos seus anexos, contém 
> informação PRIVADA, CONFIDENCIAL e de DIVULGAÇÃO PROIBIDA,e destina-se 
> unicamente à pessoa e endereço electrónico acima indicados. Se não for o 
> destinatário desta mensagem, agradecemos que a elimine e nos comunique de 
> imediato através do telefone  +351 21 359 44 00 ou por email para: 
> ge...@spautores.pt 
> 
> This electronic mail transmission including any attachment hereof, contains 
> information that is PRIVATE, CONFIDENTIAL and PROTECTED FROM DISCLOSURE, and 
> it is only for the use of the person and the e-mail address above indicated. 
> If you have received this electronic mail transmission in error, please 
> destroy it and notify us immediately through the telephone number  +351 21 
> 359 44 00 or at the e-mail address:  ge...@spautores.pt
>  

Data Import Handler (DIH) - Installing and running

[DINSD | SPAutores]

Hi,

I'm trying to install the package "data-import-handler", since it was 
discontinued from core SolR distro.


https://github.com/rohitbemax/dataimporthandler

However, as soon as the first command is carried out

solr -c -Denable.packages=true

I get this screen in web interface

Has anyone been through this, or have any idea why it's happening ?

Assinatura SPA Thanks for any help
**
*Rui Pimentel*


**
*DINSD - Departamento de Informática / SPA Digital*
Av. Duque de Loulé, 31 - 1069-153 Lisboa PORTUGAL
*T * (+ 351) 21 359 44 36 */* (+ 351) 21 359 44 00 *F* (+ 351) 21 353 02 57
<mailto:%7bmailsector...@spautores.pt> informat...@spautores.pt
<http://www.spautores.pt/>www.SPAutores.pt
<https://www.facebook.com/spautores> 
<https://www.youtube.com/user/SPAutores1925><https://plus.google.com/107542947146636584118><https://www.linkedin.com/company/spautores> 


Please consider the environment before printing this email

Esta mensagem electrónica, incluindo qualquer dos seus anexos, contém 
informação PRIVADA, CONFIDENCIAL e de DIVULGAÇÃO PROIBIDA,e destina-se 
unicamente à pessoa e endereço electrónico acima indicados. Se não for o 
destinatário desta mensagem, agradecemos que a elimine e nos comunique 
de imediato através do telefone +351 21 359 44 00 ou por email para: 
ge...@spautores.pt <mailto:ge...@spautores.pt>


This electronic mail transmission including any attachment hereof, 
contains information that is PRIVATE, CONFIDENTIAL and PROTECTED FROM 
DISCLOSURE, and it is only for the use of the person and the e-mail 
address above indicated. If you have received this electronic mail 
transmission in error, please destroy it and notify us immediately 
through the telephone number +351 21 359 44 00 or at the e-mail address: 
ge...@spautores.pt

Assinatura SPA

Re: data import handler deprecated?

[Dmitri Maziuk]

On 11/30/2020 7:50 AM, David Smiley wrote:

Yes, absolutely to what Eric said.  We goofed on news / release highlights
on how to communicate what's happening in Solr.  From a Solr insider point
of view, we are "deprecating" because strictly speaking, the code isn't in
our codebase any longer.  From a user point of view (the audience of news /
release notes), the functionality has *moved*.


Just FYI, there is the dih 8.7.0 jar in 
repo1.maven.org/maven2/org/apache/solr -- whereas the github build is on 
8.6.0.


Dima

Re: data import handler deprecated?

[David Smiley]

Yes, absolutely to what Eric said.  We goofed on news / release highlights
on how to communicate what's happening in Solr.  From a Solr insider point
of view, we are "deprecating" because strictly speaking, the code isn't in
our codebase any longer.  From a user point of view (the audience of news /
release notes), the functionality has *moved*.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Nov 30, 2020 at 8:04 AM Eric Pugh 
wrote:

> You don’t need to abandon DIH right now….   You can just use the Github
> hosted version….   The more people who use it, the better a community it
> will form around it!It’s a bit chicken and egg, since no one is
> actively discussing it, submitting PR’s etc, it may languish.   If you use
> it, and test it, and support other community folks using it, then it will
> continue on!
>
>
>
> > On Nov 29, 2020, at 12:12 PM, Dmitri Maziuk 
> wrote:
> >
> > On 11/29/2020 10:32 AM, Erick Erickson wrote:
> >
> >> And I absolutely agree with Walter that the DB is often where
> >> the bottleneck lies. You might be able to
> >> use multiple threads and/or processes to query the
> >> DB if that’s the case and you can find some kind of partition
> >> key.
> >
> > IME the difficult part has always been dealing with incremental updates,
> if we were to roll our own, my vote would be for a database trigger that
> does a POST in whichever language the DBMS likes.
> >
> > But this has not been a part of our "solr 6.5 update" project until now.
> >
> > Thanks everyone,
> > Dima
>
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com <
> http://www.opensourceconnections.com/> | My Free/Busy <
> http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
>
>

Re: data import handler deprecated?

[Eric Pugh]

You don’t need to abandon DIH right now….   You can just use the Github hosted 
version….   The more people who use it, the better a community it will form 
around it!It’s a bit chicken and egg, since no one is actively discussing 
it, submitting PR’s etc, it may languish.   If you use it, and test it, and 
support other community folks using it, then it will continue on!



> On Nov 29, 2020, at 12:12 PM, Dmitri Maziuk  wrote:
> 
> On 11/29/2020 10:32 AM, Erick Erickson wrote:
> 
>> And I absolutely agree with Walter that the DB is often where
>> the bottleneck lies. You might be able to
>> use multiple threads and/or processes to query the
>> DB if that’s the case and you can find some kind of partition
>> key.
> 
> IME the difficult part has always been dealing with incremental updates, if 
> we were to roll our own, my vote would be for a database trigger that does a 
> POST in whichever language the DBMS likes.
> 
> But this has not been a part of our "solr 6.5 update" project until now.
> 
> Thanks everyone,
> Dima

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Re: data import handler deprecated?

[Dmitri Maziuk]

On 11/29/2020 10:32 AM, Erick Erickson wrote:


And I absolutely agree with Walter that the DB is often where
the bottleneck lies. You might be able to
use multiple threads and/or processes to query the
DB if that’s the case and you can find some kind of partition
key.


IME the difficult part has always been dealing with incremental updates, 
if we were to roll our own, my vote would be for a database trigger that 
does a POST in whichever language the DBMS likes.


But this has not been a part of our "solr 6.5 update" project until now.

Thanks everyone,
Dima

Re: data import handler deprecated?

[Erick Erickson]

If you like Java instead of Python, here’s a skeletal program:

https://lucidworks.com/post/indexing-with-solrj/

It’s simple and single-threaded, but could serve as a basis for
something along the lines that Walter suggests.

And I absolutely agree with Walter that the DB is often where
the bottleneck lies. You might be able to
use multiple threads and/or processes to query the
DB if that’s the case and you can find some kind of partition
key.

You also might (and it depends on the Solr version) be able,
to wrap a jdbc stream in an update decorator.

https://lucene.apache.org/solr/guide/8_0/stream-source-reference.html

https://lucene.apache.org/solr/guide/8_0/stream-decorator-reference.html

Best,
Erick

> On Nov 29, 2020, at 3:04 AM, Walter Underwood  wrote:
> 
> I recommend building an outboard loader, like I did a dozen years ago for
> Solr 1.3 (before DIH) and did again recently. I’m glad to send you my Python
> program, though it reads from a JSONL file, not a database.
> 
> Run a loop fetching records from a database. Put each record into a 
> synchronized
> (thread-safe) queue. Run multiple worker threads, each pulling records from 
> the
> queue, batching them up, and sending them to Solr. For maximum indexing speed
> (at the expense of query performance), count the number of CPUs per shard 
> leader
> and run two worker threads per CPU.
> 
> Adjust the batch size to be maybe 10k to 50k bytes. That might be 20 to 1000 
> documents, depending on the content.
> 
> With this setup, your database will probably be your bottleneck. I’ve had this
> index a million (small) documents per minute to a multi-shard cluster, from a 
> JSONL
> file on local disk.
> 
> Also, don’t worry about finding the leaders and sending the right document to
> the right shard. I just throw the batches at the load balancer and let Solr 
> figure
> it out. That is super simple and amazingly fast.
> 
> If you are doing big batches, building a dumb ETL system with JSONL files in 
> Amazon S3 has some real advantages. It allows loading prod data into a test
> cluster for load benchmarks, for example. Also good for disaster recovery, 
> just
> load the recent batches from S3. Want to know exactly which documents were
> in the index in October? Look at the batches in S3.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Nov 28, 2020, at 6:23 PM, matthew sporleder  wrote:
>> 
>> I went through the same stages of grief that you are about to start
>> but (luckily?) my core dataset grew some weird cousins and we ended up
>> writing our own indexer to join them all together/do partial
>> updates/other stuff beyond DIH.  It's not difficult to upload docs but
>> is definitely slower so far.  I think there is a bit of a 'clean core'
>> focus going on in solr-land right now and DIH is easy(!) but it's also
>> easy to hit its limits (atomic/partial updates?  wtf is an "entity?"
>> etc) so anyway try to be happy that you are aware of it now.
>> 
>> On Sat, Nov 28, 2020 at 7:41 PM Dmitri Maziuk  
>> wrote:
>>> 
>>> On 11/28/2020 5:48 PM, matthew sporleder wrote:
>>> 
 ...  The bottom of
 that github page isn't hopeful however :)
>>> 
>>> Yeah, "works with MariaDB" is a particularly bad way of saying "BYO JDBC
>>> JAR" :)
>>> 
>>> It's a more general queston though, what is the path forward for users
>>> who with data in two places? Hope that a community-maintained plugin
>>> will still be there tomorrow? Dump our tables to CSV (and POST them) and
>>> roll our own delta-updates logic? Or are we to choose one datastore and
>>> drop the other?
>>> 
>>> Dima
> 

Re: data import handler deprecated?

[Walter Underwood]

I recommend building an outboard loader, like I did a dozen years ago for
Solr 1.3 (before DIH) and did again recently. I’m glad to send you my Python
program, though it reads from a JSONL file, not a database.

Run a loop fetching records from a database. Put each record into a synchronized
(thread-safe) queue. Run multiple worker threads, each pulling records from the
queue, batching them up, and sending them to Solr. For maximum indexing speed
(at the expense of query performance), count the number of CPUs per shard leader
and run two worker threads per CPU.

Adjust the batch size to be maybe 10k to 50k bytes. That might be 20 to 1000 
documents, depending on the content.

With this setup, your database will probably be your bottleneck. I’ve had this
index a million (small) documents per minute to a multi-shard cluster, from a 
JSONL
file on local disk.

Also, don’t worry about finding the leaders and sending the right document to
the right shard. I just throw the batches at the load balancer and let Solr 
figure
it out. That is super simple and amazingly fast.

If you are doing big batches, building a dumb ETL system with JSONL files in 
Amazon S3 has some real advantages. It allows loading prod data into a test
cluster for load benchmarks, for example. Also good for disaster recovery, just
load the recent batches from S3. Want to know exactly which documents were
in the index in October? Look at the batches in S3.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Nov 28, 2020, at 6:23 PM, matthew sporleder  wrote:
> 
> I went through the same stages of grief that you are about to start
> but (luckily?) my core dataset grew some weird cousins and we ended up
> writing our own indexer to join them all together/do partial
> updates/other stuff beyond DIH.  It's not difficult to upload docs but
> is definitely slower so far.  I think there is a bit of a 'clean core'
> focus going on in solr-land right now and DIH is easy(!) but it's also
> easy to hit its limits (atomic/partial updates?  wtf is an "entity?"
> etc) so anyway try to be happy that you are aware of it now.
> 
> On Sat, Nov 28, 2020 at 7:41 PM Dmitri Maziuk  wrote:
>> 
>> On 11/28/2020 5:48 PM, matthew sporleder wrote:
>> 
>>> ...  The bottom of
>>> that github page isn't hopeful however :)
>> 
>> Yeah, "works with MariaDB" is a particularly bad way of saying "BYO JDBC
>> JAR" :)
>> 
>> It's a more general queston though, what is the path forward for users
>> who with data in two places? Hope that a community-maintained plugin
>> will still be there tomorrow? Dump our tables to CSV (and POST them) and
>> roll our own delta-updates logic? Or are we to choose one datastore and
>> drop the other?
>> 
>> Dima

Re: data import handler deprecated?

[matthew sporleder]

I went through the same stages of grief that you are about to start
but (luckily?) my core dataset grew some weird cousins and we ended up
writing our own indexer to join them all together/do partial
updates/other stuff beyond DIH.  It's not difficult to upload docs but
is definitely slower so far.  I think there is a bit of a 'clean core'
focus going on in solr-land right now and DIH is easy(!) but it's also
easy to hit its limits (atomic/partial updates?  wtf is an "entity?"
etc) so anyway try to be happy that you are aware of it now.

On Sat, Nov 28, 2020 at 7:41 PM Dmitri Maziuk  wrote:
>
> On 11/28/2020 5:48 PM, matthew sporleder wrote:
>
> > ...  The bottom of
> > that github page isn't hopeful however :)
>
> Yeah, "works with MariaDB" is a particularly bad way of saying "BYO JDBC
> JAR" :)
>
> It's a more general queston though, what is the path forward for users
> who with data in two places? Hope that a community-maintained plugin
> will still be there tomorrow? Dump our tables to CSV (and POST them) and
> roll our own delta-updates logic? Or are we to choose one datastore and
> drop the other?
>
> Dima

Re: data import handler deprecated?

[Dmitri Maziuk]

On 11/28/2020 5:48 PM, matthew sporleder wrote:


...  The bottom of
that github page isn't hopeful however :)


Yeah, "works with MariaDB" is a particularly bad way of saying "BYO JDBC 
JAR" :)


It's a more general queston though, what is the path forward for users 
who with data in two places? Hope that a community-maintained plugin 
will still be there tomorrow? Dump our tables to CSV (and POST them) and 
roll our own delta-updates logic? Or are we to choose one datastore and 
drop the other?


Dima

Re: data import handler deprecated?

[matthew sporleder]

https://solr.cool/#utilities -> https://github.com/rohitbemax/dataimporthandler

You can import it in the many new/novel ways to add things to a solr
install and it should work like always (apparently).  The bottom of
that github page isn't hopeful however :)

On Sat, Nov 28, 2020 at 5:21 PM Dmitri Maziuk  wrote:
>
> Hi all,
>
> trying to set up solr-8.7.0, contrib/dataimporthandler/README.txt says
> this module is deprecated as of 8.6 and scheduled for removal in 9.0.
>
> How do we pull data out of our relational database in 8.7+?
>
> TIA
> Dima

data import handler deprecated?

[Dmitri Maziuk]

Hi all,

trying to set up solr-8.7.0, contrib/dataimporthandler/README.txt says 
this module is deprecated as of 8.6 and scheduled for removal in 9.0.


How do we pull data out of our relational database in 8.7+?

TIA
Dima

Re: Data Import Handler - Concurrent Entity Importing

[ART GALLERY]

check out the videos on this website TROO.TUBE don't be such a
sheep/zombie/loser/NPC. Much love!
https://troo.tube/videos/watch/aaa64864-52ee-4201-922f-41300032f219

On Tue, May 5, 2020 at 1:58 PM Mikhail Khludnev  wrote:
>
> Hello, James.
>
> DataImportHandler has a lock preventing concurrent execution. If you need
> to run several imports in parallel at the same core, you need to duplicate
> "/dataimport" handlers definition in solrconfig.xml. Thus, you can run them
> in parallel. Regarding schema, I prefer the latter but mileage may vary.
>
> --
> Mikhail.
>
> On Tue, May 5, 2020 at 6:39 PM James Greene 
> wrote:
>
> > Hello, I'm new to the group here so please excuse me if I do not have the
> > etiquette down yet.
> >
> > Is it possible to have multiple entities (customer configurable, up to 40
> > atm) in a DIH configuration to be imported at once?  Right now I have
> > multiple root entities in my configuration but they get indexes
> > sequentially and this means the entities that are last are always delayed
> > hitting the index.
> >
> > I'm trying to migrate an existing setup (solr 6.6) that utilizes a
> > different collection for each "entity type" into a single collection (solr
> > 8.4) to get around some of the hurdles faced when needing to have searches
> > that require multiple block joins and currently does not work going cross
> > core.
> >
> > I'm also wondering if it is better to fully qualify a field name or use two
> > different fields for performing the "same" search.  i.e:
> >
> >
> > {
> > type_A_status; Active
> > type_A_value: Test
> > }
> > vs
> > {
> > type: A
> > status: Active
> > value: Test
> > }
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev

Re: Data Import Handler - Concurrent Entity Importing

[Mikhail Khludnev]

Hello, James.

DataImportHandler has a lock preventing concurrent execution. If you need
to run several imports in parallel at the same core, you need to duplicate
"/dataimport" handlers definition in solrconfig.xml. Thus, you can run them
in parallel. Regarding schema, I prefer the latter but mileage may vary.

--
Mikhail.

On Tue, May 5, 2020 at 6:39 PM James Greene 
wrote:

> Hello, I'm new to the group here so please excuse me if I do not have the
> etiquette down yet.
>
> Is it possible to have multiple entities (customer configurable, up to 40
> atm) in a DIH configuration to be imported at once?  Right now I have
> multiple root entities in my configuration but they get indexes
> sequentially and this means the entities that are last are always delayed
> hitting the index.
>
> I'm trying to migrate an existing setup (solr 6.6) that utilizes a
> different collection for each "entity type" into a single collection (solr
> 8.4) to get around some of the hurdles faced when needing to have searches
> that require multiple block joins and currently does not work going cross
> core.
>
> I'm also wondering if it is better to fully qualify a field name or use two
> different fields for performing the "same" search.  i.e:
>
>
> {
> type_A_status; Active
> type_A_value: Test
> }
> vs
> {
> type: A
> status: Active
> value: Test
> }
>


-- 
Sincerely yours
Mikhail Khludnev

Data Import Handler - Concurrent Entity Importing

[James Greene]

Hello, I'm new to the group here so please excuse me if I do not have the
etiquette down yet.

Is it possible to have multiple entities (customer configurable, up to 40
atm) in a DIH configuration to be imported at once?  Right now I have
multiple root entities in my configuration but they get indexes
sequentially and this means the entities that are last are always delayed
hitting the index.

I'm trying to migrate an existing setup (solr 6.6) that utilizes a
different collection for each "entity type" into a single collection (solr
8.4) to get around some of the hurdles faced when needing to have searches
that require multiple block joins and currently does not work going cross
core.

I'm also wondering if it is better to fully qualify a field name or use two
different fields for performing the "same" search.  i.e:


{
type_A_status; Active
type_A_value: Test
}
vs
{
type: A
status: Active
value: Test
}

SOLR Data Import Handler : A command is still running...

[Doss]

We are doing hourly data import to our index, per day one or two requests
are getting failed with the message "A command is still running...".

1. Does it mean, the data import not happened for the last hour?
2. If you look at the "Full Dump Started" time has an older data, in the
below log all most 13 days, why is that so?

userinfoindex start - Wed Jan 22 05:12:01 IST 2020 {
"responseHeader":{ "status":0, "QTime":0},   "initArgs":[
"defaults",[   "config","data-import.xml"]],
"command":"full-import",   "status":"busy",   "importResponse":"A command
is still running...",   "statusMessages":{ "Time
Elapsed":"298:1:59.986", "Total Requests made to DataSource":"1",
"Total Rows Fetched":"17426", "Total Documents Processed":"17425",
"Total Documents Skipped":"0", "Full Dump Started":"2020-01-09
19:10:02"}}

Thanks,
Doss.

Re: SQL data import handler

[Friscia, Michael]

Thank you for your responses Vadim and Jörn. You both prompted me to try again 
and this time I succeeded. The trick seemed to be the way that I installed Java 
using Open JDK versus from Oracle. In addition, I imagine I accidentally had a 
lot of old versions of JAR files lying around so it was easier to start with a 
fresh VM. Now I was able to install using JDK12 and the latest Microsoft 7.4.x 
driver. Now it works out of the box as I wanted. 

Thanks again for being a sounding board for this, I primarily support 
Microsoft/dot net stuff so the Linux stuff sometimes gets away from me.

___
Michael Friscia
Office of Communications
Yale School of Medicine
(203) 737-7932 - office
(203) 931-5381 - mobile
http://web.yale.edu <http://web.yale.edu/>
 

On 9/9/19, 6:53 AM, "Vadim Ivanov"  wrote:

Hi,
Latest jdbc driver 7.4.1 seems to support JRE 8, 11, 12

https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fdownload%2Fdetails.aspx%3Fid%3D58505data=02%7C01%7Cmichael.friscia%40yale.edu%7C93626e2acbd4457d7f1608d73513f44d%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637036232130960752sdata=3bLoGx8DzsAifCW9tv64V1sCeS7mTzFU3fAazODNGYE%3Dreserved=0
You have to delete all previous versions of Sql Server jdbc driver from 
Solr installation (/solr/server/lib/ in my case)

-- 
Vadim

> -Original Message-
> From: Friscia, Michael [mailto:michael.fris...@yale.edu]
> Sent: Monday, September 09, 2019 1:22 PM
> To: solr-user@lucene.apache.org
    > Subject: SQL data import handler
> 
> I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with 
default-jre
> which installed version 11. So after a day of trying to make my Microsoft 
SQL
> Server data import handler work and failing, I built a new VM and 
installed
> JRE 8 and then everything works perfectly.
> 
> The root of the problem was the elimination of java.bind.xml in JRE 9. 
I’m not
> a Java programmer so I’m only going by what I uncovered digging through 
the
> error logs. I am not positive this is the only error to deal with, for 
all I know
> fixing that will just uncover something else that needs repair. There were
> solutions where you compile SOLR using Maven but this is moving out of my
> comfort zone as well as long term strategy to keep SOLR management (as 
well
> as other Linux systems management) out-of-the-box. There were also
> solutions to include some sort of dependency on this older library but 
I’m at a
> loss on how to relate that to a SOLR install.
> 
> My questions, since I am not that familiar with Java dependencies:
> 
>   1.  Is it ok to run JRE 8 on a production server? It’s heavily 
firewalled and
> SOLR, Zookeeper nor anything else on these servers is available off the 
virtual
> network so it seems ok, but I try not to run very old versions of any 
software.
>   2.  Is there a way to fix this and keep the installation out-of-the-box 
or at
> least almost out of the box?
> 
> ___
> Michael Friscia
> Office of Communications
> Yale School of Medicine
> (203) 737-7932 - office
> (203) 931-5381 - mobile
> 
https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fweb.yale.edudata=02%7C01%7Cmichael.friscia%40yale.edu%7C93626e2acbd4457d7f1608d73513f44d%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637036232130960752sdata=G5xMXdGQs12oK%2FDCxKy0zIn8sQ0uCpDLRGGatw45oiY%3Dreserved=0<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fweb.yale.edu%2Fdata=02%7C01%7Cmichael.friscia%40yale.edu%7C93626e2acbd4457d7f1608d73513f44d%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637036232130960752sdata=8pGHZPLlREYqKioyMrZDu2f9p3RXJBcFpkmvdKoHaZc%3Dreserved=0>

RE: SQL data import handler

[Vadim Ivanov]

Hi,
Latest jdbc driver 7.4.1 seems to support JRE 8, 11, 12
https://www.microsoft.com/en-us/download/details.aspx?id=58505
You have to delete all previous versions of Sql Server jdbc driver from Solr 
installation (/solr/server/lib/ in my case)

-- 
Vadim

> -Original Message-
> From: Friscia, Michael [mailto:michael.fris...@yale.edu]
> Sent: Monday, September 09, 2019 1:22 PM
> To: solr-user@lucene.apache.org
> Subject: SQL data import handler
> 
> I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre
> which installed version 11. So after a day of trying to make my Microsoft SQL
> Server data import handler work and failing, I built a new VM and installed
> JRE 8 and then everything works perfectly.
> 
> The root of the problem was the elimination of java.bind.xml in JRE 9. I’m not
> a Java programmer so I’m only going by what I uncovered digging through the
> error logs. I am not positive this is the only error to deal with, for all I 
> know
> fixing that will just uncover something else that needs repair. There were
> solutions where you compile SOLR using Maven but this is moving out of my
> comfort zone as well as long term strategy to keep SOLR management (as well
> as other Linux systems management) out-of-the-box. There were also
> solutions to include some sort of dependency on this older library but I’m at 
> a
> loss on how to relate that to a SOLR install.
> 
> My questions, since I am not that familiar with Java dependencies:
> 
>   1.  Is it ok to run JRE 8 on a production server? It’s heavily firewalled 
> and
> SOLR, Zookeeper nor anything else on these servers is available off the 
> virtual
> network so it seems ok, but I try not to run very old versions of any 
> software.
>   2.  Is there a way to fix this and keep the installation out-of-the-box or 
> at
> least almost out of the box?
> 
> ___
> Michael Friscia
> Office of Communications
> Yale School of Medicine
> (203) 737-7932 - office
> (203) 931-5381 - mobile
> http://web.yale.edu<http://web.yale.edu/>

Re: SQL data import handler

[Jörn Franke]

Hi Michael,

Thank you for sharing. You are right about your approach to not customize the 
distribution.

Solr supports JDK8 and it latest versions (8.x) also JDK11. I would not 
recommend to use it with JDK9 or JDK10 as they are out of support in many Java 
distributions. It might be also that your database driver does not support JDK9 
(check with Microsoft).
I don’t see it that critical at the moment to have JDK8 on this production 
server, but since it is out of support you should look for alternatives.

So if you are with Solr 8.x please go with JDK11 to have the latest fixes etc.

Best regards 

> Am 09.09.2019 um 12:21 schrieb Friscia, Michael :
> 
> I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre 
> which installed version 11. So after a day of trying to make my Microsoft SQL 
> Server data import handler work and failing, I built a new VM and installed 
> JRE 8 and then everything works perfectly.
> 
> The root of the problem was the elimination of java.bind.xml in JRE 9. I’m 
> not a Java programmer so I’m only going by what I uncovered digging through 
> the error logs. I am not positive this is the only error to deal with, for 
> all I know fixing that will just uncover something else that needs repair. 
> There were solutions where you compile SOLR using Maven but this is moving 
> out of my comfort zone as well as long term strategy to keep SOLR management 
> (as well as other Linux systems management) out-of-the-box. There were also 
> solutions to include some sort of dependency on this older library but I’m at 
> a loss on how to relate that to a SOLR install.
> 
> My questions, since I am not that familiar with Java dependencies:
> 
>  1.  Is it ok to run JRE 8 on a production server? It’s heavily firewalled 
> and SOLR, Zookeeper nor anything else on these servers is available off the 
> virtual network so it seems ok, but I try not to run very old versions of any 
> software.
>  2.  Is there a way to fix this and keep the installation out-of-the-box or 
> at least almost out of the box?
> 
> ___
> Michael Friscia
> Office of Communications
> Yale School of Medicine
> (203) 737-7932 - office
> (203) 931-5381 - mobile
> http://web.yale.edu<http://web.yale.edu/>
> 

SQL data import handler

[Friscia, Michael]

I setup SOLR on Ubuntu 18.04 and installed Java from apt-get with default-jre 
which installed version 11. So after a day of trying to make my Microsoft SQL 
Server data import handler work and failing, I built a new VM and installed JRE 
8 and then everything works perfectly.

The root of the problem was the elimination of java.bind.xml in JRE 9. I’m not 
a Java programmer so I’m only going by what I uncovered digging through the 
error logs. I am not positive this is the only error to deal with, for all I 
know fixing that will just uncover something else that needs repair. There were 
solutions where you compile SOLR using Maven but this is moving out of my 
comfort zone as well as long term strategy to keep SOLR management (as well as 
other Linux systems management) out-of-the-box. There were also solutions to 
include some sort of dependency on this older library but I’m at a loss on how 
to relate that to a SOLR install.

My questions, since I am not that familiar with Java dependencies:

  1.  Is it ok to run JRE 8 on a production server? It’s heavily firewalled and 
SOLR, Zookeeper nor anything else on these servers is available off the virtual 
network so it seems ok, but I try not to run very old versions of any software.
  2.  Is there a way to fix this and keep the installation out-of-the-box or at 
least almost out of the box?

___
Michael Friscia
Office of Communications
Yale School of Medicine
(203) 737-7932 - office
(203) 931-5381 - mobile
http://web.yale.edu<http://web.yale.edu/>

SV: data-import-handler for solr-7.5.0

[Martin Frank Hansen (MHQ)]

I made it work with the simplest of xml-files with some inspiration from 
https://opensolr.com/blog/2011/09/how-to-import-data-from-xml-files-into-your-solr-collection
 .

Data-config is now:


  

  
  
  


And the document is simply:


   
 2165432
 5
   

   
 28548113
 89
   


Now I guess I just have to add to this solution.

Thanks for your help Alex, and also thanks to Jan answering the first mail.

Best regards
Martin Frank Hansen

-Oprindelig meddelelse-
Fra: Alexandre Rafalovitch 
Sendt: 2. oktober 2018 19:52
Til: solr-user 
Emne: Re: data-import-handler for solr-7.5.0

Ok, so then you can switch to debug mode and keep trying to figure it out. Also 
try BinFileDataSource or URLDataSource, maybe it will have an easier way.

Or using relative path (example:
https://github.com/arafalov/solr-apachecon2018-presentation/blob/master/configsets/pets-final/pets-data-config.xml).

Regards,
   Alex.
On Tue, 2 Oct 2018 at 12:46, Martin Frank Hansen (MHQ)  wrote:
>
> Thanks for the info, the UI looks interesting... It does read the data-config 
> correctly, so the problem is probably in this file.
>
> Martin Frank Hansen, Senior Data Analytiker
>
> Data, IM & Analytics
>
>
>
> Lautrupparken 40-42, DK-2750 Ballerup
> E-mail m...@kmd.dk  Web www.kmd.dk
> Mobil +4525571418
>
> -Oprindelig meddelelse-
> Fra: Alexandre Rafalovitch 
> Sendt: 2. oktober 2018 18:18
> Til: solr-user 
> Emne: Re: data-import-handler for solr-7.5.0
>
> Admin UI for DIH will show you the config file read. So, if nothing is
> there, the path is most likely the issue
>
> You can also provide or update the configuration right in UI if you enable 
> debug.
>
> Finally, the config file is reread on every invocation, so you don't need to 
> restart the core after changing it.
>
> Hope this helps,
>Alex.
> On Tue, 2 Oct 2018 at 11:45, Jan Høydahl  wrote:
> >
> > > url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> >
> > Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
> >
> > --
> > Jan Høydahl, search solution architect Cominvent AS -
> > www.cominvent.com
> >
> > > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> > >
> > > Hi,
> > >
> > > I am having some problems getting the data-import-handler in Solr to 
> > > work. I have tried a lot of things but I simply get no response from 
> > > Solr, not even an error.
> > >
> > > When calling the API:
> > > http://localhost:8983/solr/nh/dataimport?command=full-import
> > > {
> > >  "responseHeader":{
> > >"status":0,
> > >"QTime":38},
> > >  "initArgs":[
> > >"defaults",[
> > >
> > > "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> > >  "command":"full-import",
> > >  "status":"idle",
> > >  "importResponse":"",
> > >  "statusMessages":{}}
> > >
> > > The data looks like this:
> > >
> > > 
> > >  
> > > 2165432
> > > 5  
> > >
> > >  
> > > 28548113
> > > 89   
> > >
> > >
> > > The data-config file looks like this:
> > >
> > > 
> > >  
> > >
> > >   > >name="xml"
> > >pk="id"
> > >processor="XPathEntityProcessor"
> > >stream="true"
> > >forEach="/journal/doc"
> > >url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> > >transformer="RegexTransformer,TemplateTransformer"
> > >>
> > >
> > >
> > >
> > >  
> > >  
> > > 
> > >
> > > And I referenced the jar files in the solr-config.xml as well as adding 
> > > the request-handler by adding the following lines:
> > >
> > >  > > regex="solr-dataimporthandler-\d.*\.jar" />  > > dir="${solr.install.dir:../../../..}/dist/"
> > > regex="solr-dataimporthandler-extras-\d.*\.jar" />
> > >
> > >
> > >  > > class="org.apache.solr.handler.dataimport.DataImportHandler">
> > >
> > >   > > name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
> > >

Re: data-import-handler for solr-7.5.0

[Alexandre Rafalovitch]

Ok, so then you can switch to debug mode and keep trying to figure it
out. Also try BinFileDataSource or URLDataSource, maybe it will have
an easier way.

Or using relative path (example:
https://github.com/arafalov/solr-apachecon2018-presentation/blob/master/configsets/pets-final/pets-data-config.xml).

Regards,
   Alex.
On Tue, 2 Oct 2018 at 12:46, Martin Frank Hansen (MHQ)  wrote:
>
> Thanks for the info, the UI looks interesting... It does read the data-config 
> correctly, so the problem is probably in this file.
>
> Martin Frank Hansen, Senior Data Analytiker
>
> Data, IM & Analytics
>
>
>
> Lautrupparken 40-42, DK-2750 Ballerup
> E-mail m...@kmd.dk  Web www.kmd.dk
> Mobil +4525571418
>
> -Oprindelig meddelelse-
> Fra: Alexandre Rafalovitch 
> Sendt: 2. oktober 2018 18:18
> Til: solr-user 
> Emne: Re: data-import-handler for solr-7.5.0
>
> Admin UI for DIH will show you the config file read. So, if nothing is there, 
> the path is most likely the issue
>
> You can also provide or update the configuration right in UI if you enable 
> debug.
>
> Finally, the config file is reread on every invocation, so you don't need to 
> restart the core after changing it.
>
> Hope this helps,
>Alex.
> On Tue, 2 Oct 2018 at 11:45, Jan Høydahl  wrote:
> >
> > > url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> >
> > Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
> >
> > --
> > Jan Høydahl, search solution architect Cominvent AS -
> > www.cominvent.com
> >
> > > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> > >
> > > Hi,
> > >
> > > I am having some problems getting the data-import-handler in Solr to 
> > > work. I have tried a lot of things but I simply get no response from 
> > > Solr, not even an error.
> > >
> > > When calling the API:
> > > http://localhost:8983/solr/nh/dataimport?command=full-import
> > > {
> > >  "responseHeader":{
> > >"status":0,
> > >"QTime":38},
> > >  "initArgs":[
> > >"defaults",[
> > >  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> > >  "command":"full-import",
> > >  "status":"idle",
> > >  "importResponse":"",
> > >  "statusMessages":{}}
> > >
> > > The data looks like this:
> > >
> > > 
> > >  
> > > 2165432
> > > 5  
> > >
> > >  
> > > 28548113
> > > 89   
> > >
> > >
> > > The data-config file looks like this:
> > >
> > > 
> > >  
> > >
> > >   > >name="xml"
> > >pk="id"
> > >processor="XPathEntityProcessor"
> > >stream="true"
> > >forEach="/journal/doc"
> > >url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> > >transformer="RegexTransformer,TemplateTransformer"
> > >>
> > >
> > >
> > >
> > >  
> > >  
> > > 
> > >
> > > And I referenced the jar files in the solr-config.xml as well as adding 
> > > the request-handler by adding the following lines:
> > >
> > >  > > regex="solr-dataimporthandler-\d.*\.jar" />  > > dir="${solr.install.dir:../../../..}/dist/"
> > > regex="solr-dataimporthandler-extras-\d.*\.jar" />
> > >
> > >
> > >  > > class="org.apache.solr.handler.dataimport.DataImportHandler">
> > >
> > >   > > name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
> > >
> > >  
> > >
> > > I am running a core residing in the folder 
> > > “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in 
> > > “C:/Users/z6mhq/Documents/solr-7.5.0”.
> > >
> > > I really hope that someone can spot my mistake…
> > >
> > > Thanks in advance.
> > >
> > > Martin Frank Hansen
> > >
> > >
> > > Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder 
> > > du KMD’s Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der 
> > > fortæller, 

SV: data-import-handler for solr-7.5.0

[Martin Frank Hansen (MHQ)]

Thanks for the info, the UI looks interesting... It does read the data-config 
correctly, so the problem is probably in this file.

Martin Frank Hansen, Senior Data Analytiker

Data, IM & Analytics



Lautrupparken 40-42, DK-2750 Ballerup
E-mail m...@kmd.dk  Web www.kmd.dk
Mobil +4525571418

-Oprindelig meddelelse-
Fra: Alexandre Rafalovitch 
Sendt: 2. oktober 2018 18:18
Til: solr-user 
Emne: Re: data-import-handler for solr-7.5.0

Admin UI for DIH will show you the config file read. So, if nothing is there, 
the path is most likely the issue

You can also provide or update the configuration right in UI if you enable 
debug.

Finally, the config file is reread on every invocation, so you don't need to 
restart the core after changing it.

Hope this helps,
   Alex.
On Tue, 2 Oct 2018 at 11:45, Jan Høydahl  wrote:
>
> > url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
>
> Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
>
> --
> Jan Høydahl, search solution architect Cominvent AS -
> www.cominvent.com
>
> > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> >
> > Hi,
> >
> > I am having some problems getting the data-import-handler in Solr to work. 
> > I have tried a lot of things but I simply get no response from Solr, not 
> > even an error.
> >
> > When calling the API:
> > http://localhost:8983/solr/nh/dataimport?command=full-import
> > {
> >  "responseHeader":{
> >"status":0,
> >"QTime":38},
> >  "initArgs":[
> >"defaults",[
> >  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> >  "command":"full-import",
> >  "status":"idle",
> >  "importResponse":"",
> >  "statusMessages":{}}
> >
> > The data looks like this:
> >
> > 
> >  
> > 2165432
> > 5  
> >
> >  
> > 28548113
> > 89   
> >
> >
> > The data-config file looks like this:
> >
> > 
> >  
> >
> >   >name="xml"
> >pk="id"
> >processor="XPathEntityProcessor"
> >stream="true"
> >forEach="/journal/doc"
> >url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> >transformer="RegexTransformer,TemplateTransformer"
> >>
> >
> >
> >
> >  
> >  
> > 
> >
> > And I referenced the jar files in the solr-config.xml as well as adding the 
> > request-handler by adding the following lines:
> >
> >  > regex="solr-dataimporthandler-\d.*\.jar" />  > dir="${solr.install.dir:../../../..}/dist/"
> > regex="solr-dataimporthandler-extras-\d.*\.jar" />
> >
> >
> >  > class="org.apache.solr.handler.dataimport.DataImportHandler">
> >
> >   > name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
> >
> >  
> >
> > I am running a core residing in the folder 
> > “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in 
> > “C:/Users/z6mhq/Documents/solr-7.5.0”.
> >
> > I really hope that someone can spot my mistake…
> >
> > Thanks in advance.
> >
> > Martin Frank Hansen
> >
> >
> > Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
> > KMD’s Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der 
> > fortæller, hvordan vi behandler oplysninger om dig.
> >
> > Protection of your personal data is important to us. Here you can read 
> > KMD’s Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how we 
> > process your personal data.
> >
> > Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information. 
> > Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst 
> > informere afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi 
> > dig slette e-mailen i dit system uden at videresende eller kopiere den. 
> > Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning er fri 
> > for virus og andre fejl, som kan påvirke computeren eller it-systemet, 
> > hvori den modtages og læses, åbnes den på modtagerens eget ansvar. Vi 
> > påtager os ikke noget ansvar for tab og skade, som er opstået i forbindelse 
> > med at modtage og bruge e-mailen.
> >
> > Please note that this message may contain confidential information. If you 
> > have received this message by mistake, please inform the sender of the 
> > mistake by sending a reply, then delete the message from your system 
> > without making, distributing or retaining any copies of it. Although we 
> > believe that the message and any attachments are free from viruses and 
> > other errors that might affect the computer or it-system where it is 
> > received and read, the recipient opens the message at his or her own risk. 
> > We assume no responsibility for any loss or damage arising from the receipt 
> > or use of this message.
>

SV: data-import-handler for solr-7.5.0

[Martin Frank Hansen (MHQ)]

Unfortunately, still no luck.

{
  "responseHeader":{
"status":0,
"QTime":8},
  "initArgs":[
"defaults",[
  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
  "command":"full-import",
  "status":"idle",
  "importResponse":"",
  "statusMessages":{
"Total Requests made to DataSource":"0",
"Total Rows Fetched":"0",
"Total Documents Processed":"0",
"Total Documents Skipped":"0",
"Full Dump Started":"2018-10-02 16:15:21",
"":"Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.",
"Committed":"2018-10-02 16:15:22",
"Time taken":"0:0:0.136"}}

Seems like it is not even trying to read the data.

Martin Frank Hansen

-Oprindelig meddelelse-
Fra: Jan Høydahl 
Sendt: 2. oktober 2018 17:46
Til: solr-user@lucene.apache.org
Emne: Re: data-import-handler for solr-7.5.0

> url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"

Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
>
> Hi,
>
> I am having some problems getting the data-import-handler in Solr to work. I 
> have tried a lot of things but I simply get no response from Solr, not even 
> an error.
>
> When calling the API:
> http://localhost:8983/solr/nh/dataimport?command=full-import
> {
>  "responseHeader":{
>"status":0,
>"QTime":38},
>  "initArgs":[
>"defaults",[
>  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
>  "command":"full-import",
>  "status":"idle",
>  "importResponse":"",
>  "statusMessages":{}}
>
> The data looks like this:
>
> 
>  
> 2165432
> 5  
>
>  
> 28548113
> 89   
>
>
> The data-config file looks like this:
>
> 
>  
>
>  name="xml"
>pk="id"
>processor="XPathEntityProcessor"
>stream="true"
>forEach="/journal/doc"
>url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
>transformer="RegexTransformer,TemplateTransformer"
>>
>
>
>
>  
>  
> 
>
> And I referenced the jar files in the solr-config.xml as well as adding the 
> request-handler by adding the following lines:
>
>  regex="solr-dataimporthandler-\d.*\.jar" />  dir="${solr.install.dir:../../../..}/dist/"
> regex="solr-dataimporthandler-extras-\d.*\.jar" />
>
>
>  class="org.apache.solr.handler.dataimport.DataImportHandler">
>
>   name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
>
>  
>
> I am running a core residing in the folder 
> “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in 
> “C:/Users/z6mhq/Documents/solr-7.5.0”.
>
> I really hope that someone can spot my mistake…
>
> Thanks in advance.
>
> Martin Frank Hansen
>
>
> Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
> KMD’s Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der fortæller, 
> hvordan vi behandler oplysninger om dig.
>
> Protection of your personal data is important to us. Here you can read KMD’s 
> Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how we process 
> your personal data.
>
> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information. 
> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst informere 
> afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi dig slette 
> e-mailen i dit system uden at videresende eller kopiere den. Selvom e-mailen 
> og ethvert vedhæftet bilag efter vores overbevisning er fri for virus og 
> andre fejl, som kan påvirke computeren eller it-systemet, hvori den modtages 
> og læses, åbnes den på modtagerens eget ansvar. Vi påtager os ikke noget 
> ansvar for tab og skade, som er opstået i forbindelse med at modtage og bruge 
> e-mailen.
>
> Please note that this message may contain confidential information. If you 
> have received this message by mistake, please inform the sender of the 
> mistake by sending a reply, then delete the message from your system without 
> making, distributing or retaining any copies of it. Although we believe that 
> the message and any attachments are free from viruses and other errors that 
> might affect the computer or it-system where it is received and read, the 
> recipient opens the message at his or her own risk. We assume no 
> responsibility for any loss or damage arising from the receipt or use of this 
> message.

Re: data-import-handler for solr-7.5.0

[Alexandre Rafalovitch]

Admin UI for DIH will show you the config file read. So, if nothing is
there, the path is most likely the issue

You can also provide or update the configuration right in UI if you
enable debug.

Finally, the config file is reread on every invocation, so you don't
need to restart the core after changing it.

Hope this helps,
   Alex.
On Tue, 2 Oct 2018 at 11:45, Jan Høydahl  wrote:
>
> > url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
>
> Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> >
> > Hi,
> >
> > I am having some problems getting the data-import-handler in Solr to work. 
> > I have tried a lot of things but I simply get no response from Solr, not 
> > even an error.
> >
> > When calling the API: 
> > http://localhost:8983/solr/nh/dataimport?command=full-import
> > {
> >  "responseHeader":{
> >"status":0,
> >"QTime":38},
> >  "initArgs":[
> >"defaults",[
> >  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> >  "command":"full-import",
> >  "status":"idle",
> >  "importResponse":"",
> >  "statusMessages":{}}
> >
> > The data looks like this:
> >
> > 
> >  
> > 2165432
> > 5
> >  
> >
> >  
> > 28548113
> > 89
> >  
> > 
> >
> >
> > The data-config file looks like this:
> >
> > 
> >  
> >
> >   >name="xml"
> >pk="id"
> >processor="XPathEntityProcessor"
> >stream="true"
> >forEach="/journal/doc"
> >url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> >transformer="RegexTransformer,TemplateTransformer"
> >>
> >
> >
> >
> >  
> >  
> > 
> >
> > And I referenced the jar files in the solr-config.xml as well as adding the 
> > request-handler by adding the following lines:
> >
> >  > regex="solr-dataimporthandler-\d.*\.jar" />
> >  > regex="solr-dataimporthandler-extras-\d.*\.jar" />
> >
> >
> >  > class="org.apache.solr.handler.dataimport.DataImportHandler">
> >
> >   > name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
> >
> >  
> >
> > I am running a core residing in the folder 
> > “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in 
> > “C:/Users/z6mhq/Documents/solr-7.5.0”.
> >
> > I really hope that someone can spot my mistake…
> >
> > Thanks in advance.
> >
> > Martin Frank Hansen
> >
> >
> > Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
> > KMD’s Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der 
> > fortæller, hvordan vi behandler oplysninger om dig.
> >
> > Protection of your personal data is important to us. Here you can read 
> > KMD’s Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how we 
> > process your personal data.
> >
> > Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information. 
> > Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst 
> > informere afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi 
> > dig slette e-mailen i dit system uden at videresende eller kopiere den. 
> > Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning er fri 
> > for virus og andre fejl, som kan påvirke computeren eller it-systemet, 
> > hvori den modtages og læses, åbnes den på modtagerens eget ansvar. Vi 
> > påtager os ikke noget ansvar for tab og skade, som er opstået i forbindelse 
> > med at modtage og bruge e-mailen.
> >
> > Please note that this message may contain confidential information. If you 
> > have received this message by mistake, please inform the sender of the 
> > mistake by sending a reply, then delete the message from your system 
> > without making, distributing or retaining any copies of it. Although we 
> > believe that the message and any attachments are free from viruses and 
> > other errors that might affect the computer or it-system where it is 
> > received and read, the recipient opens the message at his or her own risk. 
> > We assume no responsibility for any loss or damage arising from the receipt 
> > or use of this message.
>

Re: data-import-handler for solr-7.5.0

[Jan Høydahl]

> url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"

Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> 
> Hi,
> 
> I am having some problems getting the data-import-handler in Solr to work. I 
> have tried a lot of things but I simply get no response from Solr, not even 
> an error.
> 
> When calling the API: 
> http://localhost:8983/solr/nh/dataimport?command=full-import
> {
>  "responseHeader":{
>"status":0,
>"QTime":38},
>  "initArgs":[
>"defaults",[
>  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
>  "command":"full-import",
>  "status":"idle",
>  "importResponse":"",
>  "statusMessages":{}}
> 
> The data looks like this:
> 
> 
>  
> 2165432
> 5
>  
> 
>  
> 28548113
> 89
>  
> 
> 
> 
> The data-config file looks like this:
> 
> 
>  
>
>  name="xml"
>pk="id"
>processor="XPathEntityProcessor"
>stream="true"
>forEach="/journal/doc"
>url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
>transformer="RegexTransformer,TemplateTransformer"
>> 
>
>
> 
>  
>  
> 
> 
> And I referenced the jar files in the solr-config.xml as well as adding the 
> request-handler by adding the following lines:
> 
>  regex="solr-dataimporthandler-\d.*\.jar" />
>  regex="solr-dataimporthandler-extras-\d.*\.jar" />
> 
> 
>  class="org.apache.solr.handler.dataimport.DataImportHandler">
>
>   name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
>
>  
> 
> I am running a core residing in the folder 
> “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in 
> “C:/Users/z6mhq/Documents/solr-7.5.0”.
> 
> I really hope that someone can spot my mistake…
> 
> Thanks in advance.
> 
> Martin Frank Hansen
> 
> 
> Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
> KMD’s Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der fortæller, 
> hvordan vi behandler oplysninger om dig.
> 
> Protection of your personal data is important to us. Here you can read KMD’s 
> Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how we process 
> your personal data.
> 
> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information. 
> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst informere 
> afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi dig slette 
> e-mailen i dit system uden at videresende eller kopiere den. Selvom e-mailen 
> og ethvert vedhæftet bilag efter vores overbevisning er fri for virus og 
> andre fejl, som kan påvirke computeren eller it-systemet, hvori den modtages 
> og læses, åbnes den på modtagerens eget ansvar. Vi påtager os ikke noget 
> ansvar for tab og skade, som er opstået i forbindelse med at modtage og bruge 
> e-mailen.
> 
> Please note that this message may contain confidential information. If you 
> have received this message by mistake, please inform the sender of the 
> mistake by sending a reply, then delete the message from your system without 
> making, distributing or retaining any copies of it. Although we believe that 
> the message and any attachments are free from viruses and other errors that 
> might affect the computer or it-system where it is received and read, the 
> recipient opens the message at his or her own risk. We assume no 
> responsibility for any loss or damage arising from the receipt or use of this 
> message.

data-import-handler for solr-7.5.0

[Martin Frank Hansen (MHQ)]

Hi,

I am having some problems getting the data-import-handler in Solr to work. I 
have tried a lot of things but I simply get no response from Solr, not even an 
error.

When calling the API: 
http://localhost:8983/solr/nh/dataimport?command=full-import
{
  "responseHeader":{
"status":0,
"QTime":38},
  "initArgs":[
"defaults",[
  "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
  "command":"full-import",
  "status":"idle",
  "importResponse":"",
  "statusMessages":{}}

The data looks like this:


  
 2165432
 5
  

  
 28548113
 89
  



The data-config file looks like this:


  

  



  
  


And I referenced the jar files in the solr-config.xml as well as adding the 
request-handler by adding the following lines:







  C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml

  

I am running a core residing in the folder “C:/Users/z6mhq/Desktop/nh/nh/conf” 
while the Solr installation is in “C:/Users/z6mhq/Documents/solr-7.5.0”.

I really hope that someone can spot my mistake…

Thanks in advance.

Martin Frank Hansen


Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
KMD’s Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der fortæller, 
hvordan vi behandler oplysninger om dig.

Protection of your personal data is important to us. Here you can read KMD’s 
Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how we process your 
personal data.

Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information. Hvis 
du ved en fejltagelse modtager e-mailen, beder vi dig venligst informere 
afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi dig slette 
e-mailen i dit system uden at videresende eller kopiere den. Selvom e-mailen og 
ethvert vedhæftet bilag efter vores overbevisning er fri for virus og andre 
fejl, som kan påvirke computeren eller it-systemet, hvori den modtages og 
læses, åbnes den på modtagerens eget ansvar. Vi påtager os ikke noget ansvar 
for tab og skade, som er opstået i forbindelse med at modtage og bruge e-mailen.

Please note that this message may contain confidential information. If you have 
received this message by mistake, please inform the sender of the mistake by 
sending a reply, then delete the message from your system without making, 
distributing or retaining any copies of it. Although we believe that the 
message and any attachments are free from viruses and other errors that might 
affect the computer or it-system where it is received and read, the recipient 
opens the message at his or her own risk. We assume no responsibility for any 
loss or damage arising from the receipt or use of this message.

Re: Data Import Handler with Solr Source behind Load Balancer

[Emir Arnautović]

Hi Thomas,
Is this SolrCloud or Solr master-slave? Do you update index while indexing? Did 
you check if all your instances behind LB are in sync if you are using 
master-slave?
My guess would be that DIH is using cursors to read data from another Solr. If 
you are using multiple Solr instances behind LB there might be some diffs in 
index that results in different documents being returned for the same cursor 
mark. Is num doc and max doc the same on new instance after import?

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 12 Sep 2018, at 05:53, Zimmermann, Thomas  
> wrote:
> 
> We have a Solr v7 Instance sourcing data from a Data Import Handler with a 
> Solr data source running Solr v4. When it hits a single server in that 
> instance directly, all documents are read and written correctly to the v7. 
> When we hit the load balancer DNS entry, the resulting data import handler 
> json states that it read all the documents and skipped none, and all looks 
> fine, but the result set is missing ~20% of the documents in the v7 core. 
> This has happened multiple time on multiple environments.
> 
> Any thoughts on whether this might be a bug in the underlying DIH code? I'll 
> also pass it along to the server admins on our side for input.

Data Import Handler with Solr Source behind Load Balancer

[Zimmermann, Thomas]

We have a Solr v7 Instance sourcing data from a Data Import Handler with a Solr 
data source running Solr v4. When it hits a single server in that instance 
directly, all documents are read and written correctly to the v7. When we hit 
the load balancer DNS entry, the resulting data import handler json states that 
it read all the documents and skipped none, and all looks fine, but the result 
set is missing ~20% of the documents in the v7 core. This has happened multiple 
time on multiple environments.

Any thoughts on whether this might be a bug in the underlying DIH code? I'll 
also pass it along to the server admins on our side for input.

Re: Child=true does not work for data import handler

[omp...@rediffmail.com]

But in my case i see output as below


  0
  0
  
*:*
on
xml
1533734431931
  


  
IT
1
1
1608130338704326656
  
Data
1
2
1608130338704326656
  
omkar
1
1608130338704326656
  
ITI
2
3
1608130338712715264
  
Entry
2
4
1608130338712715264
  
ashwin
2
1608130338712715264





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Child=true does not work for data import handler

[Mikhail Khludnev]

This is how nested docs look like. These are document blocks with parent in
the end. Block Join Queries work on these blocks.

On Wed, Aug 8, 2018 at 12:47 PM omp...@rediffmail.com <
omkar.pra...@gmail.com> wrote:

> Thanks a lot Mikhail. But as per documentation below nested document
> ingestion is possible. Is this limitation of DIH?
>
>
> https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-NestedChildDocuments
>
>
> Also can block join query be used to get expect relationship for data i
> have
> ingested using DIH?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Child=true does not work for data import handler

[omp...@rediffmail.com]

Thanks a lot Mikhail. But as per documentation below nested document
ingestion is possible. Is this limitation of DIH?

https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-NestedChildDocuments


Also can block join query be used to get expect relationship for data i have
ingested using DIH?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Child=true does not work for data import handler

[Mikhail Khludnev]

It never works like you expect. You need to search for parents and then
hook up [child]. I see some improvements are coming, but now that is.

On Mon, Aug 6, 2018 at 9:11 PM omp...@rediffmail.com 
wrote:

> Thanks Mikhail verbose did help. _root_ field was missing in schema also in
> make some changes in child entity. Like i created id as alias to emp_id (
> in
> child query) which is id column of parent table.
>
>  query="SELECT id,name
> FROM emp">
> 
> 
>  name="child"  query="SELECT dept,emp_id as id
> FROM emp_details where emp_id='${parent.id}' ">
>  column="dept" name="dept" />
> 
> 
>
>
> Data seems to be returning correctly as below. but it show child documents
> and parent documents are shown as individual document. i was expecting 2
> documents and 2 child document for each doc.
> Any inputs will be helpful
>
>
>  "response":{"numFound":6,"start":0,"docs":[
>   {
> "dept":"IT",
> "id":"1",
> "_version_":1608073809653399552},
>   {
> "dept":"Data",
> "id":"1",
> "_version_":1608073809653399552},
>   {
> "name":"omkar",
> "id":"1",
> "_version_":1608073809653399552},
>   {
> "dept":"ITI",
> "id":"2",
> "_version_":1608073809667031040},
>   {
> "dept":"Entry",
> "id":"2",
> "_version_":1608073809667031040},
>   {
> "name":"ashwin",
> "id":"2",
> "_version_":1608073809667031040}]
>   }}
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Child=true does not work for data import handler

[omp...@rediffmail.com]

Thanks Mikhail verbose did help. _root_ field was missing in schema also in
make some changes in child entity. Like i created id as alias to emp_id ( in
child query) which is id column of parent table.




 





Data seems to be returning correctly as below. but it show child documents
and parent documents are shown as individual document. i was expecting 2
documents and 2 child document for each doc.
Any inputs will be helpful


 "response":{"numFound":6,"start":0,"docs":[
  {
"dept":"IT",
"id":"1",
"_version_":1608073809653399552},
  {
"dept":"Data",
"id":"1",
"_version_":1608073809653399552},
  {
"name":"omkar",
"id":"1",
"_version_":1608073809653399552},
  {
"dept":"ITI",
"id":"2",
"_version_":1608073809667031040},
  {
"dept":"Entry",
"id":"2",
"_version_":1608073809667031040},
  {
"name":"ashwin",
"id":"2",
"_version_":1608073809667031040}]
  }}



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Child=true does not work for data import handler

[Mikhail Khludnev]

DIH has debug modes. Have you tried to use them?

On Mon, Aug 6, 2018 at 4:11 PM omp...@rediffmail.com 
wrote:

> Thanks Mikhail, i tried changing conf but that did not help
>
> 
>  driver="com.mysql.jdbc.Driver"
>   url="jdbc:mysql://localhost:3306/test"
>   user="root"
>   password=""
>   session.group_concat_max_len = '7'
>   />
>
>
>
>  transformer="RegexTransformer"
>  query="SELECT id,name  FROM emp">
> 
> 
>
>  
>  name="dept" />
>  name="childpk" />
>  
>
> 
>
>   
>
> 
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Child=true does not work for data import handler

[omp...@rediffmail.com]

Thanks Mikhail, i tried changing conf but that did not help


  
 
   
   




 


 
 


  
  





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Child=true does not work for data import handler

[Mikhail Khludnev]

Hi, Omkar.

Could it happen that child docs as well as parents are assigned same "id"
field values implicitly and removed due to uniqueKey collision?

On Sat, Aug 4, 2018 at 10:12 PM omkar.pra...@gmail.com <
omkar.pra...@gmail.com> wrote:

> I am using similar db-data config as below for indexing this parent-child
> data. solr version 6.6.2
>
> SELECT   id as emp_id,   name FROM emp;
> +++
> | emp_id | name   |
> +++
> |  1 | omkar  |
> |  2 | ashwin |
> +++
> 2 rows in set (0.00 sec)
>
> select  * from emp_details ;
> +--++---+
> | id   | emp_id | dept  |
> +--++---+
> |1 |  1 | IT|
> |2 |  1 | Data  |
> |3 |  2 | ITI   |
> |4 |  2 | Entry |
> +--++---+
> 4 rows in set (0.00 sec)
>
> 
>  driver="com.mysql.jdbc.Driver"
>   url="jdbc:mysql://localhost:3306/test"
>   user="root"
>   password=""
>   session.group_concat_max_len = '7'
>   />
>
>
>
>  transformer="RegexTransformer"
>  query=" SELECT   id, name FROM  emp">
>
> 
> 
>
>  
>  name="dept" />
>  
>
> 
>
>   
>
> 
>
>
>
> {
>   "responseHeader":{
> "status":0,
> "QTime":0,
> "params":{
>   "q":"*:*",
>   "indent":"on",
>   "wt":"json",
>   "_":"1533325469162"}},
>   "response":{"numFound":2,"start":0,"docs":[
>   {
> "name":"omkar",
> "id":"1",
> "dept":"IT",
> "_version_":1607809693975052288},
>   {
> "name":"ashwin",
> "id":"2",
> "dept":"ITI",
> "_version_":1607809693978198016}]
>   }}
>
>
> I am expecting multi child documents. so i added child=true
>
>  
>
> but output of indexing is as below and it does not process any doucment
>
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> Requests: 3 , Fetched: 6 , Skipped: 0 , Processed: 0
> Started: less than a minute ago
>
> can you helping me if there is any issue with db or solr config
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
Sincerely yours
Mikhail Khludnev

Child=true does not work for data import handler

[omkar.pra...@gmail.com]

I am using similar db-data config as below for indexing this parent-child
data. solr version 6.6.2

SELECT   id as emp_id,   name FROM emp;
+++
| emp_id | name   |
+++
|  1 | omkar  |
|  2 | ashwin |
+++
2 rows in set (0.00 sec)

select  * from emp_details ;
+--++---+
| id   | emp_id | dept  |
+--++---+
|1 |  1 | IT|
|2 |  1 | Data  |
|3 |  2 | ITI   |
|4 |  2 | Entry |
+--++---+
4 rows in set (0.00 sec)


  
 
   
   





 

 
 


  
  




{
  "responseHeader":{
"status":0,
"QTime":0,
"params":{
  "q":"*:*",
  "indent":"on",
  "wt":"json",
  "_":"1533325469162"}},
  "response":{"numFound":2,"start":0,"docs":[
  {
"name":"omkar",
"id":"1",
"dept":"IT",
"_version_":1607809693975052288},
  {
"name":"ashwin",
"id":"2",
"dept":"ITI",
"_version_":1607809693978198016}]
  }}


I am expecting multi child documents. so i added child=true

 

but output of indexing is as below and it does not process any doucment

Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 3 , Fetched: 6 , Skipped: 0 , Processed: 0 
Started: less than a minute ago

can you helping me if there is any issue with db or solr config 




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

How to use tika-OCR in data import handler?

[Yasufumi Mizoguchi]

Hi,

I am trying to use tika-OCR(Tesseract) in data import handler
and found that processing English documents was quite good.

But I am struggling to process the other languages such as
Japanese, Chinese, etc...

So, I want to know how to switch Tesseract-OCR's processing
language via data import handler config or tikaConfig param.

Any points would be appreciated.

Thanks,
Yasufumi

Re: How to know the name(url) of documents that data import handler skipped

[Yasufumi Mizoguchi]

Hi, Rahul.

Thank you for your reply.
I already tried that, and I could see what files were read(via
FileDataSource) and what files were added(via UpdateLog).
So, by checking both, I could determine bad files.
But I want to know bad files directly.

Thanks,
Yasufumi

2018年7月9日(月) 12:47 Rahul Singh :

> Have you tried changing the log level
> https://lucene.apache.org/solr/guide/7_2/configuring-logging.html
>
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
> On Jul 8, 2018, 8:54 PM -0500, Yasufumi Mizoguchi ,
> wrote:
> > Hi,
> >
> > I am trying to indexing files into Solr 7.2 using data import handler
> with
> > onError=skip option.
> > But, I am struggling with determining the skipped documents as logs do
> not
> > tell which file was bad.
> > So, how can I know those files?
> >
> > Thanks,
> > Yasufumi
>

Re: How to know the name(url) of documents that data import handler skipped

[Rahul Singh]

Have you tried changing the log level
https://lucene.apache.org/solr/guide/7_2/configuring-logging.html


--
Rahul Singh
rahul.si...@anant.us

Anant Corporation
On Jul 8, 2018, 8:54 PM -0500, Yasufumi Mizoguchi , 
wrote:
> Hi,
>
> I am trying to indexing files into Solr 7.2 using data import handler with
> onError=skip option.
> But, I am struggling with determining the skipped documents as logs do not
> tell which file was bad.
> So, how can I know those files?
>
> Thanks,
> Yasufumi

How to know the name(url) of documents that data import handler skipped

[Yasufumi Mizoguchi]

Hi,

I am trying to indexing files into Solr 7.2 using data import handler with
onError=skip option.
But, I am struggling with determining the skipped documents as logs do not
tell which file was bad.
So, how can I know those files?

Thanks,
Yasufumi

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

I have add debug and I have this error:

null:java.lang.NullPointerException
at 
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429)
at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at 
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)

what mysql jdbc connector version I need ?





-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] 
Envoyé : jeudi 26 avril 2018 13:13
À : solr-user@lucene.apache.org
Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404

Hello,

Where I add that? In the Solr start command?

I have add -verbose:class in the /etc/default/solr.in.sh file but they logs are 
they sames.

Thanks,

-Message d'origine-
De : Mikhail Khludnev [mailto:m...@apache.org] Envoyé : mercredi 25 avril 2018 
15:40 À : solr-user <solr-user@lucene.apache.org> Objet : Re: SolrCloud DIH 
(Data Import Handler) MySQL 404

Can you share more log lines around this odd NPE?
It might be necessary to restart jvm with -verbose:class and look through its' 
output to find why it can't load this class.

On Wed, Apr 25, 2018 at 11:42 AM, msaunier <msaun...@citya.com> wrote:

> Hello Shawn,
>
> I have install SolrCloud 7.3 on an other server and the problem not apear.
> I create a Jira Ticket ?
>
> But I have an 

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

Hello,

Where I add that? In the Solr start command?

I have add -verbose:class in the /etc/default/solr.in.sh file but they logs are 
they sames.

Thanks,

-Message d'origine-
De : Mikhail Khludnev [mailto:m...@apache.org] 
Envoyé : mercredi 25 avril 2018 15:40
À : solr-user <solr-user@lucene.apache.org>
Objet : Re: SolrCloud DIH (Data Import Handler) MySQL 404

Can you share more log lines around this odd NPE?
It might be necessary to restart jvm with -verbose:class and look through its' 
output to find why it can't load this class.

On Wed, Apr 25, 2018 at 11:42 AM, msaunier <msaun...@citya.com> wrote:

> Hello Shawn,
>
> I have install SolrCloud 7.3 on an other server and the problem not apear.
> I create a Jira Ticket ?
>
> But I have an other problem:
>
> Full Import 
> failed:org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to PropertyWriter implementation:ZKPropertiesWriter
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:330)
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:411)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:474)
> at org.apache.solr.handler.dataimport.DataImporter.
> lambda$runAsync$0(DataImporter.java:457)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at org.apache.solr.handler.dataimport.DocBuilder.
> loadClass(DocBuilder.java:935)
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:326)
> ... 4 more
>
> I regard to solved the problem.
>
> Cordialement,
>
>
>
>
>
> -Message d'origine-
> De : Shawn Heisey [mailto:elyog...@elyograg.org] Envoyé : mardi 24 
> avril 2018 17:39 À : solr-user@lucene.apache.org Objet : Re: SolrCloud 
> DIH (Data Import Handler) MySQL 404
>
> On 4/24/2018 2:03 AM, msaunier wrote:
> > If I access to the interface, I have a null pointer exception:
> >
> > null:java.lang.NullPointerException
> >   at
> > org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandler
> > Ba
> > se.java:233)
>
> The line of code where this exception occurred uses fundamental Java 
> methods. Based on the error, either the getClass method common to all 
> java objects, or the getPackage method on the class, is returning 
> null. That shouldn't be possible.  This has me wondering whether there 
> is something broken in your particular Solr installation -- corrupt 
> jars, or something like that.  Or maybe something broken in your Java.
>
> Thanks,
> Shawn
>
>
>


--
Sincerely yours
Mikhail Khludnev

Re: SolrCloud DIH (Data Import Handler) MySQL 404

[Mikhail Khludnev]

Can you share more log lines around this odd NPE?
It might be necessary to restart jvm with -verbose:class and look through
its' output to find why it can't load this class.

On Wed, Apr 25, 2018 at 11:42 AM, msaunier <msaun...@citya.com> wrote:

> Hello Shawn,
>
> I have install SolrCloud 7.3 on an other server and the problem not apear.
> I create a Jira Ticket ?
>
> But I have an other problem:
>
> Full Import 
> failed:org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to PropertyWriter implementation:ZKPropertiesWriter
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:330)
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:411)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:474)
> at org.apache.solr.handler.dataimport.DataImporter.
> lambda$runAsync$0(DataImporter.java:457)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at org.apache.solr.handler.dataimport.DocBuilder.
> loadClass(DocBuilder.java:935)
> at org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWriter(DataImporter.java:326)
> ... 4 more
>
> I regard to solved the problem.
>
> Cordialement,
>
>
>
>
>
> -Message d'origine-
> De : Shawn Heisey [mailto:elyog...@elyograg.org]
> Envoyé : mardi 24 avril 2018 17:39
> À : solr-user@lucene.apache.org
> Objet : Re: SolrCloud DIH (Data Import Handler) MySQL 404
>
> On 4/24/2018 2:03 AM, msaunier wrote:
> > If I access to the interface, I have a null pointer exception:
> >
> > null:java.lang.NullPointerException
> >   at
> > org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBa
> > se.java:233)
>
> The line of code where this exception occurred uses fundamental Java
> methods. Based on the error, either the getClass method common to all java
> objects, or the getPackage method on the class, is returning null. That
> shouldn't be possible.  This has me wondering whether there is something
> broken in your particular Solr installation -- corrupt jars, or something
> like that.  Or maybe something broken in your Java.
>
> Thanks,
> Shawn
>
>
>


-- 
Sincerely yours
Mikhail Khludnev

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

Hello Shawn,

I have install SolrCloud 7.3 on an other server and the problem not apear. I 
create a Jira Ticket ?

But I have an other problem:

Full Import 
failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to 
PropertyWriter implementation:ZKPropertiesWriter
at 
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:330)
at 
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474)
at 
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at 
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
at 
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:326)
... 4 more

I regard to solved the problem.

Cordialement,





-Message d'origine-
De : Shawn Heisey [mailto:elyog...@elyograg.org] 
Envoyé : mardi 24 avril 2018 17:39
À : solr-user@lucene.apache.org
Objet : Re: SolrCloud DIH (Data Import Handler) MySQL 404

On 4/24/2018 2:03 AM, msaunier wrote:
> If I access to the interface, I have a null pointer exception:
>
> null:java.lang.NullPointerException
>   at 
> org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBa
> se.java:233)

The line of code where this exception occurred uses fundamental Java methods. 
Based on the error, either the getClass method common to all java objects, or 
the getPackage method on the class, is returning null. That shouldn't be 
possible.  This has me wondering whether there is something broken in your 
particular Solr installation -- corrupt jars, or something like that.  Or maybe 
something broken in your Java.

Thanks,
Shawn

Re: SolrCloud DIH (Data Import Handler) MySQL 404

[Shawn Heisey]

On 4/24/2018 2:03 AM, msaunier wrote:

If I access to the interface, I have a null pointer exception:

null:java.lang.NullPointerException
at 
org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBase.java:233)


The line of code where this exception occurred uses fundamental Java 
methods. Based on the error, either the getClass method common to all 
java objects, or the getPackage method on the class, is returning null.  
That shouldn't be possible.  This has me wondering whether there is 
something broken in your particular Solr installation -- corrupt jars, 
or something like that.  Or maybe something broken in your Java.


Thanks,
Shawn

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

I have modify DIH definition to simplify but sames errors:

## indexation_events.xml











##

Maxence,





-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] 
Envoyé : mardi 24 avril 2018 10:04
À : solr-user@lucene.apache.org
Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404

If I access to the interface, I have a null pointer exception:

null:java.lang.NullPointerException
at 
org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBase.java:233)
at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:187)
at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:163)
at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:80)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)





-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] Envoyé : mardi 24 avril 2018 09:25 À 
: solr-user@lucene.apache.org Objet : RE: SolrCloud DIH (Data Import Handler) 
MySQL 404

Hello Shawn,
Thanks for your answers. 

#
So, indexation_events.xml file is:














































#
And the config file is the configoverlay.xml, it's in cloud:

{
  "updateProcessor":{},

  "runtimeLi

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

If I access to the interface, I have a null pointer exception:

null:java.lang.NullPointerException
at 
org.apache.solr.handler.RequestHandlerBase.getVersion(RequestHandlerBase.java:233)
at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:187)
at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:163)
at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:80)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)





-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] 
Envoyé : mardi 24 avril 2018 09:25
À : solr-user@lucene.apache.org
Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404

Hello Shawn,
Thanks for your answers. 

#
So, indexation_events.xml file is:














































#
And the config file is the configoverlay.xml, it's in cloud:

{
  "updateProcessor":{},

  "runtimeLib":{
"mysql-connector-java":{
  "name":"mysql-connector-java",
  "version":1},

    "data-import-handler":{
  "name":"data-import-handler",
  "version":1}},

  "requestHandler":{"/test_dih":{
  "name":"/test_dih",
  "class":"org.apache.solr.han

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

Hello Shawn,
Thanks for your answers. 

#
So, indexation_events.xml file is:














































#
And the config file is the configoverlay.xml, it's in cloud:

{
  "updateProcessor":{},

  "runtimeLib":{
"mysql-connector-java":{
  "name":"mysql-connector-java",
  "version":1},

    "data-import-handler":{
  "name":"data-import-handler",
  "version":1}},

  "requestHandler":{"/test_dih":{
  "name":"/test_dih",
  "class":"org.apache.solr.handler.dataimport.DataImportHandler",
  "runtimeLib":true,
  "version":1,
  "defaults":{"config":"DIH/indexation_events.xml"}}}
}

I go to regard the solr.log

Thanks,
Maxence





-Message d'origine-
De : Shawn Heisey [mailto:apa...@elyograg.org] 
Envoyé : lundi 23 avril 2018 18:28
À : solr-user@lucene.apache.org
Objet : Re: SolrCloud DIH (Data Import Handler) MySQL 404

On 4/23/2018 8:30 AM, msaunier wrote:
> I have add debug:
>
> curl
> "http://srv-formation-solr:8983/solr/arguments_test/test_dih?command=f
> ull-im
> port=true=true"
>name="responseHeader">500 name="QTime">588 name="runtimeLib">true1 name="defaults"> name="config">DIH/indexation_events.xml

Re: SolrCloud DIH (Data Import Handler) MySQL 404

[Mikhail Khludnev]

ot;http://srv-formation-solr:8983/solr/arguments_test/test_
> dih?command=full-im
> port=true=true=reload-config"
> 
> 
> 500 name="QTime">647 name="msg">java.util.Arrays$ArrayList cannot be cast to
> java.lang.Stringjava.lang.ClassCastException:
> java.util.Arrays$ArrayList cannot be cast to java.lang.String
> at
> org.apache.solr.handler.dataimport.RequestInfo.
> init(RequestInfo.java
> :52)
> at
> org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataI
> mportHandler.java:128)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.
> java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:
> 361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:
> 305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler
> .java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143
> )
> at
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java
> :226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java
> :1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:
> 185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:
> 1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141
> )
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHand
> lerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.
> java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:1
> 34)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> RewriteHandler.java:
> 335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:1
> 34)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConne
> ction.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:
> 93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceC
> onsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(
> ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProd
> uceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:
> 671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:5
> 89)
> at java.lang.Thread.run(Thread.java:748)
> 500
> 
>
>
> -Message d'origine-
> De : msaunier [mailto:msaun...@citya.com]
> Envoyé : lundi 23 avril 2018 14:47
> À : solr-user@lucene.apache.org
> Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404
>
> I have correct url to : curl
> http://srv-formation-solr:8983/solr/arguments_test/test_
> dih?command=full-imp
> ort
>
> And change overlay config
> "/configs/arguments_test/DIH/indexation_events.xml" to "
> DIH/indexation_events.xml"
>
> But I have a new error:
>
> Full Import
> failed:org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable
> to PropertyWriter implementation:ZKPropertiesWriter
> at
> org.apache.solr.handler.dataimport.DataImporter.
> createPropertyWri

Re: SolrCloud DIH (Data Import Handler) MySQL 404

[Shawn Heisey]

On 4/23/2018 8:30 AM, msaunier wrote:

I have add debug:

curl
"http://srv-formation-solr:8983/solr/arguments_test/test_dih?command=full-im
port=true=true"


500588true1DIH/indexation_events.xml

This is looking like a really nasty error that I cannot understand, 
possibly caused by an error in configuration.


Can you share your dataimport handler config (will likely be in 
solrconfig.xml) and the contents of DIH/indexation_events.xml?  There is 
probably a database password in that file, you'll want to redact that.


You should look at solr.log and see if there are other errors happening 
that didn't make it into the response.


Thanks,
Shawn

Re: SolrCloud DIH (Data Import Handler) MySQL 404

[Shawn Heisey]

On 4/23/2018 6:12 AM, msaunier wrote:

I have a problem with DIH in SolrCloud. I don't understand why, so I need
your help.

Solr 6.6 in Cloud.

##

COMMAND:

curl http://srv-formation-solr:8983/solr/test_dih?command=full-import

RESULT:


   
 
 Error 404 Not Found
   
   HTTP ERROR 404
 Problem accessing /solr/test_dih. Reason:
   Not Found
   



This looks like an incomplete URL.

What exactly is test-dih?  If it is the name of your collection, then 
you are missing the handler, which is usually "/dataimport". If 
"/test-dih" is the name if your handler, then you are missing the name 
of the core or the collection.


With SolrCloud, it's actually better to direct your request to a 
specific core for DIH, something like collection_shard1_replica1.  If 
you direct it to the collection you never know which core will actually 
end up with the request, and will have a hard time getting the status of 
the import if the status request ends up on a different core than the 
full-import command.


A correct full URL should look something like this:

http://host:port/solr/test_shard1_replica2/dataimport?command=full-import

Looking at later messages, you may have figured this out at least 
partially.  The exception in your second message looks really odd.  (and 
I really have no idea what you are talking about with an overlay)


Thanks,
Shawn

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

ionHandler.java
:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java
:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:
185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:
1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141
)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHand
lerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.
java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:1
34)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:
335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:1
34)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConne
ction.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:
93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceC
onsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(
ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProd
uceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:
671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:5
89)
at java.lang.Thread.run(Thread.java:748)
500



-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] 
Envoyé : lundi 23 avril 2018 14:47
À : solr-user@lucene.apache.org
Objet : RE: SolrCloud DIH (Data Import Handler) MySQL 404

I have correct url to : curl
http://srv-formation-solr:8983/solr/arguments_test/test_dih?command=full-imp
ort

And change overlay config
"/configs/arguments_test/DIH/indexation_events.xml" to "
DIH/indexation_events.xml"

But I have a new error:

Full Import
failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
to PropertyWriter implementation:ZKPropertiesWriter
at
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImp
orter.java:330)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja
va:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474
)
at
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImport
er.java:457)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
at
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImp
orter.java:326)
... 4 more

Cordialement,





-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] Envoyé : lundi 23 avril 2018 14:12
À : solr-user@lucene.apache.org Objet : SolrCloud DIH (Data Import Handler)
MySQL 404

Hello,

 

I have a problem with DIH in SolrCloud. I don't understand why, so I need
your help.

 

Solr 6.6 in Cloud.

 

##

COMMAND:

curl http://srv-formation-solr:8983/solr/test_dih?command=full-import

 

RESULT:



  



Error 404 Not Found

  

  HTTP ERROR 404

Problem accessing /solr/test_dih. Reason:

  Not Found

  



 

 

##

CONFIG:

1.  I have create with the command the .system collection

2.  I have post in the blob the DataImportHandler jar file and the MySQL
connector jar

3.  I have add data-import-handler and mysql-connector-java runtimeLib
on the configoverlay.json file with the API

4.  I have create the DIH folder on the cloud with zkcli.sh script

5.  I have push with zkcli the DIH .xml configuration file

 

CONFIGOVERLAY CONTENT :

{

  "runtimeLib":{

"mysql-connector-java":{

  "name":"mysql-connector-java",

      "version":1},

"data-import-handler":{

  "name":"data-import-handler",

  "version":1}},

  "requestHandler":{"/test_dih":{

  "name":"/test_dih",

  "class":"org.apache.solr.handler.dataimport.DataImportHandler",

  "runtimeLib":true,

  "version":1,

 
"defaults":{"config":"/configs/arguments_test/DIH/indexation_events.xml"}}}

}

 

 

Thanks for your help

RE: SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

I have correct url to : curl
http://srv-formation-solr:8983/solr/arguments_test/test_dih?command=full-imp
ort

And change overlay config
"/configs/arguments_test/DIH/indexation_events.xml" to "
DIH/indexation_events.xml"

But I have a new error:

Full Import
failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
to PropertyWriter implementation:ZKPropertiesWriter
at
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImp
orter.java:330)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja
va:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474
)
at
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImport
er.java:457)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
at
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImp
orter.java:326)
... 4 more

Cordialement,





-Message d'origine-
De : msaunier [mailto:msaun...@citya.com] 
Envoyé : lundi 23 avril 2018 14:12
À : solr-user@lucene.apache.org
Objet : SolrCloud DIH (Data Import Handler) MySQL 404

Hello,

 

I have a problem with DIH in SolrCloud. I don't understand why, so I need
your help.

 

Solr 6.6 in Cloud.

 

##

COMMAND:

curl http://srv-formation-solr:8983/solr/test_dih?command=full-import

 

RESULT:



  



Error 404 Not Found

  

  HTTP ERROR 404

Problem accessing /solr/test_dih. Reason:

  Not Found

  



 

 

##

CONFIG:

1.  I have create with the command the .system collection

2.  I have post in the blob the DataImportHandler jar file and the MySQL
connector jar

3.  I have add data-import-handler and mysql-connector-java runtimeLib
on the configoverlay.json file with the API

4.  I have create the DIH folder on the cloud with zkcli.sh script

5.  I have push with zkcli the DIH .xml configuration file

 

CONFIGOVERLAY CONTENT :

{

  "runtimeLib":{

"mysql-connector-java":{

  "name":"mysql-connector-java",

  "version":1},

"data-import-handler":{

  "name":"data-import-handler",

  "version":1}},

  "requestHandler":{"/test_dih":{

  "name":"/test_dih",

  "class":"org.apache.solr.handler.dataimport.DataImportHandler",

  "runtimeLib":true,

  "version":1,

 
"defaults":{"config":"/configs/arguments_test/DIH/indexation_events.xml"}}}

}

 

 

Thanks for your help

SolrCloud DIH (Data Import Handler) MySQL 404

[msaunier]

Hello,

 

I have a problem with DIH in SolrCloud. I don't understand why, so I need
your help.

 

Solr 6.6 in Cloud.

 

##

COMMAND:

curl http://srv-formation-solr:8983/solr/test_dih?command=full-import

 

RESULT:



  



Error 404 Not Found

  

  HTTP ERROR 404

Problem accessing /solr/test_dih. Reason:

  Not Found

  



 

 

##

CONFIG:

1.  I have create with the command the .system collection

2.  I have post in the blob the DataImportHandler jar file and the MySQL
connector jar

3.  I have add data-import-handler and mysql-connector-java runtimeLib
on the configoverlay.json file with the API

4.  I have create the DIH folder on the cloud with zkcli.sh script

5.  I have push with zkcli the DIH .xml configuration file

 

CONFIGOVERLAY CONTENT :

{

  "runtimeLib":{

"mysql-connector-java":{

  "name":"mysql-connector-java",

  "version":1},

"data-import-handler":{

  "name":"data-import-handler",

  "version":1}},

  "requestHandler":{"/test_dih":{

  "name":"/test_dih",

  "class":"org.apache.solr.handler.dataimport.DataImportHandler",

  "runtimeLib":true,

  "version":1,

 
"defaults":{"config":"/configs/arguments_test/DIH/indexation_events.xml"}}}

}

 

 

Thanks for your help

Re: Solr Search Problem with Multiple Data-Import Handler

[Josh Lincoln]

I suspect Erik's right that clean=true is the problem. That's the default
in the DIH interface.


I find that when using DIH, it's best to set preImportDeleteQuery for every
entity. This safely scopes the clean variable to just that entity.
It doesn't look like the docs have examples of using preImportDeleteQuery,
so I put one here:




On Wed, Jun 21, 2017 at 7:48 PM Erick Erickson 
wrote:

> First place I'd look is whether the jobs have clean=true set. If so the
> first thing DIH does is delete all documents.
>
> Best,
> Erick
>
> On Wed, Jun 21, 2017 at 3:52 PM, Pandey Brahmdev 
> wrote:
>
> > Hi,
> > I have setup Apache Solr 6.6.0 on Windows 10, 64-bit.
> >
> > I have created a simple core & configured DataImport Handlers.
> > I have configured 2 dataImport handlers in the Solr-config.xml file.
> >
> > First for to connect to DB & have data from DB Tables.
> > And Second for to have data from all pdf files using TikaEntityProcessor.
> >
> > Now the problem is there is no error in the console or anywhere but
> > whenever I want to search using "Query" tab it gives me the result of
> Data
> > Import.
> >
> > So let's say if I last Imported data for Tables then it gives me to
> result
> > from the table and if I imported PDF Files then it searches inside PDF
> > Files.
> >
> > But now when I again want to search for DB Tables values then It doesn't
> > give me the result instead I again need to Import Data for
> > DataImportHandler for File & vice-versa.
> >
> > Can you please help me out here?
> > Very sorry if I am doing anything wrong as I have started using Apache
> Solr
> > only 2 days back.
> >
> > Thanks & Regards,
> > Brahmdev Pandey
> > +46 767086309 <+46%2076%20708%2063%2009>
> >
>

Re: Solr Search Problem with Multiple Data-Import Handler

[Erick Erickson]

First place I'd look is whether the jobs have clean=true set. If so the
first thing DIH does is delete all documents.

Best,
Erick

On Wed, Jun 21, 2017 at 3:52 PM, Pandey Brahmdev 
wrote:

> Hi,
> I have setup Apache Solr 6.6.0 on Windows 10, 64-bit.
>
> I have created a simple core & configured DataImport Handlers.
> I have configured 2 dataImport handlers in the Solr-config.xml file.
>
> First for to connect to DB & have data from DB Tables.
> And Second for to have data from all pdf files using TikaEntityProcessor.
>
> Now the problem is there is no error in the console or anywhere but
> whenever I want to search using "Query" tab it gives me the result of Data
> Import.
>
> So let's say if I last Imported data for Tables then it gives me to result
> from the table and if I imported PDF Files then it searches inside PDF
> Files.
>
> But now when I again want to search for DB Tables values then It doesn't
> give me the result instead I again need to Import Data for
> DataImportHandler for File & vice-versa.
>
> Can you please help me out here?
> Very sorry if I am doing anything wrong as I have started using Apache Solr
> only 2 days back.
>
> Thanks & Regards,
> Brahmdev Pandey
> +46 767086309
>

Solr Search Problem with Multiple Data-Import Handler

[Pandey Brahmdev]

Hi,
I have setup Apache Solr 6.6.0 on Windows 10, 64-bit.

I have created a simple core & configured DataImport Handlers.
I have configured 2 dataImport handlers in the Solr-config.xml file.

First for to connect to DB & have data from DB Tables.
And Second for to have data from all pdf files using TikaEntityProcessor.

Now the problem is there is no error in the console or anywhere but
whenever I want to search using "Query" tab it gives me the result of Data
Import.

So let's say if I last Imported data for Tables then it gives me to result
from the table and if I imported PDF Files then it searches inside PDF
Files.

But now when I again want to search for DB Tables values then It doesn't
give me the result instead I again need to Import Data for
DataImportHandler for File & vice-versa.

Can you please help me out here?
Very sorry if I am doing anything wrong as I have started using Apache Solr
only 2 days back.

Thanks & Regards,
Brahmdev Pandey
+46 767086309

Data import handler and no status in web-ui

[Thomas Porschberg]

Hi,

I use DIH in solr-cloud mode (implicit route) in solr6.5.1.
When I start the import it works fine and I see the progress in the logfile.
However, when I click the "Refresh Status" button in the web-ui while the 
import is running
I only see "No information available (idle)". 
So I have to look in the logfile the observe when the import was finished.

In the old solr, non-cloud and non-partitioned, there was a hourglass while the 
import was running.

Any idea?

Best regards
Thomas

RE: Using the Data Import Handler with SQLite

[Dheeraj Kumar Karnati]

Hi Zac,
  I think you have added entity closing tag 2 times. that might be
causing an issue. It been a long time . not sure whether you are still
working on it or not. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Using-the-Data-Import-Handler-with-SQLite-tp2765655p4336690.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data Import Handler on 6.4.1

[Walter Underwood]

Also, upgrade to 6.4.2. There are serious performance problems in 6.4.0 and 
6.4.1.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Mar 15, 2017, at 12:05 PM, Liu, Daphne <daphne@cevalogistics.com> 
> wrote:
> 
> For Solr 6.3,  I have to move mine to 
> ../solr-6.3.0/server/solr-webapp/webapp/WEB-INF/lib. If you are using jetty.
> 
> Kind regards,
> 
> Daphne Liu
> BI Architect - Matrix SCM
> 
> CEVA Logistics / 10751 Deerwood Park Blvd, Suite 200, Jacksonville, FL 32256 
> USA / www.cevalogistics.com T 904.564.1192 / F 904.928.1448 / 
> daphne@cevalogistics.com
> 
> 
> -Original Message-
> From: Michael Tobias [mailto:mtob...@btinternet.com]
> Sent: Wednesday, March 15, 2017 2:36 PM
> To: solr-user@lucene.apache.org
> Subject: Data Import Handler on 6.4.1
> 
> I am sure I am missing something simple but
> 
> I am running Solr 4.8.1 and trialling 6.4.1 on another computer.
> 
> I have had to manually modify the automatic 6.4.1 scheme config as we use a 
> set of specialised field types.  They work fine.
> 
> I am now trying to populate my core with data and having problems.
> 
> Exactly what names/paths should I be using in the solrconfig.xml file to get 
> this working - I don’t recall doing ANYTHING for 4.8.1
> 
>   regex=".*\.jar" />  
>   regex="solr-dataimporthandler-.*\.jar" /> ?
> 
> And where do I put the mysql-connector-java-5.1.29-bin.jar file and how do I 
> reference it to get it loaded?
> 
>
> ??
> 
> And then later in the solrconfig.xml I have:
> 
>  class="org.apache.solr.handler.dataimport.DataImportHandler">
>  
>db-data-config.xml
>  
> 
> 
> 
> Any help much appreciated.
> 
> Regards
> 
> Michael
> 
> 
> -Original Message-
> From: David Hastings [mailto:hastings.recurs...@gmail.com]
> Sent: 15 March 2017 17:47
> To: solr-user@lucene.apache.org
> Subject: Re: Get handler not working
> 
> from your previous email:
> "There is no "id"
> field defined in the schema."
> 
> you need an id field to use the get handler
> 
> On Wed, Mar 15, 2017 at 1:45 PM, Chris Ulicny <culicny@iq.media> wrote:
> 
>> I thought that "id" and "ids" were fixed parameters for the get
>> handler, but I never remember, so I've already tried both. Each time
>> it comes back with the same response of no document.
>> 
>> On Wed, Mar 15, 2017 at 1:31 PM Alexandre Rafalovitch
>> <arafa...@gmail.com>
>> wrote:
>> 
>>> Actually.
>>> 
>>> I think Real Time Get handler has "id" as a magical parameter, not
>>> as a field name. It maps to the real id field via the uniqueKey
>>> definition:
>>> https://cwiki.apache.org/confluence/display/solr/RealTime+Get
>>> 
>>> So, if you have not, could you try the way you originally wrote it.
>>> 
>>> Regards,
>>>   Alex.
>>> 
>>> http://www.solr-start.com/ - Resources for Solr users, new and
>> experienced
>>> 
>>> 
>>> On 15 March 2017 at 13:22, Chris Ulicny <culicny@iq.media> wrote:
>>>> Sorry, that is a typo. The get is using the iqdocid field. There
>>>> is no
>>> "id"
>>>> field defined in the schema.
>>>> 
>>>> solr/TestCollection/get?iqdocid=2957-TV-201604141900
>>>> 
>>>> solr/TestCollection/select?q=*:*=iqdocid:2957-TV-201604141900
>>>> 
>>>> On Wed, Mar 15, 2017 at 1:15 PM Erick Erickson <
>> erickerick...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Is this a typo or are you trying to use get with an "id" field
>>>>> and your filter query uses "iqdocid"?
>>>>> 
>>>>> Best,
>>>>> Erick
>>>>> 
>>>>> On Wed, Mar 15, 2017 at 8:31 AM, Chris Ulicny <culicny@iq.media>
>> wrote:
>>>>>> Yes, we're using a fixed schema with the iqdocid field set as
>>>>>> the
>>>>> uniqueKey.
>>>>>> 
>>>>>> On Wed, Mar 15, 2017 at 11:28 AM Alexandre Rafalovitch <
>>>>> arafa...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> What is your uniqueKey? Is it iqdocid?
>>>>>>> 
>>>>>>> Regards,
>>>>>>>   Alex.
>>>>>>> 
>>>>>>>

RE: Data Import Handler on 6.4.1

[Liu, Daphne]

For Solr 6.3,  I have to move mine to 
../solr-6.3.0/server/solr-webapp/webapp/WEB-INF/lib. If you are using jetty.

Kind regards,

Daphne Liu
BI Architect - Matrix SCM

CEVA Logistics / 10751 Deerwood Park Blvd, Suite 200, Jacksonville, FL 32256 
USA / www.cevalogistics.com T 904.564.1192 / F 904.928.1448 / 
daphne@cevalogistics.com


-Original Message-
From: Michael Tobias [mailto:mtob...@btinternet.com]
Sent: Wednesday, March 15, 2017 2:36 PM
To: solr-user@lucene.apache.org
Subject: Data Import Handler on 6.4.1

I am sure I am missing something simple but

I am running Solr 4.8.1 and trialling 6.4.1 on another computer.

I have had to manually modify the automatic 6.4.1 scheme config as we use a set 
of specialised field types.  They work fine.

I am now trying to populate my core with data and having problems.

Exactly what names/paths should I be using in the solrconfig.xml file to get 
this working - I don’t recall doing ANYTHING for 4.8.1


   ?

And where do I put the mysql-connector-java-5.1.29-bin.jar file and how do I 
reference it to get it loaded?


??

And then later in the solrconfig.xml I have:


  
db-data-config.xml
  



Any help much appreciated.

Regards

Michael


-Original Message-
From: David Hastings [mailto:hastings.recurs...@gmail.com]
Sent: 15 March 2017 17:47
To: solr-user@lucene.apache.org
Subject: Re: Get handler not working

from your previous email:
"There is no "id"
field defined in the schema."

you need an id field to use the get handler

On Wed, Mar 15, 2017 at 1:45 PM, Chris Ulicny <culicny@iq.media> wrote:

> I thought that "id" and "ids" were fixed parameters for the get
> handler, but I never remember, so I've already tried both. Each time
> it comes back with the same response of no document.
>
> On Wed, Mar 15, 2017 at 1:31 PM Alexandre Rafalovitch
> <arafa...@gmail.com>
> wrote:
>
> > Actually.
> >
> > I think Real Time Get handler has "id" as a magical parameter, not
> > as a field name. It maps to the real id field via the uniqueKey
> > definition:
> > https://cwiki.apache.org/confluence/display/solr/RealTime+Get
> >
> > So, if you have not, could you try the way you originally wrote it.
> >
> > Regards,
> >Alex.
> > 
> > http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> >
> >
> > On 15 March 2017 at 13:22, Chris Ulicny <culicny@iq.media> wrote:
> > > Sorry, that is a typo. The get is using the iqdocid field. There
> > > is no
> > "id"
> > > field defined in the schema.
> > >
> > > solr/TestCollection/get?iqdocid=2957-TV-201604141900
> > >
> > > solr/TestCollection/select?q=*:*=iqdocid:2957-TV-201604141900
> > >
> > > On Wed, Mar 15, 2017 at 1:15 PM Erick Erickson <
> erickerick...@gmail.com>
> > > wrote:
> > >
> > >> Is this a typo or are you trying to use get with an "id" field
> > >> and your filter query uses "iqdocid"?
> > >>
> > >> Best,
> > >> Erick
> > >>
> > >> On Wed, Mar 15, 2017 at 8:31 AM, Chris Ulicny <culicny@iq.media>
> wrote:
> > >> > Yes, we're using a fixed schema with the iqdocid field set as
> > >> > the
> > >> uniqueKey.
> > >> >
> > >> > On Wed, Mar 15, 2017 at 11:28 AM Alexandre Rafalovitch <
> > >> arafa...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> What is your uniqueKey? Is it iqdocid?
> > >> >>
> > >> >> Regards,
> > >> >>Alex.
> > >> >> 
> > >> >> http://www.solr-start.com/ - Resources for Solr users, new and
> > >> experienced
> > >> >>
> > >> >>
> > >> >> On 15 March 2017 at 11:24, Chris Ulicny <culicny@iq.media> wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > I've been trying to use the get handler for a new solr cloud
> > >> collection
> > >> >> we
> > >> >> > are using, and something seems to be amiss.
> > >> >> >
> > >> >> > We are running 6.3.0, so we did not explicitly define the
> > >> >> > request
> > >> handler
> > >> >> > in the solrconfig since it's supposed to be implicitly defined.
> We
> > >> also
> > >> >> > have the update log enabled with the defaul

Data Import Handler on 6.4.1

[Michael Tobias]

I am sure I am missing something simple but

I am running Solr 4.8.1 and trialling 6.4.1 on another computer.

I have had to manually modify the automatic 6.4.1 scheme config as we use a set 
of specialised field types.  They work fine.

I am now trying to populate my core with data and having problems.

Exactly what names/paths should I be using in the solrconfig.xml file to get 
this working - I don’t recall doing ANYTHING for 4.8.1


   ?
   
And where do I put the mysql-connector-java-5.1.29-bin.jar file and how do I 
reference it to get it loaded?


??

And then later in the solrconfig.xml I have:


  
db-data-config.xml
  



Any help much appreciated.

Regards

Michael


-Original Message-
From: David Hastings [mailto:hastings.recurs...@gmail.com] 
Sent: 15 March 2017 17:47
To: solr-user@lucene.apache.org
Subject: Re: Get handler not working

from your previous email:
"There is no "id"
field defined in the schema."

you need an id field to use the get handler

On Wed, Mar 15, 2017 at 1:45 PM, Chris Ulicny  wrote:

> I thought that "id" and "ids" were fixed parameters for the get 
> handler, but I never remember, so I've already tried both. Each time 
> it comes back with the same response of no document.
>
> On Wed, Mar 15, 2017 at 1:31 PM Alexandre Rafalovitch 
> 
> wrote:
>
> > Actually.
> >
> > I think Real Time Get handler has "id" as a magical parameter, not 
> > as a field name. It maps to the real id field via the uniqueKey
> > definition:
> > https://cwiki.apache.org/confluence/display/solr/RealTime+Get
> >
> > So, if you have not, could you try the way you originally wrote it.
> >
> > Regards,
> >Alex.
> > 
> > http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> >
> >
> > On 15 March 2017 at 13:22, Chris Ulicny  wrote:
> > > Sorry, that is a typo. The get is using the iqdocid field. There 
> > > is no
> > "id"
> > > field defined in the schema.
> > >
> > > solr/TestCollection/get?iqdocid=2957-TV-201604141900
> > >
> > > solr/TestCollection/select?q=*:*=iqdocid:2957-TV-201604141900
> > >
> > > On Wed, Mar 15, 2017 at 1:15 PM Erick Erickson <
> erickerick...@gmail.com>
> > > wrote:
> > >
> > >> Is this a typo or are you trying to use get with an "id" field 
> > >> and your filter query uses "iqdocid"?
> > >>
> > >> Best,
> > >> Erick
> > >>
> > >> On Wed, Mar 15, 2017 at 8:31 AM, Chris Ulicny 
> wrote:
> > >> > Yes, we're using a fixed schema with the iqdocid field set as 
> > >> > the
> > >> uniqueKey.
> > >> >
> > >> > On Wed, Mar 15, 2017 at 11:28 AM Alexandre Rafalovitch <
> > >> arafa...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> What is your uniqueKey? Is it iqdocid?
> > >> >>
> > >> >> Regards,
> > >> >>Alex.
> > >> >> 
> > >> >> http://www.solr-start.com/ - Resources for Solr users, new and
> > >> experienced
> > >> >>
> > >> >>
> > >> >> On 15 March 2017 at 11:24, Chris Ulicny  wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > I've been trying to use the get handler for a new solr cloud
> > >> collection
> > >> >> we
> > >> >> > are using, and something seems to be amiss.
> > >> >> >
> > >> >> > We are running 6.3.0, so we did not explicitly define the 
> > >> >> > request
> > >> handler
> > >> >> > in the solrconfig since it's supposed to be implicitly defined.
> We
> > >> also
> > >> >> > have the update log enabled with the default configuration.
> > >> >> >
> > >> >> > Whenever I send a get query for a document already known to 
> > >> >> > be in
> > the
> > >> >> > collection, I get no documents returned. But when I use a 
> > >> >> > filter
> > >> query on
> > >> >> > the uniqueKey field for the same value I get the document 
> > >> >> > back
> > >> >> >
> > >> >> > solr/TestCollection/get?id=2957-TV-201604141900
> > >> >> >
> > >> >> > solr/TestCollection/select?q=*:*=iqdocid:2957-TV-20160414
> > >> >> > 1900
> > >> >> >
> > >> >> > Is there some configuration that I am missing?
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Chris
> > >> >>
> > >>
> >
>

Re: Data Import Handler, also "Real Time" index updates

[Damien Kamerman]

You could configure the dataimporthandler to not delete at the start
(either do a delta or set the preimportdeltequery), and set a
postimportdeletequery if required.

On Saturday, 4 March 2017, Alexandre Rafalovitch  wrote:

> Commit is index global. So if you have overlapping timelines and commit is
> issued, it will affect all changes done to that point.
>
> So, the aliases may be better for you. You could potentially also reload a
> cure with changes solrconfig.XML settings, but that's heavy on caches.
>
> Regards,
>Alex
>
> On 3 Mar 2017 1:21 PM, "Sales"  >
> wrote:
>
>
> >
> > You have indicated that you have a way to avoid doing updates during the
> > full import.  Because of this, you do have another option that is likely
> > much easier for you to implement:  Set the "commitWithin" parameter on
> > each update request.  This works almost identically to autoSoftCommit,
> > but only after a request is made.  As long as there are never any of
> > these updates during a full import, these commits cannot affect that
> import.
>
> I had attempted at least to say that there may be a few updates that happen
> at the start of an import, so, they are while an import is happening just
> due to timing issues. Those will be detected, and, re-executed once the
> import is done though. But my question here is if the update is using
> commitWithin, then, does that only affect those updates that have the
> parameter, or, does it then also soft commit the in progress import? I
> cannot guarantee that zero updates will be done as there is a timing issue
> at the very start of the import, so, a few could cross over.
>
> Adding commitWithin is fine. Just want to make sure those that might
> execute for the first few seconds of an import don’t kill anything.
> >
> > No matter what is happening, you should have autoCommit (not
> > autoSoftCommit) configured with openSearcher set to false.  This will
> > ensure transaction log rollover, without affecting change visibility.  I
> > recommend a maxTime of one to five minutes for this.  You'll see 15
> > seconds as the recommended value in many places.
> >
> > https://lucidworks.com/2013/08/23/understanding-
> transaction-logs-softcommit-and-commit-in-sorlcloud/ <
> https://lucidworks.com/2013/08/23/understanding-
> transaction-logs-softcommit-
> and-commit-in-sorlcloud/>
>
> Oh, we are fine with much longer, does not have to be instant. 10-15
> minutes would be fine.
>
> >
> > Thanks
> > Shawn
> >
>

Re: Data Import Handler, also "Real Time" index updates

[Alexandre Rafalovitch]

Commit is index global. So if you have overlapping timelines and commit is
issued, it will affect all changes done to that point.

So, the aliases may be better for you. You could potentially also reload a
cure with changes solrconfig.XML settings, but that's heavy on caches.

Regards,
   Alex

On 3 Mar 2017 1:21 PM, "Sales" 
wrote:


>
> You have indicated that you have a way to avoid doing updates during the
> full import.  Because of this, you do have another option that is likely
> much easier for you to implement:  Set the "commitWithin" parameter on
> each update request.  This works almost identically to autoSoftCommit,
> but only after a request is made.  As long as there are never any of
> these updates during a full import, these commits cannot affect that
import.

I had attempted at least to say that there may be a few updates that happen
at the start of an import, so, they are while an import is happening just
due to timing issues. Those will be detected, and, re-executed once the
import is done though. But my question here is if the update is using
commitWithin, then, does that only affect those updates that have the
parameter, or, does it then also soft commit the in progress import? I
cannot guarantee that zero updates will be done as there is a timing issue
at the very start of the import, so, a few could cross over.

Adding commitWithin is fine. Just want to make sure those that might
execute for the first few seconds of an import don’t kill anything.
>
> No matter what is happening, you should have autoCommit (not
> autoSoftCommit) configured with openSearcher set to false.  This will
> ensure transaction log rollover, without affecting change visibility.  I
> recommend a maxTime of one to five minutes for this.  You'll see 15
> seconds as the recommended value in many places.
>
> https://lucidworks.com/2013/08/23/understanding-
transaction-logs-softcommit-and-commit-in-sorlcloud/ <
https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-
and-commit-in-sorlcloud/>

Oh, we are fine with much longer, does not have to be instant. 10-15
minutes would be fine.

>
> Thanks
> Shawn
>

Re: Data Import Handler, also "Real Time" index updates

[Sales]

> 
> You have indicated that you have a way to avoid doing updates during the
> full import.  Because of this, you do have another option that is likely
> much easier for you to implement:  Set the "commitWithin" parameter on
> each update request.  This works almost identically to autoSoftCommit,
> but only after a request is made.  As long as there are never any of
> these updates during a full import, these commits cannot affect that import.

I had attempted at least to say that there may be a few updates that happen at 
the start of an import, so, they are while an import is happening just due to 
timing issues. Those will be detected, and, re-executed once the import is done 
though. But my question here is if the update is using commitWithin, then, does 
that only affect those updates that have the parameter, or, does it then also 
soft commit the in progress import? I cannot guarantee that zero updates will 
be done as there is a timing issue at the very start of the import, so, a few 
could cross over. 

Adding commitWithin is fine. Just want to make sure those that might execute 
for the first few seconds of an import don’t kill anything. 
> 
> No matter what is happening, you should have autoCommit (not
> autoSoftCommit) configured with openSearcher set to false.  This will
> ensure transaction log rollover, without affecting change visibility.  I
> recommend a maxTime of one to five minutes for this.  You'll see 15
> seconds as the recommended value in many places.
> 
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>  
> 

Oh, we are fine with much longer, does not have to be instant. 10-15 minutes 
would be fine.

> 
> Thanks
> Shawn
> 

Re: Data Import Handler, also "Real Time" index updates

[Shawn Heisey]

On 3/3/2017 10:17 AM, Sales wrote:
> I am not sure how best to handle this. We use the data import handle to 
> re-sync all our data on a daily basis, takes 1-2 hours depending on system 
> load. It is set up to commit at the end, so, the old index remains until it’s 
> done, and, we lose no access while the import is happening.
>
> But, we now want to update certain fields in the index, but still regen 
> daily. So, it would seem we might need to autocommit, and, soft commit 
> potentially. When we enabled those, during the index, the data disappeared 
> since it kept soft committing during the import process, I see no way to 
> avoid soft commits during the import. But soft commits would appear to be 
> needed for the (non import) updates to the index. 
>
> I realize the import could happen while an update is done, but we can 
> actually avoid those. So, that is not an issue (one or two might go through, 
> but, we will redo those updates once the index is done, that part is all 
> handled.

Erick's solution of using aliases to swap a live index and a build index
is one very good way to go.  It does involve some additional complexity
that you may not be ready for.  Only you will know whether that's
something you can implement easily.  Collection aliasing was implemented
in Solr 4.2 by SOLR-4497, so 4.10 should definitely have it.

You have indicated that you have a way to avoid doing updates during the
full import.  Because of this, you do have another option that is likely
much easier for you to implement:  Set the "commitWithin" parameter on
each update request.  This works almost identically to autoSoftCommit,
but only after a request is made.  As long as there are never any of
these updates during a full import, these commits cannot affect that import.

No matter what is happening, you should have autoCommit (not
autoSoftCommit) configured with openSearcher set to false.  This will
ensure transaction log rollover, without affecting change visibility.  I
recommend a maxTime of one to five minutes for this.  You'll see 15
seconds as the recommended value in many places.

https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

Thanks
Shawn

Re: Data Import Handler, also "Real Time" index updates

[Sales]

> On Mar 3, 2017, at 11:30 AM, Erick Erickson  wrote:
> 
> One way to handle this (presuming SolrCloud) is collection aliasing.
> You create two collections, c1 and c2. You then have two aliases. when
> you start "index" is aliased to c1 and "search" is aliased to c2. Now
> do your full import  to "index" (and, BTW, you'd be well advised to do
> at least a hard commit openSearcher=false during that time or you risk
> replaying all the docs in the tlog).
> 
> When the full import is done, switch the aliases so "search" points to c1 and
> "index" points to c2. Rinse. Repeat. Your client apps always use the same 
> alias,
> the alias switching makes whether c1 or c2 is being used transparent.
> By that I mean your user-facing app uses "search" and your indexing client
> uses "index".
> 
> You can now do your live updates to the "search" alias that has a soft
> commit set.
> Of course you have to have some mechanism for replaying all the live updates
> that came in when you were doing your full index into the "indexing"
> alias before
> you switch, but you say you have that handled.
> 
> Best,
> Erick
> 

Thanks. So, is this available on 4.10.4? 

If not, we used to gen another core, do the import, and, swap cores so this is 
possibly similar to collection aliases since in the end, the client did not 
care. I don’t see why that would not still work. Took a little effort to 
automate, but, not much. 

Regarding the import and commit, we use in data-config.xml readonly so this 
sets autocommit the way I understand it. Not sure what happens with 
opensearcher though. If that is not sufficient, how would I do hard commit and 
opensearcher false during that time? Surely not by modifying the config file?

Re: Data Import Handler, also "Real Time" index updates

[Erick Erickson]

One way to handle this (presuming SolrCloud) is collection aliasing.
You create two collections, c1 and c2. You then have two aliases. when
you start "index" is aliased to c1 and "search" is aliased to c2. Now
do your full import  to "index" (and, BTW, you'd be well advised to do
at least a hard commit openSearcher=false during that time or you risk
replaying all the docs in the tlog).

When the full import is done, switch the aliases so "search" points to c1 and
"index" points to c2. Rinse. Repeat. Your client apps always use the same alias,
the alias switching makes whether c1 or c2 is being used transparent.
By that I mean your user-facing app uses "search" and your indexing client
uses "index".

You can now do your live updates to the "search" alias that has a soft
commit set.
Of course you have to have some mechanism for replaying all the live updates
that came in when you were doing your full index into the "indexing"
alias before
you switch, but you say you have that handled.

Best,
Erick

On Fri, Mar 3, 2017 at 9:22 AM, Alexandre Rafalovitch
 wrote:
> On 3 March 2017 at 12:17, Sales  
> wrote:
>> When we enabled those, during the index, the data disappeared since it kept 
>> soft committing during the import process,
>
> This part does not quite make sense. Could you expand on this "data
> disappeared" part to understand what the issue is.
>
> The main issue with "update" is that all fields (apart from pure
> copyField destinations) need to be stored, so the document can be
> reconstructed, updated, re-indexed. Perhaps you have something strange
> happening around that?
>
> Regards,
>Alex.
>
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced

Re: Data Import Handler, also "Real Time" index updates

[Sales]

> 
> On Mar 3, 2017, at 11:22 AM, Alexandre Rafalovitch  wrote:
> 
> On 3 March 2017 at 12:17, Sales  
> wrote:
>> When we enabled those, during the index, the data disappeared since it kept 
>> soft committing during the import process,
> 
> This part does not quite make sense. Could you expand on this "data
> disappeared" part to understand what the issue is.
> 

So, the issue here is the first part of the import handler is to erase all the 
data, so, there are no products left in the index (it would appear based on 
what we see, after the first softcommit), and, a search returns no result at 
first, but, ever increasing number of records while the import is happening. We 
have 6 million indexed products.

I can't find a way to stop soft commits during the import?

Re: Data Import Handler, also "Real Time" index updates

[Alexandre Rafalovitch]

On 3 March 2017 at 12:17, Sales  wrote:
> When we enabled those, during the index, the data disappeared since it kept 
> soft committing during the import process,

This part does not quite make sense. Could you expand on this "data
disappeared" part to understand what the issue is.

The main issue with "update" is that all fields (apart from pure
copyField destinations) need to be stored, so the document can be
reconstructed, updated, re-indexed. Perhaps you have something strange
happening around that?

Regards,
   Alex.


http://www.solr-start.com/ - Resources for Solr users, new and experienced

Data Import Handler, also "Real Time" index updates

[Sales]

I am not sure how best to handle this. We use the data import handle to re-sync 
all our data on a daily basis, takes 1-2 hours depending on system load. It is 
set up to commit at the end, so, the old index remains until it’s done, and, we 
lose no access while the import is happening.

But, we now want to update certain fields in the index, but still regen daily. 
So, it would seem we might need to autocommit, and, soft commit potentially. 
When we enabled those, during the index, the data disappeared since it kept 
soft committing during the import process, I see no way to avoid soft commits 
during the import. But soft commits would appear to be needed for the (non 
import) updates to the index. 

I realize the import could happen while an update is done, but we can actually 
avoid those. So, that is not an issue (one or two might go through, but, we 
will redo those updates once the index is done, that part is all handled.

So, what is the best way to handle “real time” updates (10-15 minutes is fine 
to see the updates in a searcher), yet, also allow dataimport handler to do a 
full clear and regen without losing products (what we index) during the import, 
we don’t want searchers not seeing the data! Have not seen any techniques for 
this. 

Steve

Re: Solr Data Import Handler

[Alexandre Rafalovitch]

They don't have to wait because they are not synchronized on anything
AFAIK. I do believe they need to be registered as different request
handler names in solrconfig.xml

But you need to be careful if they both/all wipe out data before
indexing new one. Those calls could overlap.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 12 February 2017 at 16:14, Dave  wrote:
> That sounds pretty much like a hack. So if two imports happen at the same 
> time they have to wait for each other?
>
>> On Feb 12, 2017, at 4:01 PM, Shawn Heisey  wrote:
>>
>>> On 2/12/2017 10:30 AM, Minh wrote:
>>> Hi everyone,
>>> How can i run multithreads of DIH in a cluster for a collection?
>>
>> The DIH handler is single-threaded.  It used to have a config option for
>> multiple threads, but it was removed since it didn't actually work.
>>
>> If you create multiple DIH handlers and start an import on them all at
>> the same time, then you'll have multiple threads.
>>
>> Thanks,
>> Shawn
>>

Re: Solr Data Import Handler

[Dave]

That sounds pretty much like a hack. So if two imports happen at the same time 
they have to wait for each other?

> On Feb 12, 2017, at 4:01 PM, Shawn Heisey  wrote:
> 
>> On 2/12/2017 10:30 AM, Minh wrote:
>> Hi everyone,
>> How can i run multithreads of DIH in a cluster for a collection?
> 
> The DIH handler is single-threaded.  It used to have a config option for
> multiple threads, but it was removed since it didn't actually work.
> 
> If you create multiple DIH handlers and start an import on them all at
> the same time, then you'll have multiple threads.
> 
> Thanks,
> Shawn
> 

Re: Solr Data Import Handler

[Shawn Heisey]

On 2/12/2017 10:30 AM, Minh wrote:
> Hi everyone,
> How can i run multithreads of DIH in a cluster for a collection?

The DIH handler is single-threaded.  It used to have a config option for
multiple threads, but it was removed since it didn't actually work.

If you create multiple DIH handlers and start an import on them all at
the same time, then you'll have multiple threads.

Thanks,
Shawn

Solr Data Import Handler

[Minh]

Hi everyone,
How can i run multithreads of DIH in a cluster for a collection?

Re: problem with data import handler delta import due to use of multiple datasource

[amylindan]

Did you solve the problem? I'm stuck with exactly the same problem now. Let
me know if you already had a solution,please.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/problem-with-data-import-handler-delta-import-due-to-use-of-multiple-datasource-tp4093698p4314273.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data Import Handler - maximum?

[Shawn Heisey]

On 12/11/2016 8:00 PM, Brian Narsi wrote:
> We are using Solr 5.1.0 and DIH to build index.
>
> We are using DIH with clean=true and commit=true and optimize=true.
> Currently retrieving about 10.5 million records in about an hour.
>
> I will like to find from other member's experiences as to how long can DIH
> run with no issues? What is the maximum number of records that anyone has
> pulled using DIH?
>
> Are there any limitations on the maximum number of records that can/should
> be pulled using DIH? What is the longest DIH can run?

There are no hard limits other than the Lucene limit of a little over
two billion docs per individual index.  With sharding, Solr is able to
easily overcome this limit on an entire index.

I have one index where each shard was over 50 million docs.  Each shard
has fewer docs now, because I changed it so there are more shards and
more machines.  For some reason the rebuild time (using DIH) got really
really long -- nearly 48 hours -- while building every shard in
parallel.  Still haven't figured out why the build time increased
dramatically.

One problem you might run into with DIH from a database has to do with
merging.  With default merge scheduler settings, eventually (typically
when there are millions of rows being imported) you'll run into a pause
in indexing that will take so long that the database connection will
close, causing the import to fail after the pause finishes.

I even opened a Lucene issue to get the default value for maxMergeCount
changed.  This issue went nowhere:

https://issues.apache.org/jira/browse/LUCENE-5705

Here's a thread from this mailing list discussing the problem and the
configuration solution:

http://lucene.472066.n3.nabble.com/What-does-quot-too-many-merges-stalling-quot-in-indexwriter-log-mean-td4077380.html

Thanks,
Shawn

Re: Data Import Handler - maximum?

[Bernd Fehling]

Am 12.12.2016 um 04:00 schrieb Brian Narsi:
> We are using Solr 5.1.0 and DIH to build index.
> 
> We are using DIH with clean=true and commit=true and optimize=true.
> Currently retrieving about 10.5 million records in about an hour.
> 
> I will like to find from other member's experiences as to how long can DIH
> run with no issues? What is the maximum number of records that anyone has
> pulled using DIH?

Afaik, DIH will run until maximum number of documents per index.
Our longest run took about 3.5 days for single DIH and over 100 mio. docs.
The runtime depends pretty much on the complexity of the analysis during 
loading.

Currently we are using concurrent DIH with 12 processes which takes 15 hours
for the same amount. Optimizing afterwards takes 9.5 hours.

SolrJ with 12 threads is doing the same indexing within 7.5 hours plus 
optimizing.
For huge amounts of data you should consider using SolrJ.

> 
> Are there any limitations on the maximum number of records that can/should
> be pulled using DIH? What is the longest DIH can run?
> 
> Thanks a bunch!
> 

Data Import Handler - maximum?

[Brian Narsi]

We are using Solr 5.1.0 and DIH to build index.

We are using DIH with clean=true and commit=true and optimize=true.
Currently retrieving about 10.5 million records in about an hour.

I will like to find from other member's experiences as to how long can DIH
run with no issues? What is the maximum number of records that anyone has
pulled using DIH?

Are there any limitations on the maximum number of records that can/should
be pulled using DIH? What is the longest DIH can run?

Thanks a bunch!

Re: Scheduling Data Import Handler (DIH) Delta Imports

[Walter Underwood]

First, try to do it with something like Apache Camel. That moves the whole
database import process outside of Solr where it can be more easily controlled.

http://camel.apache.org/ 

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 28, 2016, at 12:03 PM, Jamie Jackson  wrote:
> 
> One last bump before I get crackin'...
> 
> On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson 
> wrote:
> 
>> Hi Folks,
>> 
>> I have DIH cores that are being indexed by my Lucee application. That
>> works, but I'd like to make some improvements:
>> 
>> 
>>   - Make a standalone scheduler that's not part of a larger application.
>>   (FYI, I want to Dockerize the import-triggering service.)
>>   - Prevent import requests from stacking up. Some of my cores' delta
>>   imports run every 15 seconds, and they do so blindly/ignorantly. If there
>>   is contention, very occasionally, import jobs will run long and stack up,
>>   so I want to make the scheduler nicer/more intelligent. Maybe the service
>>   would check the import status to see if there's a job already running
>>   before requesting a new one.
>> 
>> I can write such a thing myself, but does anybody have a Linux or
>> cross-platform solution written already?
>> 
>> Thanks,
>> Jamie
>> 

Re: Scheduling Data Import Handler (DIH) Delta Imports

[Jamie Jackson]

One last bump before I get crackin'...

On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson 
wrote:

> Hi Folks,
>
> I have DIH cores that are being indexed by my Lucee application. That
> works, but I'd like to make some improvements:
>
>
>- Make a standalone scheduler that's not part of a larger application.
>(FYI, I want to Dockerize the import-triggering service.)
>- Prevent import requests from stacking up. Some of my cores' delta
>imports run every 15 seconds, and they do so blindly/ignorantly. If there
>is contention, very occasionally, import jobs will run long and stack up,
>so I want to make the scheduler nicer/more intelligent. Maybe the service
>would check the import status to see if there's a job already running
>before requesting a new one.
>
> I can write such a thing myself, but does anybody have a Linux or
> cross-platform solution written already?
>
> Thanks,
> Jamie
>

Scheduling Data Import Handler (DIH) Delta Imports

[Jamie Jackson]

Hi Folks,

I have DIH cores that are being indexed by my Lucee application. That
works, but I'd like to make some improvements:


   - Make a standalone scheduler that's not part of a larger application.
   (FYI, I want to Dockerize the import-triggering service.)
   - Prevent import requests from stacking up. Some of my cores' delta
   imports run every 15 seconds, and they do so blindly/ignorantly. If there
   is contention, very occasionally, import jobs will run long and stack up,
   so I want to make the scheduler nicer/more intelligent. Maybe the service
   would check the import status to see if there's a job already running
   before requesting a new one.

I can write such a thing myself, but does anybody have a Linux or
cross-platform solution written already?

Thanks,
Jamie

Re: Problem with Password Decryption in Data Import Handler

[William Bell]

Protocol.c
>> onnectWithoutProxy(AbstractConnectProtocol.java:763)
>> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.
>> java:469)
>> at org.mariadb.jdbc.Driver.connect(Driver.java:104)
>> ... 16 more
>>
>>
>>
>> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <jamieja...@gmail.com>
>> wrote:
>>
>> > It happens to be ten characters.
>> >
>> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
>> arafa...@gmail.com
>> > > wrote:
>> >
>> >> How long is the encryption key (file content)? Because the code I am
>> >> looking at seems to expect it to be at most 100 characters.
>> >>
>> >> Regards,
>> >>Alex.
>> >> 
>> >> Newsletter and resources for Solr beginners and intermediates:
>> >> http://www.solr-start.com/
>> >>
>> >>
>> >> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>
>> >> wrote:
>> >> > I haven't tried this but is it possible there is a new line at the
>> end
>> >> in
>> >> > the file?
>> >> >
>> >> > If you did something like echo "" > file.txt then there would be a
>> new
>> >> > line. Use echo -n "" > file.txt
>> >> >
>> >> > Also you should be able to check how many characters are in the file.
>> >> >
>> >> > Kevin Risden
>> >> >
>> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi Folks,
>> >> >>
>> >> >> (Using Solr 5.5.3.)
>> >> >>
>> >> >> As far as I know, the only place where encrypted password use is
>> >> documented
>> >> >> is in
>> >> >> https://cwiki.apache.org/confluence/display/solr/
>> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> >> >> under the "Configuring the DIH Configuration File", in a comment in
>> the
>> >> >> sample XML file:
>> >> >>
>> >> >> 
>> >> >>
>> >> >> Anyway, I can encrypt just fine:
>> >> >>
>> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> >> >> enter aes-128-cbc encryption password:
>> >> >> Verifying - enter aes-128-cbc encryption password:
>> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>> >> >>
>> >> >>
>> >> >> I can also decrypt just fine from the command line.
>> >> >>
>> >> >> However, if I use the encrypted password and encryptKeyFile in the
>> >> config
>> >> >> file, I end up with an error: "String length must be a multiple of
>> >> four."
>> >> >>
>> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d4
>> 3ade5e4d9
>> >> >>
>> >> >> How do I get this working?
>> >> >>
>> >> >> Thanks,
>> >> >> Jamie
>> >> >>
>> >>
>> >
>> >
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Problem with Password Decryption in Data Import Handler

[William Bell]

dbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> />
> ...
>
>
> By the way, /var/docker/solr_stage2/credentials/ is mapped to
> /opt/solr/credentials/ in the docker container, so that's why the paths
> *seem* different (but aren't, really).
>
>
> *Authentication error when data import is run:*
>
> Exception while processing: question document :
> SolrInputDocument(fields:
> []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to execute query: select 'foo' as bar;Processing
> Document # 1
> at org.apache.solr.handler.dataimport.DataImportHandlerException.
> wrapAndThrow(DataImportHandlerException.java:69)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.(JdbcDataSource.java:323)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:283)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:52)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> initQuery(SqlEntityProcessor.java:59)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextRow(SqlEntityProcessor.java:73)
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
> nextRow(EntityProcessorWrapper.java:244)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:475)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
> at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> connect: Access denied for user 'root'@'ICZ2002912' (using password:
> NO)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> ExceptionMapper.java:123)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> ExceptionMapper.java:71)
> at org.mariadb.jdbc.Driver.connect(Driver.java:109)
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:192)
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:172)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getConnection(JdbcDataSource.java:503)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.(JdbcDataSource.java:313)
> ... 12 more
> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> not connect: Access denied for user 'root'@'ICZ2002912' (using
> password: NO)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> authentication(AbstractConnectProtocol.java:524)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> handleConnectionPhases(AbstractConnectProtocol.java:472)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> connect(AbstractConnectProtocol.java:374)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> connectWithoutProxy(AbstractConnectProtocol.java:763)
> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(
> Utils.java:469)
> at org.mariadb.jdbc.Driver.connect(Driver.java:104)
> ... 16 more
>
>
>
> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <jamieja...@gmail.com>
> wrote:
>
> > It happens to be ten characters.
> >
> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
> arafa...@gmail.com
> > > wrote:
> >
> >> How long is the encryption key (file content)? Because the code I am
> >> looking at seems to expect it to be at most 100 characters.
> >>
> >> Regards,
> >>Alex.
> >> 
> >> Newsletter and resources for Solr beginners and intermediates:
> >> http://www.solr-start.com/
> >>
> >>
> >> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>
> >> wrote:
> >> > I haven't tried this but is it possible there is a new line at the end
> >> in
> >> > the file?
> >> >
> >> > If you

Re: Problem with Password Decryption in Data Import Handler

[Jamie Jackson]

question document :
> > SolrInputDocument(fields:
> > []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> > Unable to execute query: select 'foo' as bar; Processing
> > Document # 1
> > at org.apache.solr.handler.dataimport.DataImportHandlerException.
> > wrapAndThrow(DataImportHandlerException.java:69)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator.(JdbcDataSource.java:323)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > getData(JdbcDataSource.java:283)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > getData(JdbcDataSource.java:52)
> > at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > initQuery(SqlEntityProcessor.java:59)
> > at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > nextRow(SqlEntityProcessor.java:73)
> > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(
> > EntityProcessorWrapper.java:244)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > buildDocument(DocBuilder.java:475)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > buildDocument(DocBuilder.java:414)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > doFullDump(DocBuilder.java:329)
> > at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > DocBuilder.java:232)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > doFullImport(DataImporter.java:416)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > runCmd(DataImporter.java:480)
> > at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > DataImporter.java:461)
> > Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> > connect: Access denied for user 'root'@'ICZ2002912' (using password:
> > NO)
> > at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> > ExceptionMapper.java:123)
> > at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> > ExceptionMapper.java:71)
> > at org.mariadb.jdbc.Driver.connect(Driver.java:109)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> > call(JdbcDataSource.java:192)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> > call(JdbcDataSource.java:172)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > getConnection(JdbcDataSource.java:503)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator.(JdbcDataSource.java:313)
> > ... 12 more
> > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> > not connect: Access denied for user 'root'@'ICZ2002912' (using
> > password: NO)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> > authentication(AbstractConnectProtocol.java:524)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> > handleConnectionPhases(AbstractConnectProtocol.java:472)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(
> > AbstractConnectProtocol.java:374)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> > connectWithoutProxy(AbstractConnectProtocol.java:763)
> > at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
> > at org.mariadb.jdbc.Driver.connect(Driver.java:104)
> > ... 16 more
> >
> >
> >
> > On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <jamieja...@gmail.com>
> > wrote:
> >
> > > It happens to be ten characters.
> > >
> > > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
> > arafa...@gmail.com
> > > > wrote:
> > >
> > >> How long is the encryption key (file content)? Because the code I am
> > >> looking at seems to expect it to be at most 100 characters.
> > >>
> > >> Regards,
> > >> Alex.
> > >> 
> > >> Newsletter and resources for Solr beginners and intermediates:
> > >> http://www.solr-start.com/
> > >>
> > >>
> > >> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>
> > >> wrote:
> > >> > I haven't tried this but is it possible there is a new line at the
> end
> > >> in
> > >> > the file?
> > >> >
> > >> > If you did something like echo "" > file.txt then there would be a
> new
> > >> > line. Use echo -n "" > file.txt
> > >> >
> > >> > Also you should be able to check how many characters are in the
> file.
> > >> >
> > >> > Kevin Risden
> > >> >
> > >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com
> >
> > >> wrote:
> > >> >
> > >> >> Hi Folks,
> > >> >>
> > >> >> (Using Solr 5.5.3.)
> > >> >>
> > >> >> As far as I know, the only place where encrypted password use is
> > >> documented
> > >> >> is in
> > >> >> https://cwiki.apache.org/confluence/display/solr/
> > >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> > >> >> under the "Configuring the DIH Configuration File", in a comment in
> > the
> > >> >> sample XML file:
> > >> >>
> > >> >> 
> > >> >>
> > >> >> Anyway, I can encrypt just fine:
> > >> >>
> > >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> > >> >> enter aes-128-cbc encryption password:
> > >> >> Verifying - enter aes-128-cbc encryption password:
> > >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> > >> >>
> > >> >>
> > >> >> I can also decrypt just fine from the command line.
> > >> >>
> > >> >> However, if I use the encrypted password and encryptKeyFile in the
> > >> config
> > >> >> file, I end up with an error: "String length must be a multiple of
> > >> four."
> > >> >>
> > >> >> https://gist.github.com/jamiejackson/
> 3852dacb03432328ea187d43ade5e4
> > d9
> > >> >>
> > >> >> How do I get this working?
> > >> >>
> > >> >> Thanks,
> > >> >> Jamie
> > >> >>
> > >>
> > >
> > >
> >
>

Re: Problem with Password Decryption in Data Import Handler

[Fuad Efendi]

aImporter.java:416)  
> at org.apache.solr.handler.dataimport.DataImporter.  
> runCmd(DataImporter.java:480)  
> at org.apache.solr.handler.dataimport.DataImporter$1.run(  
> DataImporter.java:461)  
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not  
> connect: Access denied for user 'root'@'ICZ2002912' (using password:  
> NO)  
> at org.mariadb.jdbc.internal.util.ExceptionMapper.get(  
> ExceptionMapper.java:123)  
> at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(  
> ExceptionMapper.java:71)  
> at org.mariadb.jdbc.Driver.connect(Driver.java:109)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.  
> call(JdbcDataSource.java:192)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.  
> call(JdbcDataSource.java:172)  
> at org.apache.solr.handler.dataimport.JdbcDataSource.  
> getConnection(JdbcDataSource.java:503)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$  
> ResultSetIterator.(JdbcDataSource.java:313)  
> ... 12 more  
> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could  
> not connect: Access denied for user 'root'@'ICZ2002912' (using  
> password: NO)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.  
> authentication(AbstractConnectProtocol.java:524)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.  
> handleConnectionPhases(AbstractConnectProtocol.java:472)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(  
> AbstractConnectProtocol.java:374)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.  
> connectWithoutProxy(AbstractConnectProtocol.java:763)  
> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)  
> at org.mariadb.jdbc.Driver.connect(Driver.java:104)  
> ... 16 more  
>  
>  
>  
> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <jamieja...@gmail.com>  
> wrote:  
>  
> > It happens to be ten characters.  
> >  
> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <  
> arafa...@gmail.com  
> > > wrote:  
> >  
> >> How long is the encryption key (file content)? Because the code I am  
> >> looking at seems to expect it to be at most 100 characters.  
> >>  
> >> Regards,  
> >> Alex.  
> >>   
> >> Newsletter and resources for Solr beginners and intermediates:  
> >> http://www.solr-start.com/  
> >>  
> >>  
> >> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>  
> >> wrote:  
> >> > I haven't tried this but is it possible there is a new line at the end  
> >> in  
> >> > the file?  
> >> >  
> >> > If you did something like echo "" > file.txt then there would be a new  
> >> > line. Use echo -n "" > file.txt  
> >> >  
> >> > Also you should be able to check how many characters are in the file.  
> >> >  
> >> > Kevin Risden  
> >> >  
> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>  
> >> wrote:  
> >> >  
> >> >> Hi Folks,  
> >> >>  
> >> >> (Using Solr 5.5.3.)  
> >> >>  
> >> >> As far as I know, the only place where encrypted password use is  
> >> documented  
> >> >> is in  
> >> >> https://cwiki.apache.org/confluence/display/solr/  
> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,  
> >> >> under the "Configuring the DIH Configuration File", in a comment in  
> the  
> >> >> sample XML file:  
> >> >>  
> >> >>   
> >> >>  
> >> >> Anyway, I can encrypt just fine:  
> >> >>  
> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt  
> >> >> enter aes-128-cbc encryption password:  
> >> >> Verifying - enter aes-128-cbc encryption password:  
> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=  
> >> >>  
> >> >>  
> >> >> I can also decrypt just fine from the command line.  
> >> >>  
> >> >> However, if I use the encrypted password and encryptKeyFile in the  
> >> config  
> >> >> file, I end up with an error: "String length must be a multiple of  
> >> four."  
> >> >>  
> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4  
> d9  
> >> >>  
> >> >> How do I get this working?  
> >> >>  
> >> >> Thanks,  
> >> >> Jamie  
> >> >>  
> >>  
> >  
> >  
>  

Re: Problem with Password Decryption in Data Import Handler

[Jamie Jackson]

> call(JdbcDataSource.java:192)
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:172)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getConnection(JdbcDataSource.java:503)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.(JdbcDataSource.java:313)
> ... 12 more
> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> not connect: Access denied for user 'root'@'ICZ2002912' (using
> password: NO)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> authentication(AbstractConnectProtocol.java:524)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> handleConnectionPhases(AbstractConnectProtocol.java:472)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(
> AbstractConnectProtocol.java:374)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> connectWithoutProxy(AbstractConnectProtocol.java:763)
> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
> at org.mariadb.jdbc.Driver.connect(Driver.java:104)
> ... 16 more
>
>
>
> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <jamieja...@gmail.com>
> wrote:
>
> > It happens to be ten characters.
> >
> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
> arafa...@gmail.com
> > > wrote:
> >
> >> How long is the encryption key (file content)? Because the code I am
> >> looking at seems to expect it to be at most 100 characters.
> >>
> >> Regards,
> >> Alex.
> >> 
> >> Newsletter and resources for Solr beginners and intermediates:
> >> http://www.solr-start.com/
> >>
> >>
> >> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>
> >> wrote:
> >> > I haven't tried this but is it possible there is a new line at the end
> >> in
> >> > the file?
> >> >
> >> > If you did something like echo "" > file.txt then there would be a new
> >> > line. Use echo -n "" > file.txt
> >> >
> >> > Also you should be able to check how many characters are in the file.
> >> >
> >> > Kevin Risden
> >> >
> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
> >> wrote:
> >> >
> >> >> Hi Folks,
> >> >>
> >> >> (Using Solr 5.5.3.)
> >> >>
> >> >> As far as I know, the only place where encrypted password use is
> >> documented
> >> >> is in
> >> >> https://cwiki.apache.org/confluence/display/solr/
> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> >> >> under the "Configuring the DIH Configuration File", in a comment in
> the
> >> >> sample XML file:
> >> >>
> >> >> 
> >> >>
> >> >> Anyway, I can encrypt just fine:
> >> >>
> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> >> >> enter aes-128-cbc encryption password:
> >> >> Verifying - enter aes-128-cbc encryption password:
> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >> >>
> >> >>
> >> >> I can also decrypt just fine from the command line.
> >> >>
> >> >> However, if I use the encrypted password and encryptKeyFile in the
> >> config
> >> >> file, I end up with an error: "String length must be a multiple of
> >> four."
> >> >>
> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4
> d9
> >> >>
> >> >> How do I get this working?
> >> >>
> >> >> Thanks,
> >> >> Jamie
> >> >>
> >>
> >
> >
>

Re: Problem with Password Decryption in Data Import Handler

[Fuad Efendi]

aracters.
>>
>> Regards,
>> Alex.
>> 
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>
>> wrote:
>> > I haven't tried this but is it possible there is a new line at the end
>> in
>> > the file?
>> >
>> > If you did something like echo "" > file.txt then there would be a new
>> > line. Use echo -n "" > file.txt
>> >
>> > Also you should be able to check how many characters are in the file.
>> >
>> > Kevin Risden
>> >
>> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
>> wrote:
>> >
>> >> Hi Folks,
>> >>
>> >> (Using Solr 5.5.3.)
>> >>
>> >> As far as I know, the only place where encrypted password use is
>> documented
>> >> is in
>> >> https://cwiki.apache.org/confluence/display/solr/
>> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> >> under the "Configuring the DIH Configuration File", in a comment in the
>> >> sample XML file:
>> >>
>> >> 
>> >>
>> >> Anyway, I can encrypt just fine:
>> >>
>> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> >> enter aes-128-cbc encryption password:
>> >> Verifying - enter aes-128-cbc encryption password:
>> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>> >>
>> >>
>> >> I can also decrypt just fine from the command line.
>> >>
>> >> However, if I use the encrypted password and encryptKeyFile in the
>> config
>> >> file, I end up with an error: "String length must be a multiple of
>> four."
>> >>
>> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>> >>
>> >> How do I get this working?
>> >>
>> >> Thanks,
>> >> Jamie
>> >>
>>
>
>

Re: Problem with Password Decryption in Data Import Handler

[Fuad Efendi]

ources for Solr beginners and intermediates:  
>> http://www.solr-start.com/  
>>  
>>  
>> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com>  
>> wrote:  
>> > I haven't tried this but is it possible there is a new line at the end  
>> in  
>> > the file?  
>> >  
>> > If you did something like echo "" > file.txt then there would be a new  
>> > line. Use echo -n "" > file.txt  
>> >  
>> > Also you should be able to check how many characters are in the file.  
>> >  
>> > Kevin Risden  
>> >  
>> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>  
>> wrote:  
>> >  
>> >> Hi Folks,  
>> >>  
>> >> (Using Solr 5.5.3.)  
>> >>  
>> >> As far as I know, the only place where encrypted password use is  
>> documented  
>> >> is in  
>> >> https://cwiki.apache.org/confluence/display/solr/  
>> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,  
>> >> under the "Configuring the DIH Configuration File", in a comment in the  
>> >> sample XML file:  
>> >>  
>> >>   
>> >>  
>> >> Anyway, I can encrypt just fine:  
>> >>  
>> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt  
>> >> enter aes-128-cbc encryption password:  
>> >> Verifying - enter aes-128-cbc encryption password:  
>> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=  
>> >>  
>> >>  
>> >> I can also decrypt just fine from the command line.  
>> >>  
>> >> However, if I use the encrypted password and encryptKeyFile in the  
>> config  
>> >> file, I end up with an error: "String length must be a multiple of  
>> four."  
>> >>  
>> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9  
>> >>  
>> >> How do I get this working?  
>> >>  
>> >> Thanks,  
>> >> Jamie  
>> >>  
>>  
>  
>  

Re: Problem with Password Decryption in Data Import Handler

[Jamie Jackson]

le.txt
>> >
>> > Also you should be able to check how many characters are in the file.
>> >
>> > Kevin Risden
>> >
>> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
>> wrote:
>> >
>> >> Hi Folks,
>> >>
>> >> (Using Solr 5.5.3.)
>> >>
>> >> As far as I know, the only place where encrypted password use is
>> documented
>> >> is in
>> >> https://cwiki.apache.org/confluence/display/solr/
>> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> >> under the "Configuring the DIH Configuration File", in a comment in the
>> >> sample XML file:
>> >>
>> >> 
>> >>
>> >> Anyway, I can encrypt just fine:
>> >>
>> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> >> enter aes-128-cbc encryption password:
>> >> Verifying - enter aes-128-cbc encryption password:
>> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>> >>
>> >>
>> >> I can also decrypt just fine from the command line.
>> >>
>> >> However, if I use the encrypted password and encryptKeyFile in the
>> config
>> >> file, I end up with an error: "String length must be a multiple of
>> four."
>> >>
>> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>> >>
>> >> How do I get this working?
>> >>
>> >> Thanks,
>> >> Jamie
>> >>
>>
>
>

Re: Problem with Password Decryption in Data Import Handler

[Jamie Jackson]

It happens to be ten characters.

On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <arafa...@gmail.com>
wrote:

> How long is the encryption key (file content)? Because the code I am
> looking at seems to expect it to be at most 100 characters.
>
> Regards,
>Alex.
> 
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com> wrote:
> > I haven't tried this but is it possible there is a new line at the end in
> > the file?
> >
> > If you did something like echo "" > file.txt then there would be a new
> > line. Use echo -n "" > file.txt
> >
> > Also you should be able to check how many characters are in the file.
> >
> > Kevin Risden
> >
> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
> wrote:
> >
> >> Hi Folks,
> >>
> >> (Using Solr 5.5.3.)
> >>
> >> As far as I know, the only place where encrypted password use is
> documented
> >> is in
> >> https://cwiki.apache.org/confluence/display/solr/
> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> >> under the "Configuring the DIH Configuration File", in a comment in the
> >> sample XML file:
> >>
> >> 
> >>
> >> Anyway, I can encrypt just fine:
> >>
> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> >> enter aes-128-cbc encryption password:
> >> Verifying - enter aes-128-cbc encryption password:
> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >>
> >>
> >> I can also decrypt just fine from the command line.
> >>
> >> However, if I use the encrypted password and encryptKeyFile in the
> config
> >> file, I end up with an error: "String length must be a multiple of
> four."
> >>
> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
> >>
> >> How do I get this working?
> >>
> >> Thanks,
> >> Jamie
> >>
>

Re: Problem with Password Decryption in Data Import Handler

[Jamie Jackson]

I tried it both ways yesterday--with a newline and without.

On Thu, Oct 6, 2016 at 12:26 PM, Kevin Risden <compuwizard...@gmail.com>
wrote:

> I haven't tried this but is it possible there is a new line at the end in
> the file?
>
> If you did something like echo "" > file.txt then there would be a new
> line. Use echo -n "" > file.txt
>
> Also you should be able to check how many characters are in the file.
>
> Kevin Risden
>
> On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > (Using Solr 5.5.3.)
> >
> > As far as I know, the only place where encrypted password use is
> documented
> > is in
> > https://cwiki.apache.org/confluence/display/solr/
> > Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> > under the "Configuring the DIH Configuration File", in a comment in the
> > sample XML file:
> >
> > 
> >
> > Anyway, I can encrypt just fine:
> >
> > $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> > enter aes-128-cbc encryption password:
> > Verifying - enter aes-128-cbc encryption password:
> > U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >
> >
> > I can also decrypt just fine from the command line.
> >
> > However, if I use the encrypted password and encryptKeyFile in the config
> > file, I end up with an error: "String length must be a multiple of four."
> >
> > https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
> >
> > How do I get this working?
> >
> > Thanks,
> > Jamie
> >
>

Re: Problem with Password Decryption in Data Import Handler

[Alexandre Rafalovitch]

How long is the encryption key (file content)? Because the code I am
looking at seems to expect it to be at most 100 characters.

Regards,
   Alex.

Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 6 October 2016 at 23:26, Kevin Risden <compuwizard...@gmail.com> wrote:
> I haven't tried this but is it possible there is a new line at the end in
> the file?
>
> If you did something like echo "" > file.txt then there would be a new
> line. Use echo -n "" > file.txt
>
> Also you should be able to check how many characters are in the file.
>
> Kevin Risden
>
> On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com> wrote:
>
>> Hi Folks,
>>
>> (Using Solr 5.5.3.)
>>
>> As far as I know, the only place where encrypted password use is documented
>> is in
>> https://cwiki.apache.org/confluence/display/solr/
>> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> under the "Configuring the DIH Configuration File", in a comment in the
>> sample XML file:
>>
>> 
>>
>> Anyway, I can encrypt just fine:
>>
>> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> enter aes-128-cbc encryption password:
>> Verifying - enter aes-128-cbc encryption password:
>> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>>
>>
>> I can also decrypt just fine from the command line.
>>
>> However, if I use the encrypted password and encryptKeyFile in the config
>> file, I end up with an error: "String length must be a multiple of four."
>>
>> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>>
>> How do I get this working?
>>
>> Thanks,
>> Jamie
>>

Re: Problem with Password Decryption in Data Import Handler

[Kevin Risden]

I haven't tried this but is it possible there is a new line at the end in
the file?

If you did something like echo "" > file.txt then there would be a new
line. Use echo -n "" > file.txt

Also you should be able to check how many characters are in the file.

Kevin Risden

On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <jamieja...@gmail.com> wrote:

> Hi Folks,
>
> (Using Solr 5.5.3.)
>
> As far as I know, the only place where encrypted password use is documented
> is in
> https://cwiki.apache.org/confluence/display/solr/
> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> under the "Configuring the DIH Configuration File", in a comment in the
> sample XML file:
>
> 
>
> Anyway, I can encrypt just fine:
>
> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> enter aes-128-cbc encryption password:
> Verifying - enter aes-128-cbc encryption password:
> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>
>
> I can also decrypt just fine from the command line.
>
> However, if I use the encrypted password and encryptKeyFile in the config
> file, I end up with an error: "String length must be a multiple of four."
>
> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>
> How do I get this working?
>
> Thanks,
> Jamie
>

Problem with Password Decryption in Data Import Handler

[Jamie Jackson]

Hi Folks,

(Using Solr 5.5.3.)

As far as I know, the only place where encrypted password use is documented
is in
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
under the "Configuring the DIH Configuration File", in a comment in the
sample XML file:



Anyway, I can encrypt just fine:

$ openssl enc -aes-128-cbc -a -salt -in stgps.txt
enter aes-128-cbc encryption password:
Verifying - enter aes-128-cbc encryption password:
U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=


I can also decrypt just fine from the command line.

However, if I use the encrypted password and encryptKeyFile in the config
file, I end up with an error: "String length must be a multiple of four."

https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9

How do I get this working?

Thanks,
Jamie

RE: Data import handler in techproducts example

[Brooks Chuck (FCA)]

Hello Jonas,

Did you figure this out? 

Dr. Chuck Brooks
248-838-5070


-Original Message-
From: Jonas Vasiliauskas [mailto:jonas.vasiliaus...@yahoo.com.INVALID] 
Sent: Saturday, July 02, 2016 11:37 AM
To: solr-user@lucene.apache.org
Subject: Data import handler in techproducts example

Hey,

I'm quite new to solr and java environments. I have a goal for myself to import 
some data from mysql database in techproducts (core) example.

I have setup data import handler (DIH) for techproducts based on instructions 
here https://wiki.apache.org/solr/DIHQuickStart , but looks like solr doesn't 
load DIH libraries, could someone please explain in quick words on how to check 
if DIH is loaded and if not - how can I load it ?

Stacktrace is here: http://pastebin.ca/3654347

Thanks,

Re: Data import handler in techproducts example

[Ahmet Arslan]

Hi Jonas,

Search for the 
solr-dataimporthandler-*.jar place it under a lib directory (same level as the 
solr.xml file) along with the mysql jdbc driver (mysql-connector-java-*.jar)

Please see:
https://cwiki.apache.org/confluence/display/solr/Lib+Directives+in+SolrConfig




On Saturday, July 2, 2016 9:56 PM, Jonas Vasiliauskas 
<jonas.vasiliaus...@yahoo.com.INVALID> wrote:
Hey,

I'm quite new to solr and java environments. I have a goal for myself to 
import some data from mysql database in techproducts (core) example.

I have setup data import handler (DIH) for techproducts based on 
instructions here https://wiki.apache.org/solr/DIHQuickStart , but looks 
like solr doesn't load DIH libraries, could someone please explain in 
quick words on how to check if DIH is loaded and if not - how can I load 
it ?

Stacktrace is here: http://pastebin.ca/3654347

Thanks,

Data import handler in techproducts example

[Jonas Vasiliauskas]

Hey,

I'm quite new to solr and java environments. I have a goal for myself to 
import some data from mysql database in techproducts (core) example.


I have setup data import handler (DIH) for techproducts based on 
instructions here https://wiki.apache.org/solr/DIHQuickStart , but looks 
like solr doesn't load DIH libraries, could someone please explain in 
quick words on how to check if DIH is loaded and if not - how can I load 
it ?


Stacktrace is here: http://pastebin.ca/3654347

Thanks,